Incident Summary: Excess Appointment Reminder Notifications

Dennis Myl

Updated June 29, 2022 22:47

🚨 Evening of June 8, 2021 between 8PM - 11PM (EST)

Yesterday evening, our customer support and development teams were notified of an issue affecting appointment reminder notifications. An isolated outage with our asynchronous reminder service caused some users to receive an excess amount of appointment reminders via email or SMS. (spam).

📅 Appointment reminders are meant to be a single email reminder, and we deeply apologize for the spam. If you've blocked or marked the sender as spam, it is now safe to remove that to continue receiving your appointment notifications. Our development team has since resolved the issue (see summary below).

📬 Our email service works in two steps, and these happen every 5 minutes:

Send out all appointment reminders based on upcoming appointments scheduled within a user's account.
Mark them as sent in our backend so our system knows not to send the same notification again.

Due to a large volume of appointment reminders being sent out, our reminder service had crashed and timed out before reaching the vital second step (marking as sent ✅). Thus, why some users continued to receive a large amount of appointment reminders, as our system never marked these notifications as delivered. This loop caused duplicate reminders to go out between 8PM - 11PM (EST).

🔧 Our development team's resolution and next steps:

Our engineering team acted swiftly to rectify the notification glitch after receiving reports from our clients and customer support team.
Furthermore, our engineering team will look to further refine our reminder service to handle larger volumes as occurred last evening.

To conclude, we are continually making improvements to our platform each and every day! In this case our teams responded quickly to put a stop to the appointment reminder spam some users experienced. We do apologize for any inconvenience this may have caused, and most importantly for the messy inbox.