Degraded performance
Incident Report for The AskCody Platform
Postmortem

Following the Platform Maintenance roll-out we started to experience slow loading times and, in some cases, timeout issues. From Monday May 23th it became clear that the impact was higher than initially perceived and overall platform performance got worse. A degraded performance incident was created and an investigation to identify the cause was started. AskCody Status - Degraded performance

Better monitoring was established specifically for this incident and the following investigation found that connection time to the database was causing problems for requests within the platform. This resulted in the long response times and some individual requests "hanging" for minutes, resulting in timeouts. Tuesday May 24th in the afternoon we scaled our solution to handle the issue concerning connection times to the database. The data bandwidth was sufficient but there was a bottleneck in actually opening new connections to the database. When this was solved the average response time dropped and platform performance was fully restored.

Following the implementation of the solution, the incident status was changed to "monitoring" and the following day it was fully closed. The Performance readout suggests that not only is overall platform performance "back to normal" but exceeds what it was before the Platform Maintenance. We sincerely regret the inconvenience this has had and continue to monitor and maintain the platform to make sure the risk of further incidents is kept to a minimum.

Posted Jun 01, 2022 - 15:31 CEST

Resolved
This incident has been resolved.
Posted May 25, 2022 - 09:50 CEST
Monitoring
A fix has been implemented and we are monitoring the results. The platform performance has drastically improved and all services is now operational. We keep the incident open for monitoring reasons.
Posted May 24, 2022 - 16:03 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted May 24, 2022 - 13:46 CEST
Investigating
We still experience performance issues and are sorry for the inconvenience in relation to this. The fix deployed yesterday has not fully solved the performance issues and we are now doing further investigations.
Posted May 24, 2022 - 09:40 CEST
Monitoring
A fix has been implemented and we are monitoring the results
Posted May 23, 2022 - 15:02 CEST
Investigating
We are currently investigating issues with slow performance and timeouts across the platform.
Posted May 23, 2022 - 13:35 CEST
This incident affected: Visitor Management (Europe) (Outlook Add-in, Visitor Management Portal, Check-in kiosk), Meeting Services (Europe) (Outlook Add-in, Meeting Services Portal, Meeting Services Finance API), Room Booking (Europe) (Outlook Add-in, Room Management Portal, Meeting Dashboards, Room Displays, Workplace Central), and Workplace Insights (Europe) (Power BI Dashboard).