[EU] Latency and timeouts from secondary services - Post Mortem

‌

This Postmortem and technical timeline follows the announcement of degraded performance on the AskCody platform, beginning on Wednesday, August 13th.

Summary

On August 13th at 13:12 CEST, parts of the AskCody platform became unresponsive with very long response times and timeout errors. The errors were periodic and were not caused by any changes made by AskCody. This report describes the timeline and remediation steps taken to resolve the issue.

Incident overview

On August 13th, 2025, at 13:15 CEST, our team was alerted to a severe drop in availability on the app.onaskcody.com domain, serving both our management portal, add-ins, and the check-in kiosks to our EU customers. The alert was in response to a steep increase in response times that had manifested a few minutes earlier.

Investigation and actions taken

Initial response

August 13, at 13:15 CEST: Detected drop in availability and influx in response times.

August 13, at 13:15 CEST: Extended team investigation into the cause and effects of the incident. No conclusive evidence.

Investigation, temporary solutions, and monitoring

May 14 - 15: Investigations extended to include Microsoft Azure services as PaaS resources seemed likely to be involved. Working with Microsoft to profile latency through their network and services.

Critical discovery

August 18, 13:00 CEST: Tracking led to the identification of periodically failing calls to a third-party service that had made changes to their ingestion on the 13th.

Resolution

August 18,. 14:00 CEST: The issue was mitigated, and the incident status was set to “monitoring”.

Root cause analysis

The root cause of the incident was a misconfigured request to a third-party service. We had received timely advice to alter the configuration, but the fact that the endpoint still periodically accepted requests made the troubleshooting more complex.

Preventative and supplementary measures

To prevent similar incidents in the future, we have:

• Extended alerting systems for availability tests.

Conclusion

Though this incident was not classified as critical, we still consider the impact to be serious. We apologize for any inconvenience caused by the service degradation and assure you that our platform is fully operational again. We remain committed to maintaining the highest security standards, following Microsoft best practices, and will continue to take proactive measures to safeguard our services.

If you have any questions or need further information, please do not hesitate to contact our support team.

Thank you for your understanding and continued trust in AskCody.

Sincerely,
AskCody