Dear Valued AskCody Customers,

We would like to provide you with a detailed post mortem following the recent service disruption experienced on October 10, 2023, between 16:43 CEST and 20:30 CEST. We understand that this incident had a significant impact on your operations, and we sincerely apologize for any inconvenience this may have caused.

Incident Summary: During the aforementioned timeframe, we received reports from several of our customers regarding errors when attempting to access the AskCody Management Portal and Add-ins. After a thorough investigation, we identified the probable root cause of this disruption.

Root Cause: The root cause of the issue was related to the depletion of allocated memory on some of our virtual machines. While there was a possibility that a recent software version upgrade on the node framework of a few of our services contributed to this problem, it is more likely that external factors were at play. Specifically, we see signs that changes made by Microsoft Azure may have impacted our services, though we cannot currently definitively confirm this as Microsoft has not yet provided any information regarding outages or alterations on their end.

Actions Taken: Upon identifying the memory depletion issue, our technical team worked diligently to resolve the problem and restore normal service operations. We increased memory allocations, optimized resource utilization, and conducted thorough testing to ensure stability and reliability. As a result, services were fully restored at 20:30 CEST on October 10, 2023.

Preventive Measures: To prevent such incidents from occurring in the future, we are implementing additional monitoring and alerting systems to provide early detection of memory-related issues. We will also maintain open channels of communication with our infrastructure providers to better understand and respond to any external changes that may impact our services.

Customer Impact: We understand the frustration and inconvenience this service disruption may have caused you and your organization. We deeply regret any disruption to your workflow and want to assure you that we are committed to improving our services to ensure greater resilience and reliability.

Customer Feedback: Your feedback is invaluable to us, and we encourage you to reach out to our support team with any further concerns or questions regarding this incident. We will also be conducting a thorough internal review to ensure we learn from this experience and continually enhance our service quality.

We appreciate your patience and understanding during this incident, and we remain dedicated to providing you with the best possible experience using AskCody. We are committed to upholding the highest standards of service reliability and will work tirelessly to prevent future disruptions.

Thank you for your continued trust in AskCody.