Post Mortem of Outage – Friday 08/18/2017

Description of Outage:

Standard SureDesks and some Custom Server SureDesks rebooted without warning. Overall outage was 30 minutes. Customers were working post reboot without any major problems.


Outage Review:

From the review of Logs and talking with Citrix the issue seems to be that the Delivery Controllers failed to reboot some of the Server this weekend during regular server maintenance on Sunday. It seems that the Delivery Controllers could finally complete the reboot that was scheduled weekly to allow for consistent performance. Nothing indicates that this was anything other than that function running at the wrong time. The message and the timing all point to the process that should take place at 3am on Sunday Morning.




This clearly was a bug in the Citrix Version. The Delivery Controllers were upgraded to the latest version to help prevent this from happening in the future. We will continue to streamline the process so that these can happen at a regular pace to prevent these types of outages from occurring.