It has been raining for the last week here in San Francisco. And when it rains it pours. And that’s a very fitting picture of the situation at Zendesk Ops the last 48 hours. It started Sunday morning (PST) where a planned hardware maintenance with our service provider exceeded the one hour service window with a whopping three hours. Monday morning we experienced unusual high email traffic, which tipped over as a client decided to push a backlog of emails to our system. More than 10,000 emails hit our servers within a few minutes. We should have been resilient to that, but weren’t. By Monday afternoon everything was back to normal, application responding properly, all emails processed, no tickets lost.
If you’re familiar with the Three Wise Monkeys, you will appreciate the Zendesk Ops version with yours truly trying not to show Anger, Frustration, Despair. Not our best Zen moment. Sorry. We will do better.