Friday May 10th at 7:25 am Pacific Daylight Time, Lithium experienced a significant network outage. However unwanted, unpleasant, and unplanned such events can be, in the world of SaaS technology, outages can be a very real possibility. But that factor doesn’t mitigate the downstream effect we know this has for our customers and their customers who depend on our technology. In the spirit of our company value of ‘being real’, I want to share with all of you what Lithium learned from the experience and where I see opportunities to continue to strengthen our performance.
The outage in our North America data center lasted just under 3.5 hours and created an intermittent experience for a number of our customers, both in the US and those that this data center serves abroad. We believe that the outage resulted from what is called a Denial of Service attack on our servers, which essentially means that someone from the outside flooded our networks with spurious traffic impacting Communities, Lithium Social Web instances and our internal email system. As a result of this attack, there is no evidence that our customers’ information stored in our systems has been compromised. While we continue to investigate exactly what triggered the attack, there are a number of immediate steps that we are taking including:
1. Fortifying our edge network with increased capacity and additional DOS prevention features to withstand traffic floods and thwart DOS attacks.
2. Improving our monitoring system so it can identify the attacks faster.
3. Practicing our ability to quickly route our traffic through our existing DOS prevention partners. Our partners have the ability to strip out the spurious traffic and route clean traffic to us giving us the ability to withstand a DOS attack.
I know many of you often hear me talk about the values of Lithium, and Friday’s situation reflected two of these values very accurately: Take Customer Success Personally and Learn Fast, Act Faster. For me, Friday was a solid testimony of how well Lithium is living up to its core values.
While the engineering and tech ops team worked urgently to solve the network issues, our sales, service and marketing teams rallied to provide transparent communication to affected customers in short order—leveraging both social channels and outbound call downs when emailed failed us. Our performance here was strong and in my eyes reflective of just the type of culture I want to see thrive at Lithium.
I want to reiterate that we are committed to being a world-class SaaS company. We are proud of the service-levels we provide customers and I want you to know just how seriously Lithium takes any disruption in our service to you and your customers. You rely on us and I can only assure you that what transpired on Friday demonstrates what you can expect from Lithium: quick recovery, full transparency and incorporating any learning quickly back into the business.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.