Incident Report: December 4, 2015

Owned by Daniel Rehner
Dec 07, 2015
1 min read

Background

The GradesFirst application was unavailable for approximately 67% of customers on December 4, 2015 from 3:56pm to 4:27pm Eastern time. Following investigation by both GradesFirst and its hosting providers, we have identified the root cause of the outage and resolved it.

The application is now fully available for all customers. If you are still experiencing any problems accessing GradesFirst, please email us at support@gradesfirst.com.

Root Cause Analysis

The GradesFirst application is hosted on Amazon’s EC2/EBS platform. GradesFirst uses several hosted database servers operating in separate server environments. At 3:56pm on December 4, one of our databases became inaccessible. Investigation in coordination with our infrastructure provider revealed the issue occurred due to a hard drive failure on a device backing one of our Elastic Block Storage (EBS) volumes.

Timeline

  • 3:56pm EST: GradesFirst infrastructure monitoring systems alerted us to an availability issue. Our infrastructure team immediately began investigating the issue.
  • 4:05pm EST: Potential root cause identified; began coordination with our infrastructure provider; started initial remediation of reconfiguring connections to point to a redundant database.
  • 4:15pm EST: Root cause confirmed; began remediation by reprovisioning the EBS volume
  • 4:20pm EST: Continued coordination with our infrastructure provider to take corrective action.
  • 4:27pm EST: Completed reconfiguration. Outage resolved. Service restored to all customers.

Corrective Action

We have reprovisioned the affected EBS volume and run diagnostic disk checks to ensure its viability. We do not expect this issue to recur.

 

We know that you depend on GradesFirst. We regret any inconvenience you may have experienced as a result of this incident.

If you have any questions, please contact your Client Success Manager or email us at support@gradesfirst.com.