Incident Report for itslearning 

On Monday 10th October, 20.24 CEST, we experienced a cascading double hardware failure on our enterprise storage array. This caused a system, which is designed to be fully redundant, to become unavailable. Investigation into the root cause is still ongoing. 

Our operations team identified the problem one hour before the system crashed, immediately contacted the vendor and began troubleshooting the issue. 

For customers hosted out of our Oslo data center the incident caused downtime from 20.26 CEST, Monday 10th October to 09.30 CEST Tuesday 11th October, totalling 13 hrs and 4 minutes of downtime. During the period from 09.30 CEST to 13:30 CEST Tuesday 11th of October these customers also experienced minor disruptions related to instant messages, notifications, and OWAS . 

Customers hosted out of our Amazon environment experienced serious disruption to their service from 20.26 CEST, Monday 10th October to 08.50 CET Tuesday 11th October. 

There was a further outage during the planned maintenance period from 03.30 – 07.30 CEST, Wednesday 13th October, to replace the faulty parts. 

For itslearning, any unplanned downtime is unacceptable. To prevent similar incidents happening in future we are fully committed to: 
  • Reviewing and improving our hardware infrastructure to ensure that all failover mechanisms work correctly. 
  • Working closely with our 3rd party service and hardware providers to ensure compliance with our SLAs. 
  • Reviewing our business continuity procedures to ensure impact of any future unexpected incidents are minimized. 
Oct 14, 2016 - 12:42 CEST




Resolved - All systems operational.

Oct 11, 16:49 CEST


Update - If you experience logon problems, please be patient and try again in 10 minutes. 
Oct 11, 09:23 CEST


Update - itslearning is now available. There might still be som delays using the Instant messaging. We will continue to monitor the system very closely. 
Oct 11, 08:56 CEST


Update 

itslearning is still unavailable for customers located in EU. New information will follow in 30 minutes at the latest. 

Oct 11, 08:25 CEST


Update

Third party technical experts and our engineering team are actively investigating the current issue. This is our Highest priority and will remain so until service is restored. Our most recent recovery attempts were unsuccessful due to hardware issues. We appreciate your patience while we are working to recover our service.

Posted 1 minute ago. Oct 11, 2016 - 07:12 CEST


New Incident Status: Monitoring

The service is available for all users but you may notice some pages are taking longer than usual to load. Our engineers will continue improving and monitoring the situation. 

Posted 3 minutes ago. Oct 11, 2016 - 00:04 CEST


Identified

We have identified the issue as a severe hardware failure. We are in the progress of restoring the service.

Posted 12 minutes ago. Oct 10, 2016 - 23:12 CEST


Update

We are experiencing technical issues. Itslearning may be temporarily unavailable for some users. We are investigating the problem, and will fix it shortly. If you are experiencing issues, please try again later.

Posted about 1 hour ago. Oct 10, 2016 - 22:18 CEST


Investigating

We are experiencing some issues for file uploads and downloads on parts of our application. Our engineers are investigating and we will fix the problem shortly.

Posted about 2 hours ago. Oct 10, 2016 - 21:23 CEST