[07:02 AM U.S. EST] - iLab Site Reliability Engineers received alert from server monitor on North American instance.
[07:15 AM U.S. EST] - iLab North American sites are confirmed inaccessible.
[07:25 AM U.S. EST] - iLab Site Reliability Engineers confirm outage is on the network service provider.
[07:29 AM U.S. EST] - iLab Site Reliability Engineers working with cloud service provider to troubleshoot site outage.
[08:00 AM U.S. EST] - Issue has been identified and is actively being worked on for a fix. Thank you for your continued patience.
[08:29 AM U.S. EST] - iLab has confirmed to be a hardware issue on the physical host.09
[09:05 AM U.S. EST] - iLab is continuing to work on a fix for this issue with the cloud service provider.
[10:00 AM U.S. EST] - iLab is continuing to work on a fix for this issue with the cloud service provider.
[01:15 PM U.S. EST] - No ETA on when service will be fully restored from cloud service provider. iLab Site Reliability Engineers are investigating alternative solutions.
[01:45 PM U.S. EST] - iLab Engineers have partially restored service to the back-end search functionality, which was the one of the main points of failure in today's incident. In order to expedite access back into iLab and let users access system features, we have temporarily restored this search functionality, understanding that not all information will be visible to you. Items like searching for service requests, calendar events will be restored over the course of the next several hours.
[03:12 PM U.S. EST] - iLab engineers have created a replacement search server and initiated re-indexing of the data. As the data is added to the new search server, it will become available on the iLab site.
[04:35 PM U.S. EST] - Network service provider continuing to reboot hardware to restore iLab sites fully. Periods of instability may occur over the weekend.
We apologize for any inconvenience that this has brought to your daily operations and if there are any questions, please email iLab Support at firstname.lastname@example.org or call 1-800-690-2957.