Increased latency resulting in slow performance
Incident Report for Layer Systems
Resolved
This incident has been resolved.
Posted Jan 18, 2023 - 17:30 GMT
Identified
Our upstream provider has provided the following statement:

Impact Statement: Starting at 11:24 UTC on 18 Jan 2023, a subset of customers using storage services in Europe may experience higher than expected latency, timeouts or HTTP 500 errors when accessing data stored on accounts hosted in this region. Additional services with dependencies on storage may also experience impact.

Current Status: We have determined that a subset of racks in a storage scale unit, hosted in one of the region datacenter experienced a drop in availability and may have rebooted. Our telemetry is showing that a power event impacted the said racks (impacting storage and network resources in the racks). Following the power event, most impacted resources auto-recovered through their normal recovery mechanisms. However, some resources required manual intervention to fully recover. Most customers should already be seeing signs of recovery and we are working on recovering the remaining impacted resources, while monitoring the health of previously recovered workloads.

An update will be provided within 60 minutes, or as events warrant.
Posted Jan 18, 2023 - 12:02 GMT
Investigating
We're currently investigating reports of latency affecting some requests. We'll update this page when we have further information.
Posted Jan 18, 2023 - 11:47 GMT
This incident affected: Services (Mail Triage) and Layer App, Layer API, Layer Exchange & Office 365 Connector, Data Manager.