On Tuesday, 10 November 2015, outbound traffic going through one of our European routers from both Google Compute Engine and Google App Engine experienced high latency for a duration of 6h43m minutes. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and
DETAILED DESCRIPTION OF IMPACT
On Tuesday, 10 November 2015 from 06:30 – 13:13 PST, a subset of outbound traffic from Google Compute Engine VMs and Google App Engine instances experienced high latency. The disruption to service was limited to outbound traffic through one of our European routers, and caused at peak 40% of all traffic being routed through this device to be dropped. This accounted for 1% of all Google Compute Engine traffic being routed from
EMEA and <0.05% of all traffic for Google App Engine.
A network component failure in one of our European routers temporarily reduced network capacity in the region causing network congestion for traffic traversing this route. Although the issue was mitigated by changing the traffic priority, the problem was only fully resolved when the affected hardware was replaced.
REMEDIATION AND PREVENTION
As soon as significant traffic congestion in the network path was detected, at 09:10 PST, Google Engineers diverted a subset of traffic away from the affected path. As this only slightly decreased the congestion, Google Engineers made a change in traffic priority which fully mitigated the problem by 13:13 PST time. The replacement of the faulty hardware resolved the problem.
To address time to resolution, Google engineers have added appropriate alerts to the monitoring of this type of router, so that similar congestion events will be spotted significantly more quickly in future. Additionally, Google engineers will ensure that capacity plans properly account for all types of traffic in single device failures.
Furthermore, Google engineers will audit and augment capacity in this region to ensure sufficient redundancy is available.