22nd January 2019

wao.io Routing issue in one of our datacenter

In the period from 2:07pm to 2:23 pm CET one of our datacenters was not available.

In the meantime, connectivity has been restored. We are still looking for the cause of the downtime.

During this period, requests routed through the datacenter were answered with an error message.

Update 3:54pm CET: We have identified as a root cause the deletion of a network route by a change to a router during maintenance work in our datacenter.

As a result, connections to origins could not be established, but browser connections were still accepted.

As a result, the automatic shutdown of the broken data center in the DNS was not triggered. Thus, users were still routed to this datacenter.

Update 4:37pm CET: As the most important step, we have identified that we will make a DNS routing adjustment to add the check that automatically removes failed data centers from load balancing robustly.