Network Disruption

Urgency: unscheduled
Affected services: BIT network, DNS, Monitoring
Expected impact: possible disruption of IPv6 traffic
Customer action required: No
Reference number: 200512

Summary:

On September 2nd at approximately 23.45h a network disruption started, specifically affecting IPv6 traffic for a select number of networks within the BIT network. This problem was fixed by BIT network engineers at 1.35h.

Details:

The disruption was limited to IPv6 traffic on several specific networks, including networks with many customer virtual machines, and networks with BIT’s DNS services and monitoring services.

The disruption started after a configuration change prior to maintenance by a vendor was done. This is a standard change that has been done many times before at similar maintenances, and is unrelated to the problems that occurred. This time, the change triggered a problem where two of BIT’s routers for these networks both became IPv6 gateway (VRRP master) for this network, instead of just one. This resulted in broken IPv6 connectivity within BIT’s network and to the internet. IPv4 traffic was unaffected.

Engineers fixed this inconsistency in VRRP state manually, after which services recovered.

Because BIT’s monitoring servers were affected, customers may have received alarms on availability of their services, even if these services were not affected by this disruption and were working properly.

A case will be opened with the router vendor to investigate the cause and to figure out how to prevent this from happening in the future.