- 05-02-24VDC and VM management environment briefly offline (solved)
- 11-10-23interruption of BIT-HOST4
- 17-09-23CephFS storage issue
- 26-07-23Load balancer failure
- 04-07-23One of the shared mysql servers temporarily unavailable
- 13-04-23Short, limited temperature increase
- 06-04-23Short interruption webmail.bit.nl
- 01-03-23Network interruption leaf-sw5.bit-2a
- 22-02-23Malfunction email platform
- 21-02-23Disruption IMAP email services
Disruption fiber BIT-1 - Equinix AM7
24-06-2019 12:36:16
- 26-06-2019 06:37:08
Status:
resolved
At 12.05h an outage started on our fiber connection between BIT-1 and Equinix AM7. This fiber carriers parts of our traffic from BIT to Amsterdam. This traffic was rerouted automatically over other fibers. The disruption may have led to a short disruption of network traffic.
At this moment the cause of this outage is unknown, we're contacting our fiber provider to get more details.
Update placed at 24/06/2019, 13:42
The outage is still under investigation, our fiber provider is looking into the cause of this outage.
Update placed at 24/06/2019, 16:08
Our fiber provider has identified a possible cause of the outage. They will be doing more tests to determine the root cause of the outage.
Update placed at 24/06/2019, 17:37
Cause of the outage is found by our fiber provider. A technical team is on the way to fix the broken fibers.
Update placed at 24/06/2019, 23:15
There are no unused fibers available in the duct. Therefor the broken fibers need to be repaired. We have not received an indication of the time it will take to fix this.
Update placed at 25/06/2019, 01:16
The repair crews of our fiber provider are unable to find the specific location of the damaged area. The location is in a unlit area on a rugged terrain. Digging in the dark can cause more damage so the repairs are on hold till tomorrow morning.
Update placed at 25/06/2019, 08:45
It turns out the fiber duct is partially damaged, resulting in some fibers in the duct being disrupted and some not. As a result, repairs by the fiber provider have to be done in an emergency window, since this will affect the working fibers as well. During the repairs, the entire fiber bundle will be cut and reconnected, removing the damaged part. The emergency maintenance window is expected to start at 20.00h this evening, so this link will remain down for a long time. We want to make clear that the only service affected by this outage is customer wavelengths between these two locations. All other network traffic has been rerouted via other paths.
Update placed at 26/06/2019, 23:40
The link was restored 25 minutes ago. We have not received confirmation from our vendor yet that repair works have been completed, so there's still a chance the fiber will be disrupted again.
Update placed at 26/06/2019, 6:30
The repairs were completed at 1.40h. We have not observed any link flaps on this fiber since 23.15h.
The incident has been resolved. If you have any questions regarding this incident, please contact the Customer Care department by phone on +31 (0)318 648 688 or by email on support@bit.nl.
At this moment the cause of this outage is unknown, we're contacting our fiber provider to get more details.
Update placed at 24/06/2019, 13:42
The outage is still under investigation, our fiber provider is looking into the cause of this outage.
Update placed at 24/06/2019, 16:08
Our fiber provider has identified a possible cause of the outage. They will be doing more tests to determine the root cause of the outage.
Update placed at 24/06/2019, 17:37
Cause of the outage is found by our fiber provider. A technical team is on the way to fix the broken fibers.
Update placed at 24/06/2019, 23:15
There are no unused fibers available in the duct. Therefor the broken fibers need to be repaired. We have not received an indication of the time it will take to fix this.
Update placed at 25/06/2019, 01:16
The repair crews of our fiber provider are unable to find the specific location of the damaged area. The location is in a unlit area on a rugged terrain. Digging in the dark can cause more damage so the repairs are on hold till tomorrow morning.
Update placed at 25/06/2019, 08:45
It turns out the fiber duct is partially damaged, resulting in some fibers in the duct being disrupted and some not. As a result, repairs by the fiber provider have to be done in an emergency window, since this will affect the working fibers as well. During the repairs, the entire fiber bundle will be cut and reconnected, removing the damaged part. The emergency maintenance window is expected to start at 20.00h this evening, so this link will remain down for a long time. We want to make clear that the only service affected by this outage is customer wavelengths between these two locations. All other network traffic has been rerouted via other paths.
Update placed at 26/06/2019, 23:40
The link was restored 25 minutes ago. We have not received confirmation from our vendor yet that repair works have been completed, so there's still a chance the fiber will be disrupted again.
Update placed at 26/06/2019, 6:30
The repairs were completed at 1.40h. We have not observed any link flaps on this fiber since 23.15h.
The incident has been resolved. If you have any questions regarding this incident, please contact the Customer Care department by phone on +31 (0)318 648 688 or by email on support@bit.nl.