A Midnight Call: Blue Lights at BIT
A Midnight Call: Blue Lights at BIT
On Thursday, 9 October 2014, I arrived at our BIT-2 data centre around 1:30 AM. From a distance, I could already see the flashing blue lights of the fire brigade. Just before entering the premises, I’d spoken to two colleagues from my department—they’d called to ask if BIT needed assistance. I’d agreed to call them back if that were the case.
Upon arrival, I noticed the heroes of the Ede fire brigade standing around calmly, chatting. They joked that BIT employees looked like “brothers” with their glasses and receding hairlines. A colleague—also bespectacled and thinning on top—briefed me on the events of the past 45 minutes. I grabbed a radio and settled at my desk in the office.
I called back the two colleagues who’d rung me ten minutes earlier, asking if they could come to the office. By then, about ten BIT staff were already on-site in the middle of the night. I phoned another colleague to fill him in and request his presence, explaining what had happened:
At 00:45, multiple sensors in server room BIT-2B detected soot particles.
00511: Don 09-Okt-14 00:44:00: VOOR-ALARM
L3 Sens. 30 – Zone21 – VIEW
Zone: VM GROEP 1 BIT-2-B
Lok: Onder Vloer Waarde: 37%
00512: Don 09-Okt-14 00:44:04: VOOR-ALARM
L3 Sens. 12 – Zone21 – VIEW
Zone: VM GROEP 1 BIT-2-B
Lok: Onder Vloer Waarde: 25%
00513: Don 09-Okt-14 00:44:04: VOOR-ALARM
L3 Sens. 13 – Zone21 – VIEW
Zone: VM GROEP 1 BIT-2-B
Lok: Onder Vloer Waarde: 31%
Since multiple sensors registered soot, the internal emergency team and fire brigade were automatically alerted.
00514: Don 09-Okt-14 00:44:06: ** ALARM **
L3 Sens. 30 – Zone21 – VIEW
Zone: VM GROEP 1 BIT-2-B
Lok: Onder Vloer Waarde: 37%
The fire suppression system activated automatically, releasing a nitrogen-argon mixture from pressurised cylinders (at 60 bar) into the room.
To prevent the server room walls from being blown out during suppression, pressure relief vents had been installed. These vents opened, releasing a blast of air outward.
Ten minutes after the alarm, the fire brigade arrived. They consulted BIT’s emergency response plan and entered the server room in full gear, initially checking for casualties and confirming whether fire or smoke remained.
After their initial inspection, BIT staff were permitted inside five minutes later. However, due to low oxygen levels, they could only enter with breathing apparatus—a procedure they’d been specially trained for. The team then assessed material damage and searched for the source of the incident.
I informed my colleague that the incident had been logged on our site around the time the first BIT employee was allowed in. After summarising events, he too headed to BIT.
The suppression system had briefly lowered the room’s temperature and created a pressure wave and sound wave, causing a limited number of hard drives to fail. BIT’s data centre managers catalogued the affected equipment, while support staff began calling impacted customers from 2:20 AM to notify them (where unaware) and discuss whether remote hands were needed.
Most clients authorised BIT to reboot their systems, restoring functionality. Some required more extensive repairs, including rebuilding RAID arrays. A handful of customers opted to visit in person.
By 6:30 AM, I left for bed. Nearly all clients confirmed their systems were operational again. Most of the gas cylinders had already been disconnected, ready for refilling. Colleagues not needed on-site were briefed and would handle follow-up actions. Proud of the BIT team, I crawled under the covers 30 minutes later.
When I returned to the office by midday Thursday, it seemed like business as usual—little hinted at the night’s drama. A colleague shared that the culprit had been identified: he showed me a photo of a disassembled tape robot.
By: Wido Potters
