Security guard cost bank millions by hitting emergency Off button Welcome to a special edition of “Who, me?” The Register’s opportunity for readers to get their worst mistakes off their chests. We’re usually here on Mondays, but with the United States Independence Day making for slow news days, we decided to keep The Register’s servers red-lining by running an extra column. When we sneak in …
although technically it was being decommissioned and we wanted to see what it did...
it came off in our hand with a piece of broken string attached, and nothing, nothing at all happened.
I guess its lucky we never did need it.
IMHO most emergency buttons provide absolutely positively no information on what, exactly, they are supposed to stop too. Very, very few have ever been pushed, some not even in commissioning. Never mind putting a Perspex cover on them or putting them next to the light switch.
Well, the prevailing tradition is that they stop everything in a room by cutting off all electricity that goes in (except for the lights).
Now, if you've found big red buttons marked STOP next to the door of a server room that didn't stop everything but the lights, then indeed you've found some sneaky ones.
As to "very few have been pushed", well <grin>, we've got loads of stories here that prove you wrong ):-D.
"Well, the prevailing tradition is that they stop everything in a room"
Not in my experience I've mainly seen them next to big dangerous machines - and they only stop that machine. Well, thats what i assumed - i didnt press them.
I am reminded of an entry in the famous Evil Overlord Checklist:
If your evil lair absolutely has to have an emergency destruct button, then there should be a very large, obvious red button marked "EMERGENCY DESTRUCT". This will be linked to the trigger of a shotgun pointed at where anyone pressing the button would have to stand. The actual destruct system should be a complex series of controls hidden in the sewage processing plant systems.
One of the very first things I do when commissioning a room is to check the EPO actually works - and stays worked when you take your hand off it (yes, there are some pretty numpty installations out there)
And that it has a cover.
And that the lights stay on.
And most importantly of all, that's the bloody thing is labelled.
As for security guards - we've had a number of incidents where they've "helpfully" turned off AC in unoccupied areas. It's worth making sure that the companies are made aware in writing that interfering with any IT or AC equipment is a contract-breach offence.
we've had a number of incidents where they've "helpfully" turned off AC in unoccupied areas
I once helped set up a computer room in Paris - full of expensive Solaris kit and some Windows servers. Everything was set up and working correctly after about 12 hours install time[1] so we turned off the lights, locked the doors and went back to the hotel for some beers.
About 5am all our pagers[2] went off with lots of alerts from the kit in the new computer room - all 'overheat protection' errors.
We hustled in and discovered (to our horror) that the building management system was configured to disable the power to *all* the AC in the building at 10pm every night in order to save power. Including our shiny new server room - which was now sitting at somewhere like 70C ambient temperature and the servers were recoding mainboard temperatures well above 100C.
All the PC kit (Compaq servers mostly) was utterly dead with mainboards fried. Only one of the Solaris boxes had died (from memory the SCSI controller had eaten itself) - the rest recovered after the temperature came down and we restarted them. We'd lost a couple of drives too but since everything was RAID5 we hadn't lost any data.
Taught me to be paranoid about server room aircon.
[1] With no lunch break! Well - we had about 15 minutes to eat some sandwiches but that was about it. Which led to some vociferous complaints from the French engineers..
[2] Which shows you how long ago this was..
> Taught me to be paranoid about server room aircon.
Similar thing happened where I used to work. Nothing got fried, but the engineer working on the AC in the datacentre forgot to put the fire suppression system into safe mode. When he switched off the AC, the rise in temperature triggered the gas.
"And most importantly of all, that's the bloody thing is labelled."
At my place of work, the emergency stop buttons are yellow and red and clearly labelled emergency stop. The ones on the machines are obvious, stops that machine. The ones on the wall in weird places... your guess is as good as mine. It's not worth my job to push them to find out what, if anything, actually stops.
DAVROS: Press it, and you will destroy this bunker and everything in it. Only this room will remain.
https://i0.wp.com/tggeeks.com/wp-content/uploads/2018/06/MV5BMThhNjgwNWYtMWU3Yi00OTRmLTk5ZjAtYThhMTk2NmM1NGIxXkEyXkFqcGdeQXVyMjMzMzk2Mjg@._V1_.jpg?resize=768%2C432&ssl=1
In the workshops i've dealt with, the machine ones, are, indeed, local to that machine. The random ones round the wall are usually wired to kill power to every machine within that area. They're for you to use when someone else is getting pulled into a lathe etc...
And as for never being tested? In a computer machine room I can see why... in a mechanical workshop, test them daily at LEAST
I once worked for a Very Small company in which the owner was slowly going mad from the stress of running his company into the ground. One day, after receiving some mildly annoying news, he picked up a wood spitting maul and whacked both E-Stop switches off the solder machine. After the splitting maul was put back in it's place and the owner was far enough away from the place, I calmly asked him why he broke the solder machine. He said "What?". I told him he broke the solder machine by whacking the E-Stops. His response was "I didn't know we needed those".
AC because I don't know where the splitting maul is now....
The pull out was only on the earlier S/38 and in this case the switch behind it was obviously removed. It should drop system power and you have to remove the console cover to reset the switch. Later S/38 had a switch on the side that you had to press in on the level to push it down to activate the EPO.
Sometimes they do not work. I once had to install a series of Dell servers plus a Mac server for the company's graphics artists. One of the requirements was "A great big Emergency Power Off button please", but realising that this is actually an invite to disaster (if it is there then someone will push it in error) I disconnected it from the server. So, nice big shiny red button with a cover over it, warning notice pasted on the wall above and trunking running down the wall to the power supply. In reality the EPO was a switch outside the server room right beside the Halon release button.
Back in the early 80's I was working for a major bank. We had 2 identical IBM System 370/145s. Our production system ran the on-line system for every branch in the state (the second system was always powered up and usually running dev workloads). Fortunately we swapped systems each day to ensure that both were fully operational and capable of running the production system.
We had a IBM customer engineer (CE) on site most of the time. One day the usual engineer bought in a new trainee to introduce to the staff. When the normal CE went on holidays the newbie was to take over for a couple of weeks.
On the first day the new CE was running solo he was walking past the production system, looked at the "Big Red Switch" (actually white with red lettering) that was at the top right hand corner of the main console labelled "Emergency Pull", commented "what's that doing in?" and, before anyone could stop him, he reached up and pulled the switch. The role of the "Emergency Pull" was to cut all power to the system immediately. It did this by tripping every circuit breaker in the system. The room went into "panic mode". We knew we had 15 minutes before the phones would start ringing off the hook (the branches had instructions to wait 15 minutes before calling the computer centre in case of an outage). All jobs on the back up system were cancelled and job queues flushed. The "on-line" system was bought up on the back up system as fast as we could and service restored just before the 15 minute deadline.
The CE was marched out the door and told to never return. Our IBM rep was called to reinforce the order. Another CE who had experience with our site was called in and spent the next 2 days cursing his colleague as he worked to get the system back up.
"I seem to remember having read somewhere that astronauts were actually explicitly tested for it"
And submarine crews. In fact, one of my supervisors at U who had been involved in designing the tests remarked that submarine crews were possibly the best adjusted set of people in the country. In later life, working with ex-submariners, I had occasion to witness this.
The Russians obviously did the same which is why WW3 didn't start when the US started depth-charging a Russian nuclear sub.
At one time I worked for the largest brewery in the US, who was facing a problem in that there was a "bubble" of a large number of brewery staff coming up for retirement. How to pass on a lifetime of knowledge to their replacements?
First hired a group of university academics to put together a program. They managed to thoroughly antagonize and alienate the brewery staff with an attitude that bordered on contempt.
Based on past experience, my VP brought on board a group of ex-nuclear sub men. They respected what the workers knew, put together a program where the workers taught their skills to the next generation and all worked out very well.
And we got to brag - rightly or wrongly - that we had the world's fourth largest nuclear navy, albeit with no actual subs.
I would never describe that group as having good impulse control or being well adjusted. But they were a great group to know and work with.
At one time I worked for the largest brewery in the US, who was facing a problem in that there was a "bubble" of a large number of brewery staff coming up for retirement. How to pass on a lifetime of knowledge to their replacements?
I'm moderately impressed that they noticed that they might have an issue. Most places I've been just sack their older staff, then wonder why things start going wrong on a regular basis.
Virtual Machine AIMentoring to Perfectly Monitored Presentations .... with ESPecial Deliveries for Open Coded Future Supply of AIMentoring Virtual Machines ..... The RAT AI Access to Virtually Enterprising LOVErs for Heavenly AIdVenturing, in a/the Most Immaculate Greater IntelAIgent Game Play for Command and Control in Stellar Command when Almighty Fake State Powers be Exploding and Imploding.
As the Answer to Everything Revealed be of Alien Instruction in Ab Fab Fabless Labs just like the Ones that Go Hunting and Exploring for Real Wise Bots Proven Unassailable in Training with Special Force Sources.
? What Drivers urWorld ? Is IT for Real? LOVE is AIMasterKeyDefended ....for Live Operational Virtual Environments Provided for Free with All Costs Priced for Payment by Systems Administrations Are there any more More Secure Top Secret Future Channels Providers Operating these Spaces in Grand Revelation of the Core AI Source Driver Keys.
Is any or all of that not hereby Proven Fact for Future Possible Missions with the Great Virtual AIMachine for Key Keenly Mastered Position for Trading as AIDerivative Future Offerings?
Yeah, it sure is that. Thanks for that. You are Proven Vital for Future AIDiscovery Channels Use with Adequate Maintenance and Exercising in Private Accommodations .... :-) Pirate Dens/Alienating Spaces ...... for Fail Safe Guaranteed COSMIC Performance.
What price that Offering? Do the Markets do AIDerivative Future Offerings yet?
The CE was marched out the door and told to never return
We had an IBM operator that very nearly lost his job when, after a long liquid lunch, he managed to purge all the spool space instead of purging a new DASD that had just been brough online.
He got moved out of operations and told that, if he ever came into work smelling of drink again, he'd be out of the door so fast that he'd break the sound barrier..
Found it was some sort of D-Stink P-O-S which somebody's using as a switch.
Yep. Been there, had that. Also had someone set up a port switch as a port mirror for the main LAN switch during some testing and then forget to remove it afterwards (or label it properly). Which made for great fun when we plugged a DSL modem into that port and brought down[1] our main LAN..
[1] Well, theoretically it was still working but was trying to stuff 10GBs via a 10/100 connection of a 7Mbit DSL line which meant our main LAN spent most of its time waiting for the switch to clear its buffers..
I had a site recently that had a power outage.
Afterwards their server "went weird".
After hours of investigation turned out that there were TWO home grade routers acting as switches hidden on the site. Sure enough they had been configured by a predecessor to not act as DHCP servers and only dumb switches. After the power went out, they both booted up with a factory reset and started handing out IP addresses (192.168.x.x & 10.0.0.x) while the ACTUAL DHCP server was trying to hand out another range altogether and got quite sulky about the whole situation.
I murdered the two routers and restart the DHCP service and everything calms down again...
I think the idea of having the EPO by the door is so that you can bang the EPO as you flee the burning room. Now when you couple that with electric door locks interesting things may happen.
One instance I recall was a courier was delivering parts to a computer room that had powered sliding doors the operator let the driver into the room to drop off the parts, the driver then asks the distracted operator how to get back out and the operator utters the fatal words "Just press the button by the door"..... room goes quiet. Customer attempted to blame the hardware vendor that was having the parts delivered for the outage.
Another one happened at one of by customers, following and narrowly averted disaster our branch manager decided to visit and was being shown around the room after being shown around and he was ready to exit the operator tells him to press the button by the door and then turns in time to see the branch manager reaching for the EPO and yells "Not that button!" The manager had been a large system service rep himself before becoming a manger and should have known what th big red button with "Emergency" in white letters was for.
Where I'm working now the fire alarm disables all the electronic door locks. If the first fire drill i experienced had been a real fire we'd have been in serious risk as we all headed for the front door then had to turn round and head out through the basement. At least now there's a little sign on the special button that opens the door when the alarms on. Fucking idiotic system.
Thinking about it, par for the course for their software too....
Not sure which of them should be fired ...
But surely not the security guard scapegoated in the first story. When you smell and see fire, you don't hesitate, you use the emergency button to shut down kit that could turn it into something much, much bigger and altogether more catastrophic.
@Nick - I came to say the exact same thing. What if he hadn't pushed the emergency stop and the entire place had burned down?
And generally in what appears to be an emergency with smoke, fumes and flames, people don't tend to stop and have a quick chat first - they act.
No way that poor chap should've been fired.
"it's just common sense to ask ...
That's the common sense that was applied at Chernobyl."
Actually... no. It's a lot worse than that.
1 there were some Big Red Buttons on various consoles
2 no-one, I repeat, no-one in the building had any idea what the hell they did
3 none of the available documentation said what they did, either
4 every single staff engineer had been literally trained on the job by those who came before them
5 management decided that it would be a good idea to find out what those buttons did and scheduled 'training' on one reactor which was not in use
6 they found out what the buttons did. Ooops.
See 'Ablaze' by Piers Paul Read for more detail.
Actually... no. While what they were doing was not particularly advisable, it is established fact that the inherent and extremely dangerous instability of reactors of that design at very low power levels was not known at the time, and certainly wasn't known to anyone working at the plant. You might want to do some more reading...
" inherent and extremely dangerous instability of reactors of that design at very low power levels "
Virtually all light water reactors are unstable and have a nasty positive void coefficient at very low power levels, snapping into prompt-criticality at the drop of a hat, which is why this kind of operation is specifically avoided (It was the resulting steam explosion which blew the roof off at Chernobyl). The russians had a number of those in the 1950s and the americans deliberately experimented with them after the SL-1 accident to understand how they happen.
It's a fundamental flaw of using water as the moderator. If the operators didn't know this, then soviet operating procedures were incredibly bad, as prompt criticality risks were well documented on both sides of the Iron Curtain
"If the operators didn't know this, then soviet operating procedures were incredibly bad,"
I've worked with Russian scientists. "Safety" is a word that they haven't even heard of. Truck load of science equipment arrives, how to get it off the truck, get the driver to drive back and forth jerkily until it falls off the back. Need to move a 6 tonne telescope, everyone grab a bit of the frame and lift.
I come in after crashing my bike, coz a big lump of wood fell off the truck in front of me, I'm dripping blood all over their carpet, only one of them is there, slowly eating his sandwich. I ask where the first aid kit is, he slowly finishes chewing his current mouthful, swallows, then looks up at me and says "I'm still eating lunch.", then takes another bite and ignores me.
>When you smell and see fire, you don't hesitate
Actually, you should take time to assess the situation - otherwise you risk putting yourself and others in even more danger.
I recall a visit to a site, where I was in the main equipment room and smelled burning, and noticed wisps of smoke near the ceiling. I was about to go into full Corporal Jones mode, when I realised that none of the regular staff seemed bothered. A few moments and a few lungfuls of air later, and I noticed the air had a rather agricultural aroma. Then one of the staff told me that this always happened when the farmer started a fire in the field next door and the wind was blowing towards the ventilation inlets.
"Then one of the staff told me that this always happened when the farmer started a fire in the field next door and the wind was blowing towards the ventilation inlets."
There are certain chemical fertiliser processes that produce that distinctive "burning circuit board" small we are all familiar with. It can be a bit concerning when driving near one of those places and the wind is blowing "just right".
A similar thing can happen at road junctions in the UK... The new 'colour coded' road coatings used to differentiate things like pedestrian crossings, cycle areas, etc... apparently are bonded with a type of cement (maybe phenol based?) that, in warm weather, REEKS of overheated electronics.
I have, on several occasions, panicked and wondered what electronic part of my car is burning out and on a couple, stopped to investigate
That's the common sense that was applied at Chernobyl.
Chernobyl was designed without any containment, with fuel rods that are lowered in (so fail 'hot' rather than fail safe). IIRC, the engineers were running a standard load test or similar, it was failures of the design that caused the fire and meltdown.
"IIRC, the engineers were running a standard load test or similar, it was failures of the design that caused the fire and meltdown."
It was worse than that; they were testing to see how a generator failure mode was handled. In principle it was the right thing to do, but due to a combination of unforeseen circumstances the thing ran out of water.
It was not due to someone pressing the wrong red button. It was apparently due in part to people being given management jobs due to connections and suitably proletarian background rather than being competent nuclear engineers.
Substitute "aristocratic" for "proletarian" and you can see why we have major problems too, though usually rather with things like government IT projects and military programs rather than nuclear reactors. I guess you could call it the Pyotr Principle.
@Voyna i Mor
I've always been taught that water acted as a moderator (can't remember if that's the exact term) to slow the particles down so if they'd let the water boil away and hadn't added more, the disaster would quite literally have run out of steam before the reactor could explode.
That's why they don't let people like you and me run nuclear reactors.
The Chernobyl (means wormwood by the way and has the same root word as black or devil) reactor had graphite moderator, like the infamous Windscale reactor that caught fire. It was water cooled. When the cooling water went low, the temperature built up until things literally melted down.
The reason for the graphite moderator was that this design produces plutonium for bombs, like the original Chicago reactor.
It's a little bit like those criminals who used to make their own nitroglycerine over a kitchen stove, not all of whom survived. Some things are just so dumb godawful stupid you wonder how intelligent people could get involved. And then you realise that there was an injection of politician into the mix.
The effect of removing water from a reactor core depends on its void coefficient. Briefly, the cooling water can both slow neutrons (increasing reactivity) and absorb/remove them (reducing reactivity). The overall effect is a function of reactor design.
RBMK had a positive void coefficient, meaning water loss increased reactivity.
ArrZarr, do some Googling on Chernobyl. It is the opposite with the water. The RBMK reactor has a positive void coefficient. When the water boiled and steam bubbles appeared this caused an INCREASE in power output from the reactor. I also dont believe that the operators added any more water - indeed how can you add more water in what is highly likely to be a closed loop system.
That's the common sense that was applied at Chernobyl.
More accurately the common sense that was applied for Piper Alpha
Turn off the automatic sea water fire pumps because divers might be injured - health and safety you know
Have other rigs feed masses of high pressure gas into the blazing rig because they hadn't been ordered by senior management in Houston to stop production - you never want to stop production
I think you mean control rods, not fuel rods.
James Mahaffey's book "Atomic Accidents" goes into the deficiencies of the RBMK reactor design and the personnel running it in some detail. ISTR he says somthing like its graphite-moderated, water-cooled design was unique in combining the potential for a steam explosion AND a graphite fire.
Walk into an empty room and smell burning, you hit the emergency button. Walk into a room where people are working, and see that they haven't done it, it's just common sense to ask "why not" before taking extreme action.
Not necessarily. people on a tight deadline are generally poor judges of risk. I have been in situations of having to forcible drag people from test rooms after a fire alarm has gone off because "they just needed to wait for the test to complete"
Another scenario would be that a fire/smoke was detected, the security guard came to sort the issue out, told to go away and the fire spreading and causing greater damage or loss of life. Yes it was very sad that the bank lost a few hours processing, but this needs to be balanced against the risk of loss of life.
It sounds like the security guard was used as a scapegoat because they were non-essential and cheap
Different professions will give different answers for this situation. I happen to be from an IT background, but spent some time doing security and safety audits, and had a chance to talk with many security guards. One thing they are taught, for good reason, is that human life comes first, everything else is secondary. They also know very well that fires in datacenters spread quicker than most people realize, and that some of the equipment can release very toxic smoke even before the big fires start blazing. We worry about equipment and downtimes, because that's usually the worst we had to deal with. Some of those guys witnessed first-hand people injured or dying because fire or toxic smoke inhalation. And they also know that a lot of us, the geeks, have very little understanding of the dangers, that's why they are compelled to step in firmly - often to save us from our own mistakes.
I think you need to put this i context.
However annoying it was for the team, the bank and its customers., no one was injured or killed, the system was brought up OK.
Yeah maybe a few million virtual pounds were lost, but I sure the bank recovered that in a few minutes of trading or reduced their interest rate by 0.000001%
The question is next time an event occurs will the staff hesitate in fear they might be blamed?
"When you smell and see fire, you don't hesitate"
Smoke is not fire. I didn't see any mention of flames - or even smoke, just a loud noise and a bad smell.
When everyone else int he room is screaming at you not to do something, you should probably be pausing to think "Should I really be pressing that button?"
Computer power supplies are in steel boxes - and furthermore, those are inside further steel boxes before (usually) being in a steel rack. If one burns up, it's contained.
Smoke is annoying and messy, but it's not fire and doesn't spread fire.
You only want the EPO or a gas dump if there's an actual _uncontained_ FIRE. The factor of damage of disk damage from the noise (or permanent hearing damage for meatsacks) along with the very real risk of injuries from flying floor tiles should be borne in mind on the latter, on top of the £1000 per bottle cost of Inergen - and a small data room (5*3 metre) room uses 5 of them.
>When you smell and see fire, you don't hesitate
Actually, you should take time to assess the situation - otherwise you risk putting yourself and others in even more danger.
Would you empty a fire extinguisher on a candle? Even if there was someone standing behind it about to make a wish? No, you would assess the situation first.
A number of years ago I worked in IT support for a local council authority. A new guy started (working for one of the systems guys next door) and immediately set a bad impression by asking how he could get copies of some of the software we had.
After a few days, he was tasked with installing some monitoring software for the UPS systems in our server racks. A short while later, PowerChute broadcast messages started being sent out to every PC in the building warning of a system shutdown. New guy insists to my boss that these are in error, nothing is being shut down, it's just a test of the messaging system. My boss is still annoyed about the fact that these broadcasts are being sent every few minutes to all 400+ machines on the domain.
Ten minutes later however, we have a strangely quiet rack, plus a lot of phones ringing due to our mail system, file server, plus several other systems all being down. My boss is fuming, new guy is suddenly very quiet, and about 10 minutes later the new guy is suddenly very unemployed.
Boss had SSH'd in to a mac mini server somewhere in Switzerland on behalf of a client. It needed rebooting for some patch, or something.
Boss was obviously distracted, as he typed...
sudo
shutdown
-h
now
As his finger was reaching for the <return> key, and think cliched "hollywood slow motion", I shouted Noooooooooooooo!
Only for him to actually press it...
We eventually found a cleaner in the building, after a few hours, to hit the power button on the back...
Sometimes IPMI is the problem.
So a long time ago, it was one of those 3am moments in a well-known datacentre, working for a well-known and large ISP. Being the era of unstoppable growth, we had to install some new muxes, routers and power to comms racks. All planned, sysadmins informed that we were going to power off a rack, and encouraging them to attend, just in case.. Which they declined.
So one rack contained a Livingston Portmaster. A 16 port tty box that let you telnet through it and connect to the consoles of the servers attached to it. And being in our comms rack, one of the boxes we'd informed would be powered off.
So snag was.. when they were powered back on, they sent a break to all attached ports, which sent a break to all attached servers, which halted them. So mail and DNS servers stopped, as did a box labelled "x.root-servers.net". Which was a bit embarrassing, but there are 13 in that coven, so was all good. The on-call sysadmin did get woken up by a lot of outage notifications though, and a bollocking the next day for not attending.
But I've heard this story from three separate people at different times, so I believe it's true.
A man starts a job at a large crisp manufacturer in the UK. Let's call them Strollers. At the time (I don't know if they still do it, because of this) all new recruits in the head office were sent to one of the manufacturing sites to see how the products were made, and what their core business was about.
They're taken through the site induction, given security passes etc, then given a tour of the offices, where they meet and greet all the local staff, before suffering a morning full of the usual corporate nonsense and lectures. Then, just before lunch, they're told that they're visiting the plant.
So they get kitted out in the relevant PPE, and *stroll* into the building. As they go in the entrance, someone in a hi-vis waistcoat explains the safety procedures, and what happens if the fire alarm sounds.
Tired after a morning full of someone else talking, and no doubt looking forward to lunch, our hero gazes off into the distance and stops listening. Somewhere along the line, he hears the words "Press the red button". Looking to his left, he sees such a red button. Pointing to it, he asks "Like this one?"
"Yes" comes the reply.
He pushes the button. There's the slow inevitable sounds of EVERYTHING turning off. Fans stop whirring, conveyor belts stop moving, ovens stop baking etc...
Then there's uncomfortable silence.
"Why did you press that?"
"Because you told me to."
"I said, 'If there's an emergency, press the red button'"...
Yes, our hero shut down *Strollers* Crisps manufacturing plant for a couple of days, as once the machines are down, everything has to be drained, cleaned, and made ready to start up again.
*names changed to protect the guilty
Training a group of volunteers in how to check over CO2 inflation life jackets:
Me: 'Check that the green clip is present - if you want to be doubly sure all is well DON'T pull the red toggle, we'll take the cylinder off. first.....
Sound effect of dramatic hiss and inflating jacket
Student: 'Oh - did you say don't?'
me: 'Well now you know what happens when you pull the red toggle'
"Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry." ― Terry Pratchett, Thief of Time
Obligatory request for a Sir Pterry icon.
I think that sending Foul Ole Rons smell over should be enough to convince them, or we could try some headology!
You must NOT under any circumstance add a Pterry icon!
and for the small minority of you who don't get the Pterry bit I suggest reading up on Pyramids.
I was working for an organisation that was having a new building designed. Unfortunately the architect was an egocentric &%^$ who designed a building that was totally at odds with our requirements. To demonstrate that the building was unfit for purpose I had to build and render a 3D model of it. In those far off days it could take 12 hours to render and plot an A2 drawing. This drawing had to be completed by 11am the following day, so our CEO could take it to London and show it to our very, VIP President. When I finally left work that night the rendering had completed and the plotting had begun...
You guessed, the cleaner came in and turned off the PC and the plotter at the mains, despite the fact that the plug switch was taped open with yellow and black tape with a large warning sign above saying DO NOT SWITCH OFF!!! End result was obviously initial panic, followed by a painful replotting process and a motorbike courier racing up the A3 just in time to get the drawing to the meeting before it concluded. We never did see the cleaner again.
I once worked in an office, when suddenly the power went off. That was in the days when unsaved work was lost (nowadays a Mac just reboots when it comes back and everything is back). An electrician comes into the room to inform us that he turned the power off. Well, no need to tell us, we knew the power was off.
@ gnasher729
A similar thing happened here. We share a large industrial building with a ball bearing factory (I know, it sounds like a line form Hogan's Hero's!) A while back we had a sudden power outage. Our data center has good UPS backup power; enough for about two hours for all of the systems including cooling.
Senior management called the power company, and they said they would investigate, but that it would likely be the end of the day before they had an answer. So, management sent almost everyone home (almost, meaning I had to stay to shut down all of the production systems when we got close to UPS exhaustion).
So, I had just finished shutting everything down, when my boss and another VP came by and said "let's take a walk next door, and see if they know anything that we don't?" As we are walking around the side of the building, an electrician comes out of a side door. We stop him and ask him what he knows about any of this. He suddenly has a horrified look on his face. He then makes some half-assed excuse about having to check something, gets in his truck, and literally leaves long black skid-marks across the parking lot.
It turns out that the stupid jackass had turned power off to the whole building! There are four very large electrical boxes at the end of the building. Two are for us, and two for the neighbor. None of them are marked, of course. We turned them all back on, since if someone was actually working on something, they would be properly locked-out, right? Needless to say, our boxes are all padlocked now.
Our CEO sent the bill for all of the lost work to the company next door. He suggested they pass it on to the electrician's insurance company.
It took me the rest of the day to get all of our systems back up and running.
In a previous job, our sysadmin was putting some infrequently used stuff back on a tall shelf. He was standing on a stepladder to do so and overreached himself. Feeling the ladder wobble he grabbed at anything handy to keep his balance, and unfortunately what he caught was the lever for the main power breaker, placed nice and high up the wall where no-one could bump it accidentally.
The red button was safely placed way across the room beside the door...
A number of decades ago, I was working on a bit of telco kit with a proprietary operating system (admittedly in a dev environment), and ran an application I was developing, from the system terminal. I realised there was a problem with the application and went to kill it with CTRL-C. Hit CTRL-X instead. That initiated a system restart. Which took a long time, back in the late 80s, with disk drives more like gyroscopes.
I did the only thing I could and went for lunch.
Reminds me of the ABC (Australian Broadcasting Corporation, Aussie government media) and a fun incident in their Darwin studio. They had some construction workers in one part of the studio doing something or other. During the evening news, which was broadcast live, the lights went dark, but the cameras, microphones, and transmission equipment kept working. The entire city clearly heard one of these workers yelling out "Argh, the lights are fucked now!".
A friend of mine marched into City Hall of a major US city and dramatically turned off the power to their mainframe without going through any shutdown procedures. The operators were somewhat stunned.
He would have been in a considerable amount of trouble were he not a member of the police department's computer investigations team investigating and later proving a citywide fraud.
We'd been talking to a London hospital some years back about a data collection system to streamline updating bed availability and allocation. We'd been given the Big Tour and seen all the wards with serious amounts of boxen and blinken lighten.
We were doing the handshake bit at the end, standing near the door of a perfectly ordinary ward; no boxen, no blinken lighten, just rows of oldish guys in beds. While I was looking, one of the guys at the far end leaped out of bed and started loudly declaiming the Ode to the Haggis. One by one the other old guys joined in until almost the whole ward was doing it.
I asked our guide what that had all been about and was it some kind of psychiatric ward.
"No" he said. "Its the Serious Burns Unit."
-
Sorry.
Things had been going pretty much as expected, then the one managing the process said to me "you should get ready to press that red button, just in case", immediately followed by "no, press it now!". So I did. It produced quite impressive results.
I'd barely let go before the crash team had burst through the door with a variety of equipment, immediately doubling the room's head count, and set about improving the oxygen intake of a recently arrived, worryingly blue-grey infant.
More of an "on" than an "off" button, fortunately.
recently arrived, worryingly blue-grey infant
That was me <mumble> years ago - well before red buttons were invented. A combination of Rhesus incompatibility (mother was O-, I'm O+ and this was well before RH+ injections) and having the umbilical cord wrapped 3 times around my neck..
"So I did. It produced quite impressive results."
Surprised the button wasn't blue instead of read, since it seems you triggered what many hospitals term a "Code Blue," meaning a life-threatening emergency, usually a cardiac arrest (in your case, it was respiratory arrest and cyanosis on a newborn). Given the promptness of the reply, the alarm was probably local to the ward you were in because a crash team was already in place.
Indeed, they were in fact quite handily stationed in an adjacent room, probably doing little but waiting for someone to press any of the emergency buttons in any of the birthing rooms. I'm pretty sure the button was red, though.
[Ob: isn't the NHS wonderful, 70 years today, etc etc]
Whilst working for a previous employer, me and my PFYTS'er at the time were in Datacentre one of a large company who manufactured and sold liquid refreshments. The comany had 2 identical datacentres, 1 in each of their offices at aiehter side of a large City.
Their opperation ran on an AS/400 setup. We were on site installing their first Windows NT Servers, with 2 DC's and 4 servers running multi port RS232 cards connected to a new batch of 48 Rack Mounted US Robotics modems, as a remote access solution for the sales people, to get into the AS/400 whilst out and about.
We were working away, and it was the first time the PFYTS'er had been in a data centre or even seen and AS/400 I was very strict on him not touching anything bnut what we were working on.
We get the servers installed, but meantime and IBM CE was on site carrying out maintenance to the AS/400. Hes being cocky and mouthing off that our "pisshy wee windows boxes are shite" and such other delights.
He finishes his work and steps back with and exlamation of "job Done time to hit the pub"
The room goes black and silent apart from fans and disk on spin down or clunking stops.
I whisper to the PFYTS'er "what di dyou touch?"
"nothing" he replies
Emergency lighting comes on and its immediatly apparent that MR IBM'er has stepped back into the unprotected stop button.
the site guys arrrive and go trhough the statup Procedure.
Whats do you know our "Pisshy wee windows boxes" startup and run perfectly when power on.
The AS/400 not so much.
2 weeks, a few disk enclosures, a couple of processors, some disks, a system board, and a fair bit of RAM later the AS/400 on this site is back to full fucntionality.
During this time we had replicated our Windows RAS solution into the 2nd DC.
Never swa the IBM'er again after that day.
And within 5 years the company was fully windows.
Cutting budget on training is what cost the bank millions, whither it was by the bank itself or a sub-contracted security firm(more than likely).
It's not the Security Guards fault that he sees an electrical problem and goes to kill the power supply. Firing his boss, yes I can see why, but I see no reason for him to be fired unless he'd been warned already. Given a warning yes but it's hardly his fault he wasn't trained properly.
it's commonplace to fire the company and give the contract to a new one.
Who quite often end up hiring the same security guard
Certainly my experience in firing cleaning companies for not actually cleaning anything - you end up with a new company that employs the same cleaner
Emergency Power Off, Normally Closed, and Normally Open.
At the company where I worked (in the last two years) the data center once went Very Quiet (tm) and it took a while to figure out. The EPO button was properly behind a plastic guard and no one was within five feet of it when things went Very Quiet. It developed that the EPO button was a normally-closed button (think of a light switch where the light is always on). A break in the circuit would cause the EPO to engage.
Yeah, in decades of preventative maintenance no one had ever removed the Big Red Button from the wall to check to assure its terminals were still tight. Years of people walking past and slight vibrations had loosened the terminals so that the next vibration momentarily broke the circuit and down everything went.
The electricians said they had seen the Big Red Button wired both as NC or NO and it was our choice. We had it rewired as NO so you had to push the button to engage the emergency power off function.
EPOs tend to be NC. If you truly need an *emergency* stop: as in: "this function must do nothing for years and years, but absolutely must work in an actual emergency to prevent people from losing important body parts", then you go with a NC circuit. The alternative is that the terminals work lose, wires get cut, E-stop power supplies fail, connectors aren't properly terminated at the factory, etc. and the emergency stop will not work when called upon.
For systems where an accident would leave people looking into the resulting crater you normally have NC - so that any wiring break will trigger the alarm.
One place I worked had a continually sounding alarm, a bell every 2 seconds so that you knew the alarm was working. If the alarm stopped - you ran
I've seen much the same NC/NO issue with the door release circuits. It's important to get it right.
Two of my colleagues were in a small data centre in Zurich when a fire alarm test resulted in the halon (it was in the 90's) pre-release alarm sounding. Instead of the alarm condition releasing the door lock it disable the door release button and they couldn't get out. Whilst it turned out they were not in danger they just about crapped themselves. Had it been a real emergency they would probably have died trapped in the room.
Early AS/400 systems where in 19" racks that each had a recessed EPO switch and they where all tied together so that any one of them would power off the entire system., and most of these system where in rooms with a raised floor. One of my co-workers had a customer who complained that the system was powering off when they walked by it, and at first we did not believe it, but my colleague saw it happen, so an outage was scheduled to investigate, and I went onsite to look for the problem. In this room the floor would shift slightly when you walked across it and it was enough that when you walked past the AS/400 the tops of the rack would lightly contact each other. The switch behind the EPO was normally closed toggle switch with solder lugs on it and heat shrink over the connection. On the EPO that was causing the problem the wires where never soldered and just had heat shrink over them and walking by caused enough vibration to cause a momentary break in the connection.
3 classic WTFs recently where I work:
- A new engineer pressing the EPO rather than the door release button. Failover to the standby DC - failed (Is that one WTF or two?) .
- An email mistakenly sent to a global distribution list, causing an email storm of "please remove me from this mailing list" and "stop clicking reply-all!" emails.
- An uncustomised email signature with "Your Name" left in it.
And one with a modern twist:
- A company email requiring staff to take GDPR training, but sent to staff by an external trainer who'd been given our email addresses, personnel numbers, full names & managers names.
Latest WTF moment in our company is HR and Marketing sending out their regular company-wide emails with no text, just an image of their pretty A4 full colour + photos and graphics missives as a single inline image file. Apart from being impossible to read on a phone screen by all of us out on the road, they keeps triggering spam filters. The solution was to whitelist them, not call the instigators ignorant dicks (which was our preferred fix)
Just last week a council jobsworth decided the best way to shut down a music event held in the market square was to pull the plug on the generator powering everything. Apparently someone had complained about the noise (for an event that happens every year) at about 6pm.
End result, several thousand pounds worth of damage to the PA system that was running all the sound.
Lucky he didn't get an impromptu arc welding lesson; I assume this was a CeeForm, not Cam \ Powerlock?
Pulling a fully loaded neutral out of a set of Cam locks is a fuck up I'm pleased only to have read about, not tried, however even a 63/3 isn't rated for disconnection under load.
what is the reason for so many pieces of hardware failing when the power is cut?
They simply don't like a change of state*. Keep 'em unpowered or keep 'em powered and they'll last for years, but flick backwards and forwards between the two every now and again and they'll be scrap by the week after next.
*This is the wisdom of decades. I'm clever, me.
"*This is the wisdom of decades. I'm clever, me."
Many years ago a directive was issued to save energy by powering off users' video terminals at the end of the working day. Very soon the failure rate for said terminals caused a reversal of the directive.
More recently one of my friends rang to say his desktop gaming PC wouldn't power up. I asked him to try again with the psu mains switch and watch for which lights came on. After a few seconds he said "A big blue flash!". He had impatiently rocked the psu power switch on/off several times in quick succession. When the PC was stripped down - the psu fuse had vaporised and disks, cards, floppy etc were dead. Only the cpu fan still worked.
The majority of electrical/electronic equipment has a 'wound component' (Inductor) somewhere in its mains input circuitry.
When current ceases to flow through said item, the magnetic field collapses rapidly, producing an EMF (the same principle is used in old tellys and monitors to produce 25KV for the CRT). So, if the power supply to the equipment is cut off the resulting EMF is fed back into the 'outside world' via the AC mains supply lead.
Now one piece of equipment on its own probably wouldn't do much harm, but when you've got a room chock full of 'em, the effect tends to add up... if not multiply :)
Another reason is electrolytic capacitors, even 'computer grade' ones, tend to dry out, and their effective internal resistance goes up, meaning that they are less able to cope with sudden surges - it's a double whammy, really.
p.s. I'm so pleased that after donkey's years of reading the register, I've finally found a question that I can answer.
In a job so far in the past even my linkedin account has no space for it, I worked in a bakery.
The big red button I happened to push was on the doughnut (or donut for our celebrating colonial cousins) machine - stick freshly deep fried doughy lumps of sugary loveliness on (very sharp) pointy bits on the front, press red button, inject jam into said doughnuts. Remove and repeat. Very boring and monotonous until lack of concentration makes the job boring (into your hands) and mainlining (jam) - (I told you the pointy bits were very sharp) .
Could have been worse - at least it wasn't a lemon curd run....
Once, In the midst of a rather bad case of morning grogginess, I spaced out and ran dhclient on a shared web hosting server. Instant network deconfiguration, and for lack of IPMI or similar on the box, a reboot ticket to the datacenter... fortunately, no major consequences for anyone involved.
Why would this even possibly happen? I've been using Linux for many years, but I'm often two lazy to set up my networking stuff statically or make sure it happens automatically, so running dhclient after a reboot had become a part of my personal routine.
The best part was the response from a higher level systems administrator to a trouble ticket I submitted, which detailed the situation thoroughly, and suggested to remove dhclient from our many servers, as it is yet another 'big red button' waiting to happen. The response was basically a whiny remark along the lines of "just don't run the command!", despite that having been the entire subject and general tone of my ticket. This apparently resulted in them getting "talked to".
...the story about the clueless VP who, while he was conducting a tour of the data center where all the SWIFT and bond trading computers were located, concluded the visit by hitting the Big Red Switch located near the door. Apparently this guy thought that the bank's data center was so high-tech--and that his tour group would be duly impressed--that the doors would `magically' open when he hit that button. Instead of the doors making a `swoosh' sound (a la the Enterprise) the sudden silence probably had just the opposite effect on his guests. Still a popular tale told during team lunches.
We once had a dual cooling system failure (*really, one at a time, but the customer de-prioritized repairing the first because backups never fail, amirite?) in a mid-sized server room, and we had to keep things running for several hours while repairs were effected. This involved trucking in and lugging literally hundreds of pounds of ice into the server room in large plastic tubs (and blowing large fans onto the ice), removing and emptying the tubs when the ice melted, wash, rinse (literally), repeat. This actually functioned quite well - the servers stayed happy and we got paid overtime rates to get in a good workout.
And yes, "tons" in A/C equipment cooling ratings refers to how many tons of melting ice it replaces....
Working in an office overlooking the M1 about a mile north of J14, and one day there’s a commotion outside. Two pigeons are either fighting or being amarous, but it’s enough of a display whatever it is to attract our attention.
About five seconds later the pigeons decide to fuse themselves against the transformer of an overhead power line and there’s a bright flash!
My first concern is whether or not the lines will separate and end up showering electrical goodness across six lanes of motorway traffic. Thankfully not...
The second concern was what was I going to do for the rest of the day, as computers all started switching off due to a lack of power.
Apparently the feed for our office came from a substation fed by that pigeon-coated transformer!
Again reminds me of my life in Darwin. Where every now and then the power would flicker. Often enough for the long term residents to ignore it. Walking around the city you could sometimes see the reason why, hanging off the power lines. The black corpse of a flying fox (rather large bats), their wingspans where big enough to touch one wire while sitting on the other.
When I worked for a major hotel chain's data center, we once had a security guard push the emergency off button for no reason in particular. He said he just wanted to see what would happen (and it was strongly suspected that he was stoned at the time). The end result was similar, likely costing millions in lost revenues. On the plus side, we found out the hard way that the individual UPS units for our servers were absolute crap.
Mid 90s and very early in my career I was a lowly operator, on call for first line support. One morning in the early hours, my pager woke me, indicating a Pyramid minicomputer had gone down. Shortly after, another message about another Pyramid, then another, then another. Knowing that these machines were all next to each other, I assumed a power issue. I reluctantly got out of bed and made my way into the office.
Sure enough, once I arrived, the affected machines had no power. On the wall at the end of each row of machines was a switch that controlled that phase? (My terminology may not be correct, electrickery has never been my forte. But everyone used to call it a phase). The switch had a green on button, a red off button, and a light indicating its status. The light on this switch was out.
Standard procedure at this point would be for me to page the on call electrician. But given it was about 2 am, I would have to wait around for at least 30 mins for the electrician to get on site, who may just cycle the power then scowl at me wondering why I'd woke him for such a simple thing. I was also tired, young, over confident and a little laissez faire; so I decided to first just try cycling the power myself.
I pressed the green button first - nothing happened. So then, figuring it was off anyway, it maybe needed the red button to be pressed first to 'reset' things before it could be turned on. So that's what I did. At that point, things got a little quieter as another five or so minicomputers in that row which unbeknownst to me had been fine until that point, suddenly powered off gracelessly.
At this point I called the sparky. And the Pyramid team on call person. Fortunately things were up and running fairly soon and with no big issues. Whatever the power issue was it was something that needed the sparky. The reason the phase switch seemed to be off was... there was no bulb in the indicator light...
I got away pretty much unscathed because it was the early hours of the morning so little impact on business; I was on pretty good terms with everyone involved; and I think the sparkies recognised it was pretty stupid/dangerous to have no bulb in the indicator light on the power switch.
Near the end of the previous millennium I was a small part of a small part of the fire protection for a petroleum tank farm and truck filling facility. Gasoline was the major product, with diesel close behind. Not surprisingly, fire was thought of as a bad thing.
The first time I went to training I was dazzled by a few dozen red strobe lights almost illuminating the entire facility. The strobe lights were reminding the people at the facility that the automatic extinguishing system was disabled. Our unit's training involved putting out fires in tanks of gasoline and diesel and they didn't want the system to go off.
The sensors were so sensitive that at an earlier date, someone started welding in front of another facility up the road and it triggered the system. Instantly, 2000 lbs (almost 1000 Kg) of sodium bicarbonate (NaHCO3) dropped from the ceiling of the truck filling area, making five tank trucks and the entire area look like a blizzard.
Regrettably, the system had been tested and they didn't need confirmation. The employees at the tank farm then had to get another 2000 lbs of NaHCO3 from stock and dump it in bins twelve feet off the ground. That involved having a small number of people doing a great deal of work high off the ground as a tall forklift delivered the bags of soda.
My mistake is I actually believed them, those that said software was needed to keep hardware working longer and do all those things that hardware couldn't.
but instead it's more of a big fraud that just keeps being committed
some dudes try to do a good job but the rest are only in it for the $$$Money
So you can go back to the analog anytime you like
Visiting the data center of a global bank, one of UKs "big 4" I observed the big red button had a clothes beg round the shaft making the press a two-step process - remove the peg first. The IEE guys I was with frowned at this (as they did with just ONE item of electrical equipment in the battery room lacking the normal gas-tight isolation normal in that kind of environment).
On reflection no worse than having a perspex lid server the same purpose but it did make me wonder if this indicated that it was a response to an "inadvertent activation"
But I've been told similar versions of this story by two unrelated people that worked for the same bank so I think it may be true:
A large new data centre for a UK bank opened mid nineties just outside the Yorkshire town the bank was named for. It had new fangled doors with swipe cards to get through, though sometimes there was a 'push to exit' button at the side of a door to get it open.
A new cleaner was working at the site one evening and walked up to one of the secure doors. Looked at both sides for a card reader but didn't see one, though there was a big red button to one side.
Pushed the button to open the door but it was still locked? Apparently they got all the systems back up and running by close of business the next day!
One of the guys that told me the above also told me about the time they tested the fire suppression system for the archived paper storage at the same banks head office (a large office building in the middle of the town the bank is named for). They had installed a new halon suppression system in the basement, and the final test was of the extract to clear the halon if someone got caught in the area being suppressed (so they didn't suffocate). To do this they fill the area with smoke then turn on the extract to see how long it would take to get all the smoke out. They did this and the smoke cleared nice and quickly.
Walking out of the building, on the way to a liquid lunch after everything had gone well, there was lots of noise and fuss outside. It turned out the basements ventilation system was linked to the modern art statue in front of the offices, and for the last 10 minutes plumes of smoke had been poring out - obviously the fire brigade had been called.
So many years ago in a remote Australia town, I was asked to check the battery status of a large APC UPS - the likes of which I hadn't seen before at that time. (This might have been the first rack I'd seen.) I was the only permanent IT guy in town.
The only means of communication appeared to be a serial port on the back. So I figured I'd just plug my laptop into it with a serial or null modem cable and have a chat at 9600/n/8/1 if it was in the mood.
The moment I connected that cable, the UPS immediately turned off. Along with a few racks of servers. (I think it was a digital concentrator, or whatever it was that replaced analogue dialup modem racks back then.)
Nice bit of evil design there, thanks APC.
Where to start?
Those with a Novell background will recall that one used RCONSOLE to start a remote session to a Netware server, from a DOS prompt. And one closes a DOS prompt with "exit".
Whilst managing the IT for the UK's largest contributor to GDP (at the time) via an outsourcing agreement,
I get a phone call at death o'clock that the main site server has gone down. Our remote first-line Helpdesk in Birmingham was doing some low-level sysadmin work one evening, duly fired up rconsole to do the work. When he had finished, he duly typed "exit"... and then inadvertently followed the directive "you must down the server to exit" typed DOWN.
He confessed that as soon as he pressed return he realised what he had done. Fortunately, it didn't impact production, and didn't really impact our uptime SLA. The client was OK about it given that he had immediately realised his error, and owned up to it; and they were also happy that no disciplinary action was needed as one could guarantee he would never ever make that mistake again.
And no, I didn't mind 1st line staff working in servers as I had trained them beyond that level, to improve the service offering. And human error can never be fully mitigated:
A few years later, managing a telco's NMC, faced with an embarrassed looking 3rd line network engineer who has shut down a major customer's site: connected to the appropriate router, did the required config work, then shutdown the network interface. The far end interface. Which neccesitated a truck roll to restore. Again, owning up to it went a long way to making amends, together with the comment "no need to look at remedial action, you can guarantee he'll never make that mistake again"
Myself and a colleague were tasked with cabling up a newer bigger room for the Social Services Dept Server in a large County Council in UK, and then moving everything over. We installed and patched up the router, laid all the new power cables, and then it was time to move. Being short of time, we figured we could run a long cat6 cable into the server from the new rooom, then unplug the UPS and server together, with the UPS keeping the server up, carry it all over, plug the mains back in and everything would stay up.....this we did. However, the UPS was five years old, and didnt all all feel like providing power to anything once deprived of mains. Upon unplugging the UPS, the server went straight down. The ensuing chaos as all social services databases suddenly stopped responding, locking up PC's all over the county was impressive in its speed. We plugged everything back in, rebooted, and denied all knowledge, blaming it on the Telco.....
It's about ten minutes ago (this is virtual live reporting) since I can enter the premises, after I locked myself out, leaving the keys inside. With this being the first piece of reporting I see on the internet. A cock-up that set me back a nice €242,-, which I could not directly afford. (There is still poverty on this planet, how strange it might seem to all you billionaires.) So the professional burglar had a TITSUP (Total Inability To Show Uninhibited Politeness), telling me that I could know when I engaged services like his, one would have to pay? Non! Vraiment? How could I have missed that after months of careful planning of closing the door with the keys on the onside. O wait, no, I frigged up, and thanks to people like me he has a living.
In a few months time, 2 at the most, I get an inheritance, and I need to get rid of at least €5.000 before the end of the year, for insurance reasons. Any El Reg reader (or El Reg itself) coming up with a good cause to give €242,- to to counter dickishness is welcome to make a suggestion (if that's okee with El Reg-moderators...). Equal rights is a suggestion... Thanks, enjoy your weekend, and keep track of your keys, folks!
A friend of mine worked for BT and told me of an incident from about 15 years ago. As all the digital telephone exchanges are unmanned, any alarms from exchanges when things go wrong, appear on screens at a central control centre. The control engineer at the remote centre then contacts a field engineer in that area, who will then drive to the exchange and fix the problem.
At the end of the day from about 5 to 5:30 pm, a small number of night staff take over in the control centre. If any alarms appear from then on, the night staff ring the appropriate engineer at home. They do get a disturbance allowance payment if control ring at say 3 in the morning !
Sometimes, engineers will be working overtime past 5:30 pm and so will be bringing up false alarms. These are noted by the control day staff as 'engineer working' and noted by the night staff on changeover. When the engineer working overtime has finished, he will clear the alarms and inform the night staff that he is going home.
Well one particular evening when handing over to the night control staff, they were informed that X power engineer would be working overtime at the main Bristol telephone exchange in the city centre. This was a huge 8 story building with every type of telecomms equipment imaginable. Unfortunately, there was some sort of breakdown in communications between this power engineer and the central control centre. He cleared all the alarms when he finished, and went home.
About ten minutes later, some contractors, presumably also working overtime, severed a huge power cable in the road feeding the city centre, and the whole of the city centre was going to be without power for at least 24 hours. But not to worry, because the main telephone exchange had a massive generator the size of a pair of semi's, and a back up of the same size in case the first one went kaput. Just one snag, on this occasion the first generator didn't kick in, and neither did the second one ! Apparently so my friend told me, all telephone exchanges have back up batteries for just such an occurrence and although they're to the same spec as the ones used on the moon landings, they only last for an hour.
But back to the night control staff and alarms coming up all over the place. "Blimey" they thought, "X the power engineer must be working like mad to bring this lot up !" What they didn't realise was that these alarms were real ones ! Eventually when the alarms started to tell them that the back up batteries had kicked in and there were just 40 minutes left before the whole building went dead, they frantically phoned X on his mobile to ask him what he was up to. They got the shock of their lives when his wife answered and said he was in the bath.
By the time X got back to this main exchange, everything was definitely dead. Despite other power engineers rushing in from everywhere, no one could get either generator working again. Mobiles, computer links, 999 services, you name it, everything was dead as a Dodo. An emergency was declared and with managers now on site tearing their hair out, a huge mobile generator was ordered to make the journey asap from London on a massive Pickfords truck.
It eventually arrived and as it was backing into the yard, someone noticed something. It wasn't going to clear the large brick archway. According to my friend, the reaction of everyone there would be totally unrepeatable.
Tyres were deflated on the Pickfords truck but all to no avail. An urgent request went out to every local BT stores and also local electrical suppliers to see if they had any appropriate cable, two pieces, each about as thick as someone's leg and at least 100 metres long.
The mobile generator eventually roared into action and as it was by now 6 am the next day, this was no mean feat. It took another 3 hours to get the exchange up and running again but the original breakdown in communications was never satisfactorily resolved. However, my friend said that the generator problem was. It seemed that despite being regularly tested, maintained, and run, by a million to one chance a different part failed on each one at the same time that could never had been anticipated. To get those parts would have taken far longer than bringing in a mobile generator.
It was a classic case of 'Sod's Law' i.e. if something can go wrong, it will. This was a zillion to one chance because three things went wrong at the same time. The breakdown in communications; both generators failing at the same time; and the brick archway being too low.
The name of this incident has been changed to Bristol, but the city where it happened was even bigger apparently.
Not a Big Red Button incident, but Mother Nature having her way. Company's UPS was a bit dodgy, maintenance had been at it several times. A new card pass system was put in place to allow entry to the building, and the physical keys were taken back from the lower ranks. Storms hit hard, and my boss called me to report server room issues. Not surprising, since every time storms came round one of the phone links would die. Except I got there and couldn't get in, and my boss had to come out anyway, since he was one of the only ones who still had a physical key. Power was off to the entire server room. The UPS had cut in when power went out, but didn't cut back out when power came back, and when the battery bank was done, it all went dark. Including the server than ran the key-card system. The other network admin and I both got our keys back after that.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
