Network engineer chose humiliation over a night on the datacenter floor
To avoid lock-in, it helps if you remember your keys
Who, Me? Welcome once again to Monday morning at the coalface, which as Reg readers know is when we publish reader-submitted tales of tech support gone awry under the banner of Who, Me?
This week our hero, if that is the appropriate term, is a reader we'll Regomize as "Erik" who works as a network engineer and was once asked to install a set of new switches for a client that was rebuilding its network topology.
The client had several rooms in which vital networking equipment was located and had installed appropriate security for the purpose – including both physical and electronic keys. In some of the rooms, Erik found a collection of shiny new uninterruptible power supplies into which the new gear was to be plugged. In one room he found a sole, lonely, router whose purpose was unclear except that it was not part of this project particular upgrade.
To install the new stuff Erik was obviously going to have to unplug things, so an after-hours maintenance window was locked in. Erik was provided with the appropriate keys that would let him into all the right rooms to get the job.
The job went well and by around 1:00AM Erik had done all the necessary unplugging, reconnecting, and software updates.
All seemed to be working brilliantly, so he returned to the main lab, which had been his base of operations, to begin packing up.
Just then the phone rang. The client employed monitoring services to keep an eye on its network remotely from another building across town (for security reasons). And according to that remote monitor, nothing was working. At all.
Erik immediately understood what that lonely router sitting by itself in a room must be for. He reasoned that if he rebooted it, all would be well.
He therefore sprang up, left the main lab, and made his way through the empty corridors to the room where the router could be found.
And as the door closed behind him, he made a horrible realization: he had left the keys behind in the main networking lab with the rest of his gear.
He was therefore locked in the client's premises, and could not escape.
- Undergrad thought he had mastered Unix in weeks. Then he discovered rm -rf
- The sad tale of the Alpha massacre
- Relocation is a complete success – right up until the last minute
- I made this network so resilient nothing could possibly go wro...
He figured he had two choices:
- Total humiliation, in the form of waiting until morning when the client would arrive and free him;
- Partial humiliation, in the form of calling his boss in the middle of the night to hopefully ask if the client had supplied an extra set of keys.
Erik chose partial humiliation, and luckily his hunch was right. Extra keys had been provided, the boss rode to the rescue, and the client was none the wiser.
Incidentally, Erik's other hunch – that restarting the monitoring company's router would fix everything – also proved well founded. The job was done, and Erik was liberated by 2:00AM and even managed a half-decent night's sleep.
Have you ever come up with a quick-thinking solution and then realized, like Erik, that perhaps thinking a bit more slowly might have been better? That's just the kind of story we love to read. Tell us all about it in an email to Who, Me? and we may feature your exploits on some future Monday. ®