Hot, sweaty builders hosed a server – literally – leaving support with an all-night RAID repair job As two precious days of free time loom, The Register offers a two-minute taste of the energy-draining drama that makes weekends so important in another instalment of On-Call, our weekly reader-contributed tale of the messes IT people are asked to address. This week, meet "Oswald" who sent a story of pulling an all-nighter to …
The setup I inherited in my current role was that the server cupboard (it was too small to be called a room) had two heat pumps, not even proper aircon. At the time we were located in a small factory unit with office on the front so rather than do a proper job the "exterior" units for the heat pumps were installed inside the factory, just under the metal roof, above a hot plastics extruder. Of course that meant that just at the time they were most needed they couldn't cope. I did have to make an emergency trip to the garden centre for a large plant trough to catch the drips from the units!
Eventually they failed completely and were replaced with a proper aircon unit that was somewhat overspecced for the job.
With something like a cooling unit there is a very large zone before it becomes over-specced although I understand what you mean. The biggest risk with having spare capacity is that someone has a bright idea on how to make use of that excess and in no time at all you need to uprate everything.
"The biggest risk with having spare capacity is that someone has a bright idea on how to make use of that excess and in no time at all you need to uprate everything."
I'm not a mechanical engineer, but I know a couple who get paid to design server room air conditioning systems. From what they've told me, oversized air conditioners tend to short cycle (turn on, quickly overshoot the cool set point, and quickly turn off again), which shortens their life considerably.
If you insist on oversizing, they are pretty clear that they'll be happy to quote you a replacement when it dies.
Modern systems are variable rate - the compressor is inverter driven and will change speed to match demand. ISTR that's been a requirement (energy efficiency standards or something) for some years now.
But without that, ANY system which is not perfectly matched to demand will either under-perform, or will have to cycle. Only if the system is grossly over-specced will the short cycling be any issue.
I'm not a mechanical engineer either, but my immediate solution to that would be, rather than having one massive over-specced unit, to have several smaller units (maybe 5-10 of them) which can cut in individually as required, and maybe act as fail-over for each other as well. Sure, the control systems might be a bit more complex, but that's just microcontrollers and wiring, and cheap compared to the infrastructure.
our data centers have two large units for fail-over; one unit can handle the room with a full load of servers, and while the compressors will vary their speed, the blower fans are almost always full blast.
We had an incident where one of the units was set to 'soft off' by the failover controller, but in a power failure condition, both units would come back on when power was restored, resulting in a very chilly room until someone went in and reset the controller.
I once did a stint at an insurance company during a hot western European summer. At the time, wearing business suit was mandatory, even for IT. So I wore the jacket on my arm, got into my "office" (the server room), put my jacket on, buttoned it, and toiled away. And the end of the day I would take my jacket off and go home... Fortunately it was just for a pair of weeks, but it felt really awkward.
When ICL release their S39 range of mainframes the temperature window for the CPU's was very narrow, we often had to install an underfloor air dam to reduce the airflow through the CPU cabinet as they were getting too chilled and pulling errors. They had been designed for the expected DC temperature range but in a big DC the sir through the flow would often start off very cold at one end in order to keep the other end cool enough
>> insane aircon in the comms room. You need a jacket
Reminds me of a server room at one site that installed a nice Mitsubishi AC unit right next to the main rack.
Worked fine all thru summer and fall but when winter came... it was running all the time, became VERY cold by the rack.
I finally noticed there was a heating duct in the ceiling a bout six feet away from the AC, constantly blowing hot air at the AC unit.
OF course the heat was controlled by a thermostat in a different office space with a lot of windows would eventually require a contractor to resolve.
Temporary workaround by IT... block the heat duct with towels.
In the very early days of DEC VAX development, the prototype hardware was in a room with its own aircon. The building itself had been built in the 1860s (yes, 19th Century - they made blankets for the Union Army in the Civil War) and was NOT air conditioned. It was sweltering in the cubicles and we would look forward to some excuse to go to one of the machine rooms ("gotta hang this tape..."). There was no raised floor, so the entire room was kept cold. The machines (each the size of two or three refrigerators) were carefully positioned just so over the massive *wooden* floor beams because the building engineer had said that was the only place that could support the weight. The floors in general were rated "25 pounds per square foot" according to the signs. SO don't stand with your feet together...
You just described what is still probably about 20% of working spaces in the UK. Victorian buildings aren't uncommon, and although you'll probably find most of them have been retrofitted with air conditioning, it’s still not nearly as common in the UK as in many other countries.
TWO heat pumps? You were lucky!
When our new server cupboard was built as part of a factory extension, either the builder or the architect had assumed that a bathroom vent fan would be adequate cooling!
State of the server racks if I hadn't complained up the manglement chain --->
I've seen a few of those over the years, trying to cool equipment......
One - they are not designed for 24/7 operation and may have a short life.
Two - in order to suck hot air out, there has to be an inlet to allow cooler air in. Something that "designers" or builders do not seem to understand.
Server cupboard at home, vent in ceiling, air pulled out over a heat exchanger which transfers some warmth to incoming air which is then distributed around the house (same system also extracts from kitchen, utility room and bathroom, supplies go to bedrooms, living room, study etc.). Door to cupboard has about a 10mm gap at the bottom and cool(er) air from the hallway is drawn into the room via this. Works reasonably well with a couple of NAS units, Raspberry Pis, modem and laser printer, but I wouldn't guarantee it if there was a chunky Xeon or Threadripper-based server in there :-)
M.
While I applaud your re-use of the heat throughout the premises, having the air-gap at the bottom could lead to ingress of dust and other material. Obviously the air is cooler at the bottom, but you've got to factor in the air quality too. You might be better off with a small wall vent close to the floor with a replaceable/cleanable filter.
Yeah, could do that, but this is domestic stuff and not a lot of it so I'm happy to shut them down and give them a good seeing-to with the air duster every now and then, and it makes the cupboard look like all the other doors in the place, all of which have a small gap at the bottom for the same reason - though the kids seem pretty determined to leave the doors open anyway, which doesn't really work in the bathroom...
<digression>
Previous houses, large family, single bathroom, have all suffered from black mould from time to time in the bathroom. Traditional 4" extractor fans, timed, light switch-controlled or humidity stat-controlled made very little difference and were noisy. The best solution was often to open the window following bathtime and leave the door open so that there was a through-draft. Not very good for energy efficiency, but the room dried out much more quickly and thoroughly.
With this whole-house ventilation system the best plan is (normally) to keep the window closed, keep the door closed, and let the fan do its job. The thing runs continuously at low-level (silent, to all intents and purposes) and has two "boost" levels, neither of which is as noisy as a typical 4" bathroom fan. Drier air from the house is drawn under the door into the bathroom and finds its way up to the extract terminal, taking warm, moist air with it.
At the heat exchanger the extracted air cools, so the moisture condenses out and is drained out of the building.
Because there is a continuous flow of air the bathroom dries out quite thoroughly (except perhaps for a corner of the shower cubicle), all without wasting heat, and smells dissipate reasonably quickly.
M.
"in order to suck hot air out, there has to be an inlet to allow cooler air in. Something that "designers" or builders do not seem to understand."
THIS!
May I recommend: "Hot Air Rises and Heat Sinks", a delightful book which clearly explains the basic facts of cooling to those who need to know. As an EE, I have seen far too many "I put a heatsink on it, isn't that enough?" and fans blowing in with no exit port.
Icon, because it's a large fan.
If I wasn't retired I would maybe get that book.
Extra info that hopefully is in it. Air conditioners have to be made to work in cold temperatures. I think it was they can need a compressor heater so the oil doesn't get to thick to start up in the winter.
I was working in a small room next to a big wire wrap machine that generated lots of waste heat. So our room got hot even in the winter. So they had to get a heater ad on or whole new unit up on the roof because the original AC unit wasn't built to run at -20 F.
And the babies, bless, have no idea what the Save icon is supposed to be...
Mine do (youngest just turned a teen), though they still can't quite get their heads around the fact that 3.5" (and 3") discs are "floppy", because that's a better description of the 5¼" variety. ("it's the disc, not the case!")
They also know how to work:
M.
I'm over 64 by a 2 bits. Finally got around to recycling my two old 17" CRT monitors at Staples office store. The lady who was working when I brought the first one in was older so knew what I had. When I brought the second one in and there was a "kid" working who said "is that a computer?".
In his defense. The last one was a big beast. A very good Hitachi 17" monitor. And I brought them in separate days because I'm getting too old to lug those behemoths around and didn't want to do two at once. Next to go is the HP laser all in one. Maybe I should reconsider using a cart.
I took my 22" 2048x1536 CRT to the electrical disposal a couple of years ago. I hate throwing away stuff which works perfectly, but took up far too much room compared to a 27" 1440p LCD. It had lasted 16 years, which was really surprising I had 3 fail and rejected 2 replacements under warranty in the year before.
I've had that be completely adequate for quite a few small businesses, the few servers they had didn't generate enough heat for it to be a problem.
OTOH, I've got one client that set up a completely reasonable server closet, but their solution to cooling was to just put a duct from the HVAC system into the closet. Works fine in the summer. Not so great in the winter.
Having struggled with lack of cooling in a DC hosting 7 mainframe systems it finally came time in the organisations capital programme to replace it. Having been 'done over' at my budget expense by the property department I was determined it was not happening this time and insisted on seeing the details of the systems they had specced up.
Needless to say they wanted to replace the 3 Huge Airedale Aircon systems with 2 much lower powered system's.
The old systems had been designed to provide a 2+1 operation with us cycling through the 3 aircon systems with only 2 operating at any one time but we had increased the number of mainframe systems, hugely increased disk storage and started migrating to UNIX / WIndows systems all 3 aircon units were often in operation to coll the workload.
If our optimistic project plan for the mainframe replacements came to fruition the load would decrease by 20-30% before the aircon was replaced, if we had a delay then the total cooling available would have been insufficient for normal operation, let alone a summers day. I stood my ground and ended up with a 2+1 set up again with an increase in cooling but a reduction in power consumption. As it turned a couple of the old mainframe CPU's were replaced early and thew huge decrease in cooling requirement meant that we could theoretically cool the whole DC with one of the new units having 2 on standby, but that didn't give the optimum airflow for the far end of the DC. The space released by the replacement of the mainframe CPU's and a couple of aisles of DASD by and EMC array was soon taken up by racks of Windows and Unix Servers. I suspect with the increase in power and density of intel based server even the monster cooling Units I installed would now be struggling again
I came in to work one morning in the lab where I worked and being first in I opened up the server room. Opened the door in to it to be met with a wall of heat! The only time I had felt the same was going from the aircon splendor of Singapore airport to outside the terminal, just like opening your oven door! The walls were hot the doors were hot the cabs were hot EVERYTHING was hot to the touch! The Airedale of which there was just one had failed overnight, out came the fans all the doors were opened etc took a while to cool down and we didn't lose too much. Thankfully when we built a new DC it had proper 2+1 setup
Thermal chamber, Aberdeen Proving Ground, entire Stryker hot-soaking all night at 50 deg C -- hotter than the recommended output of residential water heaters -- with sun-load lamps on.
Opening the door is a wall of heat. Staying in it more than a couple minutes makes you feel sick. Touching the steel hull with naked skin (e.g.: ungloved hand) more than 10 seconds would cause an actual burn.
We were stress-testing a new, larger DC alternator for the Stryker in 2012. We had a DC load bank inside the vehicle, and while I could remotely command an idle speed, for each speed I had to go into the hot room to flip the discrete switches for the load resistors to "ride" the alternator's max output.
In a nod to the actual On Call today: the load bank had been placed with its exhaust up against the side bulkhead, so the limit switches (temp and/or pressure) kept cutting the load in and out at the highest values. Almost ruined the test. (What truly ruined it later that week was shearing the engine output shaft bolts doing road tests with the alternator / load bank at full output.)
We repeated it later at a different Army test cell where they had a facility-installed solid-state load bank with its own cooling and an upstairs control room. Much more stable.
End result: That bigger alternator became part of the Stryker A1 block upgrade.
I came in just after New Year last year and similarly found the server room to be a wall of heat, 40+ degrees. Thought the AC had failed at first, but on investigation found that some vagrants had climbed over the fence and down to the sheltered area outside our basement server room and turned the power breaker off for the AC. Apparently the master switch had to be located there rather than internally. It is now protected with a lock to prevent unauthorised switching, and better temperature monitoring...
"Apparently the master switch had to be located there rather than internally. It is now protected with a lock to prevent unauthorised switching, and better temperature monitoring..."
Sounds like the emergency isolation switch for use by the emergency services, especially fire crews. Putting a lock on it may not have gone down well at the next inspection, assuming they remembered they were supposed to check that.
"emergency services, especially fire crews. Putting a lock on it may not have gone down well at the next inspection, assuming they remembered they were supposed to check that."
Fire crews have bolt cutters, saws, angle grinders, and other tools that they love to use. A padlock on an external power disconnect won't phase them (pun intended).
The easily accessible power cut off isn't about speed as much as safety. Without one, firefighters sometimes resort to pulling power meters. That can work, but is fairly dangerous. Cutoff switches are designed to safely disconnect under load. Pulling a meter out from its base when under load can create an arc flash.
Source: conversation a co-worker (who was a volunteer firefighter) had with an employee at a power company. Coworker mentioned that they'd pull meters, power guy's reply was: "uhhh, please don't do that".
Once had a fault report of machines losing network connection at random. "Just reboot the switch" was the expected fix.
Travelled to site and signed out the key to the server room, which was a corner of the 5th floor office with metal and glass walls.
As I approached, I could feel the heat. Both of the aircon units had failed over the weekend.
It took about an hour with desk fans shifting cooler air through the door before normal service was resumed. The Compaq servers stayed up the whole time. The logs showed that the temperature had hit 55C before I arrived.
Luckily the aircon was the customer's responsibility.
I've had the "wall of heat" twice in my life. First time was at a council where some of the support staff worked inside the server room at the far end. Room was feeling a bit cool, so they temporarily turned the newer aircon unit off as it blew cold air over their desks (leaving just one older unit running at the other end of the room) and they subsequently forgot to switch it back on later. Overnight, the room slowly but surely heats up as the old functioning aircon unit struggles to cope with the load. Eventually, it packs in altogether and the support staff return the following morning to a room that resembles a sauna.
Second time was when working at a university around 2010 or so. The external heat-exchanger for the aircon system was at the back of the building just above ground level. We'd had quite a bit of snow, but no issues to start with. eventually the snow began to melt, a huge heap of it slid off the roof and completely buried the heat-exchanger.
At the place where I started work the mainframe was in the office space above a paint warehouse.
The aircon compressors were used to warm the huge space below.
We knew whenever they took on a fresh forklift driver, as the temperature in the computer room would suddenly rise beyond "too hot". Cue a rush down to get someone to shift the pallets stacked in the place obviously marked "KEEP CLEAR".
Eventually the bearings in the units gave up the ghost, rattling the fixings of the units loose. The aircon company's request to weld the mounting brackets back together was refused by the Fire Officer.
"a proper aircon unit that was somewhat overspecced for the job."
There was this one customer I used to visit every now and then who had a freezing cold "server room". It was quite a large room, maybe 50 feet by 30 feet. It contained 7 servers, a couple of switches and the usual ancillary gubbins. One of the servers was in the rack with the switches and patch panels. The others were floor standing beasts from before it was standard to but kit that fitted in racks. And it was bloody FREEZING in that very large, almost empty room! Even at the height of an exceptionally hot (for the UK) summer, I always took a coat with me :-)
We had a new data center constructed a few years ago, as it neared completion some of our engineers went out for a final inspection before we began the process of moving all our on-site servers into it. They found it a little warm inside, eventually discovered that the builders had somehow connected the shiny new aircon system up backwards. It was cooling the outside air and dumping the exhaust heat into the building. Great for the planet I'm sure but not a happy place for 30 racks of servers to live.
You were lucky!
builders were explicitly told that they would need to cover the very expensive kit that was due to be shipped to the end customer soon before starting work over the weekend
come Monday morning everything was covered with a thin layer of dust
(luckily it was just surface dust and the kit was powered down and had enough protection to prevent damage)
At a previous job we managed internet provision for a science park. One time I went into the room where the services came in, I found a thick layer of dust on the equipment and racks, footprints in the dust, and the tops of the cabinets caved in. Yup, some BT gorillas had chosen to stand on the cabinets while drilling a big hole for a new cable.
Unfortunately, I've found that this seems to be a common attitude amongst trades over the years - even when I've made sure the vacuum cleaner is there and plugged in, they'll still cut holes in the ceiling and just let the carp land on your desk.
Did a migration about 15 years ago for a re-insurance company. We moved their servers into cages in a pair of co-lo data centres.
At this point in time though, thanks to BT there was no connectivity to the backup site.
One day I was sat in the office in a meeting when the calls went out that "everything had stopped".
A quick check to see and a walk to the data centre (it was only 400 metres down the road) and when I went up to the cage it was immediately apparent what was wrong.
With no permission / notification of works, the owners of the data centre were installing a new cable tray above our cage. The builders, being careful, had draped blankets over the entire row of racks. Of course there was no airflow and eventually due to heat they shut themselves down.
Strong words were had.
Even earlier than that - c1997 - and another server room move for a large shipping company in Southampton. We got word from the builders that the new server room was ready. I was but a junior engineer so above my pay grade, the decision was made to ship the racks.
Only the room wasn't quite finished - there was no suspended floor. The posts for it were there but no posts. The builders had apparently taken a gamble they'd arrive before the kit and lost. And no one on our side of the project had taken time to check what they were saying was factual. Oops. We drove down to be there at 6.30am only to have to turn around and come home!
Most people underestimate the pressure drop through a pipe, even engineers. Many years ago, I was working for a company that made kit to separate separate liquid droplets from a fast flowing gas. Putting it at its most basic, it worked by temporarily slowing the flow (by introducing a much larger cross-section) and allowing the liquid droplets to coalesce and fall out to a drain. The clean(er) gas was then vented to a safe area - through what everyone assumed was a large pipe. When I ran some check calculations (taking into account the bends needed to get around existing plant) the residual back pressure was enough to bring the whole system down and divert flammable gas to a very unsafe area - where a large “bang” would be very likely. I only signed it off after the vent line was made considerably larger.
We once bought several pre-built racks of servers.
They were shipped to us by a moving company which claimed to be 'expert' computer movers.
When the racks arrived we discovered that they were all damaged very badly. The movers had not tied the (wheeled) racks in place on the van so the racks had been flung around inside and crashed into each other like bumper cars!
Not had that, but at least two occasions where the delivery would need stair crawlers and four people, we paid for that, ie "delivery to room". The delivery company had subbed the final leg of the delivery out to a company and only paid them for "delivery to kerb". The two of us there to commission the kit Were Not Happy, to say the least.
I've been involved in several new built DC's large and small
My three lessons are:
1/ lift every floor tile and check the whole roof void. In one case the builders had swept all the concrete dust, fag packets and other refuse into the center of the floor before fitting the tiles, I've also found lengths of unused cable, packaging and insulation trimmings in roof voids
2/ Sink test the aircon before installing any kit, bring in heaters and stress the aircon, I've had under powered aircon units installed in error, a blocked exit vent (it was hidden in a decorative clock tower and the builders sealed it to keep pigeons out)
3/ Confirm all breaks in the walls ceiling and floors of the 'cube' are properly sealed with the correct fire resistant /expanding sealants' all easy things to fix when the DC is empty and absolutely painful one fully populated.
Reminds me of back in early 90s I was involved in a project to install a telecoms/datacoms network, much earlier in the project they had built some large communication hub buildings, what the designers forgot was about sealing the cable ducts entering the building and they ended up as holes under raised floor, the holes were below ground level and most of the sites had permanent water under the floor in comms room (most equipment was fortunately in different area)
About 15 or so years ago I was involved with installing a pair of HP san racks. The racks were going into separate building with the data mirrored between them, the sensible approach was to have them in one location, let them replicate and then take one to the other building.
The vendor engaged a “specialist” transport company.
They turned up with a van without a tail lift and neither building had a loading dock so they had to manhandle a populated san into the back of the van.
At the other end they arrived with a not very long ramp, and placed this next to the building doors, in front of the marble floor with glass doors behind it. When the rack got to the top of the ramp we got out the way as 200kg of rack started accelerating down the ramp. Somehow it stayed upright and didn’t take the building doors out….
Words were had with the courier and the company that engaged them…
The system did survive the move without any disk failures but not an experience I would want to repeat.
Many years ago, I ran IT for a 9-5 M-F SME and slowly dragged them from "we've never had an IT guy before" to a broad approximation of best-practices. I managed to get them to stop entering data directly on to the server, then to stop using the 3m x 3m room with the servers as an office and dedicate it to servers, to lock the now declared server room (keeping out the cleaner and stopping her putting stuff in there), to rack mount stuff and then to undertake items like removal of carpet tiles and seal it.
After a while, having pushed a long time for aircon I was finally granted an extractor fan which, in concession to keeping the room warmer in winter, had an automatically retracting cover and was hooked to a thermostat to keep shut when the room was under 19c.
Having gone home one Friday night (1hr away) I noted at 9pm my external services were unavailable. I went back on Saturday and found a black scorch mark 16ft up the wall from the outlet of the fan. Turns out the cover had failed to retract, the fan had superheated and caught fire and the fire had created a pile of airborne plastic particles which had then filled the room. My poor three DL360s had started to warm up and in turn ramped the fans to "full 747 takeoff" power before they'd shut off due to heat. Every wall was black. When I opened the servers they were packed solid with plastic flakes.
I spent a day cleaning the server room, vacuumed my servers, network kit and SAN inside and out and it all powered up with no losses.
Surprisingly, my request for an aircon (humidity controlled) and raised floor was granted the day after.
Tasked to investigate why a medical practice suffered slow connectivity first thing in the mornings & especially on Monday mornings.
Turn up, after finding a parking space on a very wet Friday afternoon of my last day.
Walk in & make the determination of root cause within minutes.
The modem & router was in a cupboard full of medical supplies, for ease of working they left the doors open during the day, but at night.......
At an old job we had a cobbled together server room. It was reasonably secure, didn't look totally awful, and had "plenty" of power (one 20A circuit dedicated to each rack, and the room lights and wall outlets for their own circuits).
The air conditioning was provided by an elderly wall mount unit. Said unit sat in an Ill fitting hole in a south-facing exierior wall, so it baked in the summer sun. Oh, and the cold side of the AC wasn't located in the server room, but was about 5 feet (sorry, about 10 Linguine) from the server room wall. The "cold" air was directed in with corrugated tubing and some custom engineereed cellulosic materials (a mess of cardboard and tape)
It worked every bit as well as you would imagine. Was great fun trying to explain to the boss that the high rate of hard drive failures and the low life of UPS batteries may be related to temperature.
Builders and IT - where to start !
At a previous job, we were the sole tenant in what had originally been a farm building - converted to offices, factory, warehouse etc. The farmer was actually a fairly decent chap, but his trades people were ... variable.
From little things like : I'd removed a data socket, coiled up the cable neatly, and tucked it away where it couldn't possibly get damaged - yet somehow they'd got it out from it's safe space, draped it all over the floor, and cut it by dropping a concrete block on it.
To bigger things like : I had a bunch of cables attached to a steel catenary wire for support. One of the end brackets for the catenary wire was in their way so they just removed it. And we tended to run bunched of cables in the wall purlins - but they were extending the offices and these were in the way of some timber work for a window. The builder "couldn't be bothered" to just leave a gap, so they just hiked the cables out and left them dangling like a load of washing lines.
Gawd, builders!
At one site I'd done the first fix electrics, routing all the power cables as per protected zones (straight lines in line with outlets) ready for plastering. The next day I found the plasters had somehow chopped through a cable, and in order to refix it had pulled it on a straight diagonal and just managed to tightly rewire it. So it was now "invisibly" routed (in an area I knew would end up with wall fixings being drilled in), and with strained connections. I sighed, chopped out the damaged cable, re-chased their plaster, and replaced it with a new length correctly routed.
I remember hearing a story from someone who related that after builders had been doing some work in their office over a weekend they found their LAN wasn't working very well. After a bit of investigation they found the reason. The builders had been moving things around and found a thick yellow cable in the way (for younglings this was original style "thick" ethernet cabling) so had cut it, moved cabinets through the gap, then clearly spent some time soldering the strands together before taking over the joint and assuming no-one would ever notice!
When I first started my job, the company were switching their network from using miles of co-ax to using structured cabling with patch panels and RJ45s. While we had multiple networks for different purposes, the installers had helpfully used grey cables for all. They also hadn't left us with any plans or schematics, but after a few weeks, I had a fairly good idea of which sockets were in which rooms.
One day, my boss, having had enough of trying to work out which network was going where, sent me to our patch rooms, with a box of assorted cables, all different colours and asked me to sort it out. It took nearly a week of me looking at the socket number in the patch room, then running to the relevant office, checking what was on the other end, running back to the patch room and plugging the correct colour cable in. These "rooms" were cupboards, each barely bigger than the patch panels they contained. It was hot enough that by the end of each day (it took nearly a week for me to do this in addition to my other duties) that the clothing I wore (thankfully, the company allowed us to wear casual clothing, as long as it wasn't too tatty, so t-shirts were fine) was soaking with sweat at the end of each day.
As always, with all the tales of overeating, you lot have now got me worrying about the Ivar cupboard at home that houses the patch panel, modem, switch and PiHole!
I can *just* feel it is warmer in there than the rest of the study but still find myself poring over aircon websites.
What is the word for being hypochondriac over the hardware?
I've found that it's much better to clear everything out of a construction zone rather than trying to work around something. Especially so when it comes to something like a server installation. The intricacies of a large computer system are nearly always going to be beyond the ken of your average construction worker. If they have any sort of issue that would affect that thing that's trying to be worked around, the issue will come up all of a sudden and somebody will come up with a solution that sound perfectly fine to the 1 or 2 other people they tell about it. Of course, none of them know enough about the thing to be in a position to spot that one huge problem they are going to create.
Bottom line, move the server setup off-site under the building crew has turned over the property ready for use. If everything were to go perfect by keeping it in place, that would be great, but nothing ever goes perfectly. The cost to relocate the servers temporarily should be looked at just like insurance. If you are uninsured, it's not a problem if nothing happens, but it can be a huge downside if something does so you weight the risks and often opt for the insurance. I'd expect that the build would have gone much faster and saving would have been made there.
Had a few over the years.
One tightass client had their server rack set up inside a shower cubicle, with tiles, plumbing etc still in place. They had installed a splitty AC up on the wall to keep things cool. Problem was, the server rack was mounted sideways, so unracking servers was impossible once the rack was slid into place.
It all worked, but was sooo dodgy. No UPS, around 3 switches between one room and another etc.
Another site had a splitty up on the wall with racks directly below it, Condensation drain outside blocked up, so it dripped all over the racks and took a $500mil org offline for a day or two until replacement kit could be sourced.
Another site had the chillers fail/trip off for their server room, it hit around 60-65C on the door handle to the room, according to the fireys who fired an infrared thermometer at the handle. Big red switch was pushed. PDUs have since had temperature monitors installed, and high temp cutouts enabled. AC is still woeful though, with AC coming from the ducts in the ceiling, and no way for the hot air to get out, other than rising to some return vents.
Had a server room air con fail because it was running too cold and the condensor coil turned into a solid block of ice :) In a multi storey office block so building maintenance were none too happy that we were dumping our hot outlet into their building AC to resolve.
We had to run one of those big portable refrigerated units that had a water container for the condensate, that had to be emptied every 6 hours. So for a weekend I had to drive to the office every 6 hours and empty the container. Fortunately it was only a 20 minute drive and the traffic was good over the weekend, but it felt like I was doing the new father bit again. I did get paid a handy bit of overtime though :)
I did some contracting work for Arthur Anderson a few months before everything hit the fan for them and one warm Friday I noticed a procession of nice young ladies heading into the Server room to cool down after lunch.
I could not understand why the two IT guys had the biggest grins on their faces ever until said young ladies walked out of the Server room suitably refreshed.
I'm retired now, but in my first post-university job (UK, mid 1980s) there was a "machine room" with air conditioning, temperature and humidity controls.
Local rumour had it that by proper (i.e. highly improper) settings of the temperature and humidity controls (presumably something like "temperature all the way down, humidity all the way up") it was possible to get it to snow in the machine room.
I am not aware of anyone having tried it. At least no-one came back the following week to report the results.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
