Developer's default setting created turbulence in the flight simulator Welcome once again, gentle reader, to another instalment of Who, Me? – the Monday missive in which Reg readers share stories of occasions on which their prowess didn't quite meet tech repair challenges. This week, meet a reader we'll Regomize as "Shirley" whose very first job as a software developer was for a firm that built …
"Oh, cut the bleeding heart crap, will ya? We've all got our switches, lights, and knobs to deal with, Striker. I mean, down here there are literally hundreds and thousands of blinking, beeping, and flashing lights, blinking and beeping and flashing; they're flashing and they're beeping. I can't stand it anymore. They're blinking and beeping and flashing. Why doesn't somebody pull the plug."
I can second that recommendation. Though I might suggest for anyone who has never seen "Airplane" before to watch "Top Secret" first.
If you want something homegrown with a barrage of gags coming thick and fast, see if you can find "A Touch of Cloth" (starring John Hannah and Suranne Jones) on DVD or streaming.
For those who confused because they didn't spend Saturday mornings in 70s watch wrestling on ITV - Shirley Crabtree
From the 1980's, watched opened mouthed as Royal Navy matelots manually dragged a PDP system out of the onshore DC I was about to enter and loft it onto a flatbed truck. They tied it down, put a canvas over it and set off. I suspect it was not functional when it arrived and was offloaded to whatever destination it was going to.
I did raise my concerns with the person who was hosting me on site but they just shrugged their shoulders and said it was not part of their IT domain.
Not just dropped off the flatbed, but bounced up a flight of stairs, only to discover it is to be installed in the basement, and so bounced back down two flights. The heavy goods lift was out of order at the time. After re-setting all the cards (just in case), it was with great trepidation that I tried booting it for the first time. I shouldn't have bothered worrying.
Happened at SLAC in about 1985. Last time I checked, the equipment was still functional and used fairly regularly.
"If this was the 1980s, it was very likely it was obsolete and decommissioned anyway."
I could show you PDP kit still in active duty. I have contracts for a couple bits & bobs that supervise, monitor and/or record some rather large and expensive equipment that isn't going to be upgraded any time soon, if ever.
And if anyone is likely to still be using a PDP system for something, it is the military/defense world (or in the case of the Royal Navy, defence) It wasn't that long ago the US updated its ICBM launch system from floppy disk. The old school 8" floppy disk, the ones that really were "floppy"!
The one's I worked on (USAF--classified), 1988-1990 were most definitely still functional.
Although, one DID stop working when one id10t decided it was a good idea to insert the bit of a styrofoam cup under one of the cards & screw it back in.
After I showed that to my supervisor, removed the card, picked out the styrofoam, and put things back, however it worked fine.
And yeah, said id10t, who had already lost a star & a stripe then received an entirely new rank and a controlled tour to a base in NE Kansas...
Before I left the Army I went to a cold place on a comms exercise. The generators were put onto ISO pallets, covered in plastic sheet and shipped over. When the generators were switched on the majority failed to start, having ingested an amount of salt water. This was pointed out before they departed the UK but it wasn't the persons job who was shipping them to make sure they worked on arrival, just to make sure they arrived.
but it wasn't the persons job who was shipping them to make sure they worked on arrival, just to make sure they arrived
I had a similar experience sending some Sun servers to Bergen with Air Chance France - they didn't even do the "make sure that they arrived" bit. So we had an enjoyable 3 days there waiting for them to arrive (had ended up in Munich apparently) and, when they did, trying to get them to work. From the state of the pallet that they had been shipped on, I'd say that they literally fell out of the back of the cargo truck taking them from the airplane..
Second hand story.
I was working in final testing of "small" mainframes in the early 80's. The boss told us one of our Sperry Univac 1100/60's was dropped when unloading from the plane that took it over seas. They needed a new machine.
Maxim 11: Everything is air-droppable at least once.
When I worked at [ISP] back in 2000, we had a couple times where the Stupid Shipping Gang had ben called in for our stuff. The first was a Juniper M40 that was dropped off the back of a truck somewhere, landed on it's side, speared with a forklift and then delivered to us. We were not amused at the boot prints on the side of the box, the neat rectangular hole, the dent and scratch in the back of the unit from where the forklift tine was in contact with it, and all the Shock Watch and "Tip 'n Tell" indicators were tripped. Two MILLION US kopecks it had cost us, and to it's credit, it survived and served in our lab there for a time. I credit the wooden crate it was shipped in and the copious amount of foam packing inside it that it survived.
The second time? was a pair of Cisco ASR10000 units, fully loaded. The bare chassis was 50 grand at least, and the cards were probably more than a house. These both were also dropped off a truck, and since Cisco shipped them wrapped in cardboard (two layers!) and expanded foam, they were unusable when delivered- the packaging was held together with tape, the chassis was all bent to hell, we couldn't get any of the cards to move, and the department director, along with our Cisco rep, had some really nasty words for the shipping company, and by nasty words meaning "filled out a couple pages in the book of the obscene and profane" nasty. Words that were banned by the Vatican nasty. :)
Worst I've seen was a Sun rack with some T3 disk arrays in the bottom of it. The guy unloading it from the van didn't get it properly on the tail-lift and as the tail lift was lowered, it caught the truck floor and made a lovely "flip over" manoeuvre onto the ground... Not sure on the total value, but this was about 2003 so it wouldn't have been cheap.
My mate who was taking delivery of them had to go through a post incident safety briefing because of the potential for injury/death if some poor soul had been in the way... Given that the delivery driver should have been more experienced in the safety precautions necessary, I think they were chasing the wrong person.
This was pointed out before they departed the UK but it wasn't the persons job who was shipping them to make sure they worked on arrival, just to make sure they arrived
As a taxpayer this sort of thing makes me angry. Well I'm a US taxpayer not UK but I'm under no illusions our military doesn't do stuff as dumb or dumber.
Its this sort of "just follow the letter of the order, no questioning orders even if they are moronic" thinking that's encouraged that makes me glad I never served. I would not have lasted long before I pissed off my superiors and got shipped off to a "cold place on a comms exercise" for the remainder of my enlistment (whereupon I would likely have quickly figured out the best route to getting a dishonorable discharge rather than suffer for the stupidity of those whose orders I questioned)
I have heard plenty of worse stories from friends who did serve, they were better able to look the other way when things like that happened. I guess that takes a certain amount of patience for the stupid that I didn't develop growing up with a Berkeley PhD father and a college graduate mother who skipped two grades and won several statewide awards in high school. I asked my dad once how he handled being in the army for two years (back in the 50s, mostly in Germany) He said "the first couple months I peeled a lot of potatoes [common punishment back then I gather], then I figured out if I got really drunk at night and was hungover on duty I didn't care as much about what anyone was doing"
Military service does rely quite a lot on obeying orders and not stopping to question them. Unless they're war crimes. For that matter, your commander may have a grudge against computers, so it's deliberate. But, yes, terrible things happen to innocent computing equipment. But it's probably for launching bombs or something unpleasant like that, and so the world may be a better place if that doesn't happen.
As a taxpayer this sort of thing makes me angry
It seems to be a perennial problem. A slightly different circumstance, but nonetheless breathtaking. During WWII my father was growing up in a South Wales port which happened to be one of the places the US Army used for shipment of supplies, food mainly I believe. He and his mates discovered that a lot of supplies were coming off ships - bear in mind these things had run the gauntlet of the Atlantic U-boat fleets and were destined to feed the troops - and being sent more or less immediately to landfill. The US Army had a policy that if the outer packaging was damaged, the entire contents were suspect. Even tins. Once sent for disposal these items weren't under so much security...
Not that tins were terribly interesting to my dad. He apparently had his school desk full of sweets and chocolate and did a roaring trade, while his schoolbooks languished in his satchel.
On a totally different scale we have the modern scourge of the "use by" date and "not suitable for freezing" packaging. We do work with a local foodbank which takes (among other donations) "expired" food from some local supermarkets. Bagged salads are a constant problem (but we have a compost heap) and sandwiches (which can't be put in the compost), but why would one supermarket's carbonara ready meal (for example) have "not suitable for freezing" while another's says "can be frozen"? Food safety rules means that even on the use-by date, the latter can be frozen and used within 3 months while the former should be binned.
M.
A long time ago , we used a PC as part of a <redacted><redacted> test on board one of her majesty's ships.
We installed all the gear... wished the sailors good luck and returned to our lair.
Few weeks later the test was over and said sailors said they'd removed the gear and it was on an aircraft heading for RAF Brize Norton.... cue a 5 hr trip in the works van to pickup a load of big heavy stuff including a PC.
After arrival.... said ground staff said our big heavy kit was over in the warehouse corner.......... a slightly second hand moniter..... "Wheres the rest?" my superior asked.. "Thats it mate"
We spent the trip home grumbling that we should have used a mac. what a waste of a day...sheesh.
I remember that someone wrote a nifty script that went checked something every second (for example if a file has changed, notify me so I can do something). This was a great little tool and soon every one was using it. Soon every one (100s of people) were using it on the server. People left this running overnight when they were not at work.
The system got slower and slower as more and more CPU was used. The root cause was this little program. When they set the default interval between checks to 10 minutes the CPU problem went away, and a check every 10 minutes was fine for the end users.
I forget the exact details, but I once suffered from a post-articles-to-usenet cronjob interacting badly with an unreliable nfs mount. I think what was happening was that article in out.going could be read and was posted, but then never got deleted due to the nfs issue, so that as a result it got posted over and over again. There was quite the little forest of stuck processes when I eventually found out and looked.
I must say the email from the moderator of sci.physics.research, alerting me to this problem, was remarkably polite.
One likely pitfall with cron is the “thundering herd” effect. Say you have a number of housekeeping tasks to run once a day. Maybe you schedule them all for midnight. Then, the time rolls around, and your system suffers a noticeable pause as all these tasks start at once.
The savvy sysadmin will offset each task by a minute or so, to spread the load. (Pity cron doesn’t let you specify times in fractions of a minute.)
Luckily, more modern scheduling systems don’t require manual offsetting. For example, systemd timers let you specify a “jitter”, which will be used to add a pseudorandom (i.e. different, yet repeatable) delay to each task.
《I've heard of people getting into trouble with cron, but it's always a story about someone else. I've never actually seen it in person, leading me to think it's apocryphal at best.》
An empty /etc/cron.allow on most systems keeps it apocryphal. :) Users can show a hideous if unsuspected creativity with cron.
The original pre y2k BOFH would have replaced 'crontab' with 'rm -rf $HOME'
I was puzzled why the who-me was regomized as Shirley when a sentence or so later Shirley is clearly male - I do recall Alf Garnet referred to his son-in-law as Shirley (a UK thing?) From the comments I conclude its a reference to movie comedy/farce involving an aircraft with an incompetent (male) pilot called Shirley. Missed that movie.
When we were preparing to roll out IPv6 a number of years ago we had a consultant in to assist. She'd been doing some work at home over the weekend, and when she came into the office on Monday she continued checking how gracefully our equipment would handle NDP cache exhaustion in larger subnets but couldn't quite get anything to trigger the way she wanted to. After a couple of hours her husband called from home and complained about the Internet connection dropping out constantly. It was then it dawned on her that she hadn't changed the target IP subnet back from her home router on returning to the office, so she'd been DDoSing her husband for a few hours.
If it's designed to go 0-100% nothing would happen before then anyway. Mechanical fail-safes can be tricky to make, set, calibrate and maintain.
For excessive vibration, it's far easier, more accurate and more reliable to have a transducer monitoring conditions. The associated electronics will determine whether things have gone out of control. After all, the operator might have become incapable for medical reasons.
Yes, accessible to the people running the simulator, not to the people training in the simulator - as the article and the earlier comments both allude to, these simulators are designed to mimic the real cockpit environment down to the last nut and bolt, so if in the real thing there isn't a big red "slap me to shut everything down" button available to the pilots, then there won't be one in the simulator either. And once you're sat in the pilots seat, trying to make it safely out to the simulator control panel in order to slap that big red button, when the simulator is simulating the most severe turbulence it's physically able to simulate, would be more an "if all else fails" move than a "this seems like the most sensible idea to resolve the problem" one.
TBH though, the real issue here seems to be the lax H&S procedures at this company, allowing two seperate incidents to occur which could have been resolved with far less fuss and potential harm had they simply operated a "no solo working on the simulator" policy. One such incident is unfortunate, two smacks of a failure to use the first as a valuable learning opportunity.
"then there won't be one in the simulator either"
And how far do you take this quest for realism? flame throwers to simulate the cockpit being disintegrated with burning jet fuel?
The school of thought that "oh they cant have an emergency stop button on this big metal dancing cow because real pilots dont have that" is ludicrous
Its not like they are going to end up issuing pilots licences to pilots who think that button is a real world option.
Are you trying to convince me that it would be legal to strap someone inside a heavy moving machine with a multitude of flammable components and high pressure hydraulics without giving this person a quick way to shut down said machine quickly if it fails because it would "break immersion"?
The big red button will be there, located somewhere that is easily accessible but not used in simulator training. I would personally use the space reserved for cup-holders.
The sim operator will also have a button, and there will be buttons in the room where the simulator is located.
Anything else would be a multi million dollar / pound / Euro lawsuit waiting to happen. Do you want to sit inside the sim and hear the screams of the poor sod that is being chewed to pulp underneath you and not be able to do anything but wait for the sim operator to shut things down (oops, it IS the sim uperator that is screaming).
"Are you trying to convince me that it would be legal to strap someone inside a heavy moving machine with a multitude of flammable components and high pressure hydraulics without giving this person a quick way to shut down said machine quickly if it fails because it would "break immersion"?"
And yet millions of people do that every time they board an airliner, or a theme park ride. And to a lesser extent, when they board a bus, train etc - i.e the myriad of everyday scenarios where people very much do place their lives in the combined hands of a trained operator and a well designed piece of machinery, because there's nothing they can do to influence the outcome themselves once they're underway.
yet millions of people do that every time they board an airliner
Yes people trust in other people or machines sometimes . When They Have To !
Theres no point blindly doing it when you dont have to.
The two situations are completely different.
Why not implement a virtually free , possibly invaluable safety measure?
back when I were a lad, learning software coding, the instructors stated often that all variables should be per-initialised. It saved me a few times from cockups when the default value showed a logic error, usually after multiple successful runs. Is initialising variables not done anymore ?
Yup. That's exactly what happens.
You are apparently unaware that the people who spend time in the simulator _also_ spend time in another environment that includes absolutely no big red buttons? Or is it your contention that the other environment is a "multi million dollar / pound / Euro lawsuit waiting to happen".
Still, the difference between the two environments is that the simulator has limits that it cannot exceed. The real environment is limited only by the physical world.
As for solo operating, it was crunch time - so to speak.
Let me know if you'd like me to look up an old science fiction story after spoiling it here. A future in which space pilots are tremendously trained on their model of rocket (no computer pilots). When it is obsolete, do are they. One ex pilot sadly rides as a passenger on a slow ride to Mars. Things happen on the way and he's going to have to land the ship. But he cannot drive this model.
So he spends a lot of time taking the controls apart and rebuilding them into the control room design that he knows. There are dials that he just painted on. But he needs them to be there.
I don't remember the ending, but I think one of his landing simulations has the ship stop moving about 100 vertical meters after hitting the ground, which would not be a success.
I liked the story, before I heard about the Boeing 737 Lawndart which exists because pilots etc. can't be retrained.
"the Boeing 737 Lawndart which exists because pilots etc. can't be retrained."
There seems to be widespread misunderstanding of this. Boeing didn't make it handle like the old one because it's cheaper, or because the pilots can't be retrained, but because retraining pilots carries its own set of risks; training isn't automatically successful. You can argue about whether they struck the right balance of risks, but it's not unreasonable to suggest that we'd have seen at least as many accidents (and possibly more) on a significantly different platform due to (re)training failures than we have seen this way. The air industry has stats on this sort of thing, and I would imagine they show that Boeing got that part of it right.
It's also worth pointing out that there were multiple MCAS incidents, and only two of them resulted in crashes. It's no coincidence that they were in parts of the world where pilot training and experience are not up to rich-world standards. In all the other cases, the pilots successfully managed the situations.
https://www.nytimes.com/2019/09/18/magazine/boeing-737-max-crashes.html
That's not to say Boeing didn't do anything wrong, but understanding what really went on is important in preventing future crashes - which Boeing-bashing doesn't actually achieve.
" In all the other cases, the pilots successfully managed the situations."
In the cases pilots knew about MCAS and had instructions to disable it. Boeing at first decided that pilots don't need to know. After couple of crashes they were forced to tell.
Boeing fans again trying to blame pilots for Boeing killing people *only* to make more profit. Literally no other reasons exist.
I forget the system name, it may even have been bespoke in house but a modest workshop I had a summer job at had a fancy conveyor (rubber cords running in grooves) system to dispense and distribute prepicked trays of components to the assembly workstations. It used small pneumatic pistons to push the trays from one central conveyor on to a perpendicular one and, as the system aged, the creaky pistons stopped doing their job reliably. In response to the grumbling, someone volunteered that the regulators could be adjusted for a little more power. After lunch, there was quite a commotion when trays of components started flying through the air. It seems that someone on another distribution line was upset at not getting their line adjusted too and had copied the procedure without understanding the method. They'd simply wound the regulator fully open and the result helped a lot with the post-lunch sleepiness.
The first several times I got to play in an RAF flight simulator it had the pneumatic (hydraulic?) Cockpit, both fixed wing and Heli cockpits were available,but the interior view was provided by a tiny camera on an arm that moved throughout a rather large physical scale model of the Welsh mountains. I was given to believe that the army also had a not dissimilar setup for tank training.
This was also the method for the original Concorde simulator. I visited Filton in 1991 and had a go in the simulator (take-off, circle, land at JFK airport) with the more recently installed computer generated display. I only needed a little help with the landing.
The computer room was mostly taken up with the older physics simulator part of the system (complete with paper tape readers) with all of the graphics being handled by a single rack in the corner.
I knew one of the blokes who maintained the simulator in Leuchars.
That one even had a power station with twin chimneys at the bottom of a valley. If you got the angle right you could get the "plane" between them, if not you came to a jarring stop and had to buy many beers.
These were probably Link / Air Trainers / Redifon simulators. Early versions (1940’s) had a camera probe operating over an horizontal endless rubber belt with rudimentary scenery painted and glued to it. The Army may well have used a variant of the design. Control over the kit was pure analog with servos doing the moving. My input comes from Redifon/Airtrainers 1969/70. From memory the last of these belt simulators in use were Hawker jet (ended up in India) and Lightning jet - scrapped.
Redesigned for the jet age by the mid 60’s the camera probe operated parallel to a wall mounted model of town, country and airfield scenery. The wall was getting on for 30 foot high and 100 foot long. A (same size) parallel wall of fluorescent tubes illuminated the model and the camera probe was moved around between lights and model by cockpit control movements. The interface was through a thermionic tube (valve) computer that was partially analog, using voltage comparitor circuits .These enormous contraptions were ridiculously expensive to run and created a very warm workplace in winter. The wall models were connected to 6 axis cockpit simulators that were built using the Aircraft manufacturers original blueprint drawings. I heard some 6 axis sims’ were later upgraded to full digital control using early computer graphics to replace the wall model.
" Thankfully there was no damage to either the machine or Shirley – but he didn't mention the incident to the boss."
I'm no engineer, but Shirley, when they put this kit together someone would have calculated or at least estimated the maximum possible exerted force and the maximum mechanism tolerable, force-- and made sure that one was a margin less than the other.
My cousin worked for one of the first Xbox developers. A few months before the launch of the original Xbox, he bought one of their test kits home so we could play some games. Microsoft only had Xboxs configured for 120 V, and he forgot the step down transformer, and just plugged a UK power lead in (IIRC, it had a figure of 8 IEC connector). The xbox lasted about 30 seconds, and he had some serious explaining to do to his boss when he got in to work. He was under an NDA. He was not even supposed to talk about the xbox, let alone show it to us.
That's why this post is anonymous.
My employer once got a whole shipment of laser printers which (we later discovered) were supposed to go to Japan. I just have this habit of reading labels, so naturally I called one of our hardware types over to check it when I saw something odd like “110V” on the label. Other departments were a bit less obsessive-compulsive, and a few of their new printers got fried.
Programming/software can be fun, but absorbing real-world data/sensors and especially controlling real-world things, motors. lights, ... (basically robotics) is next-level fun.
However that interaction with the real world also comes with responsibility - you can cause real-world harm. Human-harm, or just destroy expensive hardware.
If your prototype- or development-software is controlling anything that could cause human harm, or could cause damage costing more than a day or two's worth of salary... always always insist there is a physical emergency-stop button within reach of the operator - and potentially for onlookers.
The emergency-stop may just kill the power, or may invoke lower-level hardware or software to bring the thing to a safe stop. But you must have one.
More often than not there's a couple of seconds between hitting "run", having the "oh sh*t" moment, and damage actually being caused. You don't want to be flustering with a stupid touch-pad in that second.
Whether it's scientific equipment, dangerous lasers, precision optical translation stages costing £10k+, electronics switching kilowatts of power (e.g. software controlled motor-control loop), vibration-test rigs that could shake themselves apart, things that could make literally deafening noises, produce lethal voltages, or robotic arms worth £50k+ throwing kilograms of metal about (and very much a human-impact hazard) ... you need an emergency stop. I'm in my late 40's. I've worked on nearly all of those.
A key case when an emergency-stop "saved" me was a robotic arm which I was making oscillate for testing purposes... I hit upon a resonance at a few Hertz where it would have thrown itself over. The amplitude was building... but I hit the stop, and saved the situation.
For something as big as a flight-simulator, some modern low-cost accelerometers and some wholly-independent "monitoring" software that cuts out the system if a given acceleration is exceeded, or if some acceleration*time threshold is exceeded would seem to be prudent.
When I was working on the development of some new stuff for the Navy, part of my duties included keeping a record of the equipment as it progressed, and writing and updating the Instruction Manual. One day we had the complete equipment in the Test Area for a heat run, which was to be observed by the Customer. Lots of gold braid and stripes on sleeves, and the test was started.
As I didn't have any involvement in the actual heat run, I took the opportunity to take a few photographs of the complete, buttoned up, suit as it stood in the Test Area. Whilst trying to get the whole suite in frame, I backed off int a corner of the area, and suddenly the whole department went very quiet and dark - I had inadvertently backed into one of the Big Red Buttons and shut everything down,
Howls of fury and protest from the testers, and the Scrambled Egg Brigade were unceremoniously led away. The whole test had to be rescheduled for the next day as it had to start from cold. Needless to say, I was NOT allowed into the Test Area for that or any subsequent heat runs, officially observed or not.
Probably a real story, but artistic licence, authors embellishment or contributors poor memory or lack of pertinent detail seems to leave something not quite right about this story.
Specifically "The sim that Shirley worked on had been promised to a client for $16 million. And Shirley was one of 30 developers working on it, so it was in use pretty much around the clock, with limited time to test things out."
Why are devs working on a $16m simulator about to go out to a customer that is in pretty much constant use? Shirley devs work on a simulator to get it ready for use? Or is it normal for the manufacturer to be using a simulator while modding it for sale to a client?
Sim development on each device certainly used to run on shifts 24/7 as there would be that much demand for time on it and you generally can't test or fiddle with two different things at the same time for various reasons.
Each one is different and each one is/was fully built, developed, tested and certified in the factory before being broken down & shipped, then rebuilt & weeks more testing before starting a life of near 24/7 use.
Even the 'mass production' of these things, even for a single aircraft type, was bespoke every time so lots and lots of effort.
$16million isn't even expensive...
BTW sims do have a big red stop button (often more than one) but if you're single handedly running a civil one it can be hard to reach if you're in the pilot seat not the instructor chair.
And if you roll the thing over on the motion after messing it up it can be fun climbing out. Though not as hard as trying to get out of a genuine fast jet canopy when it's left with functional locks, the power trips, and it's a weekend & you're working alone despite procedure...
What is a bit wrong is I don't think anyone did pneumatic motion systems; to get the performance (multiple tonnes of sim cab, accelerating in 6 axis at 2g, with the right motion cues & vibrations) originally demanded a big high powered Moog valve controlled hydraulic system, eventually moving to hybrid electrohydraulic then pure electric actuators.
Also despite seeing motion systems do all sorts of mad things I can't think of any motion platform where you'd get a vibration slider, let alone have it do anything functional. Maybe on a motion seat but that's a different beast - I've seen a Moog one get miss-driven into doing some nasty shaking but that was still based on the motion model doing some odd things, not anything as prosaic as cranking a vibration setting because such a thing didn't exist.
"Sim development on each device certainly used to run on shifts 24/7 as there would be that much demand for time on it and you generally can't test or fiddle with two different things at the same time for various reasons."
Thanks for that! The way it's worded in the article, it sounded like it was in use by pilots for training :-)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
