Hardware inspector fired for spotting an error he wasn't trained to find Welcome again to On Call, The Register's weekly column in which readers share stories of earnestly trying to fix broken tech, and end up feeling broken afterwards. This week, meet a reader we'll Regomize as "Tex," who briefly worked at a semiconductor company in the second-largest state of the USA. "I was working in their …
Good question! I'm wondering the same. Although, I did come across the odd SMD that were mounted and partially(!) soldered upside down - directly out of the soldering oven. And pre-SMT chips with their pins bent upside and then mounted upside-down on purpose (it wasn't me!), for some prototype.
Happened to me in a group project at uni but it wasn't our fault - the lab tech who had been tasked with producing the PCBs got it wrong, though only (as I remember it) for us; every other group got correct boards. Our acetate was clearly marked. Nobody noticed until I came to put the thing together a week or so later, and the lecturer didn't have the time or the budget for a re-run. Ended up soldering the ICs on the trace side (single sided board, all through-hole). Learned some good technique that day. Can't for the life of me remember if I put the Rs and Cs on the same side as the ICs (would have been tidier) or on the side they were originally meant for.
M.
Then there was the Land Of The Giants PCB (unpopulated, of course). What they'd forgotten to do was photo-reduce it from the original 2x or 3x by-hand layout.
(Not the place I was at. It was a friend of a friend's employer, or the fabricator they'd farmed the job out to -- I forget which. Only the initial sample run, but still...)
Ah yes. We even had a dip adaptor, as a production lot of double sided pcbs was so expensive and would take so long to re-make, and the assemblers had installed most of the leaded parts before someone had an Oh Bugger moment. An adaptor was single-sided phenolic screen printed - we had them in a day.
I've installed a few mirror ICs as well. In the old days each new pcb designer had to learn about putting text on every layer the hard way. Luckily the pcb makers were more than willing to help us get that expensive lesson.
In my defense, this was a home project while I was in High School. So 1973-74 time period, no PCB software for your home PC. And a high power home PCs was a Altair 8800 with an Intel 8080.
So I made my PCB by drawing the circuit out by hand on paper (for easy erasing). Then I used an awl to mark the IC pin and part holes for drilling using the paper as a template. I then used a sharpie to make the traces. You had to be sure you had a good sharpie so it left a thick line or you would get holes in your traces. Then it was ready to etching.
This usually worked out but I forgot to "mirror" it at least that one time. I think I just mounted the chips on the bottom side without drilling the holes.
I recall a project I did in college where I used clear mylar I think and then there were sticky templates for IC's and round donut templates for R's & C's. Then black tape to make connections. Somehow I got several moderately complex boards correct first time. I was in school so I checked like a dozen times to make sure they were correct. Now DRC/LVS is big standard for pcb. I cannot imagine doing double sided smt boards with tape and templates. It'd never work.
"And pre-SMT chips with their pins bent upside and then mounted upside-down on purpose (it wasn't me!), for some prototype."
Somebody may have discovered that the PCB layout was back to front for that part(s). New boards will take time, but there's still a bunch of testing that can be done as long as the board gets shredded when the new ones arrive. I've built some very interesting prototypes so I could test things even as new boards were in the works to take care of the most egregious screwups. The tools these days make that much harder as long as the component models are correct.
C'mon! You don't bend them upside down. Ever! You use an additional small breadboard to map the pins to the PCB for permanent use. Or, temporarily, a solderless breadboard. Or use wire wrap and have the chip suspended in air for permanent but prone to fail application.
And I never did any of these either! And if I did, I forgot.
I'm guessing that the idiot who soldered had zero notion of electricity and just basically soldered the back of the chip to the motherboard, regardless of the fact that there were pins and emplacements on the motherboard.
So, not only an idiot, but a myopic one as well.
I can't imagine any other solution to this unbelievable situation.
And, if Tex was fired because he pointed out the obvious, then it was obviously not a company worth working for.
zero notion of electricity
Mike Cook, he of the "Bodybuilding" series in Micro User (I think, or was it Acorn User?) once wrote a piece about getting a kit of parts back from an irate subscriber complaining that his circuit didn't work. Each component was mounted very neatly in the correct orientation in the correct place on the board...
...with superglue.
M.
Back in the day, Acorn User published a circuit board design program in its Yellow Pages. The idea was you printed out the resultant diagram and all the component pin holes should be in the right location. The AU staff got an irate call from one reader who said none of their components would fit. AU staff asked what printer did they use. Reader didn't have a printer, so overlaid a sheet of paper on their TV/monitor and traced the diagram...
I duess that an IC with a very low profile enclosure being pressed into solder paste upside down could result in the solder reaching up to the "wrong" side of the pins. If anything can be done wrong, then it will.
Just this spring we had a landscaping firm plant plants upside down (roots pointing upwards), so the stupidity of someone "just doing their job" should not be underestimated.
My favorite story of that nature dates back to the 1960s when I was in high school.
The school had just opened a new wing of classrooms and I was sitting in class when I happened to look up and notice an electrical outlet smack dab in the middle of the ceiling, well out of reach except by way of a ladder.
I asked the teacher about it and he told me that it was an "overhead projector" outlet, which the electrical contractor took a bit too literally and placed it. . . well. . . overhead. . . and not in the floor where it belonged.
I worked for a few days in a room that was designed for people to work on hardware projects. It had power cords permanently dangling from the ceiling on spring-retract reels. Want power in the middle of the room? Just reach up and grab some. (When not in use, the cords hung low enough to be within reach, but not low enough to be in the way.)
Super convenient. Also safer, since an in-use cord wouldn't be running across the floor. Brilliant!
Back in the day, we did that for the power to the desks not near a wall. It seemed better than floor plates which often get broken or damaged by people not knowing how to fit the covers back into the floor box when adding/removing plugs. They also allow for more repositioning of desks than floor plates.
Our local livestock market has an exhibition space, and any number of overhead sockets.
I attend a flea market held in that kind of building. Keeping the electrical outlets out of the combustible material like straw, and away from liquids which might cause a short, as well as being out of the reach of curious or bored animals are all reasons to do this. It also means that reconfiguring the interior for all its uses but keeping power available to those who need it (and paid extra for it, on occaision) is simple.
If one didn't know what an overhead projector was, "overhead projector outlet" would be an easy phrase to mis-parse.
I deplore that hyphens have gone out of style; their lack makes for a lot of such ambiguities -- or at best, sentences that you have to stop and puzzle over, but whose meanings could have been crystal clear with better punctuation.
I was working a on a hardware lab fit out project one time, and a manager wanted to mount all the network panels in the ceiling instead of the back of the benches so the layout would be easier to change. He was promptly disabused of this notion, since it would make it very difficult to hook up systems on the benches, and we never changed layouts after the benches were installed
I don't see your argument. With a higher minimum wage, some people still will leave school earlier to work (about three o'clock according to one story), but be paid more?
But if you have something like Scotland's "Education Maintenance Allowance" - paying students who stay in school - then more of them will. Let's hope that it is worth the money.
Some "chips" are capped with metal cans[0]. I have seen a badly adjusted pick & place machine flipping a chip over just before running the motherboard through the wave solderer. Robotic manufacturing, especially in the early days, can cause all kinds of havoc ... but Ive never seen a complete run of prototypes shipped to a customer with this kind of fuck-up. It would be caught by the operator after the first one or two units.
[0] For instance a couple of bare ICs and a few surface-mount devices and perhaps a few printed & etched resistors wired together with something like a gold-ball bonder, all on a substrate, which is then capped with a metal can, sometimes in a vacuum, sometimes in nitrogen or other non-reactive gas. The cap can be epoxied on, or welded, depending on application. The wires for this little circuit are bonded to the external pins before the cap is installed, the pins, in turn, can be soldered to a standard socket. This was sometimes called a hybrid integrated circuit, or hybrid for short.
Metal can on a single die was not unknown. A long time ago the circuit board from an IRA remote controlled bomb had one of those. All the text had been scraped off all the ICs but someone took the top off the can and brought it over for me to check under a microscope. There was the ID, plainly readable and I remembered seeing it advertised in Wireless World. And given that the 8 pin package clearly wasn't an op-amp it must be a 555 so the rest of the board was fairly easily worked out.
I've seen a few chips with (by-design) exposed metal portions on their tops. The old-style wave-soldering machines would not have cared. Capillary action would have drawn the molten solder up through the plated through-holes, where it could adhere to the exposed metal of the upside-down chip-tops.
I've seen that type of soldering. You need "a bit" more solder to get a contact, but it works to keep the part in place. I've also seen it done deliberately with a layout not being mirrored and without through-holes board, but that was somewhere 'round the early 90's and not for mass-production stuff.
Sometimes the pick & place machine will flip a chip while picking it up, and it looks the right shape so gets mounted anyway. It stays in place after reflow because the flux from the solder paste glues it in position.
It can happen several times in a row if the pickup targeting is slightly off.
You only hold a soldering iron wrong once. Or you need a new hobby.
I was looking at what I was going to solder and just reached for the soldering iron laying on the bench. I dropped it fast enough to not get a deep burns.
I bought a soldering iron holder soon after that.
The glue is what holds them in place as they go into the oven.
No, it's not that simple for 148 solder-ball connections, and yes, as the solder melts surface tension is supposed to pull the chip into alignment,
But yes, you can glue chips onto the board upside down if they are loaded incorrectly, and yes, there were also some through-hole pards that needed to be glued down before going through the wave-soldering oven.
From what I understood it sounds as though it was half through hole on one side and surface mounted on the other. Through hole is when theres a hole actually in the motherboard that a post or pin is to go into a be soldered in. Surface mounting is just how it sounds the legs of various chips are soldering to the surafce or top of a motherboard. Anything that sticks through like big capacitors is through hole and the little caps and resistors you see are the ones that are surface mounted.
I'm not a huge believer in conspiracy theories and talk of brown envelopes changing hands* but I've also worked in manufacturing for a great many years and I'd be very suspicious of why the manager didn't want someone knowledgeable on the inspection team. The phrase, "What's in it for him?" springs to mind.
Full disclosure, my Dad ran an inspection department (largely before it was renamed to QC or QA) for many years and yes, there were inducements on offer to be lenient, in at least one case negotiated by the chief buyer without his knowledge. He would never compromise principles however and would treat all suppliers equally regardless.
*It's rarely that overt, it's donations to an election fund or holidays disguised as a business trips or similar.
Years ago, my father worked in QA (for a while) for an engineering company that made various metal components.
Where quite a lot of QA involved getting e.g. your micrometer on the go & checking everything was in tolerance. Some of the tolerance levels were fairly tight.
My father, being diligent & honest, failed quite a lot of stuff he tested for being out of spec (nothing wrong with his testing, it was the components).
His manager tried to get him to be less rigorous on his testing - my father said (obv. me paraphrasing as he is not alive to ask) if you want the tolerances I test against to be less strict then put that in the specifications of what I have to test against as I will not pass something that obviously fails the spec - no way is my name going down as passing something thet failed.
..The company did not change the spec (as spec to be tested against was what a customer demanded & paperwork to be signed off on QA tests listed what the tested against spec was).
Instead my father was shifted off the shop floor (I'm guessing his replacement was keener to turn a "blind eye") to an office role where he was pretty much twiddling his thumbs so he looked for a job elsewhere and quit.
.. So, obvious inference is company was dodgy, wanting to cut costs by passing on some of their out of tolerance parts as legit.
> I'd be very suspicious of why the manager didn't want someone knowledgeable
When I was mildly interested in a job at Radio Shack, the local manager (who knew me too well) said Tandy did NOT like staff who "knew electronics". They tended to geek out and steer customers to lower-profit products. Tandy could train about any warm body to work their script and sell anything to anybody.
Whenever someone at Radio Shack tried following that script, I left and bought whatever it was elsewhere. I'm really not that surprised they failed to make it as a going concern after they went extra-hard on the 'hard sell' techniques, while everyone else was competing on price and/or customer service. It felt a bit like going to a car dealership.
Third rule is work in a place with some level of employer protection where firing somebody for such a frivolous reason would hit the company hard (plus unfair dismissal).
Sadly, there are far too many dickwads like that in management, because the correct response would have been "nice catch, I was wondering if you'd spot that" (while quietly thinking "dammit, why didn't I spot that").
The state I live in (Western Australia) is about 70% bigger than Alaska, or 3.7 times as big as Texas.
Wikipedia informs me that Sakha in Russia is about 20% bigger again, but WA is second on the list. Which is pretty good I suppose if that's something you find interesting. In practical terms it just means there's a lot of pretty empty country and it can take forever to get anywhere.
https://en.wikipedia.org/wiki/List_of_first-level_administrative_divisions_by_area
I’m retired now, but I spent a measurable part of my three decades at my last employer’s talking back to piss-poor mangers in public. At least prior to the current US administration, that was not a firing offense. Indeed, it appeared to improve morale among both the decent managers and all the rest of us.
Oh, I spent quite a few years making our micromanaging boss look like a tool. Rare enough skill set, good experience, adhering to all the bureaucracy and genuinely not giving a crap whether they fire me or not helped a lot.
Left on my own terms in the end, after teaching the next generation how to utilize malicious compliance to the fullest extent and put all the BS in writing.
Several years ago I used to work production line jobs, and spent a while making implantable medical devices with built-in electronics at a client manufacturer. We used to receive the PCB's from the supplier upstream already encased in their little plastic resin-filled tubes. Each one had a internal cost of approximately £1000 US dollars.
Unfortunately the US based client who owned the product had only contracted one upstream supplier who could be best described as "cottage industry" who had no room or interest in scaling things up.
The client contracted a large manufacturer to set up a line and boost production. $2 million later they shipped their first batch of 150 to us for processing. Client insisted we skip incoming inspection steps and concessions were drafted to remove all visual inspection stages up until end of line QC, just prior to packing for the sterilisation house.
It was at the end of line QC we discovered that this entire batch of 150 had been encapsulated into their tubes upside down, and although the electronics communicated as normal their essential functions didn't. Cue much screaming from the client about "why hadn't this been spotted sooner" despite them having signed off concessions and demanded we not check anything visually....
"Client insisted we skip incoming inspection steps and concessions were drafted to remove all visual inspection stages up until end of line QC, just prior to packing for the sterilisation house."
Wow! On medical implant devices? Sounds very "American". Skip steps in the hope it will save money, even when peoples lives might be at stake.
This post has been deleted by its author
That puts me in mind of a review I once watched for a hot-air rework station. One of the guy's complaints about the thing was that its fuse was on the neutral wire, not the hot one -- a safety issue. (This was the fuse in the unit itself, not one in the A/C plug as in the UK.) The reviewer decided to look into it further.
Now, you'd expect such an error to be due to an inattentive (or just clueless) assembler getting the brown (hot) and blue (neutral) wires backwards when installing the power cord. But no.
It turned out the problem was the molded power cord itself! It presented live power on the blue wire and neutral on the brown one. Yikes!
The ultimate blame for the unsafe rework station was with its maker, of course. But seriously, who would think to check that one's power-cord supplier hadn't made such an egregious screw up?
(Caveat: this was some years ago, so I'm reconstructing the details. And as a LeftPondian I'm used to a different colour scheme entirely, so I might be reconstructing them wrong. But I trust that I have the gist right.)
It isn't clear from the description exactly where in the chain he fit, but I recall working for an engineering company in my first "corporate" job that a new line of products rolling out from an overseas division (I think in Italy...memory is hazy) Only problem was, they didn't work right - some software problems that made them error prone. They had committed to customers, and stockholders, for a launch by the end of the year, so they dutifully shipped thousands of products in the weeks leading up to Christmas. Not one of them functioned.
I don't remember the details, but they insured a component was installed wrong so that it would power up, but not operate. Customers returned those products, and they relied the calendar working out so two work weeks were burned due to holidays. They had engineers working overtime during the holidays to fix the real problem, and since they'd "broken" the products in a simple way it was easy to rework them and ship them back out in early January apologizing for the manufacturing error claiming a bad lot or something.
They would have got away with it but one of their executives got in a bit of legal trouble so he spilled the beans hoping for a deal. No idea what happened to him, but it caused a lot of higher ups in that division to be fired and maybe prosecuted (no clue there either) and the SEC got involved in the US since it turned out the top people here knew about it. I don't think they could prove anything but there was a shareholder lawsuit I got mail about for years because I'd held a few shares from a company stock purchase plan during that time.
The (extremely large IT) company I used to work for did something similar back in the day. Distributed thousands of corporate-branded USB sticks as part of a large scale marketing promotion; the sticks were huge plastic monstrosities (shark shaped, for some unfathomable reason) with 32MB capacity. MB not GB, in the days where 4GB sticks were the norm.
And they didn't work. Not a single one.
Somebody in Marketing had clearly scored a bulk deal for next to no money on some defective swag that was going to be junked, "saved" the company a fortune, probably got an award for it, and the result was that thousands of potential customers decided to steer well clear of us as we were clearly incompetent. Not a good look.
I doubt many customers changed their opinion of your company based a non functional USB key, unless they knew a bunch of others who got them and they all compared notes and found they all didn't work. Even back in the day dodgy USB keys were common, very especially when they were giveaways.
For a moment I thought I remembered a shark USB key then I realized it was a surfboard. Dunno what company I got it from, and I don't think I ever even tried to use it since it was like 2 1/2" long which made it really unwieldy and would probably have snapped if I put it in my laptop bag's pocket where I used to keep that stuff along with keys, phone, etc.
"I doubt many customers changed their opinion of your company based a non functional USB key"
Those who were already fully for or against the company probably wouldn't have changed their minds, true, but if you were on the fence (or shortlisting) it could very well have been one of those things that tipped the balance in favor of the competition. Bear in mind: (1) the swag was likely for a trade show (I forget, but probably) where many attendees would have been in the position of at least some purchasing power, (2) it wasn't just the non-functional nature of the keys, but also the relative size. 32MB when multi-GB keys were common. Poor quality, poor QA and cheap; not exactly making the case for your business.
"Probably" got an award? That twit in Marketing definitely got a nice fat bonus for that and that's been all the consequences for them. Marketing is never to be held responsible for their errors!
Back in my QC days we had four 40' containers of 50 m cable reels ordered by a genius in Marketing that were an immediate fire hazard. They did fire the people responsible - for not informing the higher-ups. Other similar debacles were blamed on QC and suppliers...
And now I recalled getting a 32MB USB stick from the big blue, at a time when 4GB would be normal...
But it did work, and was still working the last time I found it in a drawer and checked it a few months ago, so it might not be the same.
Our new boss decided that the product could be delivered much faster if we could eliminate some of the antiquated procedures we had adopted. Like inspection and pressure-testing before assembly. He placed the order for the materials from an overseas supplier for 'finished product' i.e. inspected, tested, certified, documented and fully compliant with our requirements. A big saving in price. And early to the schedule. An early adoption of 'Move fast and break things'. What could possibly go wrong?
As instructed but against our instincts, we carried out no validation of our own, assembled these major components into our machines and prepared them for customer-inspection and performance test. My instrumentation immediately started raising alarms which were initially brushed off as operator incompetence; a label I never dispute, at least at first. We had all the supplier-documentation which confirmed compliance. However, as the puddles on the floor suggested, these containers were not pressure-tight and never had been, despite the certification, and certainly not suitable to prevent water reaching the high voltage insides..... To his credit, the boss didn't ask me to 'review' the indicators.
It took ages to rectify the problem involving a lot of dismantling, assembling, retesting on multi-repeat. The schedule went away on a handcart. Rework and late delivery meant extra costs....
One would NOT hope so. The better solution would be to ask The Boss to sit inside the box and watch to see where the leaks were on the inside, giving him a paint marker so he can mark exactly where the leak is. Tell him that if they can solve the problem that way he'll become the hero of this disaster movie instead of the villain.
Then you power the box up and turn the taps on.
"OH NO, THERE'S BEEN A TERRIBLE ACCIDENT!!!!!"
I hope the power supply inside the device wasn't one of those prissy, modern, tiny, switching power supplies with dead-short protection and suchlike.
It works best when it's a large, heavy, old-school, iron-core transformer-based power supply: the kind which keep on giving the amps, no matter what!
Regretfully, no.
I had a lot of trouble with him although to be fair, he was trying to get things processed more quickly. We were a relatively new company, built with 'old hands' who had lots of experience. Unfortunately, this is often heard as 'stuck-in-the-mud' to those with 'vision'.
Typically the shareholders sue the company and the board/management. Suing the company might seem self defeating but it would be a limited class - shareholders of record at the time perhaps who due to particular transactions lost money due to the scandal/investigation. So it is sort of "future shareholders paying past shareholders".
I remember getting some small award, because it was below a certain amount I could accept it in shares or via check and I took the latter. Don't think it would even buy lunch, but I only had a few thousand dollars in company stock.
Most likely, Tex was not so much fired for pointing out the error per se, but for pointing out a bleeding obvious error the manager has managed to miss. It reeks of an organization suffering from injelititis or palsied paralysis, first studied by C. Northcote Parkinson, where high levels of incompetence and jealousy in management mean that anyone showing any degree of competence is removed, as they are perceived as a threat.
During the Y2K project I was in, my team was correcting an application consisting of almost a 100 programs chained together (just as COBOL likes) and we were made aware that that particular batch chain would break every single night, meaning an hefty on-call bill. And is had been going like that for years.
So I analysed the failure point and it was seemingly simple to correct - IIRC there were numeric conversions that would fail under certain conditions, conditions that would almost certainly be met once or twice per run.
That correction was deemed out-of-scope by our client rep, who, curiously, was also that application's IT boss.
So let me guess, even more curiously this IT boss/client rep had a financial incentive to keep the on-call bill nice and padded? I think I might have made a nice anonymous email to someone up the foodchain. Call me petty.
My previous employer decided that they wanted to bring in an MSP.
We were INCREDIBLY understaffed and this was their solution to "Well, actually, you're just too busy to spend time babysitting the boss when he can't open Teams, so we need more" and rather than hire, they decided to try to get an MSP. With an unstated and repeatedly-denied purpose of replacing in-house IT, I'd like to add. It was so blatantly obvious.
Anyway, they brought on an MSP, despite my objections, and that MSP proceeded to lecture me on my job, and claim to know better than me about everything, and tried to take over everything (it was clear that their brief was "take over the IT" while that was always denied in meetings, etc.).
So... I let them. Not without objections, and clarifications, and pre-warnings and I-told-you-so's, but I let them do it.
It turned out hilarious.
One of their "network team" (they had a "team" for everything, but those teams were always busy with whatever they did, which meant that the slightest query always went back to the MSP, lingered for days, then came back half-hearted with no time spent on it, a bill for doing that properly, and "no, our other guy you're paying can't do that, it has to be the network team") literally lectured me on NTP servers. One of the most trivial and relatively unimportant things ever... we had no need of time sync beyond basic domain operations. But they decided we were "wrong" with our "non-standard" deployment, because we were using a local NTP server and one of our remote NTP servers was one they hadn't heard of.
What they failed to take account of? The local NTP was a literal Bodet NTP radio clock sync device, the kind used in stock exchanges and railway operations. It was designed for site-wide sync and someone (*cough* me *cough*) had bought it as part of an all-site tannoy-like system because it synced time for free to all the units, used GPS and radio for timesync, and provided a local network NTP server that was certified to some ridiculous accuracy. We never needed it, but it was already there and cost us nothing... why not use it?
No, apparently, we had to use time.windows.com.
Then they argued about the remote service and demanded we replace it with NTP Pool servers. They gave me some huge bluff piece on it, and there was a LOT of time wasted on this, especially for something that we absolutely did not need. That's why they tried to lecture me on how NTP works and why the pool was better, and how to configure our NTP. In the middle of which they specified settings which were both insecure but also... that included a particular NTP pool server (in an incorrect way to address it, I'd like to point out). I let them argue with me some more. Then I told them. That's my server. It's literally mine. I operate it. I joined it to the pool. It's been there for over a decades. It's one of the more reliable in the pool. It handles more NTP traffic than the entire commercial network we were using for that employer. Every day. It's literally my personal server that I operate outside of work. You're telling me to use MY server. Then you're telling me how MY server is configured and how it works and that I should use it.
And the fact of the matter is... we already were, via the use of NTP pool. They just didn't understand how it worked.
I had similar run-ins with them on all kinds of issues. They replaced our perfectly-functional intra-site VPN with one that literally didn't work. I know why it didn't work. I told them. I had even pre-warned them, and dropped hints at every opportunity and told them explicitly half a dozen times. But their "network team" never understood that they had to route additional subnets over the VPN or those subnets wouldn't work on the remote site. They were trying to pretend they could interpret Wireshark traces. They were trying to pretend our networks were undocumented (I literally pointed them at the existing, working configuration in plain text). They were trying to pretend that we were doing something completely impossible (It's bloody working already!!!!!). Etc. Etc. But if you don't route those additional subnets over the VPN... then the VPN isn't going to bother to route that traffic.
Everything on the main subnet... fine. Everything on another VLAN/subnet... never transited the VPN. Access control stopped working. Telephones. Printers. Digital signage. Anything on another subnet didn't work on the remote site from the second they put in the totally-unnecssary VPN of theirs. The irony was... their VPN box was literally just a VPN box. To operate, it had to sit behind the routers at each site that... had been running the VPN between those sites. So we had had to turn off the VPN functionality on the router, install two expensive boxes behind them, configure port-forwarding etc. for VPN ports on both ends, and then have the boxes route the VPN traffic... badly and incompletely.
After six months, my employer got tired of the constant arguments and people complaining to me about stuff not working at the remote site (and I just filed tickets with the MSP... not my problem!) and told me to back it out. Ten minutes later (I had saved the config), the VPN worked and all traffic routed properly and we threw away the VPN boxes they'd made us buy. I mean... I had literally told them what was necessary and told them that we routed several subnets over there... not once did they ever put in any additional config to route those other subnets.
Similarly, they installed an new high-availability router device. Massive, expensive rack-mount thing. I asked how they intended to deploy it. They said it had to sit directly behind the main gateway. Okay. Well... we have two gateways, you know. Because we have two leased lines to the Internet for redundancy. And we use both in failover. They said it had to sit directly behind the main one. But what happens when we're in failover? We get no internet, that's what. Again, "the network team know better than peons like you and they've spent months designing this and you're just a guy we intend to put out of a job".
(shrug). Okay.
Because of power and other problems, that site would failover about once a week. There was a reason we had two leased line, two routers, at either end of the site, on independent power supplies. So within weeks, it failed over to the other device. I asked why it wasn't working was still in place. But obviously they couldn't contact their device. Everything else was still working (because we designed the network to work like that, and they were well aware of that) but the router was now entirely out of the loop, sitting on a dead gateway. Worthless.
There were MONTHS of that. Literally MONTHS of accusations flying around about how we must have turned it off deliberately (I honestly didn't need to sabotage the idiots, they were doing quite well by themselves!). But ultimately, they realised... this wasn't going to work. Not only had they spent months putting in a device in a terrible configuration, but even the "HA" portion of it literally never worked. Not in a single demo. Not once. Never. They even made us run 100Gbps fibre between the routers SPECIFICALLY for HA heartbeat. Fibre was fine. HA never worked.
They did something else similar. Bought an IDS/IPS. Attached it to one gateway. It never detected anything. Literally nothing. I kept complaining. And complaining. And complaining. And it kep getting escalated but they assured me it was all working. I got it in writing. MONTHS this went on.
It was at that point that I pointed out that the device they were supposed to install was still sitting in the rack, uncable and unpowered. They'd racked it. And that was it. Then they tried to blame me, but I had not only a trail of evidence, but I'd deliberately pointed it out to my boss who had - sensibly - not said a word and let them drop themselves in it.
When they did cable it and turn it on, it didn't work. Why? It was only monitoring one gateway. Fine while that was the active one. Useless when it wasn't. Our network was unprotected 50% of the time. They claimed that it was fine and they'd checked with the manufacturer and it was a supported config. Strange that. Because I had a written statement from the manufacturer (thanks BlueDog!) who had agreed to talk to me when they realised I was the customer, that they had SPECIFICALLY warned that MSP (the reseller) that they would need at least two such devices, and it would be worthless without. They sent me the email chain. Where the MSP dismissed that and just ignored it, repeatedly, against BlueDog's advice.
Whoops.
But apparently, I "don't know what I'm doing" and their "expert network team" were geniuses. The same geniuses who trashed our SAN in the middle of the day by stomping over IP addresses on a reserved subnet, the same geniuses who couldn't deploy a webfilter onto a Chromebook over WEEKS and said - quite literally - that it was impossible to do with the kit that THEY THEMSELVES had bought, and implied it wasn't possible with any kit. They said they'd never seen a project "so set up to fail" and basically accused me of sabotage (again!). (Presented with this fact by my boss, I took out a fresh Chromebook out and configured it in five minutes in front of the senior management and showed them that it was working... they took it away and confirmed it themselves and presented it to the MSP who were forced to admit they were wrong). The same geniuses, in fact, who remotely booted our in-house team out of remote desktop while we were working on the servers, logged in themselves as the user we'd given them, and for some insane reason decided to APPLY all checkpoints on the VM cluster. Not delete... not housekeeping... not tidying... they APPLIED them. Without warning, reason or permission. Rolling the entire network back months in the middle of the working day. Then denied it. By which point I was already on the phone to my boss telling them what I was watching happening as we spoke and they came and witnessed it. Restore from backup was required. Of the 2-node S2D cluster that they'd forced us to migrate to. Oh, with 1Gbit networking that was run off the motherboard network port, which then failed.
But apparently... little old me, with my strange ways, and my lack of certifications (just a degree and 25+ years experience) was the dumb one who just "didn't understand anything" and their "expert dedicated teams" for security, networking, servers, implementation, etc. did. I wasn't "trained" like their teams of experts. No.. I just built and ran the exact servers they were lecturing me on using, I just designed the network and told them exactly how and why it worked and watched them fail repeatedly. I just realised that if you have two paths to the Internet, then you need to cover both. And so on.
So after we'd dropped some £200k+ on these idiots, they were finally let go after an absolute screaming match where they decided it was a good idea to yell at our only senior management who understood IT and had been brought in specifically to mediate and clear up the confusion (i.e. me telling and demonstrating that these people were idiots, and the MSP playing their roles as idiots perfectly). He calmly replied things like "No, that's absolutely not true, though, is it? That's not how it works, or could ever work." (on technical matters) which basically convinced the entire senior team that actually I'd been right all along.
I asked for a reasonable raise. Was denied. Got a 20% raise elsewhere within HOURS. Went back to them. Nothing. Quit.
I hear they're now employing another MSP, under the charge of the senior manager who had a clue about IT. At significantly more cost than ever before. And they have had to seriously dial down their expectations of the network that were given them when I was running it. You think that a tiny team of in-house staff running EVERYTHING with response times in the minutes was your problem? Okay, see how you deal with an MSP with 40 staff that won't even respond to a ticket same-day most of the time, and who just pass stuff off or back saying it's not to do with them all the time.
Little ol' untrained me. Now working at a bigger place, earning more, actually working less (hours, holidays, systems that were budgeted for appropriately, etc.) and under comparatively zero pressure, with no sign of an MSP (and no intention to get one).
Indeed, very much wow.
But whilst a long read, it's exactly the entertaining diversion that this column (and "Who, Me?") used to regularly serve up in the main article (as well as in the comments).
And certainly better than work on a Friday morning.
Most certainly a well earned one of these for Lee >>>>>>>
Working in an network and timing adjacent area I had to come up with a concept for management to consider. Part of the deal was working with the local supplier of magic-boxes to get their OK that the concept would work and that $$$ weren't going to be flushed down the dunny.
All went well, apart from a bit of a dispute/disagreement about VLANs [why is it always VLANs?]. Supplier (notably not the OEM) swore black & blue that VLANs could never be supported and were never in the standard and that I was a fool. I asked if they had a copy of the standard. They did. I asked them to read section X.Y (the bit that had mandatory use of 802.1Q priority and recommendation to use VLAN IDs too). Silence. And then some puffery about how "it didn't really mean that and that I was misinterpreting it".
Time to drop the clanger. I said "So you're saying that I don't understand ... what I wrote?". Confusion. And then I had the joy of referring them to page 6 of IEEE Std XYZ.ABC, where yours truly is listed as a Working Group participant, and the VLAN section was one of my contributions after getting burnt by dodgy vendors cutting corners retreading designs from other industries. Silence never felt so sweet. And then we got on with the discussion and everything was sweet as, and the design was endorsed.
Indeed. The inestimable Quote Investigator looked into it.
TL;DR: A precursor quote goes all the way back to the Old Testament, but the line only reached its modern form in a 1993 Usenet post. It's fascinating how these things evolve.
But the mis-attribution detracts not one whit from the line itself.
I had recourse to that yesterday, working with my manager on a knowledge transfer document to present to the rest of the team, he said it needs to be idiot proof - I said that wasn’t possible and putting that in the presentation would be seen as a challenge by some people…
But considering there is a 15 page document on how to configure a wlc, and so many far people have missed out pages such as licensing the controller, or set the option 43 address to point at a wlc half the world away (yes APs in Manchester instead of the local controller lets point it at one in India instead). I sometimes just look and go not again…
I'm reminded of the (possibly apocryphal) story of the UK company that ordered some mechanical components from Japan. In their order they stated that they would accept 5% of out-of-spec components.
A few weeks later the parts arrived accompanied by a note from QC. The note read: "We do not understand your business methods, but the out-of-spec components you ordered are wrapped in red tissue paper in the top layer of the first box."
I once worked for a company that was already in the habit of testing everything that went out of the door -- but one customer said they were prepared to accept a 1% defect rate.
For a long time, everyone was very happy with this arrangement -- until a communications error led to several crates of widgets being sent out with the faulty one in the same position in all of them, and the customer twigged to what was going on.
Once upon a time (later 20th century), Japanese product was regarded as trash, and it may have been. Fake lookalike products also were alleged. Then, probably with a well known effort, they got good.
The same was said about industrial China.
The same was said in Australia about British products, cars in particular - according to Clive James.
There's probably poor quality work everywhere.
I think the styled-like-fluorescent battery lamp I saw and carefully didn't buy, whose long opaque light box had an incandescent flashlight bulb inside, was from China. But that may mean Taiwan.
Some time in the last millennium we got our first batch of prototype PCBs back and the HW developers started to populate them, only to discover that somehow the pin assignment for one of the more important QFP ICs what not quite right, it was upside down.
The engineers then realised that by slightly bending the pins and liberal application of solder (capillary effect is your friend here), you can get proper connection to the PCB. And they worked!
But most likely this here is a badly placed component that was just sent through the reflow oven and somehow stayed on.
I once took delivery from a catalogue company of a top of the line Philips VCR which had StarText on it (an idea I'd had aged~14 but someone with an actual job had obviously thought of it at the same time; hence my desire to get my hands on one now I had a job and had some money - see how well it worked etc). Anyway, unboxed the VCR only to find a breeze block cellophane taped to a board in the bottom of an otherwise empty outer carton.
I do. A customer brought their PC in to our retail PC sales/repair shop, and the PC had a Miniscribe 3650 in it. It was a real screamer. Literally. I managed to back up the drive (shout-out to Fifth Generation Systems and their awesome FastBack program) before the spindle bearings totally froze.
I replaced the drive with a Seagate.
I also had inherited, for work purposes, a full-height Miniscribe drive at a different job. It worked fine, but had read errors on a certain set of tracks. I used it as a scratch drive, solving the read-error problem by low-level-formatting it with fewer cylinders than the drive actually had, reducing the drive's usable capacity by about 90 MB.
Having a bit of a flashback to my first "proper" job building prototype PTH style PCBs. I could solder very well, could read and understand resistor codes, knew which end of a chip had pin one etc. etc. What I failed to realize on day one was that, although at casual glance tantalum capacitors look like ceramic capacitors, they are polarized and yours truly got approximately 50% of them placed and soldered backwards. That was a lot of magic smoke on first power up.
1 2 3 4 5 6
Is a perfectly acceptable sequence from a perfect random number generator. As is 1 1 1 1 1 1.
The time to become suspicious is if those are the only sequences you ever see. But even then...
(Cue story about Apple making the iPod shuffle *less* random, as people complained about hearing two songs, one after another, the same way they appear of the LP track listing)
Decades ago I worked a "piece work" gig at a "used equipment refurbishing r/reseller" in a certain southern US State.
I came prepared with my portable PACE vacuum de-soldering unit and proceeded to being removing a DIP package similar to those depicted. After replacing a few, the "most senior" tech came over, with Boss in tow, to chastise me to "quit wasting time" by removing chips and simply to solder them into place ON TOP of the suspected defective DIP and move on.
When I pointed out this was not a great idea, and gave reasons obvious to anyone with any training at all, I was deemed "unsuited" to the position and given the booth.
I found later they were known to have a lot of customer complaints about intermitted failures on the gear they resold and just stonewalled any warranty claims.
I won't go into another place I later worked at in a Northern State, that scorned any kind of static protection when building or servicing PC and Network gear, including handling various RAM packages. The work areas were covered in synthetic carpet, to "keep things from sliding around" and the tossed DIMM's etc. with abandon. They spent a good deal of time jousting will suppliers over their "defective products". I presented "the Boss" with reprints of NASA and other White Papers that described how "anti static" measures were necessary. Not my most endearing move to management. Same guy that thought applying the "S" flag to the "everyone" group was the proper way to fix file access issues.
This post has been deleted by its author
Back in the stone age (1976 or so), I had a board designed after my hand wired breadboard "prototype". It was nice and all that, but the components were on the wrong side of the board. I went ahead and "approved". only to find out that later I had to make ALL subseuent boards the same way. I keep on kicking myself for this one screw up to this day.
Yes, the boards did work but the standard prototype boards you could get commercially were the designed "the other way" so it was hard to do prototypes without flipping things over. (*SIGH*)
A few years back I went stir crazy and began to feel like a caged animal at the desk, so I went into trades. I've since settled but at first I did a bit of hopping around to get in where I fit in. One thing I've learned along the way is that it doesn't matter how intelligent you are or what skills you bring to the table, to not a small amount of people "new" = "stupid" and you will be treated accordingly.
I worked for an agency that evaluated cryptograpic products.
As the computer guy, I was asked to look at the implementation details of a product, and lo and behold, I found the classical "reuse of the keystream" mistake.
I neatly reported this, but afterward I had no access to the report because I did not have the clearance to know about cryptographic weaknesses.
It might just be the US companies.
I once got hired by a UK branch of a US company as a Quality Assurance Lead.
I ran a proper QA exercise on their new product and found multiple serious bugs even though their carefully scripted test found none.
I was called in to see HR and was told my QA job was not to find bugs, but just to run scripts which would highlight any bugs that existed!
I was told that I was not the sort of person they employed and was not even allowed back in to collect my coffee mug.
A few months later the whole company ceased to exist!
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
