Techie fluked a fix and found himself the abusive boss's best friend Digital technology remains frighteningly finickity, which is why good tech support people are always in demand – and also the reason The Register never tires of telling your support stories each Friday in On Call, the column your generosity makes possible. This week, meet a reader we'll Regomize as "Boris" who years ago worked …
The pellet with the poison is in the vessel with the pestle, the chalice from the palace has the brew that is true, right?
Right, but there's been a change. They ... broke the chalice from the palace.
They ... broke the chalice from the palace?
... and replaced it with a flagon.
A flagon?
... with a figure of a dragon.
A flagon with a dragon.
RIGHT.
But, did you put the pellet with the poison in the vessel with the pestle?
Noooo, the pellet with the poison is in the flagon with the dragon, the vessel with the pestle has the brew that is true.
The pellet with the poison's in the flagon with the dragon, the vessel with the pestle has the brew that is true.
I was thinking to myself that I'd been in plenty of code reviews that sounded a lot like that ....... then I realised your comment was to do with the 'poisoned chalice' mentioned in the OP!
---------> It's a Friday. That's my excuse and I'm sticking to it!
For those too young to be able to remember Danny Kaye "The pellet with the poison is..."
Capt. Hans Geering:
Do you not see that if you kill him with the pill from the till by making with it the drug in the jug, then you need not light the candle with the handle on the gateau from the chateau!
Rene:
Simple plans are always the best.
Icon - Closest to the flying helmet (Wet Celery Optional)!
https://www.youtube.com/watch?v=kj5gJqy6Q-8
This post has been deleted by its author
This post has been deleted by its author
Pellet, flagon, vessel and brew sounds like the "Copper Clapper Caper". I leave it to others to get the proper URL, but it is classic from the 60's I believe (Tonight show in USA).
No, not "Four Yorkshiremen" but close.
Just looked it up:
https://www.youtube.com/watch?v=zOJGlhw5Jbo
> Have you ever found a fix despite not being an expert in the troubled tech you were asked to tend?
I've maintained code in languages I don't know, and in one case I didn't even know what the language was, but generally it's possible to reverse engineer the syntax by looking at the rest of the program. It only becomes a major problem when you need to know the type system.
Yup, happened to me too.
Client was ranting about a script that wasn't doing its job anymore. Instead of asking the coder (who had apparently left), I was the one it fell on to try and find a solution.
Thankfully, my 30 years of coding experience in another language (and ample what-the-frak-is-this experience) allowed me to pinpoint the solution, and leave the client happy.
I have never done coding so by the seat of my pants as that.
I was sat down in front of something that vaguely resembled an Osborne luggable computer and told that as I knew about computers I could make this "thing" work.
The "thing" was a programmable logic controller that drove hundreds of hydraulic rams and solenoids with an equal number of limit switches, buttons, and levers. My very rapid introduction to ladder programming.
I once made a fix to a legacy system written in FORTRAN 77 (with the inevitable vendor specific extensions), as with our sole FORTRAN guru was off work with long term illness and I was the only other person who had ever used that language. My total experience was writing a few library functions to emulate string functions in the C standard library and then using them to write a command line argument parser as a front end to batch processes. Thankfully the fix was obvious and our guru eventually returned with his health restored.
Webhosting!
I wouldn't call it bluffing, I frequently tell clients 'I can't write this stuff at all, so I'm starting from scratch with google on hand, it's really really inefficient asking me to look at this'
Why do i end up there? Because they've gone out and spent a bazillion quid on a fancy new website that runs like arse and the people that build it say "we don't usually have these issues, there's something wrong with your server init!"
What is is always down to? It's yet another Turdpress site where someone who doesn't understand all this stuff any better than I do has added every plugin under the fucking sun and at least 3 of them are sat there waiting from data from some non-existant API or stalling on a DNS lookup that just isn't going to happen.
Lost track of how many times I've had to add an admin account to the database and go rummage around.
I worked for a Japanese office equipment company and was debugging a word-processor when one of the Japanese managers walked by. I was reading the source code out aloud (doesn't everyone?) when he asked me where I learned Japanese. Funny how he immediately understood the purpose of a variable but I had to pour over the source code.
I entered the Linux Kernel's thank you list many ,many moons ago (think early 90s) for finding a missing closing "]" in the generic Broadcom network adapter's wrapper and emailing the coder about it.
It was maybe the fourth time I had to recompile a kernel and I had exactly zero coding skills at the time, but I was desperate to make those generic network adapters work and since Google was still a mere dream, I decided to look at the compiling error and take a hit at the files.
I was once asked to look at a model (I'm not a modeller) that ran on modelling software (that I had never seen) to try to help with the issue where the run times were going up exponentially as the size of the model was increased.
I looked at the model, and scratched my head. Then dug into the modelling software, more heady scratchy.
I ended up doing a setting by setting read up of each of the config settings. That's when I discovered that the initial memory was something like 50KB, and when more was needed it was increased by something like 10KB. On the off-chance that the software was spending all its time increasing the size of the memory pool I jacked those numbers up to values more in line with the resources available ....... and suddenly hours of model time shrank to mere minutes.
This, of course, led to much rejoicing in the halls, tickertape parades on Main Street, and the dancing of Ewoks!
Ive been in a similar situation. This problem had been going on for months. I was asked to go on site and be a fresh pair of eyes.
I got the joblogs, and one of the first messages was "storage pool allocations are too small". Apparently I was the first person to have noticed this.
We made the storage pool bigger, and tried it on the tests system, which suddenly went much faster. Although the storage pools were not my area of expertise, I read the doc, and looked at the stats. The stats gave recommended storage pool.
We made the values for production to match what we tested, and restarted production.
Next day the operations people rang up the system progs to say "The application was using about 10% of the CPU it used to - was there a problem?"
I stayed on site for a couple of days, and spent my time poking around other applications. Several applications had the same problem.
The outcome was they deferred a machine upgrade, because now they had spare CPU!
> I got the joblogs, and one of the first messages was "storage pool allocations are too small". Apparently I was the first person to have noticed this.
God damn... if I had $5 for each time I've gone "what do the logs say?" and gotten "no idea" or even "logs? huh?" I'd be telling Elon Musk how many months I wanted to stay on ISS.
Is and has been key to finding so many problems on critical/complex systems YET, the amount of pushback that I received from management when suggesting adding it to the system.
At one site and despite me having the whole package ready to go they said NO. That software has not been developed here so can't be used. NIH to the extreme...
I declined to extend my contract and soon found another one this time in Denmark (this was before BREXIT).
Queue panic phone calls from the very manager who had said NO.
This time, I said NO.
He asked how much to buy your code?
I gave him a silly figure he said yes.
Sadly, before the contract had been agreed, he was fired and the project canned. So near but so far.
I sent in a patch to a bit of open source software, which parsed & formatted GPS logs: it was ok for a log over an hour or so, but feeding it the week's worth, or more, from a touring holiday ground it to a crawl. All the time being spent in allocating little chunks to hold only a few records each.
Obvious patch: as you say, increase the allocation geometrically each time and it chomped through the data with ease.
The patch was accepted, but with a caveat: the maintainer had "removed the unnecessary memory alignment code". Yes, the (nextblocksize <<= 1) was gone, the shift removed, to, um, no longer block align the memory on a power of two boundary?!!
I stuck with my version.
«the run times were going up exponentially as the size of the model was increased.»
Usually a simple problem in the initial parameters setting scale etc
Climate scientists in the 80s and earlier were often computer numpties so when they decided to take their model from a 100km×100km grid over most of the South Pacific to a 1km×1km grid it was no surprise to anyone except themselves that the model (and minicomputer) crapped itself.
A little explanation that even though each cell had only six single precision reals going from the 100km grid to the 1km grid involved a factor of 10,000 (not 100) caused more than slight embarrassment.;)
Large minicomputers of that era would have had just enough ram and swap to run the original model. Probably about 10,000 cells with six 32 bit floats so about 256Kb but the finer grid would need +2Gb which was more than the total risk storage.
Even today you might have to introduce "developers" to mister big O().
Quoting Digory Kirke "...bless me, what do they teach them at these schools!” (Although I am not sure Plato be of much help here. ;)
> A little explanation that even though each cell had only six single precision reals going from the 100km grid to the 1km grid involved a factor of 10,000 (not 100) caused more than slight embarrassment.;)
For my sins, I maintain an MIS platform which can do various number-crunching things, including spreadsheet exports.
Occasionally, a beloved customer will grumble about how long it takes for certain files to be exported. And this generally boils down to:
You've set a 2-year date filter, which means it's picking up 200,000 items. Meta-data for these items is spread around several dozen tables, adding an extra hundred or so records per item. And then the final spreadsheet contains about 200 columns, most of which need to be transformed in some way.
So, that's around 20 million database records which need to be crunched together, after which the system has to render around 40 million "spreadsheet" cells.
Oddly, this doesn't happen instantaneously...
"Mister Big O()", along with,"Disc accesses are slow.", "Do not use spreadsheets to store databases or documents.", and, "Avoid inner joins." are engraved in into the stainless steel of my (L)user Adjustment Bat, such that when they look at themselves in the mirror, they can read it.
Offending (L)users soon find an item in their calendar reading, "Going on a LAB exercise."
This post has been deleted by its author
While it's perhaps not quite in the spirit of the discussion as I was at the time paid to do this stuff ...
With a previous hat on our employer used a piece of software to run the business that came with a built in report generator. It was reasonably capable, and fairly neatly integrated in that it looks like it was part of the system rather than the bought in and grafted on bit of software that it was. However, it could be "a tad inefficient", and more to the point, a tad inconsistent. You could be building up a report and it would run just fine, then suddenly the run time could expand by orders of magnitude - a seemingly innocuous change would result in it not using indexes. When you have an order line table that grows to ~1GB by the end of the year, but your machine is limited to a little over 1/2G of disk cache (OS limitation - who else remembers SCO OpenServer and it's "admin set & needs a re-link on change but with a hard upper limit" cache size ?). So make a small change, suddenly it's doing a non-indexed join onto a table twice the size of the system's disk cache, which results in everything grinding to a halt with the system on 99 to 100% WIO.
So I sat down, re-wrote the report in Informix, taking care to use indexes and some temporary tables, got the run time from "2 days and it can only be run at the weekend when hardly anyone is using the system" to "under 2 minutes and no-one notices".
Of course, if the software ran under Linux (at the time is was SCO OpenServer or AIX), then we could have just thrown a few gig of RAM at the problem so it could keep the live database in cache all the time - which would have solved some other performance issues as well. The budget wouldn't have run to AIX and the hardware to run it on, and I've no idea if that had any similar limitations anyway.
This post has been deleted by its author
Very similar to the "go onsite to the 'always failing' system, and ask for a demonstration".
The demonstrator follows the instructions as written and it does not fail.
I think we all know where the problem lies here?
Yes, it should not allow the failure if they are not following the instructions, but hey-ho!
This post has been deleted by its author
This post has been deleted by its author
We had a tester who was great at finding bugs. He would power off the machine, and make sure it recovered - and any in flight work backed out.
He was asked to write a paper for a prestigious journal about testing. So he wrote about how he would test a car. Test 1) Drive it in reverse as fast as you can. Test 2) Drive it through a forest. Check the body shop can repair any scratches - and have the right colour paint. Test 3) Drive it along a stony beach. If you get a puncture - check there is a jack - there are instructions, and there is a spare tire, which fits. Test 4) Give it to his granny - with no instruction book.
It was a great article - very clear etc. The paper was rejected by the journal as being too obvious. They were looking for things like standard deviations from the normal, and mean time to failure.
Unfortunately that's not the sort of thing journals are looking for, but that'd be a great article to publish in a book or just standalone on the internet.
It's okay to publish "obvious" things, but only if no one has published it before and you can scientifically state the obvious.
That being said, I'm studying business informatics now, and I can't stress enough how important having field experience in helpdesk and systems design is. So many "obvious" things get overlooked if you don't have the right troubleshooting mindset (or experience dealing with end users). I notice how oblivious and ivory tower-esque some of my peers are, and i feel bad for whoever they end up working for.
As a child, my father, the head of a local secondary school, took school holidays walking in the hills, Cumbria or North Wales. I was always the youngest by over a year. Anyone, one year I was sharing a bedroom with one member of staff and two sixth-formers. None of whom could get the heating to work. So I told them how to operate the machine:
Staff member: How did you know that?
Me (pointing): I read the instructions on the top.
Irritating, moi?*
*Strangely I survived childhood with very few life threatening 'accidents' or near misses. I guess being the head-master's son had some benefits, after all.
> We never had a clue WTF that was all about, but hey, if putting the fear of Christ into hardware is enough to get a fix, I'm good!
Back in the day, I was providing 4th line OOH support for a system which occasionally needed a bit of a kick. And there was something of a running joke about a particular (and occasionally cantankerous) system: it would only restart if I was involved.
It got to the point where 3rd line support had tried several times to get it restarted without any success. So they rang me, and tried again while I was getting my laptop booted.
And it restarted flawlessly. Because it knew I was watching ;)
Aside from the interrupted sleep, I was quite happy to claim the overtime, and to bask in the worship of the slightly bemused third line team...
It's a funny one that. In the past I've had a bit of a reputation for fixing things just by being there - as in : user calls to say there's a problem, you talk them through options, finally walk over to their desk, and as you get within sight of the PC it decides it wants to work properly now.
Just a comment triggered by your handle...
My wife wrote an SF novel which involved a VR system. At one point, one of the people who wrote the system comments that one of the design decisions was a "big endian vs. little endian" issue. When the galleys came back from the publisher, that had been changed to "big Indian vs. little Indian". My wife changed it back, adding a marginal note of "See J. Swift". When she mentioned it to me, I pointed out that "big endian vs. little endian" is, in fact, a hardware architecture debate. My wife was not a techie.
(So as not to leave everyone hanging... The work is "A Point of Honor" by Dorothy J. Heydt. It is out of print, but can be downloaded from her web site.)
This post has been deleted by its author
About my only exposure to Cobol was when my then boss asked me to take a look at some code she was struggling to debug.
I don't think I'd ever even seen the language before but I was trained and proficient in BASIC and Pascal*, and had a self-taught working knowledge of FORTRAN, so simply going through the code logically was easy enough. It only took a few minutes to find the obvious error, I'm not blaming my boss, it was one of those where fresh eyes were needed.
*Unlike other establishments at the time Newcastle Uni focussed on programming skills in a single language, UBC Pascal, rather than teaching simple skills in multiple languages. It may have reduced immediate employment opportunities but overall it's stood me in good stead for nearly 40 years.
I started on Algol/Basic then moved onto Fortran. These stood me in good stead for a long time, I think it's the discipline in logic that guide and help in fault-finding in any language or system (how could it be otherwise?).
I was very disappointed to learn that my new Engineering Graduates had learned no programming language at all at Uni. It shows.
The other technique I figured out when bug hunting during development was to say to yourself - If I wanted the results that I'm getting how would I code it. You then end up with a second set of code in your head that you can mentally compare to the code that you originally designed, and 9 times out of 10 there will be a line (or area) that you need to change to flick the functionality over to what you're getting rather than what you want.
So instead of asking how do I stop that behaviour, asking how would I get that behaviour has been the quickest way for me to fix bugs.
I haven't done it consciously for a long time, cos I think it became such a habit that my brain now does it in the background, which means that when a bug pops up most of the time I'll immediately just scroll to a certain place expecting to see a certain bit of code that is certainly wrong! That's for certain!
I prefer "did". I could speculate about places where it could have come from, and I often do, but that is rarely as helpful as drilling into it and figuring where I got it from, where that thing got it from, and so on until I find the first thing in the chain that gave me the wrong thing. The other risk is that I posit something as an "it could have come from here", the recipient hears it as an "the problem is here", it turns out it didn't come from there, and we have disagreements all round about whether I'm an idiot and did what I should have done.
"I was very disappointed to learn that my new Engineering Graduates had learned no programming language at all at Uni. It shows."
I must ask our new mech eng. graduate whether he learned a language...
I learned C at uni on my mech eng course, but I had to use Fortran immediately after I started work 26 years ago, and since then I've had to use awk (yes, really), Matlab, visual basic, tcl scripting (yuk!) and Python. I have only ever had to use C to alter a small piece of code from an obscure in-house FE pre-processor, to get it to compile on a GNU compiler when it was originally written and compiled on a Sun workstation. For the record, I work in mechanical engineering, I'm still occasionally using Fortran for real engineering purposes and my role is *not* IT related....
When I was getting my mechanical engineering degree, I had to talk my advisor into letting me skip FORTRAN. After all, I was almost done with a minor in computer science, having programmed in C++, Java, Assembler...
And now I work with TCL - but the other one, that almost no one has ever heard of, Taylor Control Language.
This post has been deleted by its author
I had a case where a script I had written on client site just refused to work.
I know how to write code. I've been writing code for practically all my working life. This code should have worked. But it didn't.
Try as I might, I banged my head against a brick wall for more than an hour. The I called for help. The project manager (who knew code) came to take a look. He didn't find any fault. That was fine and dandy, but it didn't solve the problem.
He called a colleague. After a full half an hour of code review and discussion, someone (not me, but can't remember who) mentioned : "hey, shouldn't those be points in the mail address ?".
Problem solved, with three sets of eyeballs after an hour of debugging.
Sometimes there are things that you just can't see for yourself.
Similarly when I was at Aston Uni the standard lanuage for the course (or at least the first 2 years) was Modula-2 (aka Son Of Pascal). Also on (almost) the first day one of the lecturers stood up and said "...we could teach you somethign commercially viable like Ada, but we're not into that..." The next year the standard language was changed to Ada...
And how much commercially-viable Ada code have you seen since then?
FWIW I was taught Pascal first, as "just a simple teaching language" instead of nasty commercial C - and then spent a good few years being paid to work with Turbo Pascal.
(and picked up C, C++ - when it finally came along - on the job)
Back in the 1980s the OU taught programming using HEKTOR, a simple computer that had a UHF modulator which you plugged into the aerial socket of a TV. The course began by using pure machine code then using a very, very simple assembler. Once you got the idea you were introduced to BASIC. That stood me in good stead at work when I encountered 6502 assembler (Commodore Pet and Raeto West)
Had an analyst who loved reading data from 12" tapes and failing to check for PEOT, just kept on reading till the tape was all wound off the spool into the innards of the MictroVAX II (Yeh - I forget the drive model but it was the one which mechanically slurped the tape off the spool into the depths of the drive for the takeup spool)
That's when I started to learn FORTRAN and VMS system calls to fix her code for her.... mostly cos I was fed up with respooling the tapes!
We got on a bit better after that, but she was a really odd one... and this is me calling someone odd!
I did that too, when I was a Ph.D. student using a VAX 11-750 as a visitor at a famous but now sadly defunct government research institute. I got a royal bollocking from the system administrator who had to fish the end of the tape out of the reservoir of the tape drive and painstakingly thread it back onto the reel. I only made that mistake once. Happy memories.
How much luck did this guy have? This was a much bigger stroke of fortune than spotting the initial problem - otherwise he would have been stuck in the position as the go-to guy who could solve all the problems, having to do all the work whilst watching sundry mediocre managers take all the glory and all the bonuses for everything working efficiently. It might seem appealing to be seen as the "resident genius" , but it can be exploited by the incompetent and cynical, and therefore being the go-to guy should mostly be considered harmful.
"being the go-to guy should mostly be considered harmful"
Only if you're not freelance. Being the go-to guy gets you the gigs without having to use the pimps agencies. Because you're only there for the length of the contract you automatically duck out before it's too late.
I talked about this recently,, but it'll sit well in this thread, so here it goes again.
I once stopped by a PFY trying to fix a user's Access application that had stopped generating email reports. He was at wit's end, having tried for hours everything he thought might possibly solve the problem. I nonchalantly asked the user if IT had done any upgrade that day. "Hm yes, they installed new printer drivers".
Ah.
After changing the default printer to PDF, the emails started once again being generated, because of course Access needs the printer driver to generate reports, even if they're not being actually printed.
At one place I worked, the accounts dept ran some software on an Apple II computer for reconciling invoices received against orders. Apparently the software was getting slower and slower to run as the amount of data increased over a year or two. It got to the stage that the monthly run was taking 2 or 3 days to process!
I was asked to take a look at the code. Essentially there were two long lists of data and it had been coded to read each entry in the first list, then search the entire second list, record by record from disk, looking for a match. As both lists had far more than 1,000 records it implied over one million records were being read and compared in all. I rewrote the code from scratch and did some pre-processing first, sorting both lists into alpha-numeric order using a Shell-Metzner sort algorithm I'd seen in a Commodore Pet programming manual. Then all that was required was to step through both lists at the same time comparing records in a sort of crab-wise motion. Only one pass of each list was required. This got processing down from several days to around 20 minutes! After that I was given lots more programming tasks.
This post has been deleted by its author
I had a similar experience. In this case it was a client-written report for a commercial ERP system. I think whenever they'd tried to run it they ended up knocking on the head after a couple of days. For reasons* we weren't allowed to set the optimiser statistics. I was asked to look at it during my first week on the gig.
The report queried a mix of big and small tables. I turned on the query explanation mechanism to get the query plan and found that it had chosen to do the initial selection out of a small table and then join it to a big table on an un-indexed column. And then another. I rewrote it query in a sensible order into temp tables. By the end of the morning I had it down to a run time of about 10 minutes.
*I suspect because the package had been written by someone familiar with the older version of the RDBMS who was used to hand optimising selections and didn't trust the new-fangled optimiser. I'd also been used to the older version so I knew such tricks existed. But I'd enough experience to trust the optimiser.
> The report queried a mix of big and small tables. I turned on the query explanation mechanism to get the query plan and found that it had chosen to do the initial selection out of a small table and then join it to a big table on an un-indexed column.
Indexes for the win!
We recently had a server which was running hot and building up a backlog. Investigation took us to a query which was using an index... but the explain plan indicated that it was matching around 8 million records. Cue a new composite index, and now the system is only matching 4 records.
Which is a slight difference...
Got called into a database slowdown problem where the overnight batch processing was getting forever longer, until it would soon take more time than the 8 hours allotted. The problem was easy to guess by simply monitoring the SQL - >99% failed inserts and rollbacks. For the first order to be processed, you need to get the next unique ID number, so insert ID = 1. If that fails with a duplicate error, then insert ID + 1. If that fails, insert then ID + 2. etc. At some point you will find the next available ID. How dumb is that? But wait. For the second order to be processed, get the next unique ID number so insert ID = 1. If that fails... Even if you spotted the obvious first error, of not selecting max or using a serial sequence value as the initial ID, it was mind boggling to think that someone had coded the the algorithm to always start at ID = 1, and then wonder why the batch run got longer, and longer, as the number of orders in the database continually increased.
I found myself in a similar situation on a New Year's eve some 20 years ago. I was working in a financial institution, in a team supporting an application that consolidated market prices which were then used by a plethora of accounting and risk systems. One end-of-day job was failing after 30 minutes and after several failed relaunches we concluded that we had to look inside the code and fix whatever was taking too long. Of course this being the last day of the financial year, this job was one link in a long chain of critical processes, so there was a certain amount of pressure to solve the issue quickly.
This was a long bash script, so by adding a few traces to stderr we quickly identified the problematic part: it was trying to match and merge 2 very long text files by grepping a keyword in the first line in File1 to the whole of File2, then moving to the 2nd line of File1, etc... Since both files had tens of thousands of lines each, this was obviously most inefficient and it was frankly surprising that this hadn't happened sooner [1]. We wrote a quick-and-dirty Perl script to do all the matching in 1 or 2 passes [2], and the job completed in a few minutes.
[1] of course, Sod's law dictated that it had to happen on a 31st of December of all days
[2] soon after, I learned about the 'join' Unix utility that would have done the job even faster
TBH, I've done stuff like that in scripts - but with enough nous to know to only do it when you know the file sizes won't get big.
At one point I tried to learn PERL, but this was in the days before there was a wide selection of inexpensive structured training courses available on t'interwebs - and my budget only ran as far as buying a book (employer "not generous" with pay, and even less generous with stuff like training).
I've had very similar recently.
Someone complained that the system was going slow because their spreadsheet took a long time to refresh.
When I examined the spreadsheet I realised they'd got VLOOKUPs on each of several thousand records to a long list on an external spreadsheet that was stored in SharePoint. Given that both lists were refreshable queries to an SQL database they only had to ask and I could have amended the base query in a few minutes. They didn't get the faster laptop that they were hoping for!
I was once asked to help out an affiliate at an old employer.
The affiliate's code was in PHP which I'd never used before but I was able to understand enough to find numerous errors in a few hours and fixed it for them. Typical offshore code written with assumptions and no thought of "what would happen if?"
Like most languages I know now, I picked up enough in a few hours to understand the code but like a golfer, I'll spend the rest of my life actually trying to be good at it.
I have on occasion found bugs in students' code in a few minutes after they had struggled with it for days. I often just scrolled up, because I knew the error wasn't where compiler or debugger flagged the problem, but a bit earlier. Students were often very impressed, but I always reminded them that the reason I knew where to look was simply because I had made that kind of mistake far more often than they had.
. . . and no semicolons in FORTRAN.
The fix seemed simple: a
Rewind All;statement in the code – one at the start and one at the end – would surely ensure the tape was always at the start point when the application ran.
Admittedly, it's been decades since I touched FORTRAN and I know it's gone through a few changes since I last did but I don't remember a statement that resembles REWIND ALL (note ALL CAPS, in which all good FORTRAN should be written, punched on tab cards on an IBM Model 029 keypunch, submitted to the hot card reader 9 edge in, face down. . .).
Someone want to help this ex-mainframer out here?
As I recall (and the last time I used Fortran in anger was something like 1973, so my memory is vague), the REWIND statement takes a logical unit number. So you have a subroutine, called at the beginning and the end of the run, that contains REWIND statements for all of the logical units used in the program).
It's a common mistake, but JCL was only used in IBM operating systems. There's no indication that this problem occurred on an IBM system.
It's a common mistake, but JCL was only used in IBM operating systems. There's no indication that this problem occurred on an IBM system.
I only have experience with IBM operating systems (at least where mainframes are concerned) and mainframe without other specifications usually (at least in my experience) means IBM.
Not in the 1960s (when the REWIND statemeent appeared in Fortran). In those days, mainframes were sold in North America by IBM, Univac, Burroughs, and Control Data, among many others; in the UK, ICL was big. The “job control languages” of these different systems not only looked different, but had very different views of the relationship of a program to the outside world.
Indeed. "JCL" is a generic term.
Our company used a much-enhanced version of ICL's George 2, indeed enhanced from ICL Dataskil's "George 2+". I was the maintainer.
George 1 and 2, and their derivatives, took a JCL document and "compiled" it, producing a small encoded form to be interpreted at runtime. Jobs with obvious JCL errors could be rejected early on.
For enhancements to the JCL I would first write the documentation and pass it around. "If I did this would it be useful?".
Naturally I had already half-coded the thing in my head, :-)
Wait, what?! Our installation had both 026s snd 029s; both models printed, in dot-matrix form, what you'd typed, along the top of the card.
Our computer-output card punches did not print on the cards. If we wanted printing on such cards, we had to run them through an "interpreter", which printed fully-formed characters across the top of the cards. And those printed characters were radically shifted right from their associated columns, making the printing largely-useless.
Perhaps some cheap manager at your installation refused to pay for new 026 ribbons...
I for some reason I claimed\got dumped (As I was working on the issue) all the tickets for non working procurement systems at GSK Dartford after some upgrade, it was going on for about 3 days & over a weekend.
I reached out & was given a "Boys Own Book Of Knowledge" help sheet by someone higher up in the application support by someone who really wasn't that interested in helping beyond that, until I discovered (IIRC) it was he had set a path in a config file something like "C:\Progra~1\JDE\Subfolder\New\Target" & for what ever reason our site in the imaging process, scripted installs had the were set up slightly differently & the path should have been "C:\Progra~1\JDE\Target".
Made the change, tested, then dropped\edited it on all the assets in the tickets & emailed instructions to the clients & to advise them how to do it themselves if any colleagues found themselves with the same issue but hadn't logged tickets.
No recognition was forthcoming, despite the fact I had restored functionality for 2 sites & 3000 employees.
Shouty people trigger my stubborn streak. I'm unlikely to go the extra bit for them. Ever.
I am likely to go the extra bit for them ... but not in the direction they expect. I am a programmer by trade, but I qualify as operator on some systems as well, so they will get the complete BOFH and BPFH treatment when I deem it is warranted.
No need for anonymous posting as those who recognize this fall into two categories:
Colleagues who completely agree and shouty people whose parents weren't even introduced to each other and who will still get screaming nightmares, leaving them unable to do anything about this.
Once I was hired as an "extra hand" to help a real estate company develop their new website where all of the content was backed by a database connected to their internal systems. It was very late and slightly over budget but mostly complete. But it was put into production nonetheless and then they started having a persistent performance problem where for many users the site would "hang" for about 15 to 30 seconds and then work normally for a few more requests and then hang again. It there were more than half a dozen users browsing the site at the same time it was almost guaranteed that everyone would experience this behaviour.
While the target audience for this company was quite small (high-end luxury estate), this was a serious problem as the user experience had to be as flawless as early-2000s technologies allowed.
The "senior" developers all were blaming database performance, server performance, low memory, etc. because profiling the production servers showed 100% CPU both on the server and the database, but very low memory usage. But even after moving everything to a more powerful server (quad processor! yay!) the problems remained.
The problem never happened even once using the test database that was fully loaded with all sorts of fake data.
Me, knowing almost nothing about the project at that time, started looking into the code and after one very long night noticed that there was a section where as the last step building the web page it would choose three random internal "ads" for other real estate from the database based on the current search criteria or estate being viewed and then complete sending the partial web page to the client.
Problem was that the routine for selecting the random items was extremely flawed. Select one random ad, then select the next one. If it was the previous one, select another. Select the next, check against the already selected list, choose another if already in the list. Repeat until the desired quantity is selected. Why this was done this way I do not fathom. You know, "senior", "enterprise" ways of doing things?
Now imagine that I want 3 ads and the database only has 2 for the search criteria... it would keep trying to select new ads to add to the list, but they will always be the same. It would "hang" the processor thread at 100% CPU utilization and also the database because it would receive thousand of useless (cached) queries per second.
The client would receive the web page almost complete and then "hang" until the server timeout was reached and the connection dropped, allowing some more browsing until the search criteria for the ads would again result in not enough items to fill the list, hanging the page again.
The problem never happened with the test database because it was loaded with thousand of fake "ads" carefully distributed into all of the possible categories and because of that the poor selection routine never was starved of records to pick from, while in production the "ad" distribution was much less random letting several of the categories with few or no items.
For the pleasure of finding the fix would involve rewriting one function of one class to use one slightly more complex SQL statement I got a lot of ugly looks from the "senior" developers and "senior" DBAs that were proposing yet another server upgrade to "fix" the problem. And my work there was complete, got paid for one day and that was it.
I think that bringing a fresh pair (or more) of eyes that never worked on some project that is experiencing persistent problems is indeed a valid approach in many cases.
My experience was with a 6502 assembler. My boss (whose name was in the tombstone in front of the company) had gotten a 6502 assembler to make up his code. It was written in (wait for it) FORTRAN and he was running it on a PDP-11/44. Nice. The assembler was written for "small" projects and its symbol table had only a few entries in it (100 maybe). He was writing MUCH larger programs and just upped the dimensions of the arrays. Further "upgrades" were made to make the arrays "virtual" and use some addressing funnies to expand them to 1000's or more. Well, as the programs grew, the assemblies took longer and longer, and were soon relegated to the overnight queue, which isn't helpful for debugging such large programs.
Being frustrated with this, I looked at the code for the assembler, and discovered that the hashing algorithm for symbol lookup was still based on the low (100 or so) number of symbols, and with the large assemblies hash hits would always happen, then revert to a linear search. So, I changed the hash algorithm to be based on the actual size of the allocated symbol table, and miracle, it would be sooo much quicker. Then I found out the algorithm for sorting the symbols at the end of the assembly (for printing purposes) was slower than a bubble sort, and changed that to one I saw in the Radio Shack (Tandy) Model 100 instruction book (shell sort), and things went much quicker.
At the end we could assemble things over lunch break, and we were MUCH more productive.
This was in the mid 1980's. The company has since gone out of business (sold to a bigger player in the application space) about a year after they "downsized" me. Oh, well.
This post has been deleted by its author
Might not have been a bad thing. Even if he wasn't able to solve an issue, the IT director might be more willing to believe him when he said "it is working the way it is supposed to" since he had solved a problem many others had failed to solve.
Worst case he could have said "well I'm not as much of an expert in xxx, can I call my boss to see if I can bring in someone else" and even if he missed an "obvious" issue if he gets it fixed the IT director is still going to see him as his go to guy and treat him with respect. Because what he really wants is his problem solved, and if he knows "if I call that guy he gets my problem solved" he probably doesn't care who actually does the solving.
I was an engineering laboratory technician, meaning that I dealt with the hazardous chemicals and such. The place had a Novell 3.11 file server that had Pegasus Email on it, and a 56k leased line to the outside world. One day the server started bogging down, and no one could figure out why. Eventually I got ordered by my boss to take a look at it, ignoring my protests that I had no clue about Novell software. I was taking night classes in C and that was good enough for him.
The server had two network cards, each with a 10Base2 coax bus connection. Looking at the network cards, one had a bright activity light while the other one was barely flickering. I unplugged the coax on the bright card, and half of the office was back to normal. Aha, now to narrow down the problem.
The server had the only network cards with indicator lights, so I couldn't just look around. I followed that coax cable and disconnected computers until the problem went away. The problematic PC was at a new engineer's cubicle and he was very put out that I had disconnected him from the network. It seems that he was waiting for an important email. The engineer proudly proclaimed that he had written a program to check the Pegasus email server so he would get it right then, not sometime in the next 5 minutes or whenever.
I asked to see this amazing program, and the engineer had set up an infinite loop that did nothing but continually check the server to see if he got any email. I rebooted his PC, hooked the network connection back up, and told him that his little program had caused the network problems, and to not do that again. 5 minutes later the problem was back, and I had my boss tell that engineer more firmly to stop with his silly program. And there was much rejoicing.
From that point on, I was the first person called whenever anyone had a computer issue. Eventually I managed to get the company to pay for certification classes and became a network admin. It's been my bread and butter since then.
The idea that someone who turns up and finds the bug that has evaded discovery by all the other experts (and fixes it) in about 10 minutes must be a great genius is only had by some moronic boss who does not realise the vast array of skill needed to design, create, manage and run a complex IT system. A proper manager would have asked 'Boris' why he succeeded and the others had failed, and understood that a mainframe hardware engineer, for example, never had a chance of understanding what we going wrong.
Some comentards here would describe this as "manglement", and rightly so. I pity that persons' underlings.
Merry Grinchmas, everybody
> The idea that someone who turns up and finds the bug that has evaded discovery by all the other experts (and fixes it) in about 10 minutes must be a great genius
Usually it just means someone with a lot of experience. You learn what to look for by having years of solving problems.
At my workplace there was a bug caused by splitting a date into days and seconds and storing each in a 16 bit unsigned integer. First developer got nowhere - it worked for him whatever he tried.
He worked nine to five. The bug happened every day if the time was taken after 17:50pm (65,536 seconds). Another programmer worked from 12am until he thought he’d done enough for the day. He found and fixed the bug at about 17:51.
A fellow postgraduate student of mine was once paid big bucks by an academic in our engineering department to find out why a FORTRAN program he had bought didn't work as expected. My acquaintance looked at the source, spotted a lot of lines starting with a D, recompiled with D-lines included and everything worked as expected. Total the on the job was about ten minutes.
Explanation: As well as comment lines, which start with a exclamation mark, VAX FORTRAN also allowed "Debug" lines which started with a D and were only included if a command line option ("/D-LINES", I think) was set at compile time,
All mine seem pedestrian by comparison.
My least fave was being given a procmail list of, I think they're called recipes - parameter sets to make procmail do it's little rat-dance by an office self-confessed "genius", one of several such who made up our SysAdmin unit.
"It's got a bug. I've looked at it on an off for a few years but can't see it. See if you can."
This was of course "go away" work, but I bought a book on procmail, opened it and lo and behold the missing semi-colon was obvious in about 10 minutes reading.
"I fixed the bug. Look, I don't have much to do around here and you are swamped. Maybe I could take over the administration of this for you?" I offered.
No reply was offered, but that afternoon the server admin account password was changed.
Then the old bugger died and his entire department was locked out of the server, which it turned out used procmail as a dispatching system for a much larger set of servers. Whoops.
Oh how I laughed.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
