Sign in / up

back to article You shrunk the database into a .gz and the app won't work? Sigh

Welcome yet again to On-Call, our unimaginatively-named regular recap of readers' recollections of jobs gone wrong. You all liked a double-barrelled story so much last week we've decided to do it again, starting with “Gary” wrote to tell us of his time working for Burroughs and the job he was sent to do for a bank with a …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Tom 7

    Replace tape

    When trying to restore a backup for someone we discovered the tape was the 2nd part of the backup.

    Where's the first tape?

    That's all there is.

    When you run the backup does it ask for the tape to be replaced?

    Yes, we pop it out and pop it back in again and after a bit it comes up with 'finished successfully'.

    Turns out this has been going on for ages and its only the customers close resemblance to Susanna Hoffs that encourages an almost certainly futile attempt to recover individual bits of disk and compare them with the corrupted database and the eventual disappointment of actually being able to stitch the database successfully back together, rewriting the backup to check that fresh tapes are being inserted, running a backup and verifying and then wending off into the night only to be punched by my trainee in harness for not spending time pretending we weren't there yet even thought running and verifying the backup twice 'just to be sure' had got the customer looking sideways in a definitely not alluring way.

    36 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Replace tape

      first job as sysadmin-in-training, couple of weeks in, now going to be responsible for changing backup tapes .... person who did it before showed me the pop the tape out, pop it in again, backup software said "finished successfully" routine.

      Me: "Is it always that quick?"

      Supervisor: "yes, not much changes on this server during normal days"

      Me, later, looking at tape to put in for next night, asking self - 'why does it say 'cleaning tape' on it?'

      49 0 Reply
      1. Guy Geens
        Reply Icon

        Re: Replace tape

        This reminds me:

        Technician is called in to fix a tape drive. After he's done, the operator pops in a tape to test if everything is OK. After a few seconds, the tape comes out again. The operator says: "Everything is OK, backup is done"

        The technician is highly suspicious and takes a look at the console:

        INITIALIZING TAPE...

        SYNTAX ERROR

        After a simple edit to the script, the backup takes considerable longer. The operator looks at a cupboard filled with tapes and realizes these are all empty...

        39 0 Reply
        1. Alan Brown Silver badge
          Reply Icon

          Re: Replace tape

          The good thing about packages like Bacula is that they actually READ the tape before writing to ensure people haven't pulled that kind of stunt.

          (They also use a database which contains SHA256 and exact location of every file on every tape so restoring XYZ file doesn't mean having to run hours of restores, AND you can tell when the file last changed.)

          1 0 Reply
      2. Chris King
        Reply Icon

        Re: Replace tape

        You probably had one of the cleanest drives out there, I inherited a really dirty one.

        I knew it was going to be trouble when the report from the DEC engineer said (and it was all-caps)

        "THIS DRIVE CONTAINED SO MUCH DIRT, YOU COULD PROBABLY GROW POTATOES IN IT !!!"

        17 0 Reply
        1. Anonymous Coward
          Reply Icon
          Anonymous Coward

          Re: Replace tape

          Clearly not the same DEC engineer who let the ash on his cigarette (yes, really, in the computer room) grow so long that it all dropped into the tape reader/punch on the PDP15/76.

          10 0 Reply
        2. Alan Brown Silver badge
          Reply Icon

          Re: Replace tape

          "THIS DRIVE CONTAINED SO MUCH DIRT"

          I just pulled a DLT drive off the bottom shelf in a room which meets this description.

          The room itself was fairly clean. People forget that drives are in cases with fans and that tends to suck every bit of airborne dust available into the case.

          Rather fine dust, finer than talc, over a 15 year period....

          1 0 Reply
    2. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Replace tape

      Please tell me that happened on a Monday.

      9 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Re: Replace tape

        The cleaning tape incident? I have to admit it was about 20yrs ago, and I don't remember day of week, but been thinking about it today and I vaguely remember taking on the task half-way through a month - the supervisor's boss had insisted on the changeover mid-month to avoid any risk to month end data.

        1 0 Reply
    3. Zippy's Sausage Factory
      Reply Icon

      Re: Replace tape

      I finished reading this and got three comments further down until I realised I wasn't paying any attention and just thinking about Susanna Hoffs.

      Saddo that I am, I was actually thinking "I know that name from somewhere, who is she?"

      Ah yes, Bangles. My one chance to see them we got lost - wandering around Hyde Park with "Walk Like An Egyptian" blaring out...

      10 0 Reply
      1. Locky
        Reply Icon
        Flame

        Re: Replace tape

        @Zippy's Sausage Factory - Famously told by an unscrupulous producer that Eternal Flame would sound better if she recorded naked

        Now try and get that out of your head....

        9 0 Reply
      2. usbac
        Reply Icon

        Re: Replace tape

        @ Zippy's Sausage Factory

        This is a little bit off-topic, but still relates to tapes...

        Way back in the mid 80's I was in school in Los Angeles to be a recording engineer. Our school had a very high end recording studio. Some of us that had earned the trust of our professor were allowed access to the studio on off hours. We had friends with bands that wanted to record demos to send in to record companies (those were the days).

        The problem was that reels of 2-inch tape were very expensive for us poor students. A reel of good quality tape could be as much as $200-300. Our professor told us where to get really good tape for cheap. He explained that the high-end recording studios sell reels of "used" tape for really cheap. The reason is that when well known bands come in to record an album, they run many takes of the same song. Since they are paying thousands of dollars per hour to use the studio, they just keep grabbing new reels of tape. When the project is over, the studio bulk erases the tape, and sells them off cheap.

        A friend and fellow student had some friends of his that wanted to record a few demos. So, he went and bought a few reels of this used tape. We arrange to use the studio ofter class one afternoon. His friends aren't quite there yet, so we start getting things ready in the control room. I take one of the reels and start loading it up on our 24-track deck. I notice that a bunch of the VU meters twitch as I'm winding the tape onto the take-up reel. We both look at each other, and realize this tape wasn't erased. So, we quickly re-patch the console for playback instead of recording.

        We start listening to the tracks, and make a rough mix. As we are listening we're thinking we've never heard this song before, and are trying to figure out who is the band. About this time Richard's friends show up. They come into the control room and say "Cool.. The Bangles... where did you get this?" It turns out we had an un-released song.

        There was one mostly usable take on the tape. We quickly unloaded the tape and put it away. I don't know what Richard ever did with the tape. These days, we could have done a decent mix-down, and posted it on the net. We were both too honest do anything unscrupulous with the tape. It would be quite an artifact to have these days, however.

        10 0 Reply
        1. JPeasmould
          Reply Icon

          Re: Replace tape

          When I worked at Wessex Studios in London in the eighties we cleared out our tape store once a year. We didn't wipe the tape and resell it as it had already been paid for.

          The first year I just took the sides off the reels and let them unwind into bin bags. I spent the next day picking the tape out of the trees all along Highbury New Park (a good half mile) due to the antics of a couple of the local intelligentsia.

          The second year I used a splicing blade to cut the tape from the reels. You can't get as much fun from 6" lengths of tape.

          I did find all the "Never Mind the Bollocks" masters in the attic, but they had a bit of masking tape round them stating that they were being kept for the court (the Sex Pistols were fighting Virgin at the time). I assume Matrix got them as they were not claimed by the time Chrysalis sold Wessex.

          I wish I'd nicked them along with the Reg Dwight demos...

          1 0 Reply
      3. William 3 Bronze badge
        Reply Icon

        Re: Replace tape

        With regards to the Susanna comments, surely only doing work because of how that woman looked was nothing more than everyday sexism, and given how left leaning and all SJW all the comments and articles are in the register these days I can only wonder why it was voted up so highly.

        Perhaps because all the people in this section of the register are actually skilled workers rather than snot nosed left wing SJW humanities degrees tossers that go looking for offence as justification for their existence that populate the other areas of the register.

        2 14 Reply
        1. Anonymous Coward
          Reply Icon
          Anonymous Coward

          Re: Replace tape

          ... given how left leaning and all SJW all the comments and articles are in the register these days ...

          You must be reading a version of TheRegister from some other, bizarre dimension.

          7 0 Reply
          1. 404
            Reply Icon

            Re: Replace tape

            OP has a valid point - mention anything negative about Obama, Hillary, <insert progressive sacred cow here>, and the downvotes multiply like rabbits. Not Reddit-level SWJism, but still present in different forms.

            1 8 Reply
            1. John Gamble
              Reply Icon

              Re: Replace tape

              Or, perhaps because the negative comments on the alleged sacred cow are about as dull as the "will it play Crysis?" comments?

              Here's a clue: if the commentard actually uses the term "SJW", you can safely assume too many drugs were consumed during the commentard's fetal development.

              6 1 Reply
              1. Anonymous Coward
                Reply Icon
                Anonymous Coward

                Re: Replace tape

                But we need to know, does Obama play Crysis?

                3 0 Reply
            2. kain preacher
              Reply Icon

              Re: Replace tape

              Could it be that most of the negatives I've seen about Obama and hillary are from a few rabid posters her that post made up stuff ? I mean if you want to slag off on Hillary for using a personal email serve by all means do it, but dont act like she was the first or because all dems are corrupt. Just remeber Collin powel used AOL.

              I for one wished that Obama had not chicken out and did not make weed legal or pardon Snowden. I would also like for him to say that Julian will not be prosecuted. If nothing more than to then deflate Julian's ego.

              1 3 Reply
    4. Stoneshop
      Reply Icon

      Re: Replace tape

      Back in the days of VAX11/780's and such, one of my colleagues was called to fix a disk drive. THE disk drive, OS, data, everything. And so he does, then proceeds to help with restoring the backup. Which is duly retrieved from the safe, but he's rather surprised to see that it's just three RX01 floppies. Which is rather at odds with the capacity of the disk, 120MB, where an RX01 can only hold a few 100kB.

      "So, how do you run your backup?"

      "Well, like the manual says, @SYS$SYSTEM:STABACKIT, and we swap floppies when prompted"

      Every Friday the customer had faithfully made a bootable floppy set for the VAX to run off, so that the system disk was not in use when you wanted to backup or restore it. No actual backup, however...

      13 0 Reply
    5. ecofeco Silver badge
      Reply Icon

      Re: Replace tape

      Back up tapes at one place I worked were somehow not working after I began testing them. One after the other. Ran verification tests and still nothing or only partial, corrupted back ups.

      Spent weeks trying to figure it out, then by pure happenstance I notice a very strong magnetic field in the vicinity of where the tapes were stored. (strange buzzing noise from a piece of equipment that should not be making buzzing noise but only when sitting in a very certain spot)

      Managed to get a building engineer to come down with a field tester and sure enough, there was an unusually strong magnetic field right in that one spot. Turns out it was close to an outside wall that was receiving very strong bounced radio waves from somewhere outside.

      You can surmise the rest. Yeah, no backups.

      2 0 Reply
      1. BostonEddie
        Reply Icon

        Re: Replace tape

        recollects me of the time the local (Mass) office of the IRS moved and consolidated their new operation center into one location. There, right in the middle of the new location, was a high voltage line feed power pole exposed for all to see in the middle of the storage area. Most unsightly. Why don't we hide that ugly thing by putting those reels of magnetic tape around it...

        Income taxes were a bit late that year.

        2 0 Reply

    6. This post has been deleted by its author

  2. Anonymous South African Coward Silver badge

    Backup tapes...

    Back yonkers one of our clients did his usual backup, and locked the tapes up in his fireproof safe (onsite). This was in the heady days of Novell 3.12...

    Came back the next day, PC's was gone. Server was gone. Safe (with tapes inside) was gone. No hard copies.

    Oops.

    And the taxman was busy with an audit on him...

    14 0 Reply
    1. allthecoolshortnamesweretaken
      Reply Icon

      Re: Backup tapes...

      "PC's was gone. Server was gone. Safe (with tapes inside) was gone. No hard copies. [..] And the taxman was busy with an audit on him..."

      What an odd coincidence.

      79 0 Reply
      1. JimC
        Reply Icon

        Re: Backup tapes...

        Why would you expect a serious robbery *not* to take a safe unless its so large as to be impractical to move? There are enough of us here and with long enough careers that most unlikely events are going to turn up...

        And theft of all IT equipment including all the tapes is very familiar. And on at least one occasion it was followed by "Oh, is that why you told us we must always get someone to take a tape home? We decided it was too much trouble." (as if we hadn't explained why they needed off site).

        12 0 Reply
        1. Chris King
          Reply Icon
          Facepalm

          Re: Backup tapes...

          An academic decided that he was going to teach "IT security" to his final year Informatics students, stressing the importance of protecting physical kit and taking regular backups.

          Next day, he plonks his laptop on a desk, turns his back, and someone runs off with it.

          Bonus points for losing three weeks of lesson plans, presentations and course materials that weren't backed up anywhere else.

          23 0 Reply
        2. DropBear
          Reply Icon

          Re: Backup tapes...

          "Why would you expect a serious robbery *not* to take a safe unless its so large as to be impractical to move?"

          Because if it isn't bolted to the floor* it's called "security theatre" not "a safe", unless you need a forklift to move it (and even then, you should know better).

          *if said safe has its own floor, complete with giant-ass submarine-style hatch door weighting many tons, you officially have my permission to not bolt it to anything and still call it a safe. Although you might prefer to call it "a vault"...

          14 0 Reply
          1. I ain't Spartacus Gold badge
            Reply Icon
            Happy

            Re: Backup tapes...

            I had a safe by my desk in a previous job. It used to be the finance manager's office - but he'd changed. Turns out the reason is that the safe door broke, so rather than remove the safe, he simply ordered a new one to be delieved to a different office, and swapped...

            8 0 Reply
        3. kain preacher
          Reply Icon

          Re: Backup tapes...

          Jim I would follow you thought process except for it was right before an audit.

          0 0 Reply
  3. GrumpenKraut
    Facepalm

    Editing a config file with an Office program

    ... similar "success". Not me, but the "IT specialist" of a not really small company.

    Had the guy read the error messages over the phone. The XML-ish tags gave it away. He wasn't able to copy the old version of the file, that is, using the 'cp' command. I had to come to the company. Charged him 3 hours and promised not to tell anyone.

    7 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Editing a config file with an Office program

      I had a job setting up some data ETL into a new and pricey proprietary system from a legacy database using a very well known and established XML standard. A simple task that ended up being far more difficult that expected. After much tinkering, transformation and experimentation I eventually used some test data from the proprietary tool to generate some of it's *own* XML which I copied to my Unix box from which the system duly refused to import as well. Aha, with my diligent troubleshooting notes and sample files in hand, I emailed the vendor's tech support email address and got a phone call the next day, from a developer no less.

      Vendor: "We've figured out your issue"

      Me: "Ah Great! What is it???"

      Vendor: "Your XML file is missing ctril-m characters from the end of each line."

      Me: "Ummmm....ctrl-m, you mean DOS line end of line characters? They are outside the tags anyway."

      Vendor: "The what?"

      Me: "Ummmm... the tags..." (explanation of XML and tags follows)

      Vendor: "Those angle bracket thingies? Why would I try to parse those when it is so much easier to parse line by line?" (I paraphrase, but only just a bit)

      Me: "........"

      Things got somewhat interesting after that as my new boss had recommended buying this thing and had a strong motivation to "Make It Work", I'll not bore you with the predictable details except astoundingly, the next tier of management agreed with my findings when it all came to a head.

      23 0 Reply
      1. Doctor Syntax Silver badge
        Reply Icon

        Re: Editing a config file with an Office program

        Sub-contracting to a sub-contractor for the main contractor (Your first guess is probably right) public sector job. We were to take XML encoded documents from main contractor for processing. Very early in the proceedings I had to take a trip to visit the onshore representative of the main contractor's favoured development house to explain how to handle apostrophes and the like in XML. Not surprising that from time to time we had outbreaks of badly formed documents sent to us.

        6 0 Reply
      2. Kiwi
        Reply Icon
        Joke

        Re: Editing a config file with an Office program

        the next tier of management agreed with my findings when it all came to a head.

        I believed you all the way up until that statement! :)

        1 0 Reply
    2. JoeF
      Reply Icon

      Re: Editing a config file with an Office program

      Had somebody edit a Linux config file with a DOS/Windows text editor which put crlf instead of lf in there.

      Was rather hard to detect because vim (vi improved, cough) "helpfully" hides control characters, so the file looked ok.

      A hexdump finally revealed the problem.

      5 0 Reply
      1. Justin Clift
        Reply Icon

        Re: Editing a config file with an Office program

        From rough memory, vim has a "binary" mode which can show them as well.

        I think the -b switch from the command line turns it on.

        eg:

        $ vim -b some_file.txt

        But yeah, hex editing utils also work. :)

        0 0 Reply
    3. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Editing a config file with an Office program

      I've had a nice one involving UTF-8 apostrophes in usernames. Lovely Irish girl had been resetting her password every day for months in order to log in.

      3 0 Reply
  4. dvd

    Went to Otley instead of Otley road?

    Pah! I worked with someone who went to the wrong African COUNTRY on a support call.

    In fairness to him it was the travel agents fault, not his. He ended up getting a taxi to the border and walking across in a Checkpoint Charlie sort of deal.

    15 0 Reply
    1. Andrew Moore
      Reply Icon

      We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre.

      30 1 Reply
      1. Fibbles
        Reply Icon

        We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre.

        Ok, just a second whilst I get the pontoons and outboard motor from the boot.

        8 0 Reply
      2. Phil W
        Reply Icon

        "We had an American colleague fly to Iceland instead of Ireland. Only realised it when he got into a taxi at Reykjavik and asked to be taken to Dublin city centre."

        If he didn't notice until he was in the taxi and asked to go to Dublin city centre, was he phenomenally ignorant/stupid? To get that far surely he musn't have noticed:

        - The fact it said Reykjavik rather than Dublin on his tickets

        - The various signs in Icelandic (I suppose he could have thought it was Gaelic, but come on)

        - The length of flight (you didn't say where he was flying from, presumably the USA. But we're talking about an extra hour or two or flight at least most likely, I would of thought it would be noticeable)

        - The Captain announcing the destination as Reykjavik on take off, and before approach.

        - How bloody cold it was! Either in the airport or when he went out to get the taxi. I know Ireland is not exactly tropical but the difference is usually fairly substantial I believe? I think Reykjavik peaks at about 13 degrees in Summer?

        7 1 Reply
        1. Doctor Syntax Silver badge
          Reply Icon

          "If he didn't notice until he was in the taxi and asked to go to Dublin city centre, was he phenomenally ignorant/stupid?"

          He was American. He'd flown over water for some time. He was somewhere furrin'. It's all the same, isn't it?

          23 1 Reply
          1. TomPhan
            Reply Icon

            Flying into Reykjavik and the only clue was when in a taxi? Wouldn't it have happened after he landed at Keflavik?

            5 0 Reply
          2. Anonymous Coward
            Reply Icon
            Anonymous Coward

            Having recently flown to America via Iceland I really have no idea how one could get it confused.

            0 0 Reply

        2. This post has been deleted by its author

    2. rh587 Silver badge
      Reply Icon

      Pah! I worked with someone who went to the wrong African COUNTRY on a support call.

      Pah. Just him, not anyone else?

      I know someone who sent a shipment of 300 new cars to Sweden instead of a Swindon dealer (large dealer, it was the order for the new plates, huge turnover in a couple of weeks).

      I wish I were joking... it shouldn't even be possible, but he managed to bork the paperwork sufficiently that the error wasn't uncovered until his Swedish importer called asking why the port was pestering him over 300 cars on their quayside with his name on that he hadn't ordered...

      11 0 Reply
    3. John Brown (no body) Silver badge
      Reply Icon

      "Pah! I worked with someone who went to the wrong African COUNTRY on a support call."

      Bollocks! You beat me! We had our south west engineer go to a call in Poole, Dorset, pre-sat-nav and smartphone era. He rang back after a 2 hour drive saying he can't find the street. He'd even asked around and none of the locals knew it either. Bloody call centre had got confused and should have sent one of the northern engineers to Poole, near Otley in North Yorks.

      1 0 Reply
    4. kain preacher
      Reply Icon

      I got one better for you. I was traveling on this road. lets Call it 1st. Now first street has the destination of north on one side and south on the other side. Now when you cross over to the next city it's east and west, and no there is no sign letting you know you changed cities.

      0 0 Reply
  5. Anonymous Coward
    Anonymous Coward

    When life imitates art...

    One of my favourite quotes on bash.org was a guy who claimed he'd managed to lose a PC inside his apartment. It responded to ping, he just couldn't work out where it was.

    Last job was as part of a network team looking after a nationwide estate for an outsourcer. Got dragged into an emergency call with management where a server had gone missing on a campus site and they wanted my help tracking it down. True to art, the server (an ancient Windows box) responded to ping, RDP and SMB, but it was missing from the rack and noone could work out where it had gone.

    An hours discussion with the team who managed the other half of the campus network on this site (a horrible legacy arrangement) revealed it was still in the same building and still in the same room, and that the network ports assigned to it hadn't been changed.

    The server was eventually located, in the rack it was meant to be. Turns out the sticker for the rack had fallen off, and the rack next to it wasn't stickered, so some well-meaning newbie had come in and taped the sticker they found on the floor onto the wrong rack, then couldn't work out why they couldn't find it when they were asked to look for it a few weeks later...

    15 1 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: When life imitates art...

      You Could have told them which switch and port to go follow the cable from.

      7 0 Reply
      1. phuzz Silver badge
        Reply Icon

        Re: When life imitates art...

        That's assuming that you could have found out which port it was plugged into, which would require a network engineer who both knew the password for the switch, and was willing to actually log in and tell you the information, and if they were that helpful, they probably wouldn't have ended up as a network engineer.

        (I'm joking network ops people. Mostly.)

        26 0 Reply
        1. pig
          Reply Icon

          Re: When life imitates art...

          "and if they were that helpful, they probably wouldn't have ended up as a network engineer"

          LMAO. Funny and true.

          12 0 Reply
          1. John 110
            Reply Icon

            Re: When life imitates art...

            Well, our network people are absolutely brilliant! (are you listening, Denise)

            5 0 Reply
        2. mr_souter_Working
          Reply Icon

          Re: When life imitates art...

          but, but, but, it's NEVER the network that's at fault!!!!

          at least according to the network engineers I regularly have "energetic" discussions with.....................

          2 0 Reply
      2. BinkyTheMagicPaperclip Silver badge
        Reply Icon

        Re: When life imitates art...

        You could have told them which switch port - if all switches are intelligent. There are plenty of installations where only core switches are intelligent, or none of them are.

        8 0 Reply
        1. Chris King
          Reply Icon

          Re: When life imitates art...

          "You could have told them which switch port - if all switches are intelligent".

          Unless of course you were on a site full of 3Com switches and half your management modules were permanently in transit to/from 3Com.

          7 0 Reply
          1. Antron Argaiv Silver badge
            Reply Icon
            Thumb Up

            Re: When life imitates art...

            ...3Com switches and half your management modules were permanently in transit to/from 3Com.

            Those wouldn't have been 3CEME modules, would they? They used integrated RTC/SRAM/Battery modules from Dallas that had solder joint issues...after a while, the SMT solder joints on the SRAM inside the modules would open up. I was at 3Com and did a redesign to eliminate them.

            Then they canceled the whole Eclipse product line.

            4 0 Reply
            1. Chris King
              Reply Icon

              Re: When life imitates art...

              "Those wouldn't have been 3CEME modules, would they?"

              These switches were bog-standard SuperStack series, nothing to do with the Eclipse series but the management modules on those things dived more often than continental footballers.

              Management was happy with the free "lifetime warranty", even though I was frothing at the mouth about six-week turnaround time and not being allowed to carry spares.

              1 0 Reply
      3. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Re: When life imitates art...

        "You Could have told them which switch and port to go follow the cable from."

        Did that, it wasn't much help as the switches were in a room 20 metres away and the cabling for it (5 long Cat5 leads ziptied together) went into the space above the ceiling tiles ;)

        I have no idea why, given that there were closer switches in the room in question. I can only assume there weren't any ports available when the server was originally stood up and someone bodged it.

        3 0 Reply
        1. Alan Brown Silver badge
          Reply Icon

          Re: When life imitates art...

          "wasn't much help as the switches were in a room 20 metres away and the cabling for it (5 long Cat5 leads ziptied together) went into the space above the ceiling tiles"

          That's where one of these (https://www.youtube.com/watch?v=PKgzPYNu6-g) comes in handy.

          There are a bunch of tracers on the market with greater and lessor ranges. If truly lost, invest in a Cat and Genny. These are somewhat bulky in a server room but if you know how to use them properly you can tell where a cable is in the wall/floor/ceiling/ground and _how far_ buried it is (it's nice to know "6 inches vs "10 feet")

          0 0 Reply
      4. whbjr
        Reply Icon

        Re: When life imitates art...

        Don't count on the switches to be intelligent. Ping (continuously) the lost host, and start unplugging cables. If ping continues to work, plug the cable back in. If it doesn't, then the lost host is somewhere at the end of that wire.

        3 1 Reply
        1. Anonymous Coward
          Reply Icon
          Anonymous Coward

          Re: When life imitates art...

          Ping (continuously) the lost host, and start unplugging cables.

          Not a recommended approach for any switch that could even potentially be carrying important traffic.

          Cue: "Our production Oracle database just failed over, the standby then failed, and the original isn't failing back properly. Were you doing something in the server room?"

          2 0 Reply
          1. Adrian 4
            Reply Icon

            Re: When life imitates art...

            Your software can't recover from a few seconds of disconnection ?

            Time to change vendors.

            0 0 Reply
    2. John Riddoch
      Reply Icon
      Facepalm

      Re: When life imitates art...

      Ah, server location... One previous place had a server which no-one quite knew where it was as there were a few comms cupboards in the place. After setting up something to ping a message to /dev/console didn't attract any attention (screen was probably in standby, or just no-one was in the room), I figured out how to use the speakers (Sun Ultra-1) to make a noise. Not just any noise, you understand, somehow I thought it would be a good idea to nab the bit from Holy Grail - "help,help, I'm being repressed". Cranked the volume up to full and set a cron job to cat the .au file to /dev/audio every 30 minutes.

      Of course, through a comms room door, the only bit passers by could hear was "help, help". Which led to security being called to find who was trapped & where. Of course, yelling "hello, hello?" at the area to identify the "trapped person" didn't elicit a response. Until the next cron run and they checked the console and tagged me. Of course, the security guard being someone I knew, he didn't call me direct, he called our mutual friend who also worked to let him know about it which gave him no end of amusement :)

      So, slight embarrassment all round, but we found the server. Which was very useful a couple of weeks later when some builders needed to get a cable out of the way and just snipped through the cat 5...

      15 0 Reply
      1. RW
        Reply Icon
        Boffin

        Re: When life imitates art...

        "some builders needed to get a cable out of the way and just snipped through the cat 5..."

        Builders (usually "contractors" in Canada and the US) do that kind of thing. The Atlantic City Music Hall contained a huge pipe organ with pneumatic action. Eventually it fell into disuse, but remained intact....until contractors working on renovations simply cut through the bundles of tubes that connected keyboard with pipes.

        [I'm paraphrasing this from memory and hope I got it right.]

        0 0 Reply
    3. Soruk
      Reply Icon

      Re: When life imitates art...

      That's almost like the story of the Netware server whose location was lost, yet it was still working. It was eventually found behind some drywall.

      14 1 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        Re: When life imitates art...

        That story about the Netware server was a VAX when I last read it.

        1 1 Reply
        1. Soruk
          Reply Icon

          Re: When life imitates art...

          http://www.theregister.co.uk/2001/04/12/missing_novell_server_discovered_after/

          1 0 Reply
    4. Richard C.
      Reply Icon

      Re: When life imitates art...

      This is where servers with speakers and CD/DVD-drives are useful... "We know it is in the server room somewhere and we have remote access it it - eject the drive and listen/watch": I've had to do this more than once....

      13 0 Reply
      1. J.G.Harston Silver badge
        Reply Icon

        Re: When life imitates art...

        Had to do that today. KVM switch to a stack of half a dozen PCs. One was booked for replacement. Ejecting the DVD drive was easier than painfully tracking all the cables.

        1 0 Reply
        1. Alan Brown Silver badge
          Reply Icon

          Re: When life imitates art...

          "Ejecting the DVD drive was easier than painfully tracking all the cables."

          Labelling the fecking things avoids the issue in the first place - using labels that won't fall off.

          Brother TZe-S641 label cassettes for the equipment (sticks well) and TZe-FX641 for cables (other colour combos are available) are worth buying and can be obtained for far less than the £25 apiece that Brother flog 'em for.

          If you have "problems" with people removing labels, then "security" labels will dissuade that practice (TZ-SE4)

          Dymo, Brady and others all sell similar labelling systems. Whatever you have make sure you have a good stock of tapes and a labeller in every area (they're not particularly expensive compared with having to trace cables) or people won't do it.

          My estimate is that it costs about 8p/label.

          0 0 Reply
  6. joeldillon

    Point of order - you mean the database's native compression, right? That's not the same as encryption!

    8 0 Reply
  7. chivo243 Silver badge
    Facepalm

    In another life

    I worked for a restaurant that had a computerized ordering system. When I started, I was told that the system needed to be backed up every night. When I was shown the procedure, I asked which floppy disk to use next I was told it didn't matter just grab one out of the drawer which had about 25 unlabeled floppies!

    17 0 Reply
  8. Nano nano

    "Shrank"

    see above.

    3 1 Reply
  9. FIA Silver badge

    Gary mis-heard the address and instead went to the bank's branch in the town of Otley, about 15 kilometres away and a half-hour's drive when the traffic is kind.

    I'm from around here and I can promise you no-one travels anywhere using one of those fancy modern fangled kilometre things. (Especially now we've left Europe), infact I'm not even sure most folk know what one is.

    We travel to our underground places of gainful excavation in Gods own miles.

    And nowt else.

    37 3 Reply
    1. Locky
      Reply Icon

      We used to dream of travelling underground....

      20 0 Reply
      1. Anonymous Coward
        Reply Icon
        Anonymous Coward

        "We used to dream of travelling underground...."

        night Night Tube ....?

        1 0 Reply
    2. Doctor Syntax Silver badge
      Reply Icon

      "our underground places of gainful excavation"

      I didn't think there were any in Otley. Maybe you're thinking of Ossett.

      1 0 Reply
  10. Jeffrey Nonken

    I grew up in Downingtown, Pa. Several Burroughs plants there and nearby. My father worked there, as did my best friend's father, the fathers of everybody in the high school computer club... except Alan, whose brother did.

    Ah, those were the days. Dad's in a nursing home these days, my best friend is the spitting image of his father back then, and I've pretty much lost track of the rest. Me? I develop firmware for a flight simulator company.

    17 1 Reply
    1. John 110
      Reply Icon

      @jeffrey

      I see you got a downvote for no apparent reason - unless it was your best friend who didn't need to be reminded that his youth is now far behind him...

      Anyway, have a balancing upvote

      8 0 Reply
  11. Anonymous Coward
    Anonymous Coward

    Takes me back to when I was in the Army and had a very nice deployment to the Sun Shine state in the US of A. Having just done by sysadmin course on the kit that I was to maintain I was surprised to hear they didn't do backups because 'they took to long'. That changed..

    There was also a security account that all emails were copied to so they could be (manually) checked for illegal words. There was a weekly job to check this which hadn't been done for a long time.. It took the whole night shift to delete them. I've no idea how many there were as the total never got calculated completely.

    4 0 Reply
  12. rjstua

    I started at a new company a while ago and did all the usual checks and measures to see how the land lies.

    Backups was first. Every day we received a "Backup Complete" email, great! Backups are being done....until you look at the amount of data backed up. Approx 1GB of data was being backed up, successfully, to the single tape backup. Alarm bells started ringing, especially as the mailstore itself totals 90GB. There's no compression algo in the world that is that efficient right?!

    After further investigation, it transpired that we were backing up a single database only. The database was essentially an index of what was supposed to be backed up, but not the actual files themselves. Yosemite Server Backup had decided that, despite not backing up well over 400GB of data, the fact it had backed up 1GB of an index was successful enough.....

    Sorted now though, with a successful backup AND restore tests!

    13 0 Reply
    1. Locky
      Reply Icon

      @rjstua - There are 2 types of companies in the world.

      Those who have yet loose data through failed backup routines, and who don't do test restores...

      14 0 Reply
    2. Sandtitz Silver badge
      Reply Icon
      Joke

      "Yosemite Server Backup had decided that..."

      What? People still use Tapeware?

      0 0 Reply
  13. Anonymous Coward
    Anonymous Coward

    MOJ

    Why am I not surprised that MOJ and the company building one of their systems thinks its OK to use production data in an offsite development environment :-(

    10 0 Reply
    1. Yet Another Anonymous coward Silver badge
      Reply Icon

      Re: MOJ

      If you can't trust a jiffy bag of CDs in the post to securely transfer confidential data - what can you trust?

      11 0 Reply
      1. David 132 Silver badge
        Reply Icon

        Re: MOJ

        To paraphrase the old joke:

        "Real government departments don't do backups, they just leave the CD of data on the train and let someone else back it up to Wikileaks"

        13 0 Reply
    2. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: MOJ

      It's not ideal, but then where do you get fake data while taking full advantage of the ability of human stupidity to confound any imaginable system?

      3 0 Reply
    3. Kiwi
      Reply Icon
      Facepalm

      Re: MOJ

      Why am I not surprised that MOJ and the company building one of their systems thinks its OK to use production data in an offsite development environment :-(

      How often have great systems turned out to be a total clusterscrew because real world data turned out to contain lots of stuff that no one ever thought to test for? eg people from Scunthorpe and Arsenal not being able to sign in to certain websites/services because of "bad word" filters....

      I wrote a little util some years back that did some work manipulating files from certain other programs. A user had a problem with it though, it created a temp file that grew in size until it had consumed his HDD space (pre GB days) then crashed. Thankfully it was a DOS system so easy to find and delete the file. What had happened was where my text editor transformed tabs into spaces, his transformed spaces into tabs, and I hadn't thought to handle any tab characters in the program - it didn't occur to me and I didn't have any data files that contained tabs. I couldn't re-produce the error untill he sent me some live data and then it took only a few minutes to figure out what was wrong. Icon for how I felt when I figured it out.

      (I do hope like hell that the MOJ made absolutely certain that the data was secure.. Ah screw it, what could go wrong? He looks legit....)

      2 0 Reply
  14. Anonymous Coward
    Anonymous Coward

    Half backups all the way

    Arrived as an external contractor sap basis at a small SMB running Sap on NT and Oracle. DB had grown over the years and they had added discs. Wanted to check backup before performing a complex Intervention. Turned out they just backup the files online (no budget for the proper do online backup tool), and had not included the new disk in the file backup. So not valid backup ever.

    Decided to not escalate but ask the sys admin to ensure we have a full, offline backup and would come back the next day.

    Next day my boss gets a call from an angry SMB owner that I did something due to which the backup takes a lot more time, doesn't fit on the tape anymore and he would refuse for the second day of consulting fees and we were responsible for the downtime.

    Next time I will just throw the local sys admin under the buss.

    Worked out well in the end, but I always struggle to snitch on incompetent local sys admins, knowing some don't get the required training. It's not really their fault. I wish they would do the same, unlike this one, who blamed me

    14 0 Reply
    1. Anonymous Coward
      Reply Icon
      Anonymous Coward

      Re: Half backups all the way

      After thirty plus years I'm not sure which stories to share - decided on this one.

      The internationally known financial services client in the US (one with a large hand in the recent 2008 unpleasantries) had called us in to look at their 12 year's out of support records management system.

      I designed a migration and replacement plan and as part of that we were discussing backup. Outsourced guy at the table says "we have that covered, just a simple shell script that runs at midnight with a copy command to an external disk". On a running, production SQL Server database with 10+GB of data. Yep, just a folder copy should be fine.

      14 0 Reply
  15. Dwarf

    Compression and encryption

    Err..

    Yes. They compressed the live production database. With something other than the database's native encryption too.

    What has encryption got to do with the compression of the file - they are very different things.

    Also, compressing something then trying to send it to tape is unlikely to result in a reduction of tape use, since all drives support on the fly compression and attempting to re-compress compressed data just makes it larger again.

    If you really want to encrypt and compress, then compress first, then encrypt. The randomisation of encryption should (if the algorithm is any good) result in virtually uncompressable data.

    6 0 Reply
    1. werdsmith Silver badge
      Reply Icon

      Re: Compression and encryption

      I'm guessing that stopping the application also meant stopping the Oracle instance, because a running instance isn't going to let you zip up its data files.

      6 0 Reply
  16. Anonymous Coward
    Anonymous Coward

    Useless backups

    I remember a new fairly large contract, where during due diligence, we were proudly shown a huge portacabin on site that was full of rusty steel cupboards packed full with various types of tape media. Not one of them was in the plastic box that protected from dirt and dust - since that took up space.

    There was no environmental control of the portacabin and none of the tapes had any labels or barcodes giving any indication of what system they were from, what date they were created, what software was used or anything that might help in working out what treasures they may have contained.

    We also learned that only a small volume of the tapes were active and the admins picked the tape they thought was next and used that, which might have been OK if there was some form of reliable rotation other than "pick one".

    Obviously, no test restores had ever been done as everyone was too busy.

    It was an interesting discussion point in the resulting report about the lack of any usable backups and one of the first tasks was to sort out that mess.

    Anon for obvious reasons !

    8 0 Reply
    1. John Brown (no body) Silver badge
      Reply Icon

      Re: Useless backups

      On a similar note, but a smaller scale, I once got sent out to a local branch office of a major UK insurance company to have a look at a failed backup tape drive. Got there, had a quick look at the logs, went to the manager and told him the tape had expired. ie it was a dated tape which the backup software would refuse to use after said expiration date. "Can I have a new one please?" says I. "A new one?" says he, "what new one, we only have the one head office sent us when the it was installed.

      A quick call to their HQ IT dept, and the guy says to me "ah good, did you find the fault? We've had 4 other offices call in with the same fault (voice in background "we just got another 5 called in"). So I tell him the reason for the backup failure and to likely expect calls from *every* local office if all the tapes came from the same batch, and the lack of any tape rotation (hard to do with just one!) and lack of spares. Cue a minutes silence. I suspect it was the time to pick up his jaw after it dropped rather than respect for the dead tape. Eventually he says they'll get on that right away and send a dozen tapes to every office and a new procedure description to do proper tape rotation. I also pointed out that the backup software config had an option to email the status report but that was currently blank. He sounded rather sheepish now he was over the shock but did explain he'd only been there a month and was still trying to get up to speed on the mess left by a previous sysadmin.

      In all that chatting back and forth, I never did ask why no one had simply remoted in and looked at the logs themselves in the first place instead of just assuming a hardware fault. Maybe he didn;t have passwords? That'd not surprise me in the least.

      5 0 Reply
    2. Alan Brown Silver badge
      Reply Icon

      Re: Useless backups

      "rusty steel cupboards"

      Yup. BTDT. The amazing thing is how frequently IT management refuse to spend money on doing backups right until you run a few quiet queries further up the food chain about what the cost of extended downtime of critical services would be (the usual answer is "It would break the company")

      0 0 Reply
  17. trollied

    A lesson some other Oracle DBAs have probably learned over the years. Do not give your redo logs a .log suffix - A sysadmin WILL think they can be archived/deleted

    7 0 Reply
  18. Anonymous Coward
    Anonymous Coward

    Disk failure

    Started a new job and created new team. IT supplied drive space and and we dutifully wrote source code and filled it.

    Two years later IT came to me and explained there had been a drive failure and they discovered that they'd not added my department's drive space to the backup schedule.

    Being head of department I'd had my own backup schedule with offsite backups. Lost nothing. Had most of the QA department's data too.

    3 0 Reply
  19. Colin 27

    Something similar....

    I had a call that was very similar in nature.

    Working one afternoon, I had a call from a customer in the US complaining that their document management system was down. Further investigation showed that the database was failing to start (which was a little odd). So, after 15 minutes I finally got to the bottom of the whole sorry state of affairs.....

    The original guy looking after the system had left, leaving a very inexperienced person with the root password to the UNIX system, and very basic notes that said...

    If the disk space fills up, delete any large files you find.

    Guess what happened to the database files.

    Oh, and the last back they took was over a week before.

    5 0 Reply
  20. Colin 27

    Oh, and the time.....

    ....where I went to restore a system from tape backups and found the tapes all useless.

    Worked at a company where backups were done it a bit of a strange way. I had to write a header file, followed by all the operational data, then the DBAs would back up the database (using code that they wrote and I had to call), and I would write a trailer file (so we could tell that we had a complete backup for that system).

    That's all fine - until one of the DBAs changed the tape device from no-rewind to a rewind without telling anyone. I backed up my stuff, he backed up his (rewinding the tape afterwards), and I wrote the trailer file....

    3 0 Reply
  21. tfewster
    Facepalm

    Not a backup error as such, but related

    Customer had a DG Aviion with a QIC drive for weekly OS backups and a DAT drive for daily data backups. A new office administrator is told the routine, and successfully backs up the system for 4 nights. Friday, 6pm, I get a panicked phone call: "I did the daily data backup and pressed the little button next to the little green light to eject the little data tape from the little drive. Then I pressed the big button next to the big green light to open the big tape drive - and everything went off!". Understandable, but the QIC drive has a latch, not an eject button. That was the power button. Unfortunately the database software has no consistency checking and is almost certainly corrupted, so a full restore is necessary. And, of course, an amendment to the Operators guide & training.

    Another customer, another Aviion, another day. A colleague is replacing a faulty console on the bench above the server. Leaning over on tiptoes to disconnect the cables from the dead monitor, he hears an ominous "click". Looking down, he sees his knee has pressed the servers power button. The slightest move, and the button will pop out and the power will be cut. No-one else around and no phone in reach. So veeery carefully, he drags the keyboard over, logs into the server (blind, remember), sends a message to everyone to log out NOW, shuts down the database and then the server, just before his trembling legs betray him and his knee slips off the button. Phew. He restarts the server and the database, and has just finished checking everything when the customers IT guy storms in to find out why the server is down. Our hero explains and, as he managed to avoid losing the customer a days worth of data input and several hours of downtime to restore the database, gets bought dinner & drinks instead of the expected bollocking.

    5 0 Reply
  22. Loud Speaker

    My SO has a Lenovo - which came with backup software, which was used regularly (though not frequently), as it needed a large bunch of DVDs, which grew and grew - not something to encourage frequent backups.

    One day, said Lenovo failed. It was replaced under warranty, and a restore attempted. No luck, same error code every time.

    A very long phone call later, the error code was explained: the software will only restore to the original PC the backup was made from. No way to recover the data to another PC, stored in a proprietary format, and encrypted.

    Within hours, Linux was installed, and all backups since done with tar to tape, and tapes rotated using GFS, with archives occasionally. Her sister is quite happy to accept the odd DAT72 tape for offsite storage. Hell, she even has an LTO tape of my stuff in her cupboard too. And yes, several files have been recovered more than once since. And now the Lenovo is due for replacement, migrating all of /home/* will be quite painless.

    Sometimes, the old tunes are the best.

    5 0 Reply
    1. David 132 Silver badge
      Reply Icon
      Pint

      the software will only restore to the original PC the backup was made from. No way to recover the data to another PC, stored in a proprietary format, and encrypted.

      Seriously??

      Wow. The stupid, it burns.

      Have a pint.

      3 0 Reply
      1. Alan Brown Silver badge
        Reply Icon

        It burns badly enough that the offending backup software should be named and shamed.

        FWIW most such "encryption" is trivial and you tend to find that the "key" is either the DOS FS serial number or the ethernet MAC.

        0 0 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like