back to article Eutelsat OneWeb blames 366th day for 48-hour date disaster

Eutelsat's OneWeb constellation suffered a date-related meltdown last week while the rest of the IT world patted itself on the back for averting the Y2K catastrophe a quarter of a century ago. The satellite broadband service fell over on December 31, 2024, for 48 hours. According to Eutelsat, "the root cause was identified as …

  1. Anonymous Coward
    Anonymous Coward

    I wonder if this was because they hired someone from Alphabet/Google who then decided to smear the 366th day instead of counting it?

    1. David 132 Silver badge
      Happy

      December 31st is an entirely respectable day, and any smears you may have heard are a sordid & baseless tabloid fabrication.

      (Yes, yes, I know the meaning of "smear" in this context, but allow me my moment of silliness...)

    2. Ivor
      Thumb Up

      smear seconds

      although smearing is for leap seconds not leap days. still excellent to get the word 'smear' in.

    3. UnknownUnknown

      If anyone was going to smear the venerable Leap Year that would be Elon Musk trolling it as a fake Vaccine con, or was complicit as a rape genocide apologist.

  2. Ken Moorhouse Silver badge

    Rocket Science

    It seems that rocket science is easier than terrestial date calculations.

    1. Irongut Silver badge

      Re: Rocket Science

      Like I always tell junior programmers, dates and times are hard. Always use the system libraries, never roll your own date/time code.

      At least then when something goes wrong you can blame Microsoft or whoever supplied said libraries.

      1. Pascal Monett Silver badge
        Mushroom

        Re: Rocket Science

        Dates would be easier if they were actual objects.

        We've been using object-oriented programming for decades now, but dates are still stored as a fucking string.

        With the seperator.

        Do you have any idea how many different seperators are used all over the world ? I have found these : / - . \ ! (and when I say found, I mean having to repeatedly debug code that was working fine until a new use case was found).

        If date/time values were stored as proper objects, with Year, Month, Day, Hour, Minute, Seconds, Milliseconds, and recoverable with the proper routines, then I strongly believe that all this kind of fuss would be eliminated.

        But, even though Redmond is on version 11 of its perenially buggy software, it would apparently be too much of a hassle to record date/time values as the objects they are.

        So we continue to get fucking strings, and having to parse them every time.

        1. Eclectic Man Silver badge

          Re: Rocket Science

          There is an international standard for date formatting, ISO 8601:

          "When dates are represented with numbers they can be interpreted in different ways. For example, 01/05/22 could mean January 5, 2022, or May 1, 2022. On an individual level this uncertainty can be very frustrating, in a business context it can be very expensive. Organizing meetings and deliveries, writing contracts and buying airplane tickets can be very difficult when the date is unclear.

          ISO 8601 tackles this uncertainty by setting out an internationally agreed way to represent dates:

          YYYY-MM-DD

          Therefore, the order of the elements used to express date and time in ISO 8601 is as follows: year, month, day, hour, minutes, seconds, and milliseconds.

          For example, September 27, 2022 at 6 p.m. is represented as 2022-09-27 18:00:00.000."

          https://www.iso.org/iso-8601-date-and-time-format.html

          1. Gene Cash Silver badge

            Re: Rocket Science

            That's fuckin' lovely if everyone would use ISO 8601... but here in the US we gotta do this stupid dd-mm-yy shit or sometimes it's mm-dd-yy shit, and of course they assume you know which one they use, because no one ever uses a different order!

        2. graemep

          Re: Rocket Science

          >We've been using object-oriented programming for decades now, but dates are still stored as a fucking string

          Where exactly are they stored as strings? If I use Python they are stored as datetime.date (or datetime.datetime objects), if they are stored in Postgres they are stored as a date time type. SQLite does store them as strings but in a very limited range of formats so you do not have that problem.

        3. AndrueC Silver badge
          Boffin

          Re: Rocket Science

          We've been using object-oriented programming for decades now, but dates are still stored as a fucking string.

          In over 30 years of programming I have never come across anyone storing a date in a string(*). It would be wasteful of storage and a pain in the arse to manipulate. Date/times are always stored as some kind of numeric offset or a collection of numbers (the former being preferable).

          At level most programmers work with those languages that support objects do indeed represent them as objects. Here's one example, C#:

          DateTime.

          Time values are measured in 100-nanosecond units called ticks. A particular date is the number of ticks since 12:00 midnight, January 1, 0001 A.D. (C.E.) in the Gregorian calendar. The number excludes ticks that would be added by leap seconds. For example, a ticks value of 31241376000000000L represents the date Friday, January 01, 0100 12:00:00 midnight. A DateTime value is always expressed in the context of an explicit or default calendar.

          (*)There are some data transfer formats, such as CSV and XML that do but that's not storage.

          1. David 132 Silver badge
            Thumb Up

            Re: Rocket Science

            Thank you. Made the point that I would have made if I wasn't nervous that I'd missed some subtlety and would make a fool of myself. C#'s DateTime structure was precisely the one I was going to offer as evidence.

            1. AndrueC Silver badge
              FAIL

              Re: Rocket Science

              To be fair where dates and times are concerned it's always best not to make assumptions and there are often gotchas that catch you out. Like a company I used to work for whose licensing periods were date based. That was fine for a while but for some reason (now lost in the mists of my mind) they switched to a time based system and chose to end the license period at midnight.

              Unfortunately they neglected to consider the impact of timezones on their international customer base. We ended up having to add a 24 hour grace period if I remember correctly which rather made a mockery of the whole 'let's be more specific about license periods' idea.

              Mind you the same company also included an example license file (it was a text format) in their help documentation which granted indefinite licensing. It must've been there for several years before I finally noticed it.

          2. Bebu sa Ware
            Windows

            Re: Rocket Science

            "12:00 midnight, January 1, 0001 A.D. (C.E.) in the Gregorian calendar."

            Presumably 0001-01-01 00:00:00.000 is proleptic here as Pope Gregory's boss would have been born then - presumably on 0001-12-25 evening.

            As previously noted times, dates and calendars are hard. Calendrical Calculations is highly recommended.

            Consulting that text the date format above appears to be based on the rata die system.

        4. munnoch Silver badge

          Re: Rocket Science

          "We've been using object-oriented programming for decades now, but dates are still stored as a fucking string"

          Not where I come from.. Times as offsets from an epoch and dates as a Julian number. Converting to and from anthropomorphic conventions like which day of the month is it are a bit of a pig but arithmetic is hard to fuck up.

          Its pretty much a core tenet of OO to decouple the internal representation from the external presentation but majority of developers I've worked with don't get that. If it comes in as a string they'll store it as a string oblivious to the possibility of anything else.

        5. An_Old_Dog Silver badge
          Holmes

          No Easy Answer

          @Pascal Monet:

          How many different ways are there to store dates and times on magnetic media? JSON? XML? Binary blobs? Which order(s) are the fields stored in? Not all compilers and interpreters, even for a single language, do these things the same way. Which types and sizes are those fields? Int? Long int? Short int? Char? BCD? Enum? If they are stored as binary words, the length will vary depending on CPU, even within the x86 family, which is not what every computer uses!

          If Program A reads a date a user types in (as a string, which needs to be parsed), and stores it in a file, Programs B, C, D, E, et. al. will have to read that from the file and have to correctly interpret it in order to load it into an in-RAM object.

        6. Anonymous Coward
          Anonymous Coward

          Re: Rocket Science

          We've been using object-oriented programming for decades now, but dates are still stored as a fucking string

          I'm not saying that date/time handling is simple, nor am I denying that it has hidden dangers, but I think that your statement says more about the quality of your software development than it does about he complexities of handling dates.

        7. John Smith 19 Gold badge
          Unhappy

          "how many different seperators are used all over the world ?"

          For this sort of question, along with telephone numbers* and 2 and 3 letter country abbreviations I find the UN is pretty helpful.

          I don't know if it was an actual standard anywhere but RPG400 installations used YYYYMMDD so date comparison (which was very frequent) worked out properly.

          The separator malarkey was down to the screen or report formatting.

          *The real limit is 20 digits. Sounds huge but UK mobiles take 5 of those for the provider, global population is already in 10 digits. Devices with their own phone numbers?

      2. gnasher729 Silver badge

        Re: Rocket Science

        Even the most stupid junior programmer should get the number of days in a year right for years from 2001 to 2099. Stupid people might fail in 2100. Halfwits might fail in 2000 as well. But getting it right for all years as far as it is currently planned is really still easy.

        This one is not due to a leap year, but due to utter idiocy.

        1. David 132 Silver badge
          Happy

          Re: Rocket Science

          > But getting it right for all years as far as it is currently planned is really still easy.

          Conversation circa 2097:

          "Sir, we have an impending crisis. Many of our critical IT systems here on Earth-1 and throughout the Musk Galactic Empire will fail in just a couple of years time!"

          "Why?"

          "As far as we can tell, they were coded by - quoting job posting literature of the period - 'stupid junior programmers', who utterly failed to calculate the correct number of days for 2100."

          (shocked pause)

          "My goodness. This will make the reprogramming we had to do in '78 to account for the Great Collapsing Hrung Incident look positively trivial..."

          1. UnknownUnknown

            Re: Rocket Science

            Wetland-Yutani-Musk.

            I’m sure that was future Elon in the belly of Prometheus.

        2. Len

          Re: Rocket Science

          Don't forget Excel's year 1900 bug that is still present in every version of Excel. Excel incorrectly assumes that the year 1900 is a leap year

          I bet scientists doing historical time series are frequently caught out by this bug.

          1. richardcox13

            Re: Rocket Science

            > Don't forget Excel's year 1900 bug that is still present in every version of Excel.

            You mean Lotus 1-2-3's "1900 is a leap year" bug that Excel copied because they wanted compatibility with the, then, dominate spreadsheet.

            This worked – who uses Lotus 1-2-3 today[1]?

            And, yes, it has been over two decades since I needed to create my own date handling (MS-DOS didn't do system libraries), but even then could use day count since epoch with a little reading (pre-internet days) and store that.

            [1] I expected there are some who take pride in their masochism.

            1. Anonymous Coward
              Anonymous Coward

              Re: Rocket Science

              "This worked – who uses Lotus 1-2-3 today?"

              I don't use that Lotus spreadsheet...but I do still have a copy of the DOS version of "As-Easy-As" - which was a 1-2-3 shareware clone from the early 1980s...and it ran rings around 1-2-3 (IMHO) and was easy to install and didn't cost a fortune !

      3. John Smith 19 Gold badge
        Thumb Up

        "dates and times are hard. Always use the system libraries, never roll your own date/time code"

        Damm right.

      4. l8gravely

        Re: Rocket Science

        Except when you need to do dates before 1970, say for a database of old Newspapers in a library collection that people might want to search on... then it's a pain. And when the library you use is abandonware and not exactly happy with newer versions of PHP without some hacking... then it's not fun.

        Ask me how I know...

  3. Anonymous Coward
    Anonymous Coward

    Laugh or cry?

    Well, the West has only had a calendar system with leap years in it for a smidge over 2000 years - such radical changes sneak up on people.

    1. Doctor Syntax Silver badge

      Re: Laugh or cry?

      Why worry over a single day when the original calendar was out by 11 days by the mid 18th century?

      1. Mage Silver badge

        Re: out by 11 days by the mid 18th century?

        Only in a couple of European countries and in the West, only Britain. E1R had been advised by John Dee to switch, but she thought she'd have too much opposition in the Privy Council. She didn't always get what she wanted.

        1. Doctor Syntax Silver badge

          Re: out by 11 days by the mid 18th century?

          That's why I said the original, i.e. Julian calendar. We're still living with the results of not switching with the rest in that the financial year runs from April 6th - instead of Lady Day.

          I suppose suspending leap seconds will stoke up a similar problem on a smaller scale.

        2. agurney
          Headmaster

          Re: out by 11 days by the mid 18th century?

          "..only Britain. E1R had been advised by John Dee to switch..."

          E1R wasn't Britain; she was only England and Ireland.

          1. druck Silver badge

            Re: out by 11 days by the mid 18th century?

            Wales?

            Or to put it another way; not Scotland.

      2. Len

        Re: Laugh or cry?

        This is very interesting reading: How the lore of New Year defeated the law of New Year – how the English state gave up on insisting the new year started on 25 March. Especially if that explains why the UK's tax year still starts 11 days later on 5 April.

  4. Andy Non Silver badge
    Coat

    Someone's

    days are numbered.

  5. Anonymous Coward
    Anonymous Coward

    TFF

    Is this the same OneWeb that was going to deliver a British Brexit GPS? (Let the lion roar, etc.)

  6. harmjschoonhoven

    Re: Laugh or cry?

    The west had it easy. Have pity with the Ottoman Empire: Tax revenues were linked to the harvest, so the solar calendar. The soldiers and the bureaucrats were paid according to the Islamitic calendar linked to the phases of the Moon.

    From time to time this discrepancy led to an empty treasury.

  7. xyz Silver badge

    Oh FFS.. It's coding 101

    That is all.

    1. Herring` Silver badge

      Re: Oh FFS.. It's coding 101

      Quite. This is why I have always insisted my team stores everything in sidereal days and converts.

      1. gnasher729 Silver badge

        Re: Oh FFS.. It's coding 101

        TAI is _the_ standard for time. Number of atomic seconds since a starting date. Conversion GPS to TAI is trivial: Add 18 seconds. Conversion TAI to UTC has caused chaos the last two times it changed and won’t change again for the next ten years (and hopefully never again)

        Anything else can be handled as localisation (translating between TAI and local time).

        1. Herring` Silver badge

          Re: Oh FFS.. It's coding 101

          To be honest, I am thinking a 1024bit number which is the representation of Planck times since the big bang. Fixes everything. Almost

  8. JWLong Silver badge

    You would think....

    ....it would fail over on 02/29.

    1. IGotOut Silver badge

      Re: You would think....

      How to say you're from the US without saying you're from the US.

      1. David 132 Silver badge

        Re: You would think....

        Obligatory XKCD.

  9. Howard Sway Silver badge

    OneWeb blames 366th day

    Now there's an idea. Add a 366th day (or 367th every 4 years) to the calendar, so programmers can use that day to fix these bugs every year. It would have to be after Jan 1 to discover the bugs, so introduce a Jan 1.5 for that purpose. The calendars would start drifting a bit without compensation for the extra day, so take a day off November each year. Who likes that month anyway?

    1. Uncle Slacky Silver badge
      Boffin

      Re: OneWeb blames 366th day

      The extra day should be January 0th, shirley?

  10. IGotOut Silver badge

    Who knew....

    .that Apple made satellites.

    1. Anonymous Coward
      Anonymous Coward

      Re: Who knew....

      Every time Apple came out with a 'new' OS / platform they entirely messed up calendaring, and in the most predictable ways. Every time. If you don't remember that, you don't know enough. Your downvotes are from ignorance.

  11. Will Godfrey Silver badge
    Facepalm

    Clearly from the "You couldn't make it up" dept.

    I think it was something like 40 years ago when I first came across time programs, and the pitfalls were well known even then.

  12. bazza Silver badge

    It's be interesting to know exactly what went wrong. For this kind of system, there is no reason for the system to work in "natural" time at all. One picks an epoch, and counts seconds from that point in time. One might even deign to use International Atomic Time TIA (if one wants to be rate-compatible with systems like GPS). If at any point in the system this system time needs translating into something friendly to humans, you write a function to provide that conversion, but you do all the calculations and storage in TAI.

    There's some very good libraries for doing this accurately, available from the International Astronomy Union - the SOFA library (SOftware Fundamenals for Astronomy). This will do accurate conversions between timescales such as International Atomic Time (TIA) and UTC (and a load more timescales) allowing for leap seconds, the lot. You have to update it when a new leap second is published, but leap seconds have been suspended for the moment.

    It's quite interesting because it highlights the impossiiblity of accurately having a Julian date/time in UTC. Which is a pretty good hint that one shouldn't use UTC (and therefore, most computers and most software libraries) for systems like this.

    Linux is actually quite favourable for such systems these days. With Linux, gpsd and a GPS receiver, Linux can maintain an accurate representation of TAI (even though the primary system clock is running on the traditional incomplete implementation of UTC), and one can program against TAI (e.g. you can ask for the TAI current date / time, instead of the local date/time). It even deals properly with leap seconds, in that when one occurs the TAI timescale remains correct.

  13. heyrick Silver badge
    Mushroom

    "An issue related to the number of days in a leap year will have many software engineers stroking their chins thoughtfully."

    Why? There are three well documented rules that have been in effect since 1582. Fifteen eighty two. Any programmer that can't work out whether a contemporary year has a February 29th and thus an extra day is, frankly, incompetent.

    [icon, because no excuse...grrrr...]

    1. AndrueC Silver badge
      Happy

      Um, well, sorta. On the other hand if you're writing software that works with historic information you also have to consider a few other years in order to also get the date correct. Some of the more notable are 1752 (Britain and colonies), 1918 (Russia and friends) and the most recent convert 2016 (Saudi Arabia).

      For example the dates 3rd September to 13th September 1752 do not exist in the British Colonies.

      But apart from that it's really quite simple :)

      1. Ivor
        FAIL

        simples

        Indeed, it's non-trivial, I had to write date code that could cope with dates back to 1582 since it had to be able to load and use historical data... and it had to manage for leap seconds.

        but still, to see a basic leap day error in modern code is utterly depressing especially by someone working with GPS technology where clearly time and date is KIND OF THE ENTIRE UNDERPINNING.

  14. Henry Wertz 1 Gold badge

    ... And that's why

    ... And that's why you don't roll your own time/date handling code. I'm so glad for the work I do that Python has lovely datetime and timezone libraries to take care of leap days and leap year, daylight savings time, etc. so I don't have to.

    1. Will Godfrey Silver badge
      Happy

      Re: ... And that's why

      I did roll my own once, but that was on an Arduino yonks ago.

      It didn't stay like that. I popped the chip out, and inserted it into the custom PCB that we made for a factory shift notification system. The time source was a GPS module. The control system was in the production manager's office and consisted of a number of very solid buttons, and an LCD screen behind a 1/4 inch thick perspex shield. About three years later the factory was bought out and closed - make what you like of that :P

  15. DS999 Silver badge

    So I guess this is new code

    Since it had to have gone into production after Jan. 1 2021 or it would have hit this issue then. Too bad, they could have blamed it on the covid vaccine and a lot of people would have believed them!

  16. Gene Cash Silver badge

    impacted the manual calculation for the GPS-to-UTC offset

    WTAF is a manual calculation doing in a satellite control system?

  17. Locomotion69 Bronze badge
    Facepalm

    Code reviewed by

    a so called "senior" with has 6 months experience after leaving college.

  18. Ken Shabby Bronze badge
    FAIL

    Excel bug

    Excel considers the year 1900 as a leap year, as such it adds an extra day, it is said that this is known and will not be fixed, for backwards compatibility with Lotus 1-2-3.

    Plus more

    https://en.m.wikipedia.org/wiki/Leap_year_problem

    1. upsidedowncreature

      Re: Excel bug

      Is it fair to call it an Excel bug then? Isn't it a Lotus 1-2-3 bug that Excel replicated intentionally?

  19. Anonymous Coward
    Anonymous Coward

    Equivalent problem 45 years ago

    Is anyone else here old enough to remember IBM's 1980 New Years bug?

    Most large companies used IBM mainframes running the MVS (Multiple Virtual Storage) OS. Telecom was handled by ACF/VTAM (Advanced Communications Function Virtual Telecommunications Access Method) running on the mainframe. The mainframe's byte-oriented I/O was not well suited to the bit-oriented operations required for telecom. The bit operations were handled by the IBM 3705, a separate computer with its own special purpose instruction set and OS, attached to the mainframe via a byte channel. The 3705 had no disk and no console, only a few front panel dials and pushbuttons. The 3705's OS, ACF/NCP (Advanced Communications Function Network Control Program), was cross compiled on the mainframe, then downloaded from the mainframe via a VTAM command. The download included a reboot step. The internal message structure for the reboot contained a YYDDD date field, computed by VTAM.

    Customers who chose to reload a 3705 on New Years Day 1980 found the downloads failed. At the time, I was working for Bank of America. Their practice was to reload all their 3705s overnight. My pager sounded soon afterwards. VTAM derived the date field from the CPU's count of ticks from a fixed base date. On 1980-01-01, it recognized that 1980 was a leap year with 366 days, and erroneously computed the date as 79366. NCP rejected the reboot command for its invalid date.

    I had been working on mainframes for only a few months, having switched over from a minicomputer system. After several hours of hunting, I found the error in IBM's microfiche listings of VTAM source code and crafted a correct fix. But my inexperience defeated me. Bank of America had multiple MVS mainframes, loosely coupled in a JES3 (Job Entry Subsystem) complex. Each 3705 was connected to a specific mainframe. I needed to specify that mainframe on my job card to ensure the patch job ran on that system. I was too green to know that. My patch was applied to the wrong system and the test failed. A few hours later, onsite IBM support staff got an equivalent fix from an IBM support center and applied it successfully.

    1. AnAnonymousCanuck

      Re: Equivalent problem 45 years ago

      Ahhh, the joys of JCL. The most powerful and most unreadable language EVER. And, I include assembler, as some of us can (or used to be able to) read hexadecimal.

      Thank you for the story and the memories. Ex-data entry clerk who "learned" JCL from "examples" and production code in the system.

      Another Anonymous Canuck

  20. awomanmanhasaname

    I monitored it myself. Cloudflare claims it was 30 hours

    An apparent failure to account for 2024 as a leap year resulted in a ~30 hour #Internet outage for customers of LEO satellite provider

    @EutelsatGroup

    OneWeb. The disruption occurred 31-Dec 00:00 UTC through 01-Jan 06:00 UTC.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like