• @xmunk@sh.itjust.works
    link
    fedilink
    127 months ago

    I have a production bug… it only happens on Saturdays ever our ops folks have no idea - this can be replicated on a test server that gets no traffic.

    Saturday why!

    • lad
      link
      fedilink
      57 months ago

      If we reject the theory that it could be someone’s elaborate revenge, Saturday may be the first day of the week that may become workday or non-workday because of incorrect assumption about the first day of the week. If everywhere but one place in your software the day numeration is correct it would be a hard bug to spot.

      Also, if it is in Java, I vaguely remember there being a lot of ways to express weekday, so a lot of ways to shoot off your foot (solely on Saturday)

      • @xmunk@sh.itjust.works
        link
        fedilink
        47 months ago

        For bonus points, this failure is in a cron job that sends out recently queued messages. It runs once every ten minutes - last weekend we had 12 failures: four were in a cluster on their own, one was in a run of two, and six were in a single continuous run.

        Please note that this server is unused by our business so no messages ever get naturally queued. Every day we sync the live production server to this server at about 9 PM - assuming an employee was queuing up a message before the snapshot is taken there might be a number of unsent messages in the snapshot - those messages will all be sent by the first cron job after the sync.

        It is a wonderfully awful problem that has me wanting to pull out my luscious locks.

        • lad
          link
          fedilink
          47 months ago

          I wish you luck. Also, maybe you could get someone else to get a fresh view of an issue

          • @xmunk@sh.itjust.works
            link
            fedilink
            47 months ago

            Yup, luck is appreciated and I’m trying to get more eyes but unfortunately I’m a senior dev that has the second highest seniority at the company so I feel guilty dragging others into it.

    • @CanadaPlus
      link
      3
      edit-2
      7 months ago

      Please tell me it doesn’t still happen when you emulate a different day of the week. Or is that non-trivial to even do because of technical debt? Either way, RIP weekends.