So today I discovered that there’s a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn’t live inside any source control. This morning it entered a terminal state, and because it overwrites its state there’s no way to revert it.

I’m currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

  • wise_pancake@lemmy.caOP
    link
    fedilink
    arrow-up
    167
    ·
    10 days ago

    What’s extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn’t live in it for no discernible reason.

      • wise_pancake@lemmy.caOP
        link
        fedilink
        arrow-up
        115
        ·
        10 days ago

        He does charge a consulting fee to “fix” these issues

        Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.

          • Agent641@lemmy.world
            link
            fedilink
            arrow-up
            19
            ·
            10 days ago

            Judgement day postponed indefinitely due to “Object reference not set to an instance of an object”

            • kiagam@lemmy.world
              link
              fedilink
              arrow-up
              2
              ·
              9 days ago

              I don’t know why but this is the first time I read this phrase and it actually makes sense.

              I knew exactly what it meant before, but it didn’t make sense until now

            • Sherry@programming.dev
              link
              fedilink
              arrow-up
              4
              ·
              9 days ago

              that might be a stupid question, but why would you running all services in tmux be a bad idea? a co-worker of mine is doing exactly that right now, which is why I’m asking.

              • swab148@lemm.ee
                link
                fedilink
                English
                arrow-up
                4
                ·
                9 days ago

                It’s not horrible, like it’ll do the job just fine, it’s just probably a better idea to use systemd and like, containers and whatnot, but I couldn’t be arsed to fiddle with all that for Jellyfin, caddy reverse proxy, and two modded Minecraft servers, so shell scripts and tmux won the day. It takes a little extra time to restart everything after an update, and maybe I’ll get the motivation to do things “correctly™” one day, but today is not that day.

              • qaz@lemmy.world
                link
                fedilink
                English
                arrow-up
                2
                ·
                edit-2
                7 days ago
                1. They’re all gone when you restart
                2. It doesn’t properly deal with logging
                3. You can’t set up dependencies between services but that doesn’t matter due to point 1

                I recommend using systemd services and/or docker compose instead. systemd services are files that describe which program / script to run and when (like after networking is active or after a certain other service is loaded).

      • koper@feddit.nl
        link
        fedilink
        arrow-up
        20
        ·
        10 days ago

        But the whole point of the doomsday machine is lost… if you keep it a secret! Why didn’t you tell the world, eh?

        • MyBrainHurts@lemmy.ca
          link
          fedilink
          English
          arrow-up
          10
          ·
          10 days ago

          It was going to be announced at his retirement party on Monday… You know the dev likes surprises.