I’ve been seeing the exact same posts, in the same order, with almost no change in up/downvotes. It’s been like this for at least 6 hours.

Did everyone agree to take an internet break and not tell me or is something not working correctly?

  • jerry@fedia.io
    7·
    1 month ago

    my apologies everyone - it was working rock solid for a few days and I got lulled into a false sense of security and took my eye off the site for a few hours :/ I am still working on some alerting to tell me when rabbitmq dies, but it’s obviously got some room for improvement.

    In any event, I am aware, I’ve fixed the current problem and the queues are draining now. I’ll work on a better monitoring and response automation system tomorrow.

      • jerry@fedia.io
        2·
        1 month ago

        that’s not so good. this is a different problem - there isn’t a backup like we’ve been seeing. I’ll see if I can tell what is going on

      • jerry@fedia.io
        1·
        1 month ago

        I’m working on it - not exactly sure what is wrong now - there aren’t any logs to indicate what is going on yet.

        • Hello_there@fedia.io
          1·
          1 month ago

          I just tried this again. When I just posted, and the last time I did it, I got an http error message. However, when I press back, the post is there. Not sure if this is a fedia thing or an mbin thing or a Lemmy thing. But that’s likely the problem.

          • jerry@fedia.io
            1·
            1 month ago

            I am seeing the same thing, and I’m not sure. I am trying to narrow down what is going on though

            • Hello_there@fedia.io
              2·
              28 days ago

              Jerry, we’ve got what you termed the rabbitmq issue happening again. Posts on new aren’t updating.

              • jerry@fedia.io
                3·
                28 days ago

                yeah, apologies. it seems to know when I am out at dinner. it’s catching up now

            • Hegar@fedia.ioOP
              1·
              28 days ago

              Yeah, as @hello_there@fedia.io pionted out, it looks like parts aren’t updating again. I’ve had the same posts on ‘hot’ for a while now.