I’ve been seeing the exact same posts, in the same order, with almost no change in up/downvotes. It’s been like this for at least 6 hours.

Did everyone agree to take an internet break and not tell me or is something not working correctly?

  • jerry@fedia.io
    link
    fedilink
    arrow-up
    7
    ·
    1 month ago

    my apologies everyone - it was working rock solid for a few days and I got lulled into a false sense of security and took my eye off the site for a few hours :/ I am still working on some alerting to tell me when rabbitmq dies, but it’s obviously got some room for improvement.

    In any event, I am aware, I’ve fixed the current problem and the queues are draining now. I’ll work on a better monitoring and response automation system tomorrow.

      • jerry@fedia.io
        link
        fedilink
        arrow-up
        2
        ·
        1 month ago

        that’s not so good. this is a different problem - there isn’t a backup like we’ve been seeing. I’ll see if I can tell what is going on

      • jerry@fedia.io
        link
        fedilink
        arrow-up
        1
        ·
        1 month ago

        I’m working on it - not exactly sure what is wrong now - there aren’t any logs to indicate what is going on yet.

        • Hello_there@fedia.io
          link
          fedilink
          arrow-up
          1
          ·
          1 month ago

          I just tried this again. When I just posted, and the last time I did it, I got an http error message. However, when I press back, the post is there. Not sure if this is a fedia thing or an mbin thing or a Lemmy thing. But that’s likely the problem.

          • jerry@fedia.io
            link
            fedilink
            arrow-up
            1
            ·
            1 month ago

            I am seeing the same thing, and I’m not sure. I am trying to narrow down what is going on though

            • Hello_there@fedia.io
              link
              fedilink
              arrow-up
              2
              ·
              28 days ago

              Jerry, we’ve got what you termed the rabbitmq issue happening again. Posts on new aren’t updating.

              • jerry@fedia.io
                link
                fedilink
                arrow-up
                3
                ·
                28 days ago

                yeah, apologies. it seems to know when I am out at dinner. it’s catching up now

            • Hegar@fedia.ioOP
              link
              fedilink
              arrow-up
              1
              ·
              28 days ago

              Yeah, as @hello_there@fedia.io pionted out, it looks like parts aren’t updating again. I’ve had the same posts on ‘hot’ for a while now.

  • dhhyfddehhfyy4673@fedia.io
    link
    fedilink
    arrow-up
    0
    ·
    1 month ago

    Yea, federation keeps breaking unfortunately. Hopefully an automated solution can be implemented.

    Btw, can someone tell me how to ping a user? Admin could probably use a heads up.

    • Drusas@fedia.io
      link
      fedilink
      arrow-up
      0
      ·
      1 month ago

      I’ve tried to look up how to DM a user and I haven’t found a way yet, so I’d like to know as well.

      • Davel23@fedia.io
        link
        fedilink
        arrow-up
        2
        ·
        1 month ago

        To DM someone open their user profile by clicking their name then click the little envelope icon. To ping someone just mention their username preceeded by an ampersand @ like so: @dhhyfddehhfyy4673

          • Davel23@fedia.io
            link
            fedilink
            arrow-up
            3
            ·
            1 month ago

            Ah, right. I wrote that lying in bed around 1 AM, surprised I got any of it correct.

  • shoulderoforion@fedia.io
    link
    fedilink
    arrow-up
    0
    ·
    1 month ago

    Is it just fedia.io or is this happening to other mbin instances too? seems like this happens every now and then, and i also find posts from fedia aren’t appearing on lemmy instances until there is server admin intervention of some sort, then it all updates at the same time. maybe the server admin isn’t available due to the jewish new year holiday?

    • jerry@fedia.io
      link
      fedilink
      arrow-up
      1
      ·
      1 month ago

      My apologies - I was off doing other things today as I thought this problem had been fixed. Things are almost complete with catching up now and I’ll be working on the problem more tomorrow.

    • doc@fedia.io
      link
      fedilink
      arrow-up
      0
      ·
      1 month ago

      Per @jerry there’s some bug in federation where an incoming message is malformed or something and it causes the message queue to silently freeze. He just has to restart it and things start moving again. Unfortunately it sounds like the mbin devs haven’t got it figured out yet so we deal with the problem now and then.

      • jerry@fedia.io
        link
        fedilink
        arrow-up
        1
        ·
        1 month ago

        My apologies - I thought I had the problem fixed, but clearly not. It’s working again, and I will get back to fixing the alerting and auto-error correcting tomorrow.