Excellent, hopefully that is included in 0.19.3 that came out overnight... will upgrade shortly.
lodion
joined 2 years ago
MODERATOR OF
Ok, there's more to this than I first thought. It seems there is a back end task set to run at a set time every day, if the instance is restarting at that time the task doesn't run... this task updates the instances table to show remote instances as "seen" by AZ. With the memory leaks in 0.19.1, the instance has been restarting when this task is running... leading to this situation.
I've updated the server restart cronjob to not run around the time this task runs... and I've again manually updated the DB to flag all known instances as alive rather than dead.
Will keep an eye on it some more...
For anyone curious, two of the bugs that are related to this:
https://github.com/LemmyNet/lemmy/issues/4288
https://github.com/LemmyNet/lemmy/issues/4039
Ok, something is busted with the lemmy API endpoint that shows current federation state. It is currently showing nearly all remote instances as dead:
But "dead" instances are still successfully receiving content from AZ, and sending back to us.
Seems to have sorted it for the most part... not sure what caused it, will do some more digging.
Ok, for some reason practically all instances were flagged as "dead" in the database. I've manually set them all to be requeued... server is now smashed as it attempts to updated the ~4000 instances I've told it are no longer dead. See how this goes...
☹️
I'll see what I can see...
You've got a way with words... are you a writer? :)
Prior to restarting the lemmy service, this showed over 2k instances as "lagging". Shortly after restart, its dropped down to single digits.
I'll leave the hourly restart going for now, it should help with federation issues... and should help with memory leaks too.
waves from aussie.zone
Testing :)
Cool, we're on 0.19.3 now.