crashdoom

joined 2 years ago
MODERATOR OF
[–] [email protected] 6 points 2 months ago (2 children)

Restarting the federation pods appears to have at least lowered the failure queue a little, but it's showing that while we've sent things to lemmit.online (as the example), it hasn't seen anything from them since 4/14/2025, 10:14:27 PM. So I'm presuming the next time they try to sync and send something to us it SHOULD update...

The federation state checker seems to be indicating that we're not behind in sending them posts anymore so outgoing should be fixed, but I'm unsure how to troubleshoot incoming further. I'm wondering if there's a bug with the latest version of Lemmy?

[–] [email protected] 7 points 2 months ago (3 children)

Morning all! It does appear like we're not getting some instances posts (you can find a list by using https://phiresky.github.io/lemmy-federation-state/site?domain=pawb.social). I'm currently investigating and I appreciate folks poking on this :3

[–] [email protected] 14 points 3 months ago

😂 We've given it a stern talking to and helped it recover through a restart, and image uploads appear to be working as normal again!

[–] [email protected] 2 points 3 months ago (1 children)

via here (I'm setting up an RSS feed monitor), via email (network[at]pawb.social), or with a DM message to highlight it. It seems like mentions in the original post don't alert the mentioned person, so I only got a notice when you replied to a message from me / mentioned me in a comment. >~<;

[–] [email protected] 2 points 3 months ago (3 children)

Media uploads should be working again. My apologies.

[–] [email protected] 2 points 3 months ago

Media uploads should now work again!

[–] [email protected] 5 points 3 months ago

We're investigating! Appreciate the report!

[–] [email protected] 2 points 3 months ago

Taking a peek now, appreciate the heads up!

[–] [email protected] 6 points 4 months ago (4 children)

The server hosting the database crashed for a still unknown reason. Sometime around 5 am MT, the database server had a kernel panic and locked up, it’s been like that since then until we woke up to the alert this morning.

We’ve got a watchdog on the server and an auto-restart in case it hangs, and we’re working to migrate it away to a known stable server while we troubleshoot.

[–] [email protected] 7 points 4 months ago (1 children)

Honestly, like you said, I genuinely have no idea what the US will do; Currently things are being so shaken up that anything is plausible right now.

For outside the US, I don't really know what reach other countries have. The UK's Online Safety Act is the current one I'm watching for since that would require "highly effective age assurance" which... if the folks writing the law had been online like, ever, they'll have realized is an utterly stupid statement unless they plan to enforce integrating ID Verification via a Credit Agency (Experian, Equifax) which does personal question-based verification (which imo is a huge breach of user privacy since while WE might not know who you are still, the credit agency does and where you signed up).

[–] [email protected] 7 points 4 months ago (3 children)

Yes, several US states along with other nations have been considering porn ban laws, along with the rather blatant anti-LGBTQ stance of the currently elected US party.

[–] [email protected] 6 points 5 months ago (1 children)

In the works! We're planning to have some info very soon, though we're still building out an improvement for the NAS to allow us to expand the storage capacity for long term, high size file storage specifically for Pixelfed!

 
  • Instance: pisskey.io
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Nazi imagery, affiliated with poa.st and other known abusive instances
  • Fediseer Action: Censured
27
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
18
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
 
  • Instance: lab.nyanide.com
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Trolling, harassment, homophobia, nazi imagery, admin / mod engaged abuse
  • Fediseer Action: Censured

Block has been applied to the entire domain.

Evidence

 

cross-posted from: https://pawb.social/post/3393854

  • Instance: cunnyborea.space
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
  • Fediseer Action: Censured

Evidence

 
  • Instance: cunnyborea.space
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
  • Fediseer Action: Censured

Evidence

 

cross-posted from: https://pawb.social/post/3337642

  • Instance: bv.umbrellix.org
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Toxicity, abusive admin, death threats, emotional abuse
  • Fediseer Action: Censured

Evidence

Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.

 
  • Instance: bv.umbrellix.org
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Toxicity, abusive admin, death threats, emotional abuse
  • Fediseer Action: Censured

Evidence

Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.

 

Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects.

We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop.

We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up.

We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.

1
liberdon.com (pawb.social)
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/fediblock
 

One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. No data, except ElasticSearch, was stored on CEPH.

When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable.

Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present.

I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.

13
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
1
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
 

We've migrated all of the Pawb.Social (Lemmy) media from within the container to the new local NAS.

We'll be performing the same updates for pawb.fun and furry.engineer over the course of the next week, minimizing downtime where possible.

We will need to take the services offline once the initial transfer is completed to perform a final check for any new data, and transition to the local NAS, but we'll post an announcement prior to the final maintenance for each instance.

These changes are to allow us longer and cheaper retention of remote and local media across all of our services, and to allow us to begin backing up media in case of a catastrophic failure.

view more: ‹ prev next ›