Dubvee Meta

As always I would like to thank the Lemmy devs for their wonderful work of making super robust database migration scripts that don't cryptically fail /s

FYI: I'm probably looking at migrating this instance to Piefed in the near future (in lieu of Lemmy 1.0). That may be an issue if you use 3rd party Lemmy apps, and I'm hoping to come up with a solution for a direct migration (if such a solution does not already exist).

The Piefed dev said their API is pretty similar, so I may be able to add a translation layer. I dunno if that's possible, but I will be looking into it. Just wanted to give a heads up on that. Bottom line is I'm fed up and just about done with Lemmy as a platform now that there are viable alternatives.

They're not enforcing their own rules as of late and letting unsourced screenshots, social media, and other non-article opinion drivel remain up. I'm getting tired of modding it locally so that the content complies with our misinformation policy.

Until they get their shit together, please use:

• !politics@sh.itjust.works
• !politics@beehaw.org

Update 10:25 PM: Back online. It's going to take a bit for the instance to catch up from being down for almost 12 hours, but more instances seem to be reporting in /catching up except Lemmy World. Federation stats say next retry is 2025-02-17 06:45 AM. Not sure if there's anything I can do to kick that into gear sooner, and once it does start catching up, it'll probably take ~8 hours to get back in sync.

Apologies for the massive downtime (10 hours is "massive" by my standards). Got 2 more inches of rain on top of a week of rain which was on top of 3 inches of rain last week. Mudslides, rockslides, and flooding incidents were all over the place today plus some trees that uprooted. Not sure which of those was responsible for my outage, but could have been any or all.

On-prem, thanks to a clogged storm drain, my generator ended up half-submerged in water. Haven't had a chance to inspect it, but it's probably going to need repairs and is down for the count until then. :(

In other fun on-prem news, my 48v 1000W sine wave inverter bit the dust when I tried to press it into service today, so I no longer have that available to extend the UPS runtime. Had that not died, I probably could have ridden out this outage, or at least most of it.

If by chance I do get a tax refund this year, I already know what it's earmarked for.

Original Announcement:

---

Running on UPS, generator partially under water. Downtime inevitable. No ETA

I'm celebrating by not celebrating and just getting drunk with some friends which is different from celebrating because we all have to work in the morning.

SSD in the load balancer is showing signs of death, and I'm going to have to deal with that today.

I've spun up a temporary, virtual LB on another host and am re-routing traffic to that, so hopefully there won't be any noticeable downtime. That said, things could go awry, so just throwing this notice out as a CYA in case.

Preface

I've run DubVee for about a year-and-a-half as of this writing. Until now, I've not really had much of a mission statement beyond being an on-ramp to the Fediverse and trying to keep things civil and non-toxic from this instance's point of view.

Recent and semi-recent events have revealed an ugly truth about many of the users in the Fediverse (and social media in general), so I've put together an actual mission statement in hopes of addressing that. It's also my last-ditch effort to not shut down this project out of disgust.

I want DubVee to be an example of what the Fediverse can be when it's not overrun with political idealogies masquerading as people, extremists, lynch mobs, and people too toxic to be anywhere else. I know for a fact that there are plenty of users that don't fit that criteria, and I dare say they are the majority. However, the above-mentioned groups are definitely drowning them out and giving Lemmy (and the Fediverse) a bad image in the process.

To that, I say: Not here. Not on DubVee.

Why DubVee and Why We're Different

DubVee is not your typical Lemmy instance. In order to achieve our goals, we have to do things a little differently compared to other instances. We want the "default" experience to be positive, welcoming, and non-toxic without requiring new users to discover and deal with the toxic parts on their own. We want everyone to be able to go to the homepage without the default experience making them say "what the fuck?!" and turn away. (That's literally what happened when I tried inviting a few Reddit colleages to Lemmy).

Some people may disagree with that; that's fine and is part of the beauty of the Fediverse. There are plenty of other instances that let you rawdog everything (and everyone) the Fediverse has to offer which you can then curate on your own.

Dubvee is not such an instance, and that is a feature we're proud of.

As an instance, we're pretty heavily moderated in order to maintain a welcoming atmosphere that fosters civil discussion. We don't "tone police" everything, but we do expect users, local and remote, to behave themselves and act in a rational, civilized manner. Those who cannot, local or federated, are quickly shown the door.

We also expect people to be here because they want to be here (the Fediverse) rather than being here because they're too toxic and/or have been banned everywhere else.

Note: None of this is throwing shade on other instances. I'm simply highlighting that DubVee recognizes a problem exists, that we are actively addressing it here, and extending an invitation to anyone who wants to be a part of it.

Inspired by Beehaw

Beehaw existed for a year or two before DubVee came online and is seen by many as the "heart" of the Reddit-like corner of the Fediverse due to their mission, philosophy and guiding principles. I've looked to them as a role-model since day 1, and that has not changed. I've always tried to make DubVee as "Beehaw-lite" as I could. In fact, many times, I've described DubVee as "Beehaw with downvotes enabled".

Join Us?

If that's the kind of environment you're looking to be a part of, we welcome you. Even if you want to create an alt here as a "vacation spot" to get away from that on other instances for a while, you're also welcome.

While we're not quite ready to throw open the doors (we're still configured as a regional instance), we're planning to open up soon to a wider audience who wants to be a part of a better, nicer Lemmyverse.

If other instances are taking similar stances against extremism and lynch-mob mentality, then please let me know so I can shake their admins' hands.

---

DubVee's Goals

1. Address the Elephants in the Room

There is one, big elephant in the room that I'm going to point out: there is a not-insignificant userbase in the Fediverse that:

• Is here only because they're too toxic for regular social media
• Has been banned everywhere else and can keep spinning up alts on new instances
• Jump straight to violent solutions to every perceived problem
• Operate in complete bad faith but aren't technically breaking any rules on the more general-purpose instances
• Just want to stir up shit (that's not unique to the Fediverse, just worth mentioning)
• Think it's some kind of absolute free speech zone where they can air their most toxic thoughts
• ...Worse (we'll just leave it at that)

DubVee has always been quick to deal with those accounts, and recent events have reinforced and escalated the need to do so more firmly and more permanently.

2. Provide a Non-Toxic, Chill, Civil, and Safe Environment

Building on the first goal, we take several measures to maintain a positive atmosphere here:

• DubVee is open-federation by default, but we do not federate with any of the "big 3" extreme instances: ml, grad, or hexbear nor any instance known for spreading hate or trolling

• Extremism of any form is not tolerated. Accounts espousing or excusing extremist rhetoric are quickly banned. This includes, but is certainly not limited to:

  • Advocating, excusing, praising, suggesting, or condoning violence against any person or group (including political violence)
  • Hypocritical takes that amount to "Fascism / authoritarianism / whatever-ism is bad, but a little bit is okay when it's my side doing it"
  • "Burn it all down" mentality
  • Stirring the pot by jumping into otherwise civil discussions and injecting extreme rhetoric
  • Pretty much any pattern of behavior that devolves into "lynch mob" mentality.
  • "Wink and a nudge" versions of any of the above

• Hate speech toward any group is prohibited.

• Trolls, bad-faith actors, and people who cannot interact civilly are quickly shown the door.

3. Focus on Quality Over Quantity

We prioritize quality over quantity, and we're not obsessed with growth nor having an endless stream of low-quality content just to fill up space.

We allow very few bots here, AI-generated slop is prohibited, and we do not federate with instances that primarily act as "bridges" reposting from other platforms. Having seen the endless stream of low/no-activity bot posts on other instances, I am very happy with how this policy has been working. This is a platform for humans to interact with one another, share cool stuff, and discuss things. Bot posts have shown to be primarily noise rather than signal.

4. Prevent Misinformation

We have a very strict misinformation policy here, and it is enforced for both local content as well as content arriving via federation. Some of the measures we take to achieve this goal include:

• No low-credibility news sources are allowed
• News articles must always be to the original source and without editorialized titles (exceptions being made for non-news/politics communities).
• Any suspect/wild claims without references are subject to a "mod first, fact check later" moderation policy.
• Any federated news-like communities that are not in compliance with these rules will be administratively removed

More details can be found in our Misinformation Policy page.

5. Ask Users to Leave Their Political Idealogy at the Door

This is a big ask on the Fediverse it seems, but it's not without a good reason. Having run this instance for a year and a half, I've found that 99% of users who make their political idealogy their entire identity tend to be 100% insufferable and cause needless drama.

Just be a person and talk to other people about cool stuff. It's really not hard to not inject political idealogy into every interaction.

This is not to say you can't express your political views - far from it. Hell, there's plenty of politics communities subscribed on this instance. Just don't make everything about it, don't inject it into every conversation, and just be a person and not an arm of your political party.

While this is more of a request to users of this instance, we do ban accounts from federated instances that do nothing but inject political ideaology into every interaction. I will say that since we defederated from .ml that has been less of an issue, but it is still necessary on occasion.

6. Respect the Right to Free Expression - Within the Bounds of Civility

We absolutely believe in the right to free expression of ideas, but it must be done in good faith, in a civilized manner, and without adding needless drama or toxicity. Trolling, hate speech, bigotry, misogyny. harassment, extremist rhetoric/calls to violence, and/or otherwise saying the worst things imaginable under the guise of free expression (aka "freeze peach") is not tolerated.

7. Not Be a Propaganda Platform

We disallow users that are clearly here with an agenda. DubVee is not a propaganda platform; it's for sharing cool stuff, staying informed, and having civil discussions. If someone is here to push an agenda, regardless of what that agenda may be, then they're outside what we consider "good faith" interactions.

Share stuff you're passionate about, sure, but don't be here just to spam your agenda.

8. No Porn

We do not federate with any of the primarily NSFW instances nor do we allow pornographic NSFW communities from general-purpose instances. No judgement if that's your thing, but porn is not our primary focus here, and neither I nor the backup admin want to moderate it or scroll through it when browsing /all.

The main frontend for DubVee, Tesseract, supports multiple profiles. As a compromise, I've unlocked the main UI to allow adding accounts from other instances. With this, you can add an account on another instance that does federate with the NSFW instances, or you can create an account on the NSFW instance(s) and switch to that profile. Many other Lemmy clients also offer the ability to add multiple accounts, so there are plenty of options. We feel this is an acceptable compromise.

News flash: Bad things can happen to people you think are bad without frothing at the mouth and turning into a lynch mob.

DubVee will not tolerate any form of extremism, vigilantism, mob justice, celebration of violence, etc. Please be sure to report any instances if you encounter them.

-- Pat

Two of the three new UPSs arrived, and I'm going to go ahead and swap them in versus waiting for the third one. I'll just have to deal with a little extra planned downtime later to install the 3rd one since there's too much potential for unplanned downtime with the two faulty ones in place now.

Shooting for 6 pm or very close to that. I'm going to cable-manage a bit, so planning for between 10 and 30 minutes of downtime.

Got some wicked winds roaring through here. Power has been flashing like crazy, and surprise, both UPS's are crapping out when I need them (even the one I replaced a few months ago 😠). They're both tripping off after 1-2 seconds.

I ordered three new UPS units Monday, but they're not scheduled to arrive until tomorrow evening. I will probably just go ahead and install them as soon as they are delivered without much warning since the current ones are in such poor shape.

I guess Tripp Lite is no longer a decent UPS brand in 2024?

Fiber's shitting the bed, apparently a cut. We're on backup WAN. ETA from ISP is 5am tomorrow

Upgraded the backup connection last week, so will leave pict-rs enabled for now and see how things go.

---

Update: Fiber came back online around 4:30 this morning and the auto fallback seems to have worked. Yay.

This is the second time today I've had to cut over to the backup WAN (which is much slower). Fiber provider is royally screwing the pooch today, and I currently have no ETA. The only silver lining is it's a widespread issue, so hopefully they're inclined to address the problem in a timely manner.

To relieve congestion on the backup connection, I've temporarily disabled external access to pictrs. All requests (GET or POST) to /pictrs will return a 404 response until the main connection is back online.

Sorry for the inconvenience. Please direct all rage at the shit fiber provider I foolishly "upgraded" to.

Status as of 5:17PM:

Service	Status	Notes
Lemmy ActivityPub	Up (Degraded)	Running on backup WAN
Lemmy API	Up (Degraded)	Backup WAN
Pict-rs	Down	Intentionally scuttled to save bandwidth
Tesseract Media Proxy	Down	Intentionally scuttled
Matrix Server	Up (Degraded)	Backup WAN
Matrix Web Client	Up (Degraded)	Backup WAN
DNS	Up (Degraded)	Operationally up, but no backup servers available
SMTP	Up (Degraded)	Backup WAN

---

Update: 4:30 PM: Tried to call and re-activate my account on my old provider since I own the equipment, and it's all still hooked up. I can even still access my old account on the web portal. Should be easy, right? Fucking wrong! They said they couldn't activate it until they sent me equipment, I returned it, and then called back to say I want to use my own. What the actual fuck?? I'm literally waving money at you, you're still charging me a $100 "install" fee (where you have to do nothing), and all you have to do is take it (and my equipment serial number). Fuck my life (and fuck those idiots, too).

Update 5:17 PM: We remain on backup WAN. ETA for fiber restoration is within 24 hours, but I'm not holding my breath. Like, how do you even fuck up this badly? (the whole goddamned state is affected including business customers)

Update 7:39 PM: Primary connection is back online (for now?). Services have successfully failed back over.

Service	Status	Notes
Lemmy ActivityPub	Up	-
Lemmy API	Up	-
Pict-rs	Up	-
Tesseract Media Proxy	Up	-
Matrix Server	Up	-
Matrix Web Client	Up	-
DNS	Up	-
SMTP	Up	-

Appalachian Power is the absolute worst about giving notifications for scheduled outages. The. Fucking. Worst.

I got an email at at 8:05 PM saying there would be a planned, scheduled outage at 8:00 PM and then 45 seconds later was plunged into darkness. Didn't bother to switch on the generator :sigh: and just decided to sit on the patio and get drunk in the waning daylight.

This is not the first time they've done that, and it probably won't be the last. Guess my next purchase is an automatic transfer switch.

Did I say "fuck Appalachian Power"? Because Fuck. Appalachian. Power.

Sorry about that. lol

Let me start off by saying, in my official admin capacity, that I really don't know if I want to be here anymore (see the problem outlined below). That has implications beyond me just deleting my account and going outside since I would not allow the instance to run unattended. I'm also unsure if my backup admin would have any interest in taking over full time.

If it does come to that, there will be a subsequent announcement and either a transition plan or a sunset plan.

That is still up in the air, but for now....

The Problem

I just wanted to take a moment here to address Lemmy's extremism problem / lynch-mob mentality and how DubVee is responding to that.

Last month, a new site rule was added expressly forbidding any form of extremism with regards to violence. Every rule is there for a reason, and this was added as a direct response to the increasingly violent rhetoric I've been seeing from users.

In full:

I don't care which end of the spectrum you're on. Any post/comment calling for or glorifying violence, especially political violence, will be removed. Offenders will be dealt with on a case-by-case basis, but bans will be the norm and swift for violating this rule.

After running reports against the database, there have been a very large number of bans issued in the last few days for both explicit and implicit violations of that rule:

Explicit Extremism

• Explicitly praising or justifying acts of violence
• Explicitly justifying / advocating for violence / "revolution" / whatever
• Suggesting, justifying, or advocating political violence, vigilantism, and/or breaking with the foundational principles of this country in order to "solve" a problem.
• Hypocritical posts/comments that amount to "Fascism / authoritarianism / whatever -ism is bad, but a little bit is okay when it's my side doing it"
• Stirring the pot by jumping into otherwise civil discussions and suggesting acts of violence (I'll admit, those were mostly new accounts and likely provocateur trolls seizing on current events, but established accounts are also guilty)

Implicit Extremism

• Patterns of upvoting or boosting posts/comments that explicitly praise, endorse, or advocate for violence (or any of the 'explicit' items above)
• Patterns of downvoting posts/comments that denounce violence or those that advocate for lawful/peaceful solutions.

Note: Before you cry "thought crime!" about issuing bans based on vote patterns, it is a well-known tactic for trolls to operate bot or sockpuppet accounts to manipulate votes to serve an agenda. Owing to the federated nature of Lemmy, this becomes easier to do and harder to detect since those can be distributed among different instances that have varying moderation, signup, monitoring, and administrative policies and procedures. So, no, I have no compunction about banning accounts that express support for extremism by way of voting; there's simply no straightforward way to tell bot from person from my point of view.

Even accounting for gallows humor, I must reiterate that extremism and violence are absolutely forbidden, and there will be zero tolerance shown.

While I'm aware there's a certain "mob mentality" inherent to social media in general, "lynch-mob mentality" will not be tolerated; we're better than that, and DubVee is not here to spread hate, violence, or fan the flames.

In Conclusion

If it turns out that we end up blocking 90% of the Fediverse because of this, then I'll just shut this whole instance down. DubVee, by way of federation, will not devolve to 4chan (or 8chan or wherever the shitty little edgelords post their green text nowadays), and I will (figuratively) burn it to the ground before it gets to that point.

What Can You Do as a User?

First and foremost, make sure you report any content that violates this rule and refrain from violating it yourself.

If you're seeing less comments than you were a few days ago or are no longer seeing content from certain users, this is most definitely why. You can check the modlog to see if someone you're expecting to see has been banned.

If you (local user) wish to challenge a ban and advocate for someone who has been blocked, you may send me a DM with your request, and I will review the account further (accounts were reviewed prior to banning, but it never hurts to take a second look).

If you have a problem with this rule and our enforcement of it, there's plenty of other instances you can choose from.

This Weekend

There will be some periodic downtime this weekend as I do some physical cleanup / rearrangement to my equipment. I'll try to keep these to a minimum, but at some point, everything is going to need to go offline for a bit in order to re-cable the UPS and power connections.

Plan is Saturday afternoon between 1 and 4 pm. Hopefully any interruptions will be brief.

Next Week

I'm switching to a new fiber provider next week (FTTH vs current FTTN), so there will be a brief disruption as I cut over to that. That downtime should only be a matter of seconds since both connections will be active, but just in case it turns into a whole thing, I'm mentioning it now. The new fiber is supposed to be installed Tuesday afternoon, and I plan to do the cutover around 7PM local time.

Next Few Months

I'll be expanding the storage server in the coming weeks, but that should not have any immediate impact on DubVee's normal operation.

With these, I hope to provide a better, faster experience. I'm also looking into setting up an Invidious instance to use with DubVee as well as exploring setting up and integrating a Peertube server.

I've been working on the latest release of Tesseract for the last few weeks, and I'm happy to announce the first beta release of the upcoming 1.4.0 series.

I daily drive my dev version throughout the development process, so all the bugs I've encountered/introduced should be fixed. The beta test on the main instance is more of a formality and final shakedown. That said, please report any bugs you encounter either through Github or by describing them in a comment here.

1.4.0 is a significant update as it drops all backwards compatibility with the 0.18.x API and introduces several new features. It's also had some additional polish since 1.3.0 and added things that have been requested for quite some time (pasting images into post/comments, etc).

As always: thanks in advance for being my beta testers. Appreciate it!

Change log for 1.4.0 if you want to see what's new: https://github.com/asimons04/tesseract/blob/1.4.0/ChangeLog.md

If you've noticed content from Lemmy World lagging by a few hours the past few days, it's not just you.

Long story short, it's a problem with how Lemmy sends activities, and it's heavily impacted by latency between sending and receiving server and creates a fixed upper limit on the number of activities per second that can be sent. Lemmy World is hosted in Finland, and DubVee on the US east coast. There's only so much I can do to work around the inherent latency of a trans-Atlantic link.

We're not alone in this. Some instances, particularly in Australia, New Zealand, and US west coast, are impacted more dramatically.

While there's always some federation delay/backlog, it only escalates to noticeable levels every so often. Not sure why, but I've gone over our infrastructure top to bottom several times and cannot find any reason, on this end, for these events. The graph in the post shows the number of activities Dubvee is lagging behind Lemmy World over the last 30 days.

Normally, it's 300-500 activities which usually corresponds to a minute or less of lag between LW sending an event and DubVee processing it. Occasionally, when LW gets busy, we see spikes into the 3000-5000 range (~5-7 minutes of lag). Every so often, though, there will be huge backlog events (the spikes from 15K to 35K) which often take 8-12 hours to catch up. A month or two ago, I think the largest spike was around 180,000 (though that was a separate issue).

I believe this is being addressed in Lemmy itself, but it'll be a while before it's ready (and I'm certainly going to let other instances kick the tires before upgrading).

In the mean time, I've talked with some other admins and have deployed a Federation proxy. I won't go into the nitty-gritty details, but it does seem to be alleviating a lot, but not all, of the congestion. We went from averaging around 10,000 backlogged events to about 2500. So, definitely an improvement.

Update: Buffer has cleared, and things seem to be coming in pretty close to real time. Occasionally the buffer kicks in, but overall, it appears to be helping. Will continue to monitor. Usually 11-12 AM eastern time is when we start to see lag increasing in activities coming from Lemmy World.

Hopefully there's some more optimization I can do in the mean time, and hopefully Lemmy addresses this limitation, but for now, this should make things less bad.

Finally got around to upgrading to 0.19.3.

Upgrade did not go smoothly due to multiple DB migration step failures and less than helpful logging from Lemmy during the DB migrations. (surprised Pikachu).

After digging through to figure out what it was complaining about, the upgrade was finally able to be completed.

You will likely need to log in again (at least I did).

I'm in the process of migrating the pict-rs database to Postgres. Pict-rs has to be offline during this process, so images here will appear to be broken for a bit. You also won't be able to upload any images for the duration of this migration.

Estimated time is about 20-30 minutes.

Currently, we host 4 UIs:

• Tesseract (default UI at https://dubvee.org/)
• Mlmym (old Reddit looking UI at https://old.dubvee.org/)
• Alexandrite (desktop UI at https://alex.dubvee.org/)
• Lemmy-UI (crappy "default" UI at https://lemmy.dubvee.org/).

Due to all the bot crawler traffic that slips through, I'm having a hard time distinguishing actual usage from bot traffic. That said, it does seem like Mlmym and Alexandrite are rarely used by actual people (there's a lot of bot traffic to filter out that spoofs its user agent, so I could easily be wrong here).

Update: Mlmym does all the Lemmy API calls server-side, so all of the actual usage traffic was actually originating from my server (which I was filtering out).

I'm looking to decommission ~~Mlmym (the old Reddit style) and~~ Alexandrite.

I just updated both of those to the latest, so apologies for the brief disruption if you were using them, but ultimately I'd like to trim down the selection to just the ones being actively used.

So, poll time: Are you using old.dubvee.org or alex.dubvee.org? If so, please make your voice heard and I'll keep them around. If not, or if I don't hear back, I'll likely decommission them this weekend.

Update: Will keep Mlmym (old Reddit-style). Alexandrite is still up for decommission unless someone is using it as well.

I'm not quite done with the 1.3.0 release of Tesseract, but the last few bits are going to take longer than I expected. So I think I'm going to hold off and add those in a 1.3.x release a bit later.

I've gutted, re-implemented, and just flat-out re-wrote large portions of the application over the last month or two. I've also been daily driving the dev version, and I guess a couple other people were too. That said, it is stable enough for daily use and ready to get some more eyes on it before an official release.

So, please bear with me if you hit any bugs (I've fixed all the ones I've found so far). If you would be willing and kind enough, either submit a Github issue if you find a bug or just throw a reply to this post with as many details as you can provide.

Hope to run a few betas here before officially releasing 1.3.0.

Apologies for the downtime.

We had some severe weather come through on Tuesday that caused extensive damage to power infrastructure throughout the area. I thought my location had been spared the worst of it, but a tree decided to fall on the transmission line servicing this area Wednesday morning (according to the power company, anyway). By that point, there were about 55,000 other outages ahead of my area in line awaiting repairs (no hard feelings there).

Since power outages are rare here, at least ones lasting longer than an hour or two, I only keep about 2 gallons of fuel on hand for the generator. At typical 3/4 load, that usually last about 4-5 hours. Again, long outages are extremely rare here, so that's usually more than enough runtime. Sadly, this outage lasted much longer (even though I was able to stretch the generator runtime by slightly hacking my UPS)

Federated content should now be coming back in, but it'll take a while to catch up.

Lessons learned:

• Mother nature is a badass bitch. I'm going to start keeping at least 2 days of fuel on hand. Worst case, I'll throw it in the car before it goes stale.
• My 48v e-bike battery and stepdown converter can run my UPS and bare-minimum servers for about 6 1/2 hours (probably more; it was only about half charged and had sat untouched on a shelf for over a year). That came in handy and was why we were able to stay online as long as we did.
• My primary network provider is pretty solid. That stayed up longer than I thought it would.
• Failover to the backup WAN works but failing back to primary does not. That's a "me" problem to fix in my watchdog script.

"Why don't you just throw it in the cloud like a sane person?" you may be asking. Well, it is and it isn't already. It's a hybrid setup. The UI and front end caches/proxies are all cloud based but the DB and API are located on prem where I can throw as much resources at them as I want for free.

I've been tempted to move those up, but it would cost more money than I want to devote to Lemmy at the moment (at least if I want to maintain the same level of performance).

Most of my VPS's are at capacity, but I am going to work on setting up a standby VPS that can scale up and keep the most recent backup dump there.

Mostly, I just don't want to have to rely on donations to keep DubVee online. Right now, all of its components are secondary payloads on my existing VPS hosts or are running on-prem on my own hardware (for free, for all intents and purposes).

I'd rather deal with an outage from time to time than have to constantly wonder if I'm going to be able to pay the cloud hosting bills. It's one of the reasons I've envisioned DubVee remaining relatively small.

Weather / power related. May go offline for a bit until power is restored. Currently on UPS.

Update 1: Yep, expect downtime. UPS ran dry and moved over to aux generator. ETA from power company is tomorrow at 11 PM EST.

Update 2: Still no change in ETA. Have had to shed some load from the generator because my UPS's refuse to charge from it and every time the fridge kicks on, a random piece of my equipment would reboot :sigh:.

Hopefully my reduced power budget will stabilize that.

On the bright side, the primary network connection has remained up and haven't had to switch over to the (much slower) backup.

Update #3: (17:06) Generator running on fumes now and will probably be going offline soon to ensure a safe shutdown and save some fuel to cool the fridge tomorrow if power is still out. Apologizes in advance. I usually have very reliable power here and typically never need the generator for more than a few hours.

Update #4 (04/04/2024 14:48): Power restored 12 hours ahead of ETA (good work AEP crew!). We are now back up and running, but there is about an 18 hour backlog of federation activity that needs to be received. I've confirmed that those are starting to resume, but it will take a while. https://dubvee.org/post/dubvee.org/977774

Lemmy World

There was an issue today with content from Lemmy World not federating to us. After pulling my hair out and testing the DubVee stack top to bottom, I got in touch with the awesome admins over at LW to try to figure out the problem. After they gave their federation service workers a good, stern talking to, we're now receiving content again.

Unfortunately, due to the amount of time (about 4 hours) the federation messages were getting stuck, there is a huge backlog that needs to catch up: about 90,000 messages. As of this writing, there are still 55,024 queued ActivityPub messages in flight (that includes backlog and current messages).

So just a heads up if you're not seeing posts/votes/comments from there or if they don't show up for a while. They will eventually arrive, though they'll be in the feed based on when they were published rather than when they arrive. On the bright side, they are receiving posts/votes/comments from us; you just might not get a reply right away.

Hopefully the queue works through overnight and things are back to normal tomorrow 🤞

Update: Backlog queue down to 37,150 now.

Update #2: 6,531 and falling. Looking like we'll be back in sync shortly.

Update #3: Backlog has finished syncing as of 23:55 EST.

mander.xyz

This one was my fault. lol. About 4 months ago, I was shoring up the firewall against some malicious traffic patterns, and mander.xyz 's server IP got caught in the CIDR block crossfire. Since I see content from their users frequently, I wasn't aware there was a problem. Turns out those were just relayed to us by the home instances of the communities. After I made a firewall allowance for mander's server, I had to reach out to their admin to reset the "last seen" date for DubVee in order to "bring us back to life" as their server had marked ours as dead.

So, all of the communities on mander.xyz should now look alive again. 🎉

My sincerest thanks to @rooki@lemmy.world, @ruud@lemmy.world, and @sal@mander.xyz for their assistance today.

If you've noticed a large number of bulk post removals performed by me in the modlog recently and are wondering if I've gone full authoritarian: no, I haven't gone mad with power. Those were posts that were already removed by other community mods or posts that were self-deleted by their creators. Some were legitimate spam or otherwise violated server rules, but the bulk of them were just cleanup.

"If they're already removed/self-deleted, why remove them again?", you might ask.

Well, Lemmy treats Pictrs (the media subsystem, basically) like a black hole - images go in but it never removes them. When posts are modded or self-deleted, any media attached to them lives forever in pictrs with no clean way to remove them later*. That wastes a huge amount of disk space on my hosting stack for media that will never see the light of day again.

I'm not okay with that for so many reasons. Yeah, object storage is cheap, but why be wasteful?

The Lemmy + Pictrs integration...well, let's just say it leaves much to be desired. "Suggest a feature enhancement" or "ping the Lemmy devs about it" you may be thinking. Haha, right.

I'm building a new API to interact with Lemmy, and the admin/mod components were the first parts that I developed. I've deployed parts of the prototype API to periodically clean up removed/deleted posts along with the media that was attached to them. I could do this silently on the backend and you'd never know, but in the interest of transparency (and also testing that the API works as expected), I've let it log its activities in the modlog as it would when it moves to production.

So, in closing, no, I have not gone mad with power. I'm just trying to keep my disk usage sane and not clutter up storage with abandoned media.

*They can be removed later, but it's a clunky external process that doesn't offer any guarantees.

Apologies for the brief ~20 minute outage. Had a loss of power and one of the two UPS's failed without warning. Had to manually move equipment over to the remaining UPS until power is restored. The cold boot takes frigging forever and was responsible for the bulk of those 20 minutes, but everything should now be back up and running.

We also lost the primary internet connection and are running on the failover, so things may be a bit sluggish.

Utility company said a tree was at fault, so will likely have both electric and network service restored in an hour or two. Unless UPS-A decides to die on me too.