this post was submitted on 24 Aug 2023
1088 points (97.4% liked)

Lemmy.World Announcements

30542 readers
1 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to [email protected] e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
1088
Recent Outages (lemmy.world)
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
 

Hey everyone, so as I'm sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS'er, and is causing these disconnects to our backend server, problematic for sure.

We've opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

  • LW Infra Team
top 50 comments
sorted by: hot top controversial new old
[–] [email protected] 267 points 2 years ago (1 children)

As always, the transparency is appreciated. Some growing pains are certainly to be expected

[–] [email protected] 236 points 2 years ago (4 children)

Whenever I get frustrated by the outages I remind myself: still better than reddit.

[–] [email protected] 88 points 2 years ago (2 children)

Thanks to lemmy I don't doom scroll anymore because either there is an outage or I read all the new content I'm subscribed to in 10-20 minutes

[–] [email protected] 30 points 2 years ago (1 children)

That's mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.

[–] [email protected] 12 points 2 years ago (1 children)

Look at this guy with his reasonable social media use. YOU THINK YOU'RE BETTER THAN ME??

[–] [email protected] 7 points 2 years ago

Scrolling a few miles or so a day isn’t too bad, is it?

[–] [email protected] 8 points 2 years ago

Huh, I could keep up with my daily feeds before, now not so much.

[–] [email protected] 9 points 2 years ago

I fallback to one of the many instances that federate with lw. Don’t get frustrated at all bro

[–] [email protected] 80 points 2 years ago

Maybe its just the times I'm accessing but its seem better this week to me compared to the last few ones.

[–] [email protected] 80 points 2 years ago (1 children)

I don't blame you for this, but the uptime records are incomplete at best. I've experienced the site being down (and confirmed with Down for Everyone or Just Me), yet status.lemmy.world showed all systems operational. As I'm writing this, status.lemmy.world is missing most data up to yesterday and dash.lemmy.world shows 16 days uptime.

I have lots of respect to you for even having these. I also remember status.lemmy.world work mostly fine some time ago. But as of right now, both uptime monitors fail to serve their purpose.

[–] [email protected] 80 points 2 years ago* (last edited 2 years ago) (1 children)

You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We'll add this info to make things clearer <3

EDIT: Added more info to our status page, thanks for the feedback Machefi!

EDIT2: Also the missing data is due to us removing and adding more specific monitors for the different infra services.

[–] [email protected] 23 points 2 years ago

Excuse me stop being so cool, you're raising the bar too high for everyone else thank you

[–] [email protected] 43 points 2 years ago

Good luck, no pressure.

[–] [email protected] 41 points 2 years ago* (last edited 2 years ago)

On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.

If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.

Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.

Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.

[–] [email protected] 37 points 2 years ago

Thanks for keeping us up to date, I don't mind downtime personally as long as I'm aware of what's going on

[–] [email protected] 36 points 2 years ago (3 children)

Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I've experienced no downtime so far

[–] [email protected] 17 points 2 years ago

The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day πŸ˜ƒ

load more comments (2 replies)
[–] [email protected] 30 points 2 years ago (1 children)

So what you're saying is we were the ddosers all along 😹

[–] [email protected] 35 points 2 years ago (1 children)

The hack is coming from inside the instance!

load more comments (1 replies)
[–] [email protected] 23 points 2 years ago

Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!

[–] [email protected] 20 points 2 years ago* (last edited 2 years ago)

Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you're getting a hell of a lot more traffic than I am but they haven't blocked Cloudflare on mine yet so that's a plus. :)

[–] [email protected] 18 points 2 years ago (3 children)

Hetzner is a pretty good provider. It would be nice to stick with them if possible.

load more comments (3 replies)
[–] [email protected] 15 points 2 years ago

I brows all, never noticed as outage lol.

[–] [email protected] 12 points 2 years ago

You guys rule. Thanks for all the hard work.

[–] [email protected] 10 points 2 years ago (2 children)

Is there anything we can do to help? Donations? Tech volunteers? Visit hosting company with a baseball bat?

[–] [email protected] 6 points 2 years ago

I imagine spreading out across instances, and having accounts in instances beyond the biggest one helps, as it reduces traffic and strain on lemmy.world and its servers.

lemmy.world still deserves support, like with donations, though helping reduce the strain is something anyone can do, albeit small, though free of charge.

[–] [email protected] 8 points 2 years ago

The transparency is appreciated.

[–] [email protected] 8 points 2 years ago

Thanks for the transparency. I think you lost some trust with the Discord debacle, but this makes me feel a lot better about things.

[–] [email protected] 5 points 2 years ago* (last edited 2 years ago) (1 children)

Thanks for keeping things running as Lemmy grows!

For what it's worth, I'm on all the time and have barely noticed the outages.

load more comments (1 replies)
[–] [email protected] 5 points 2 years ago (2 children)

So the recent reports about ddos attacks were false alarms?

load more comments (2 replies)
[–] [email protected] 4 points 2 years ago

Thanks for the update!

[–] [email protected] 4 points 2 years ago

Thanks for the updates and transparency.

[–] [email protected] 4 points 2 years ago (2 children)

The dashboard requires a login to view. At least from here it does. Is there a way to view without making an account?

load more comments (2 replies)
[–] [email protected] 4 points 2 years ago

Thanks for sharing the dash link, and for your work!

load more comments
view more: next β€Ί