All things programming and coding related. Subcommunity of Technology.
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
I've recently been wondering if Lemmy should switch out NGINX for Caddy, while I hadn't had experience with Caddy it looks like a great & fast alternative, What do you all think?
EDIT: I meant beehaw not Lemmy as a whole
Why? What's wrong with nginx?
While I can't speak for others, I've found NGINX to have weird issues where sometimes it just dies. And I have to manually restart the systemd service.
The configuration files are verbose, and maybe caddy would have better performance? I hadn't investigated it much
EDIT:
Nginx lacks http3 support out of the box
I'm running a lot of services off my nginx reverse proxy. This is my general setup for each subdomain - each in its own config file. I wouldn't consider this verbose in any way - and it's never crashed on me
service.conf
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name [something].0x-ia.moe;
include /etc/nginx/acl_local.conf;
include /etc/nginx/default_settings.conf;
include /etc/nginx/ssl_0x-ia.conf;
location / {
proxy_pass http://[host]:[port]/;
}
}
nginx was built for performace, so I doubt caddy would have any significant different in regards to that. I've not found config verbosity to be a problem for me, but I guess to each their own. I'm aware I may come across as some gatekeeper - I assure you that is not my intention. It just feels like replacing a perfectly working, battle testing service with another one just because it's newer is a bit of a waste of resources. Besides - you can do it yourself on your instance. It's just a load balancer in front of a docker image.
Isn't caddy battle tested too? And looking into alternatives is not really a waste of resources. It just feels like nginx is not as reliable and likes to drop requests. It's not just a load balancer, mind you.
I am surprised you're getting dropped requests. What do the logs say?
The problems I see with Lemmy performance all point to SQL being poorly optimized. In particular, federation is doing database inserts of new content from other servers - and many servers can be incoming at the same time with their new postings, comments, votes. Priority is not given to interactive webapp/API users.
Using a SQL database for a backend of a website with unique data all over the place is very tricky. You have to really program the app to avoid touching the database and create cached output and incoming queues and such when you can. Reddit (at lest 9 years ago when they open sourced it) is also based on PostgreSQL - and you will see they do not do live SQL inserts into comments like Lemmy does - they queue them using something other than the main database then insert them in batch.
email MTA apps I've seen do the same thing, they queue files to disk before putting into the main database.
I don't think nginx is the problem, the bottleneck is the backend of the backend, PostgreSQL doing all that I/O and record locking.
nginx 100% isn't the problem, and you're right on all counts. I'll also add that I've seen reports that Lemmy has some pretty poorly optimized SQL queries.
They need to add support for a message broker system like RabbitMQ. That way their poor postgres instance stops being the bottleneck.
You can use any reverse proxy you'd like, doesn't have anything to do with lemmy
sorry, I meant beehaw not lemmy
One more thing I forgot to mention. The nginx 500 errors people are getting on multiple Lemmy sites could improve shortly with the release of 0.18 that stops using websockets. Right now Lemmy webapp is passing those through nginx for every web browser client.
ohgod
From what I've read, the 500 errors are caused by nginx's failure mode of
"Fuck it, I'm dropping this connection"
Caddy seems to want to keep connections going even if it has to slow down.
nginx is like, the gold standard. it’s performant as heck. the issues are likely a culmination of many small sub-optimal pieces.
That's why I think Caddy should be considered, as it has less moving parts, therefore less suboptimal pieces.
If it’s not broken why change it? Are there performance benefits to switching?
I think there are, but there would need to be testing done, on the surface it seems to be a much simpler proxy than nginx. And doesn't use the same architecture as Nginx
Caddy is not going to fix anything, on the contrary, it consumes more ram. Generally the instances have been slowing down when swap gets hit by the db, so lowering ram usage and optimizing that should be the first priority.
Switching to Caddy won't change/fix anything.
Is lemmy copled to a specific web server? Can't you use whatever you want?
Nginx has nothing to do with the performance issues of Lemmy. :)
It does actually, NGINX likes to drop connections when it gets overwhelmed, Caddy prefers to slow down the connection and respond when it can.
This might be true but appservers and DBs usually give up way before nginx.
NGINX has given way on other instances too, however, when the Reddit invasion happened. I kept getting 500 errors on most instances.
Here is a caddy vs nginx benchmark test. A lot to read, but gives an idea where the strengths of both are and where not.
https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/
I used nginx for years. But I'm using Caddy since like 2-3 years now. But I didn't change because of speed.
Huh, that's interesting, thank you for linking it!
People comment a lot on performance, but I think Caddy can (and should) hold up perfectly fine. It might be worth it to experiment with running servers half on Caddy and half on NGINX, then see how the traffic is being handled by both to compare.
I do think the much cleaner config makes up for the maybe slight performance loss, though. It's just so much less work to set up and maintain compared to NGINX. The last time I've used NGINX was years ago, when I decided to drop it entirely in favor of Caddy. I do think NGINX is only "standard" because it came before Caddy, and that most applications should not prefer it over Caddy.
I, too, dislike NGINX configs, but mainly I think Caddy should be considered for the feature set and performance it has over nginx. While it is true that nginx is pretty performant, that is without talking about third party modules written in Lua. Cloudflare had an amazing post about it a while back where they said while nginx on its own is ok, when you add third party scripts into the mix it slows down to a craw.
I had no idea that NGINX has Lua plugins. You'd probably want to check if Caddy has equivalents for those plugins though, or just implement them in Go yourself.
Caddy uses go based plugins, I remember, they're called modules.
Source: https://caddyserver.com/docs/modules/ & https://caddyserver.com/docs/extending-caddy
Why is nginx preferred over Apache these days? I believe nginx was originally preferred because Apache had scaling issues with its original forking concurrency model, but that was replaced a long time ago, so…why use nginx today?
That's why I'm entertaining the idea of an alternative in this post. Although it seems there are a lot of mixed opinions on this matter
It's just matter of preference, really. You can use any reverse proxy you want in your docker compose file. With Caddy, setting up letsencrypt is a lot easier than other webservers which might help setting up your own instance a bit easier.