Thanks, I hadn't seen this one. I've updated our nginx config and AZ now passes the test on that page. No idea if it will help with GIFs etc.
With the resources available its not feasible for AZ to develop/deploy custom solutions that can be resolved by remote instances with trivial configuration changes.
I'm not going to address specific parts of your post, suffice to say I disagree on almost everything you said.
As I said previously, if you have a workable solution please do devlop it and submit a PR to the lemmy devs. I'd be happy to try your suggestion should they roll it in.
You're contradicting yourself there. By definition adding an external service is a customization to lemmy. I'm not interested in running un-vetted software from a third party.
This has been discussed previously with a request from a reputable source to batching content from LW. That setup required an additional server for AZ, close to LW. And for LW to send their outgoing federation traffic for AZ to it, which then batched and send to the real AZ server. This offer was declined, though appreciated.
I've been transparent and open about the situation. You seem to think this is the fault of AZ, and we're willfully not taking an action that we should be taking. This is not the case.
As it stands the issue is inherent with single threaded lemmy federation, which is why the devs added the option for multiple concurrent threads. Until LW enable this feature, we'll see delayed content from them when their activity volume is greater than what can be federated with a single thread. To imply this is the fault of the receiving instances is disingenuous at best, and deliberately misleading at worst.
Note I said lemmy AND the activitypub protocol, ie lemmy does not currently have this capability. If it were added to mainline lemmy I'd be open to configuring it, but its not so I can't.
The root cause of the issue is well understood, the solution is available in lemmy already: multiple concurrent outgoing federation connections to remote instances. AZ has had this configured since it was available. LW have not yet enabled this, though they're now running a version that has it available.
Appreciate the offer, but I'm not interested in customising the AZ server configuration more than it already is. If you write it up and submit a PR that the main lemmy devs incorporate, I'd be happy to look at it.
That isn't how lemmy and the activitypub protocol work. Source instance pushes metadata about new content, remote instance then needs to pull it. If we've not received the push yet, we can't pull the additional info.
And for anyone curious... blue line is traffic from a country we don't normally see much traffic from. The unusual spike, then drop when I blocked the specific sources:
The traffic stopped a few hours back, from all IPs at once. Definitely seems to have been some sort of deliberate action.
The unusual traffic all appeared to be coming from one location on the internet, with the same user agent string. Any traffic from that network will now receive a captcha from Cloudflare. I'm not aware of any lemmy instances hosted there, but will keep an eye on things.
For some context, CPU usage jumped when the traffic started... and dropped after the block was applied:
It sure doesn't feel like autumn so far
Not sure how I should feel that the bubble I live in seems far more accepting and empathy towards asylum seekers.
I don't think the people I live/work/socialise with are particularly extreme... but according to this we're collectively not representative of wider Australian views.
I choose to believe their sampling was flawed and somehow only sampled the worst of us.
My only call out this week is an uptick in storage consumption, seems to align with an increase in new user signups and general higher activity. I'm guessing this is due to the release of Sync for Lemmy.
All resource usage looking stable. Storage is the only one that is trending up, as expected.
Another week... another bunch of nerd graphs!
Not much to say here, pretty stable CPU usage wise.
The unusual memory growth appears to have been related to a minor Postgres configuration change I made last week, which was reverted on Thursday. Memory usage looking much more normal since.
As with CPU usage, network traffic is looking stable.
Storage growth has normalized, now that we've hit an equilibrium point. Though I'll be tweaking the object storage cache retention to minimise object storage pulls.
Still saving us a large volume of egress traffic. Will save even more if particular content goes viral.
Resource utilisation on the server is looking great across the board. No skyrocketing usage as we saw initially. Storage is still looking like the first trigger for another server upgrade, but as it is now a gradual increase we'll have plenty of fore warning and its looking like this will be some time away.
Questions? ๐ค
Doh! Forgot earlier in the night, so here you are... technically Saturday.
The lemmy devs have made some major strides in improving performance recently, as you can see by the overall reduced CPU load.
I need to figure out why swap is continuing to be used, when there is cache/buffer available to be used. But as you can see, the upgrade to 8GB of RAM is being put to good use.
The two large spikes here are from some backups being uploaded to object storage. Apart from that, traffic levels are fine.
A HUGE win here this week, turns out a huge portion of the database is data we don't need, and can be safely deleted pretty much any time. The large drop in storage on the 9th was from me manually deleting all but the most recent ~100k rows in the guilty table. Devs are aware of this issue, and are actively working on making DB storage more efficient. While a better fix is being worked on, I have a cronjob running every hour to delete all but the most recent 200k rows.
Cloudflare still saving us substantial egress traffic from the VPS, though no 14MB "icons" being grabbed thousands of times this week ๐
All things considered, we're in a much better place today than a week ago. Storage is much less of a concern, and all other server resources are doing well... though I need to investigate swap usage.
Longer term it still looks as though storage will become the trigger for further upgrades. However storage growth will be much more slow and under our control. The recent upward trend is predominantly from locally cached images from object storage, which can be deleted at any time as required.
As usual, feel free to ask questions.
Available here
Noting this here for my own reference, and any other lemmy server admins that don't happen to be database administrators by day ๐
I am not a DBA, if I'm doing something bad/incorrect here... please post! Yes, I've reset the password and TOTP token on the example account below.
2FA flags are stored in the local_user table, however that does not show usernames. To find the person_id for the user account you want to disable 2FA for, you'll need to check the person table. I'll use my test account here as an example:
SELECT * from person where name = 'guineapig' and local = 't';
Giving:
Note the number 781227, this is the person_id for this account on my instance. To confirm:
SELECT * from local_user where person_id = '781227';
Yep, the 2FA string has the expected username in it. Now to disable 2FA on the account we need to NULL out both totp_2fa_url and totp_2fa_secret rows:
UPDATE local_user
SET totp_2fa_url = NULL
WHERE person_id = 781227;
UPDATE local_user
SET totp_2fa_secret = NULL
WHERE person_id = 781227;
Should give output like this:
And checking the local_user table again, both TOTP fields should be empty:
The 2FA enrollment process in Lemmy isn't great at best, unintuitive and confusing at worst. Here I'll show the process to enroll from a desktop PC. The process on mobile will be different. Throwing this together quickly, hopefully no errors/omissions...
As a precaution open a second browser, log in there before starting. Just in case you encounter issues and need to disable 2FA.
I'd only suggest progressing with this if you're comfortable to do so, lemmy does not currently provide backup 2FA codes.
Log into your account, go to your account settings:
Scroll to the very bottom of your settings, locate the "Set-up 2-factor authentication tickbox":
Tick the "Set-up 2-factor authentication tickbox", note the popup at the bottom:
THIS IS THE POINT OF NO RETURN. Once you Click the "Save" button, 2FA is enabled and you'll be unable to login without it functioning. If you encounter issues, tick the "Remove 2-factor authentication" and click save again.
When you click Save you should be scrolled to the top of the page. Scroll back to the bottom, you should see the button is still ticked, but nothing else has changed:
Click refresh in your browser, or hit F5. The page will reload. Scroll to the bottom of the page again. You should now see a "2FA installation link" button/link.
Right click the "2FA installation link" button/link:
otpauth://totp/Aussie%20Zone:guineapig?secret=GFQWIYTCHEYTIYJWHA4WMZTEMQ2GIZBRGU4WCZLGGRTDQMZZGM2GKN3DMVQTONBS&algorithm=SHA256&issuer=Aussie+ZoneWhat you've pasted is the TOTP Key URI that can be used by many 2FA applications. Unfortunately this is unwieldy to copy around, so we're going to generated a QR code that you can scan on your phone.
Any questions please ask.
You're wrong, I'll leave it at that. Won't be replying any further.