bahmanm

joined 2 years ago
MODERATOR OF
 

cross-posted from: https://lemmy.ml/post/5719058

Follow up on a previous post: [DISCUSS] Recent momentary outages


I've been working on a simple opt-in solution, primarily for Lemmy end users like me (but also helpful for admins), to easily check the status/health of their favourite instance.

๐ŸŒŽ lemmy-meter.info

You can find the details of the implementation in lemmy-meter github repo.


โ“ @admins: would you be interested in adding your instance to lemmy-meter?

You don't need to do anything except confirming - I'll handle the rest. It should only take a few minutes for your instance to show up in lemmy-meter.

Out of the box it will send only 4 HTTP GET requests per minute to your instance. However that is totally configurable if it sounds too much or too little.


PS: I wasn't sure how to reach out to the admins short of messaging them individually.

 

Follow up on a previous post: [DISCUSS] Recent momentary outages


I've been working on a simple opt-in solution, primarily for Lemmy end users like me (but also helpful for admins), to easily check the status/health of their favourite instance.

๐ŸŒŽ lemmy-meter.info

You can find the details of the implementation in lemmy-meter github repo.


โ“ @admins: would you be interested in adding your instance to lemmy-meter?

You don't need to do anything except confirming - I'll handle the rest. It should only take a few minutes for your instance to show up in lemmy-meter.

Out of the box it will send only 4 HTTP GET requests per minute to your instance. However that is totally configurable if it sounds too much or too little.


PS: I wasn't sure how to reach out to the admins short of messaging them individually.

 

cross-posted from: https://lemmy.ml/post/5653264

I'm using Grafana for one of my hobby projects which is also deployed to a public-facing server.

I am the only user of Grafana as it is supposed to be read-only for anonymous access.

My current workflow is:

  1. Run Grafana locally.
  2. Make changes to local dashboards, data-sources, ...
  3. Stop local Grafana.
  4. Stop remote Grafana.
  5. Copy local grafana.db to the remote machine.
  6. Start remote Grafana.
  7. Goto (1)

However this feels terribly inefficient and stupid to my mind ๐Ÿ˜…


To automate parts of this process, I tried gdg and grafana-backup-tool.

I couldn't get the former to work w/ my workflow (local storage) as it barfed at the very start w/ the infamous "invalid cross-device link" Go error.

The latter seems to work but only partially; for example organisations are not exported.


โ“ Given I may switch to PostgreSQL as Grafana's DB in the near future, my question is, what is the best way to automate my process short of stopping Grafana and copying database files.

[โ€“] [email protected] 2 points 2 years ago

Agreed. It was a mix of too ambitious standards for up-to-date data and poor configuration on my side.

[โ€“] [email protected] 0 points 2 years ago* (last edited 2 years ago)

sane defaults and a timeout period

I agree. This makes more sense.

Your name will be associated with abuse forevermore.

I was going to ignore your reply as a ๐ŸงŒ given it's an opt-in service for HTTP monitoring. But then you had a good point on the next line!

Let's use such important labels where they actually make sense ๐Ÿ™‚

[โ€“] [email protected] 3 points 2 years ago (6 children)

beyond acceptable use

Since literally every aspect of lemmy-meter is configurable per instance, I'm not worried about that ๐Ÿ˜Ž The admins can tell me what's the frequency/number they're comfortable w/ and I can reconfigure the solution.

You can hit the endpoint /api/v3/site for information about an instance including the admins list.

Exactly what I was looking for. Thanks very much ๐Ÿ™

[โ€“] [email protected] 4 points 2 years ago

Thanks for the link. Had no idea about that.

 

I'm working on lemmy-meter which is a simple observability solution for Lemmy end-users like me, to be able to check the health of a few endpoints of their favourite instance in a visually pleasing way.

๐Ÿ‘‰ You can check out a screenshot of the pre-release landing page.


๐Ÿ’ก Currently, lemmy-meter sends 33 HTTP GET requests per minute to a given instance.

For a few reasons, I don't wish lemmy-meter to cause any unwanted extra load on Lemmy instances.
As such I'd like it be an opt-in solution, ie a given instance's admin(s) should decide whether they want their instance to be included in lemmy-meter's reports.

โ“ Now, assuming I've got a list of instances to begin w/, what's the best way to reach out to the admins wrt lemmy-meter?


PS: The idea occurred to me after a discussion RE momentary outages.

[โ€“] [email protected] 1 points 2 years ago (2 children)

I got to admit that your point about the presentation skills of the author are all correct! Perhaps the reason that I was able to relate to the material and ignore those flaws was that it's a topic that I've been actively struggling w/ in the past few years ๐Ÿ˜…

That said, I'm still happy that this wasn't a YouTube video or we'd be having this conversation in the comments section (if ever!) ๐Ÿ˜‚


To your point and @[email protected]'s RE embedded systems:

That's absolutely true that such a mindset is probably not going to work in an embedded environment. The author, w/o explicitly mentioning it anywhere, is explicitly talking about distributed systems where you've got plenty of resources, stable network connectivity and a log/trace ingestion solution (like Sumo or Datadog) alongside your setup.

That's indeed an expensive setup, esp for embedded software.


The narrow scope and the stylistic problem aside, I believe the author's view is correct, if a bit radical.
One of major pain points of troubleshooting distributed systems is sifting through the logs produced by different services and teams w/ different takes of what are the important bits of information in a log message.

It get extremely hairy when you've got a non-linear lifeline for a request (ie branches of execution.) And even worse when you need to keep your logs free of any type of information which could potentially identify a customer.

The article and the conversation here got me thinking that may be a combo of tracing and structured logging can help simplify investigations.

[โ€“] [email protected] 2 points 2 years ago

Thanks for sharing your insights.


Thinking out loud here...

In my experience with traditional logging and distributed systems, timestamps and request IDs do store the information required to partially reconstruct a timeline:

  • In the case of a linear (single branch) timeline you can always "query" by a request ID and order by the timestamps and that's pretty much what tracing will do too.
  • Things, however, get complicated when you've a timeline w/ multiple branches.
    For example, consider the following relatively simple diagram.
    Reconstructing the causality and join/fork relations between the executions nodes is almost impossible using traditional logs whereas a tracing solution will turn this into a nice visual w/ all the spans and sub-spans.

That said, logs do shine when things go wrong; when you start your investigation by using a stacktrace in the logs as a clue. That (stacktrace) is something that I'm not sure a tracing solution will be able to provide.


they should complement each other

Yes! You nailed it ๐Ÿ’ฏ

Logs are indispensable for troubleshooting (and potentially nothing else) while tracers are great for, well, tracing the data/request throughout the system and analyse the mutations.

[โ€“] [email protected] 1 points 2 years ago* (last edited 2 years ago)

I'm not sure how this got cross-posted! I most certainly didn't do it ๐Ÿคทโ€โ™‚๏ธ

 

TLDR; The author argues that free-form logging is quite useless/expensive to use. They also argue that structured logging is less effective than tracing b/c of mainly the difficulty of inferring timelines and causality.


I find the arguments very plausible.

In fact I very rarely use logs produced by several services b/c most of the times they just confuse me. The only time that I heavily use logs is troubleshooting a single service and looking at its stdout (or kubectl log.)

However I have very little experience w/ tracing (I've used it in my hobby projects but, obviously, they never represent the reality of complex distributed systems.)

Have you got real world experience w/ tracing in larger systems? Care to share your take on the topic?

6
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
 

cross-posted from: https://lemmy.ml/post/5193714

A few days DHH (from 37signals) wrote about how they moved off the cloud and how that has helped reduce their costs by a good measure.

Well, earlier today, he announced the first bit of tooling that they used as part of their cloud exit move: Kamal - which is already at version 1.0 and, according to DHH, stable.


I took a quick look at the documentation and it looks to me like an augmented and feature-rich Docker Compose which is, to no surprise, rather opinionated.

I think anyone who's had experience with the simplicity of Docker Swarm compared to K8s would appreciate Kamal's way. Hopefully it will turn out to be more reliable than Swarm though.

I found it quite a pragmatic approach to containerising an application suite with the aim of covering a good portion of a the use-cases and requriements of smaller teams.


PS: I may actually try it out in an ongoing personal project instead of Compose or K8s. If I do, I'll make sure to keep this post, well, posted.

0
submitted 2 years ago* (last edited 2 years ago) by [email protected] to c/[email protected]
 

A few days DHH (from 37signals) wrote about how they moved off the cloud and how that has helped reduce their costs by a good measure.

Well, earlier today, he announced the first bit of tooling that they used as part of their cloud exit move: Kamal - which is already at version 1.0 and, according to DHH, stable.


I took a quick look at the documentation and it looks to me like an augmented and feature-rich Docker Compose which is, to no surprise, rather opinionated.

I think anyone who's had experience with the simplicity of Docker Swarm compared to K8s would appreciate Kamal's way. Hopefully it will turn out to be more reliable than Swarm though.

I found it quite a pragmatic approach to containerising an application suite with the aim of covering a good portion of a the use-cases and requriements of smaller teams.


PS: I may actually try it out in an ongoing personal project instead of Compose or K8s. If I do, I'll make sure to keep this post, well, posted.

[โ€“] [email protected] 10 points 2 years ago

That was my case until I discovered that GNU tar has got a pretty decent online manual - it's way better written than the manpage. I rarely forget the options nowadays even though I dont' use tar that frequently.

[โ€“] [email protected] 1 points 2 years ago

I cross-posted the same questions on Matrix and got the answer there.


The hook I'm using is invoked before the minor modes are setup - that's why it's being overridden. The suggestion was to have a hook function for each minor mode that I want to control. It's not clean but gets the job done.


Here's the working snippet:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun bahman/helm-major-mode.hook ()
 (display-line-numbers-mode -1)
 (puni-mode -1))

(add-hook 'helm-major-mode-hook
         #'bahman/helm-major-mode.hook)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defvar bahman/display-line-numbers-mode.disabled-modes
 '(vterm-mode erlang-shell-mode)
 "Disable `display-line-numbers' for the specified modes.")

(defun bahman/display-line-numbers-mode.hook ()
 (when (seq-contains-p bahman/display-line-numbers-mode.disabled-modes
                       major-mode)
     (display-line-numbers-mode -1)))

(add-hook 'display-line-numbers-mode-hook
         #'bahman/display-line-numbers-mode.hook)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defvar bahman/puni-mode.disabled-modes
 '(vterm-mode)
 "Disable `puni-mode' for the specificied modes.")

(defun bahman/puni-mode.hook ()
 (when (seq-contains-p bahman/puni-mode.disabled-modes
                       major-mode)
   (puni-mode -1)))

(add-hook 'puni-mode-hook
         #'bahman/puni-mode.hook)
 

See my reply below (https://lemmy.ml/comment/3972018) for a working solution.


I've got quite a few useful minor modes enabled globally, namely puni-mode and display-line-numbers-mode.

However, I'd like to be able to disable (one or more of) them for certain major modes. For example, I don't want puni or display-line-number to be activated when using vterm.

I've been trying to make the following snippet do the job for me to no avail. What am I missing?

I'd appreciate any hints/help ๐Ÿ™


(defvar bahman/display-line-numbers-mode.disabled-modes
  '(vterm-mode erlang-shell-mode)
  "Disable `display-line-numbers' for the specified modes.")

(defvar bahman/puni-mode.disabled-modes
  '(vterm-mode)
  "Disable `puni-mode' for the specificied modes.")

(defun bahman/after-change-major-mode-hook.disable-minor-modes ()
  "Disable certain minor modes for certain major modes ๐Ÿคท.
See
  `bahman/display-line-numbers-mode.disabled-modes'
  `bahman/puni-mode.disabled-modes'"
  (with-current-buffer (current-buffer)
    (when (seq-contains-p bahman/display-line-numbers-mode.disabled-modes
                          major-mode
                          #'eq)
      (display-line-numbers-mode -1))
    (when (seq-contains-p bahman/puni-mode.disabled-modes
                          major-mode
                          #'eq)
      (puni-mode -1))))

(add-hook 'after-change-major-mode-hook
          #'bahman/after-change-major-mode-hook.disable-minor-modes)
[โ€“] [email protected] 1 points 2 years ago

Thanks all for your feedback ๐Ÿ™

I'm going to stick to "version 2" which, to my mind, reads more naturally. I'll definitely consider the iterative suggestions for the sake of performance if I ever decide to submit a patch upstream. But for now, what I've got does the job for me dealing w/ sequences w/ less than 50 elements.

[โ€“] [email protected] 22 points 2 years ago (2 children)

This is quite intriguing. But DHH has left so many details out (at least in that post) as pointed out by @[email protected] - it makes it difficult to relate to.

On the other hand, like DHH said, one's mileage may vary: it's, in many ways, a case-by-case analysis that companies should do.

I know many businesses shrink the OPs team and hire less experienced OPs people to save $$$. But just to forward those saved $$$ to cloud providers. I can only assume DDH's team is comprised of a bunch of experienced well-payed OPs people who can pull such feats off.

Nonetheless, looking forward to, hopefully, a follow up post that lays out some more details. Pray share if you come across it ๐Ÿ™

 

Update

Turned out I didn't need to convert any series to gauges at all!

The problem was that I had botched my Prometheus configuration and it wasn't ingesting the probe results properly ๐Ÿคฆโ€โ™‚๏ธ Once I fixed that, I got all the details I needed.

For posterity you can view lemmy-meter's configuration on github.


cross-posted from: https://lemmy.ml/post/5114491

I'm using blackbox_exporter to monitor a dozen of websites' performance. And that is working just fine for measuring RTT and error rates.

I'm thinking about creating a single gauge for each website indicating whether it is up or down.


I haven't been able to find any convincing resource as to if it is mathematically correct to convert such series to guages/counters - let alone how to do that.

So my questions are

  • Have I missed a relevant option in blackbox_exporter configurations?
  • Do you recommend converting series to gauges/counters? If yes, can you point me to a resources so that I can educate myself on how to do it?

PS: To my surprise, there were no communities around Observability in general and Prometheus in particular. So I went ahead and created one: [email protected]

 

My usual workflow involves quite a good deal of side-by-side windows:

For each "thing" I'm working on, viewing, doing, I use a separate desktop w/ all the required windows in there. For example when coding, for each project I usually have got an Emacs window, a Firefox window, often a Nautiuls window and occasionally an Alacirtty window open.

Managing that kind of window placement has always been a hassle for me. The default snap-to-edge in Gnome works quite well but is limited.


That's until I found out about Tactile a few days ago: it's a Gnome Shell extension that helps you place windows in a configurable tile set.

I've been using it for a couple of days now and it has really upped my workflow substantially. Here's a shout out to the developer ๐Ÿ‘

  • Practical and smart idea
  • Stable implementation
  • Easy to use - gets out of your way
  • Reasonably configurable

Definitely worth trying out โœ…

[โ€“] [email protected] 2 points 2 years ago

I think I understand where RMS was coming from RE "recursive variables". As I wrote in my blog:

Recursive variables are quite powerful as they introduce a pinch of imperative programming into the otherwise totally declarative nature of a Makefile.

They extend the capabilities of Make quite substantially. But like any other powerful tool, one needs to use them sparsely and responsibly or end up w/ a complex and hard to debug Makefile.

In my experience, most of the times I can avoid using recursive variables and instead lay out the rules and prerequisites in a way that does the same. However, occasionally, I'd have to resort to them and I'm thankful that RMS didn't win and they exist in GNU Make today ๐Ÿ˜… IMO purist solutions have a tendency to turn out impractical.

 

cross-posted from: https://lemmy.ml/post/4908824

There are two major flavours of variables in GNU Make: "simple" and "recursive".

While simple variables are quite simple and easy to understand, they can be limiting at times. On the other hand, recursive variables are powerful yet tricky.

...

There is exactly one rule to recall when using recursive variables...

๐Ÿง  The value of a recursive variable is computed every time it is expanded.

 

cross-posted from: https://lemmy.ml/post/4908824

There are two major flavours of variables in GNU Make: "simple" and "recursive".

While simple variables are quite simple and easy to understand, they can be limiting at times. On the other hand, recursive variables are powerful yet tricky.

...

There is exactly one rule to recall when using recursive variables...

๐Ÿง  The value of a recursive variable is computed every time it is expanded.

view more: โ€น prev next โ€บ