I'm curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.
As a bonus question, how have you organized your network or your services to reduce the overhead of implementing observability?
Itβs curios if you used Prometheus as well
Mimir is just the DB, no metric scrapers, but it's distributed and capable of high volume ingestion, retention and querying. Can't scrape metrics though.
Alloy has no storage persistence, so it's just the scraper part, but is massively scalable, its basically a Prometheus Agent on steroids. Can do logs, traces, Pyroscope profiles too.
So instead of deploying one server to scrape and store metrics, you deploy one to store and one to scrape.
Bezsel, simple and easy, constantly having new features added.
Prometheus, grafana, loki
Things usually break on updates unless your disk fills up or worse some hardware fails...
So it's pretty easy to fix: never upgrade automatically.
I always upgrade manually and check immediately what I have upgraded. Works fine so far.
And I was only partially joking... Yeah if a service stays broken for a long time, just uninstall it, it was useless.
I used to run dockprom but it was pretty ressource intensive and I have since moved to Beszel + Uptime Kuma.
I have yet to find a way to keep logs between container re-creation (I unsuccessfully tried to set up Loki back when I ran dockprom), both Beszel and Komodo only display logs from the container currently running in the stack.
To have logs from every container launch in journalctl, I have systemd run podman-compose without daemonizing it. To debug container logs, a shell is usually needed anyway, so I see little point in exposing logs through a web dashboard.
To add, journal-brief can e-mail you systemd logs. If your apps aren't very noisy, this can be helpful without a need to set up a complex observability stack.
Uptime Kuma if I want something to alert me (still trying to decide between ntfy & gotify)
Home Assistant is also checking some stuff
Watchdog to reboot my Raspberry Pi Zeros when they fall off the wifi
Smokeping for a general, long term feel of the network, which might answer your bonus question?
Did you ever decide on Ntfy vs Gotify?
Not yet π
Both services work well, but the (Android) app is also a consideration... Gotify appears to keep losing the connection, but as I have different alerts going via different routes I need to sit down and spend some quality time with these, which I don't have at the moment.
I asked since I recently started using Ntfy and was just curious if you found something in Gotify that's relevant to me
I have different alerts going via different routes I need to sit down and spend some quality time with these, which I donβt have at the moment.
If you mean you send notifications to different services/platforms, you should look into Apprise, which is meant to do exactly this for you but easier supposedly
Smokeping? In 2026? Jesus....
Sorry, no disrespect intended, but I'm shocked. It's like hearing someone say they drive a 1988 Toyota Cressida because it's great... It was great at the time, but we've moved on and the smokeping website should tell you how ancient it is; sponsors from 2007 by companies that don't exist anymore.
Still alive & still maintained
Alive, maybe. I think any maintenance is down to bodges to keep it running in modern environments. Neither Toby nor Niko have worked on sp in over a decade.
Smokeping is fine if all you do is look at its own graphs and you have enough traffic to see patterns in latency.
But:
I had to retire 2 smokeping monitors because their CGI implementations were security risks. That was 2015. Not a good reason for homelab, but CGI is a pretty ancient and insecure way to interact with the web server.
Smokeping is good in a big organization with lots of traffic and a few broadcast domains. It isn't really great at monitoring remote sites because ICMP doesn't tell you what segment in route is causing the issue, even with rping.
I used smokeping a lot in my career from about 2005 to 2015, when security audits made me retire it. Just even using rrdtool with an exporter and graphana would be preferable to smokeping itself.
What's insecure about CGI? I thought the main reason it isn't popular now is because runtimes are slow to start. Though IIRC when I used busybox httpd CGI with a small runtime, quickjs, speed wasn't an issue.
Sorry, no disrespect intended, but I'm shocked. It's like hearing someone say they drive a 1988 Toyota Cressida because it's great...
How did you know I had an 88 Cressida? (jk)
You have good points there, but for a homelab, it does show some interesting insights (ie, I did an upgrade of my Raspberry Pi Zeros and saw how all their latency changed after the reboot... weird)
I've been using this for years as it just worked, but a recent update has broken it (a simple "v1.10 is earlier than v1.9" issue) and I had to rollback and have been waiting for someone to fix the bug for a few weeks, so, yep, the slowness of response is a fair point.
Ah, dunno, maybe I'll look at some alternatives... one day...
No auto updates thanks. π I've just reset all my compose files to pin versions today after an unintended mariadb update cost me hours to repair. I'll look into dockhand though. Was unimpressed at portainer's announcement that new version wont have a community edition. Portainer is really just an 'at a glance' view for me since i try to do all the compose and troubleshooting stuff in terminal. Wud will probably become a thing i just review every few weeks to make sure i dont get too adrift from latest.
I ought to look into a unified dashboard as i dont have easy visibility or things like disk usage, sheduled backup success etc that id like to see alongside the container status and beszel activity graphs.
Im still quite new to it all though.
I asked this of the #homelab fediverse a few weeks back, as I was starting to deploy a family of apps to my "homelab" which is really a raspi3b with a repurposed spinny-platty laptop drive connected to some cloudflare tunnels. I settled on VictoriaLogs and VictoriaMetrics for the actual observing bits, fluentd for a log driver (I'm using docker-compose), and postgres-exporter, node-exporter, and handrolled metrics endpoints for prometheus metrics exposure (victoriametrics can do the metrics scraping itself).
If I were to change anything at the moment, I'd replace fluentd first, as some of the filters I use require the ruby interpreter and that seems a little heavy for my setup. Second I'd probably move to Grafana for dashboards. Alerting is gaping hole in the setup right now; I've used alertmanager in the past and not been particularly happy with it, but I don't have a specific alternative I would suggest at the moment. If I had a much larger deployment, especially spread across multiple nodes, I'd consider ProxMox, but I'm just not there yet.
There were other suggestions that I didn't go too deep into, especially after I got something working sufficiently, including dockhand, gatus and beszel, but so far (and to my surprise) having what I described above all running in containers on this little pi is doing the trick.
Ah good to know re:vector. When I have some time Iβll give that a look.
I've got grafana and Prometheus, but I'm currently working on a generic log pulling script that will then feed to a local ollama instance to summarize the logs and let me know if there are any potential issues I should be aware of. Grafana is great for current state, but I'm a big believer in watching the logs to see trends, but I'm too lazy to do it by hand.
Once that's up and running and steady, I'll probably tie it into ntfy to actually alert me in real time
Cool, sounds like a fun project.
Not sure if you have been looking into Jev or other System 1 models, but it might be a good fit since it can output a structured type-safe output. Most of the automation I've tried in the past with LLMs failed due to the output structure being hallucinated (ignoring my requested structure). But it sounds like System 1 models might fix that.
I've been looking at an open source model called Laya a little and it might work well for your project. Sounds like you'd want structured output.
What kind of rig do you have ollama running on?
Jev and ollaya are on my list to eventually add. I'm running ollama through an RTX 3080 with 10 gigs of RAM. Just because it's what I had lying around. I was mostly using it for gaming but it works for local model use. I'm also planning on installing Hermes in a bit once the log manager project is done, but that's going to take some careful tinkering because I'm technically underpowered for it. I'm probably just going to have some gate checks up to make sure nothing loads during a gaming session.
My main goal is to completely get off the frontier models if possible because I have concerns about the environmental impacts, especially. As a shameless pirate I mind intellectual theft less, but I really do disagree with the environmental impact, especially when a optimized local model plus a bit of scripting and filtering. Does the job 90% of the time
But we'll see, I'm still tweaking everything, but I think I'm about 90% done with the log normalizer to do most of the heavy lifting act on the logs programmatically, to keep the context window for my local models as optimized as possible.
It's definitely been a good fun learning project
Oh, one of the first things that I implemented was a hallucination checker. If it cites any data that's not actively in one of the corpus I've supplied, it automatically gets flagged and filtered out. But most of what I do can be handled with internal documentation and material I explicitly provide.
Loki?
I may eventually add that, but right now my goal isn't the actual logging itself, it's creating a digest and summarizing the logs down to the key points, and eventually getting a notification every morning if something weird showed up. I'm definitely going to add this to my backlog though, thanks for the suggestion
I use the Grafana stack but I do not self host it. Itβs the only thing I use in the cloud with their free offering which is pretty generous and good enough for homelabbing. I have also recently transition away from Alloy to Vector, mainly to give that a try and because the configuration of Vector seems easier and somewhat more intuitive.
@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
go to feed...
@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
go to feed...
If youβre wanting to go hard in the paint here, Grafana has a great OSS stack with Grafana, Loki, Mimir, Tempo, and Alloy.
save