A Guide to Monitor Linux Server using Prometheus and Grafana

3 years ago by majorswitcher to c/selfhosted

In this tutorial, I will talk about Prometheus and Grafana. You will learn to install both on CentOS/RHEL and understand how to use Prometheus and Grafana

running a lemmy instance has learned me a lot about devops already!

I love playing around with this, the lemmy instance was a great reason. lemmyfly.org doesn't have a lot of traffic yet, I think it can handle some more. Currently running on 2 vcpu/4gb ram.

After chatting to my devops colleague at work I wanted to set up a Grafana dashboard with metrics on the server. Seems it's pretty pretty doable!

I installed prometheus node_exporter on my lemmy server
I created a second server at my VPS. Very cheap, and don't think I'll be running the Prometheus/Grafana 24/7 so I'll just create a snapshot of it, destroy the server and re-create when wanted
Created an internal private network so the instances can communicate without exposing ports to the public
Installed Prometheus on the new prometheus server
Installed Grafana
log in to Grafana, admin:admin then change password
Go to the Grafana home dashboard, click on + sign and click on Import. In Import via grafana.com, put the dashboard id 1860 (preset for all prometheus node_exporter metrics) and click on Load.
Done!

Last 2 hours in the garden with a beer on the side, WFH FTW

manned_meatball 4 points 3 years ago

lol I literally did this yesterday for my job, using dashboard 1860 too. I used a docker compose stack following their guide here.

save

path: 0 259702, hotness: undefined, score: 4, children: 3

majorswitcher 1 point 3 years ago

i'm now looking into logging more specific http request data. Maybe https://github.com/...?

save

path: 0 259702 259753, hotness: undefined, score: 1, children: 2

majorswitcher 1 point 3 years ago

http_response metrics are also up, using https://github.com/...

would be nice to see the numbers go up :)

save

path: 0 259702 259753 280868, hotness: undefined, score: 1, children: 0

manned_meatball 1 point 3 years ago

I'm not looking into logging http traffic, but that seems like a popular option

save

path: 0 259702 259753 259807, hotness: undefined, score: 1, children: 0

lckdscl 3 points 3 years ago

I fell into the deep end with Grafana + Prometheus, went crazy with all the metrics and dashboard. But I got burnt out before I even began making alerts, so I just went with Netdata at the end.

save

path: 0 266555, hotness: undefined, score: 3, children: 0

mayniac 2 points 3 years ago

This would be nice to run on my home server to monitor my Lemmy instance in the cloud. It would also be nice to be able to check some Lemmy stats as well as general server stats. I'm not a grafana or Prometheus expert though.

save

path: 0 266860, hotness: undefined, score: 2, children: 0

JasonWeen 2 points 3 years ago

Uuuh I absolutely love grafana. Worked half a year with it for my job. I am happy it worked out great for you.

save

path: 0 262061, hotness: undefined, score: 2, children: 0

Wesleysnoops 1 point 3 years ago

Very cool! What's bandwidth and system consumption been like so far?

save

path: 0 713173, hotness: undefined, score: 1, children: 1

majorswitcher 1 point 3 years ago

actually I did delete the server (after creating a snapshot of it) a week or so ago. But this morning I wanted to check lemmyfly.org, couldn't load the page. Checking my Hetzner dashboard I noticed CPU was spiked at 200%?! It did drop again though, but apparently had last for 2-3 minutes. But prometheus was down, so no graphs apart from the hetzner ones. I doesn't relate to network traffic spikes, so I don't know what caused it. I've started the prometheus server again (that snapshot was really useful :) ) and will leave it on for a couple of months now.

spikes as seen on hetzner graphs:

current system consumption:

I might need to get an extra volume for storage, Lemmy is starting to eat up the root filesystem.. Does anyone know how I re-configure Lemmy to look at a different volume for storage ?

save

path: 0 713173 723062, hotness: undefined, score: 1, children: 0

Wander 1 point 3 years ago

Oh wow, I need to check this out. I might do this with LXC containers

save

path: 0 259931, hotness: undefined, score: 1, children: 2

majorswitcher 2 points 3 years ago

Let us know your findings when you did!

save

path: 0 259931 260639, hotness: undefined, score: 2, children: 1

tkohhh 1 point 3 years ago

Influxdb + grafana for me! Good stuff!