Self Hosting Fail

2 years ago by Padook to c/selfhosted

I woke up this morning to a text from my ISP, "There is an outage in your area, we are working to resolve the issue"

I laugh, this is what I live for! Almost all of my services are self hosted, I'm barely going to notice the difference!

Wrong.

When the internet went out, the power also went out for a few seconds. Four small computers host all of my services. Of those, one shutdown, and three rebooted. Of the three that ugly rebooted some services came back online, some didn't.

30 minutes later, ISP sends out the text that service is back online.

2 hours later I'm still finding down services on my network.

Moral of the story: A UPS has moved to the top of the shopping list! Any suggestions??

Kuinox 116 points 2 years ago

When you are bored, backup a VM then hard kill it and see if it manage to restart properly.
Software should be able to recover from that.
If it doesn't, troubleshoot.

path: 0 8105024, hotness: undefined, score: 116, children: 14

Deebster 19 points 2 years ago

That reminds me of Netflix's Chaos Monkey (basically in office hours this tool will randomly kill stuff).

path: 0 8105024 8127635, hotness: undefined, score: 19, children: 0

BlackAura 11 points 2 years ago

When I built my home server this is what I did with all VMs. Learned how to change the start up delay time in esxi and ensured everything came back online with no issues from a cold built.

Rip VMware.

path: 0 8105024 8125217, hotness: undefined, score: 11, children: 0

pezhore -11 points 2 years ago

While I appreciate the sentiment, most traditional VMs do not like to have their power killed (especially non-journaling file systems).

Even crash consistent applications can be impacted if the underlying host fs is affected by power loss.

I do think that backup are a valid suggestion here, provided that the backup is an interrupted by a power surge or loss.

path: 0 8105024 8105682, hotness: undefined, score: -11, children: 11

taladar 43 points 2 years ago

most traditional VMs do not like to have their power killed (especially non-journaling file systems).

Why are you using a non-journaling file system in 2024 when those were common 10+ years ago?

path: 0 8105024 8105682 8106582, hotness: undefined, score: 43, children: 9

lazynooblet 4 points 2 years ago

It's been a while since a power cut affected my services, is this why?

I remember having to troubleshoot mysql corruption following abrupt power loss, is this no longer a thing?

path: 0 8105024 8105682 8106582 8123524, hotness: undefined, score: 4, children: 1

taladar 8 points 2 years ago

Databases shouldn't even need a journaling filesystem, they usually pay attention to when to use fsync and fdatasync.

In fact journaling filesystems basically use the same mechanisms as databases only for filesystem metadata.

path: 0 8105024 8105682 8106582 8123524 8125006, hotness: undefined, score: 8, children: 0

ryannathans 4 points 2 years ago

Or even better use something like ZFS with CoW that can't corrupt on power loss

path: 0 8105024 8105682 8106582 8108973, hotness: undefined, score: 4, children: 6

emptiestplace 4 points 2 years ago

and don't fuck with sync writes

path: 0 8105024 8105682 8106582 8108973 8118709, hotness: undefined, score: 4, children: 0

taladar 1 point 2 years ago

I would still consider that generation of filesystem to be effort to use while regular journaling filesystems have been so ubiquitous that you need to invest effort to avoid using one.

path: 0 8105024 8105682 8106582 8108973 8125039, hotness: undefined, score: 1, children: 4

possiblylinux127 13 points 2 years ago

Your system should be fine after a hard kill. If its not stop using it as that's going to be a problem down the road.

path: 0 8105024 8105682 8117721, hotness: undefined, score: 13, children: 0

lemmyvore 39 points 2 years ago

IMHO you're optimizing for the wrong thing. 100% availability is not something that's attainable for a self-hoster without driving yourself crazy.

Like the other comment suggested, I'd rather invest time into having machines and services come back up smoothly after reboots.

That being said, an UPS may be relevant to your setup in other ways. For example it can allow a parity RAID array to shut down cleanly and reduce the risk of write holes. But that's just one example, and an UPS is just one solution for that (others being ZFS, or non-parity RAID, or SAS/SATA controller cards with built-in battery and/or hardware RAID support etc.)

path: 0 8105298, hotness: undefined, score: 39, children: 4

pezhore 19 points 2 years ago

I agree that 99.999% uptime is a pipedream for most home labs, but I personally think a UPS is worth it, if only to give yourself the option to gracefully shut down systems in the event of a power outage.

Eventually, I'll get a working script that checks the battery backup for mains power loss and handle the graceful shutdown for me, but right now that extra 10-15 minutes of battery backup is enough for a manual effort.

path: 0 8105298 8105362, hotness: undefined, score: 19, children: 3

shnizmuffin 9 points 2 years ago

Some of the nicer models of UPS have little servers built in for remote management, and also communicate to their tenants via USB or Serial or Emergency Power Off (EPO) Port.

You shouldn't have to write a script that polls battery status, the UPS should tell you. Be told, don't ask.

path: 0 8105298 8105362 8105519, hotness: undefined, score: 9, children: 0

JustEnoughDucks 2 points 2 years ago

The problem is that for most self-hosters, they would be working and unavailable to do a graceful shutdown in any case even if they had a UPS unless they work fully from home with 0 meetings. If they are sleeping or at work, (>70% of the day for many or most) then it is useless without graceful shutdown scripts.

I just don't worry about it and go through the 10 minute startup and verification process if anything happens. Easier to use an uptime monitor like uptimekuma and log checker like dozzle for all of your services available locally and remotely and see if anything failed to come back up.

path: 0 8105298 8105362 8137251, hotness: undefined, score: 2, children: 0

knobbysideup 1 point 2 years ago

I run nut on a pi.

path: 0 8105298 8105362 8108588, hotness: undefined, score: 1, children: 0

CameronDev 37 points 2 years ago

Did the services fail to come back due to the bad reboot, or would they have failed to come back on a clean reboot? I ugly reboot my stuff all the time, and unless the hardware fails, i can be pretty sure its all going to come back. Getting your stuff to survive reboot is probably a better spend of effort.

path: 0 8105842, hotness: undefined, score: 37, children: 8

padook 16 points 2 years ago

I didn't mean to imply that Services actually broke. Only that they didn't come back after a reboot. A clean reboot may have caused some of the same issues because, I'm learning as I go. Some services are restarted by systemctl, some by cron, some....manual. This is certainly a wake up call that I need standardize and simplify the way the services are started.

path: 0 8105842 8108452, hotness: undefined, score: 16, children: 6

CameronDev 17 points 2 years ago

We've all.committed that sin before. Its better to rely on it surviving the reboot than to try prevent the reboot.

Also worth looking into some form of uptime monitoring software. When something goes down, you want to know about it asap.

And documenting your setup never hurts :D

path: 0 8105842 8108452 8108705, hotness: undefined, score: 17, children: 4

nimmo 5 points 2 years ago

On the uptime monitoring I've been quite happy with uptime kuma, but... If you put it on the same host that's down... Well, that's not going to work :p (I nearly made that mistake)

path: 0 8105842 8108452 8108705 8108805, hotness: undefined, score: 5, children: 3

elvith 3 points 2 years ago

It's not the most detailed thing, but I just use a free account on cron-job.org to send a head request every two minutes to a few services that are reachable from the internet (either just their homepage or some ping endpoint in the API) and then used the status page functionality to have a simple second status page on a third party server.

You can do a bit more on their paid tier, but so far I didn't need that.

On the other hand, you could try if a free tier/cheap small vps on one of the many cloud providers is sufficient for an uptime Kuma installation. Just don't use the same cloud provider as all other of your services run in.

path: 0 8105842 8108452 8108705 8108805 8109549, hotness: undefined, score: 3, children: 1

CameronDev 1 point 2 years ago

Same, Uptime Kuma is fantastic. I put it on my most critical server, if Kuma is down, everything is down :D

path: 0 8105842 8108452 8108705 8108805 8108863, hotness: undefined, score: 1, children: 0

iknowitwheniseeit 2 points 2 years ago

I reboot every box monthly to flush out such issues. It's not perfect, since it won't catch things like circular dependencies or clusters failing to start if every member is down, but it gets lots of stuff.

path: 0 8105842 8108452 8108916, hotness: undefined, score: 2, children: 0

fuckwit_mcbumcrumble 13 points 2 years ago

Yeah an unclean reboot shouldn’t break anything as long as it wasn’t doing anything when it went down. I’ve never had any issues when I have to crash a computer unless it was stuck doing an update.

path: 0 8105842 8106103, hotness: undefined, score: 13, children: 0

Static_Rocket 26 points 2 years ago

I present to you the holy hardware compatibility table:

https://networkupstools.org/stable-hcl.html

Anything not listed there is not worth buying.

path: 0 8105277, hotness: undefined, score: 26, children: 8

catloaf 13 points 2 years ago

A lot of stuff on there isn't worth buying either, like anything from APC. If you want good stuff, just get Eaton.

But also you have to understand that UPSes aren't set and forget. The batteries need replacement every 3-5 years. And they're not for extended outages, they're mostly to bridge the gap between mains power going out and a generator starting up.

Personally I just have everything running from docker-compose, so I run one command and everything not running gets started. I don't worry about stuff being down for a bit.

path: 0 8105277 8105372, hotness: undefined, score: 13, children: 7

anamethatisnt 9 points 2 years ago

Eatons batteries are usually really simple to switch, see
https://www.eaton.com/...

For me they are meant for allowing a graceful shutdown in a powerout scenario and to protect the hardware behind them from power surges.

path: 0 8105277 8105372 8105507, hotness: undefined, score: 9, children: 0

acockworkorange 5 points 2 years ago

That's why you integrate with NUT. So you can automate a graceful shutdown when battery levels drop to a set level.

path: 0 8105277 8105372 8106713, hotness: undefined, score: 5, children: 0

wreckedcarzz 4 points 2 years ago

Tl;dr apc bad? I have 4 cyberpower so no experience with them

path: 0 8105277 8105372 8108003, hotness: undefined, score: 4, children: 1

corsicanguppy 5 points 2 years ago

They got bought. They started to suck.

Now they've been 'sploited, they're overpriced, and they still schlepp the same bad software.

path: 0 8105277 8105372 8108003 8110068, hotness: undefined, score: 5, children: 0

clmbmb 2 points 2 years ago

What's wrong with APC? I have one for 6-7 years. I've changed the battery once and I think I'll have to change it again this year. I didn't have any problems with it.

path: 0 8105277 8105372 8109041, hotness: undefined, score: 2, children: 2

Appoxo 1 point 2 years ago

Trash software.
At least their service is good.

path: 0 8105277 8105372 8109041 8109042, hotness: undefined, score: 1, children: 1

Static_Rocket 4 points 2 years ago

Can confirm. Software is trash. Wanted me to connect it to the internet and setup a cloud access account. Like, dude, you're a glorified battery pack I'm not adding a backdoor because you want to tell APC when my warrenty is about to expire so I can get marketing emails.

path: 0 8105277 8105372 8109041 8109042 8117531, hotness: undefined, score: 4, children: 0

Deebster 18 points 2 years ago

A general tip on buying UPSes: look for second hand ones - people often don't realise you can just replace the battery in them (or can't be bothered) so you can get fancier/larger ones very cheap.

path: 0 8128250, hotness: undefined, score: 18, children: 3

elucubra 9 points 2 years ago

Also, a larger capacity one is better, and it's likely you'll find a secondhand one with more capacity/features for a similar price.

path: 0 8128250 8128517, hotness: undefined, score: 9, children: 2

ElderWendigo 1 point 2 years ago

Why? If the power has gone out there are very few situations (I can't actually think of any except brownouts or other transient power loss) where it would be useful to power my server for much longer than it takes to shut down safely.

path: 0 8128250 8128517 8144459, hotness: undefined, score: 1, children: 1

Deebster 2 points 2 years ago

Longer means you're more likely to be able to ride out a power cut, and gives you more options if you want/need to complete something more involved than saving and shutting down.

path: 0 8128250 8128517 8144459 8145813, hotness: undefined, score: 2, children: 0

AnarchistArtificer 17 points 2 years ago

Though I wonder if even besides adding an uninterruptible power supply (UPS) (writing acronym out for anyone else who would've had to Google it), this might be a useful exercise recovering from outages in general. This is coming from someone who hasn't actually done any self hosting of my own, but you saying you're still finding down services reminds me of when I learned the benefit of testing system backups as part of making them.

I was lucky in that I didn't have any data loss, but restoring from my backup took a lot more manual work than I'd anticipated, and it came at an awkward time. Since then, my restoring from backup process is way more streamlined.

path: 0 8130141, hotness: undefined, score: 17, children: 0

HaywardT 16 points 2 years ago

Laptops

path: 0 8111269, hotness: undefined, score: 16, children: 7

BreakDecks 22 points 2 years ago

My favorite part about using an old laptop as a 24/7/365 plugged-in server is the anticipation of when the lithium battery will explode from overcharging.

path: 0 8111269 8119037, hotness: undefined, score: 22, children: 6

skilltheamps 45 points 2 years ago

"overcharging" doesn't exist. There are two circuits preventing the battery from being charged beyond 100%: the usual battery controller, and normally another protection circuit in the battery cell. Sitting at 100% and being warm all the time is enough for a significant hit on the cell's longetivity though. An easy measure that is possible on many laptops (like thinkpads) is to set a threshold where to stop charging at. Ideal for longetivity is around 60%. Also ensure good cooling.

Sorry for being pedantic, but as an electricial engineer it annoys me that there's more wrong information about li-po/-ion batteries, chargers and even usb wall warts and usb power delivery than there's correct information.

path: 0 8111269 8119037 8121198, hotness: undefined, score: 45, children: 3

Aganim 3 points 2 years ago

Isn't dendrite formation and the shorts they can cause a much bigger concern when dealing with old batteries that are being charged 24/7? Asking a genuine question here, so please don't shoot me if I'm wrong. 🙂 I'd love to hear more about the most common failure modes and causes for li-po/ion batteries.

path: 0 8111269 8119037 8121198 8128316, hotness: undefined, score: 3, children: 1

skilltheamps 6 points 2 years ago

Those are symptoms of sitting at that operation point permanently, and they are a of course a concern. What I'm after is that people think that energy gets put in to the battery, i.e. it gets charged, as long as a "charger" is connected to the device (hence terms like "overcharged"). But that is not true, because what is commonly referred to as "charger" is no charger. It is just a power supply and has literally zero say in if, how and when the battery gets charged. It only gets charged if the charge controller in the device decides to do that now, and if the protection circuit allows it. And that is designed to only happen if the battery is not full. When it is full, nothing more happens, no currents flow in+out of the battery anymore. There's no damage due to being charged all the time, because no device keeps on pumping energy into the cell if it is full.

There is however damage from sitting (!) at 100% charge with medium to high heat. That happens indipendently from a power supply being connected to the device or not. You can just as well damage your cells by charging them to 100% and storing them in a warm place while topping them of once in a while. This is why you want to have them at lower room temperature and at ~60%, no matter if a device/"charger" is connected or not.

(Of course keeping a battery at 60% all the time defeats the purpose of the battery. So just try to keep it cool, charged to >20% and <80% most of the time, and you're fine)

path: 0 8111269 8119037 8121198 8128316 8131083, hotness: undefined, score: 6, children: 0

486 2 points 2 years ago

For many li-ion laptop batteries, the manufacturer's configuration of a 100 % charge is pretty much equivalent to overcharging. I've seen many laptops over the years with swollen batteries, almost all of them had been plugged in all the time, with the battery kept at 100 % charge.

As an electrical engineer you should know that technically there is no 100 % charge for batteries. A battery can more or less safely be charged up to to a certain voltage. The 100 % charge point is something the manufacturer can choose (of course within limits depending on cell chemistry). A manufacturer can choose a higher cell voltage than another to gain a little more capacity, at the cost of longterm reliability. There are manufacturers that choose a cell voltage of 4250 mV and while that's possible and works okay if charged only occasionally, if plugged in all the time, this pretty much ensures killing the batteries rather quickly. I would certainly call that overcharging.

Since you already mentioned charging thresholds, I just want to say, anyone considering using a laptop as a server should absolutely make use of this feature and limit the maximum charge.

path: 0 8111269 8119037 8121198 8195521, hotness: undefined, score: 2, children: 0

heisenbug4242 7 points 2 years ago

If you say it quickly enough it may sound plausible to some but this is not how battery technology works, as explained by @skilltheamps@feddit.de

path: 0 8111269 8119037 8127026, hotness: undefined, score: 7, children: 0

lambda -1 points 2 years ago

Amen. I appreciate my UPS for sure!

path: 0 8111269 8119037 8120655, hotness: undefined, score: -1, children: 0

recapitated 12 points 2 years ago

I'm a big fan of running home stuff on old laptops for this reason. Most UPSs give you a few minutes to shut down, laptops (depending on what you run) could give you plenty of extra run time and plenty of margin for a shutdown contingency.

path: 0 8146184, hotness: undefined, score: 12, children: 1

Drewelite 9 points 2 years ago

Small, good value, quiet, power efficient, built in battery backup and server terminal. Laptops are dope for home labs!

path: 0 8146184 8147333, hotness: undefined, score: 9, children: 0

JoMiran 12 points 2 years ago

A UPS should always be your first or second purchase if only for power conditioning and brown-out protection.

path: 0 8105732, hotness: undefined, score: 12, children: 4

atzanteol 4 points 2 years ago

You should buy a UPS if those things are concerns for you. If not, then don't.

path: 0 8105732 8120012, hotness: undefined, score: 4, children: 0

jkrtn 2 points 2 years ago

They will do power conditioning? My modem is such a sensitive baby I cannot plug anything else in next to it or it starts dropping packets. Would a UPS help with that? Unfortunately I cannot replace the modem, that's the only one the ISP will give me.

path: 0 8105732 8108840, hotness: undefined, score: 2, children: 2

catloaf 3 points 2 years ago

Yes. An online/double-conversion UPS will be the most effective, because it actually runs off the battery the whole time, so it's disconnected from any line quality issues.

A line-interactive UPS is cheaper, but doesn't do full power conditioning.

An offline UPS doesn't do any at all, only comes online when power drops.

https://community.fs.com/...

path: 0 8105732 8108840 8109038, hotness: undefined, score: 3, children: 1

Catsrules 2 points 2 years ago

FYI Few downside of an online/double conversion UPS will use extra power if that is something your trying to avoid.

Also some of them will have a 24/7 fan so there will be extra noise.

path: 0 8105732 8108840 8109038 8133166, hotness: undefined, score: 2, children: 0

Swarfega 10 points 2 years ago

This is why I gave up self hosting. It's great when it works but it just becomes an expensive second job. I still have Plex/Jellyfin etc but for emails and password vaults I just pay for external services.

path: 0 8106385, hotness: undefined, score: 10, children: 4

cypherix93 15 points 2 years ago

if you don't want it to feel like a second job, you could always quit your first!

path: 0 8106385 8107243, hotness: undefined, score: 15, children: 1

padook 11 points 2 years ago

I could have the best self hosted setup.... living in a van, down by the river!

path: 0 8106385 8107243 8108726, hotness: undefined, score: 11, children: 0

atmur 6 points 2 years ago

I like to host as many services as possible and I’m fine with it being a second job at times since this is my main hobby, but I actually agree with you on your examples. The three things I won’t self-host are:

Emails - I am not willing to put in the effort on this. Plus, my ISP blocks those ports so I’d already be into using a VPS even if I wanted to host this. I’d rather just pay someone else, like Proton.
Password manager - I actually did self-host Bitwarden for a long time, but after thinking about it for a while, I decided to take the pay someone else approach here too. I’m pretty sure I’m doing everything correctly, but I’m not a security expert. I’d rather be 100% sure my passwords are in safe hands rather than be 95% sure that I’m doing everything right on this one.
Lemmy - I’ve heard about (luckily never seen) CSAM attacks on Lemmy/Kbin and will not risk that kind of content being downloaded because I’m federated with an instance dealing with those attacks. I’m happy to throw a couple bucks at lemmy.world’s Patreon and let them handle that.

path: 0 8106385 8120079, hotness: undefined, score: 6, children: 0

jelloeater85 3 points 2 years ago

I self host stuff that I feel the need to. But TBH, you don't really need to self host much, outside of media collections. PhotoPrism and JellyFin are about the only two I need, aside from a PiHole. Most folks would be fine with a beefy NAS.

path: 0 8106385 8107221, hotness: undefined, score: 3, children: 0

piefedderatedd 10 points 2 years ago

UPS, good idea.
backups too.

path: 0 8105801, hotness: undefined, score: 10, children: 0

ChojinDSL 9 points 2 years ago

UPS with usb allows you to configure a script to properly shutdown your server when a power outage happens and the UPS battery is about to run out.

path: 0 8105784, hotness: undefined, score: 9, children: 0

pezhore 9 points 2 years ago

This is why I have about five of these bad boys: CyberPower CP1500PFCLCD.

One is in my utility room for my cable modem and our chest freezer, three back up my homelab and wifi AP, and one is for my office.

They've been bulletproof through storms, and when we've lost power, but not Internet I can't keep on working.

The big thing to look for is number of battery+surge outlets vs just surge outlets. Typically they top out at 1500VA - the more overhead for what you're powering, the longer you can go without mains power.

A screen/display is helpful for at-a-glance information like expected runtime, current output, etc.

path: 0 8105327, hotness: undefined, score: 9, children: 2

Catsrules 4 points 2 years ago

Never heard of someone using a UPS on a Fridge/Freezer. Does it make a difference? Seems like the UPS would just died after 10-20 minutes and not really make much difference to your freezer.

path: 0 8105327 8133246, hotness: undefined, score: 4, children: 1

pezhore 3 points 2 years ago

I didn't intend to use it on the chest freezer - it was mostly for the modem, but since I had spare battery capacity and outlets I thought what the heck.

The power load is practically nothing until it cycles, and even then it's fairly efficient - my current runtime is estimated to be about 18 hours, more than enough to come up with an alternative if we lose power in a storm.

path: 0 8105327 8133246 8134358, hotness: undefined, score: 3, children: 0

Appoxo 8 points 2 years ago

Not APC. (At least for Windows) trashy software.

path: 0 8108891, hotness: undefined, score: 8, children: 8

Moonrise2473 6 points 2 years ago

It doesn't need the software, it gets recognized by the system as a battery

path: 0 8108891 8109663, hotness: undefined, score: 6, children: 3

Appoxo 1 point 2 years ago

Still bad hardware. At least the Back-UPS line.

path: 0 8108891 8109663 8120445, hotness: undefined, score: 1, children: 2

cryptix 1 point 2 years ago

Had a old APC that worked for many years. Brought a new back-ups 1100va few years ago , wasn't the quality I expected.

path: 0 8108891 8109663 8120445 8120574, hotness: undefined, score: 1, children: 1

Appoxo 1 point 2 years ago

It might work but the software is still trash.

path: 0 8108891 8109663 8120445 8120574 8125245, hotness: undefined, score: 1, children: 0

ripcord 2 points 2 years ago

Or TrippLite. Owners used it as a massive money laundering front for right-wing donations and bribes.

path: 0 8108891 8115068, hotness: undefined, score: 2, children: 0

lambda 1 point 2 years ago

Have a suggestion?

path: 0 8108891 8120682, hotness: undefined, score: 1, children: 2

the_third 1 point 2 years ago

path: 0 8108891 8120682 8120943, hotness: undefined, score: 1, children: 0

Appoxo 1 point 2 years ago

Not really. I saw multiple mentions of eaton.
Might be worthwile to look online like on reddit or other forums how the perception is there.

path: 0 8108891 8120682 8125236, hotness: undefined, score: 1, children: 0

cyborganism 7 points 2 years ago

Yeah if you self host, a UPS is very important.

path: 0 8107286, hotness: undefined, score: 7, children: 0

Shimitar 7 points 2 years ago

I use a laptop and external jbod covered with a low power ups. As other said, the point is to bridge powergaps now long term working powerless. I live in the countrisied, so small powergaps happens specially when my photovoltaic don't produce (no, i have no battery accumulators, too expensive)

path: 0 8121877, hotness: undefined, score: 7, children: 0

anamethatisnt 6 points 2 years ago

Figure out how much power your servers use on average with the help of a wattage meter, then enter that number and how many minutes battery backup you want in Eatons UPS Power Calculator to find a suitable unit. I'm sure other vendors have similar tools too.

path: 0 8105398, hotness: undefined, score: 6, children: 0

oldfart 3 points 2 years ago

Two pitfalls I had that you can avoid:

look at efficiency. It's not always neglible, was like 40% of my energy usage because I oversized the UPS. The efficiency is calculated from top power the UPS can supply. 96% efficient 3kW UPS eats 4% of 3kW, 120 watts, even if the load you connected is much smaller than 3kW
look at noise level. Mine was loud almost like a rack server, because of all the fans.

I replaced that noisy, power hungry beast with a small quiet 900W APC and I couldn't be happier

path: 0 8154834, hotness: undefined, score: 3, children: 0

mhredox 3 points 2 years ago

Exact same thing happened to me the other day. Like exactly. Maybe we live in the same area.

path: 0 8205805, hotness: undefined, score: 3, children: 0

bin_bash 3 points 2 years ago

when you say some services on your network are you talking about machines or softwares?

for machines yes ups makes sense for softwares writing some scripts to run on start up should be enough another alternative can be setting up wake on Lan that way you can bring all up again wherever you are

path: 0 8140373, hotness: undefined, score: 3, children: 0

knobbysideup 2 points 2 years ago

In addition to ups, an LTE failover. I've had my Comcast crap be offline for hours.

path: 0 8108530, hotness: undefined, score: 2, children: 8

ripcord 2 points 2 years ago

I'd like that, but also a really long-running UPS. multi-hour power outages are surprisingly common in my area.

path: 0 8108530 8115108, hotness: undefined, score: 2, children: 4

towerful 1 point 2 years ago

Thats no longer a UPS.
You could get something like a powerwall, something designed to power things from batteries for a long time.
Or get a generator with an automatic failover. The UPS then covers the downtime between powerfailure and generator taking load

path: 0 8108530 8115108 8121343, hotness: undefined, score: 1, children: 3

ripcord 1 point 2 years ago

Why is that no longer a UPS?

path: 0 8108530 8115108 8121343 8133624, hotness: undefined, score: 1, children: 2

towerful 1 point 2 years ago

Generally, UPS (lead acid) batteries are not designed for long-cycle deep discharge.
They are designed to hold their rated load for a minute or so until the power is restored (generators start, power-uncuts) or the servers have a chance to shut down.
But maybe thats dated information, and modern UPSs are designed to run from batteries for a few hours.

path: 0 8108530 8115108 8121343 8133624 8135304, hotness: undefined, score: 1, children: 1

bitwolf 1 point 2 years ago

Does this require a lot of gear? Or does it simply act as another gateway?

path: 0 8108530 8108902, hotness: undefined, score: 1, children: 2

knobbysideup 3 points 2 years ago

There are devices like the Netgear lm1200 that can do it inline by themselves.

I have that device, but configured as a second gateway. My firewall manages the failover based on primary packet loss and latency.

path: 0 8108530 8108902 8109044, hotness: undefined, score: 3, children: 0

themoonisacheese 1 point 2 years ago

It requires an LTE capable gateway and a data plan. As for the rest you can simply write your routing tables so that if the main gateway doesn't work, use the secondary gateway with lower prio.

path: 0 8108530 8108902 8109014, hotness: undefined, score: 1, children: 0

Decronym 2 points 2 years ago

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters	More Letters
CSAM	Child Sexual Abuse Material
DNS	Domain Name Service/System
NAS	Network-Attached Storage
PiHole	Network-wide ad-blocker (DNS sinkhole)
Plex	Brand of media server package
RAID	Redundant Array of Independent Disks for mass storage
SATA	Serial AT Attachment interface for mass storage
VPS	Virtual Private Server (opposed to shared hosting)
ZFS	Solaris/Linux filesystem focusing on data integrity

8 acronyms in this thread; the most compressed thread commented on today has 11 acronyms.

[Thread #567 for this sub, first seen 3rd Mar 2024, 16:05] [FAQ] [Full list] [Contact] [Source code]

path: 0 8105320, hotness: undefined, score: 2, children: 0

notannpc 2 points 2 years ago

Could also be a good opportunity to add a service monitor like Uptime Kuma. That way you know what services are still down once things come back online with less manual discovery on your part.

path: 0 8159875, hotness: undefined, score: 2, children: 0

key 1 point 2 years ago

path: 0 8105018, hotness: undefined, score: 1, children: 0

redcalcium 1 point 2 years ago

I feel your pain. Just the other day the disk on my home assistant machine died after a power outage and I had to replace it with another disk and restore from backup.

path: 0 8105167, hotness: undefined, score: 1, children: 0

JovialSodium 1 point 2 years ago

My suggestion just changes your threat model, so may not be a good one based on your wants.

Perhaps consolidate systems? Managing less devices = less points of failure. But adds the risk of any given failure being more severe.

path: 0 8105250, hotness: undefined, score: 1, children: 1

padook 1 point 2 years ago

This thought came to me this morning. I have 4 machines both because the BEAST grows organically, and because we're always trying to avoid that single point of failure. Then a scenario comes along that makes you question your whole way of thinking, diversifying may actually create more problems

path: 0 8105250 8108597, hotness: undefined, score: 1, children: 0

selfhosted

@lemmy.world

login for more options

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

go to feed...

selfhosted

@lemmy.world

login for more options

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

go to feed...