In preparation for the Reddit June 12-14 Black-out

3 years ago by TheDude to c/main

As the reddit mods gets ready for the June 12-14 black-out, there some anticipation that an influx in user base will shift over to many of the lemmy instances as user seek out a home to post their internet memes and discuss their interests.

In anticipation of this increased volume I will be growing our current instance from

16 CPU
8 GB ram

to

24 CPU
64 GB ram

This server is currently equipped with SSDs that are configured in a raid 10 array (NVMEs will come in the next gen that get deployed)

Earlier today I also configured some monitoring that I'll be watching closely in order to have a better understanding on how the lemmy platform does under stress (for science!)

I'll be sharing graphs and some other insights in this thread for everyone that is interested. Feel free to ask anything you might be interested in knowing more of!

EDIT: I'll be posting and updating the graphs in this main post periodically! Last updated: 6:21AM ET June 12th

CPU - 48 hours

Memory - 48 hours

Network - 48 hours

Load Average - 48 hours

System Disk I/O - 48 hours

ArtisanalRuntime 31 points 3 years ago

For the icon: How about a Satisfied Seal? Because when sh.itjust.works, you know that feel! satisfied_seal

path: 0 88489, hotness: undefined, score: 31, children: 3

TheDude 12 points 3 years ago

I love this!

path: 0 88489 88661, hotness: undefined, score: 12, children: 2

ArtisanalRuntime 9 points 3 years ago

Let me know if you want me to tweak the design any and I can send an SVG as well. No copyright/usage restrictions - free to use or modify as you wish.

path: 0 88489 88661 89840, hotness: undefined, score: 9, children: 0

ArtisanalRuntime 9 points 3 years ago

Here’s a link to the SVG of the Satisfied Seal so you can tweak as needed. I’ll also post the link in the comment thread. Glad people seem to like it!

https://svgshare.com/s/u8T

path: 0 88489 88661 107722, hotness: undefined, score: 9, children: 0

nutomic 23 points 3 years ago

Impressive setup! Would you be interested to have your instance recommended on the Lemmy project website?

https://lemmy.ml/post/1171717

path: 0 21518, hotness: undefined, score: 23, children: 9

TheDude 8 points 3 years ago

That would be great! I checked your link and will work on completing the steps shortly :)

path: 0 21518 22413, hotness: undefined, score: 8, children: 7

noodle 4 points 3 years ago

let's goo! Great instance that deserves some growth! 🚀

path: 0 21518 22413 22487, hotness: undefined, score: 4, children: 0

Acheron 4 points 3 years ago

Just joined you from that featured site, it just works. People are complaining about how hard it is to switch, I genuinely don’t know what the difficulty is

path: 0 21518 22413 27277, hotness: undefined, score: 4, children: 5

planish 3 points 3 years ago

This is one of the easier instances to join; most of even the big ones have a manual approval step/application process that slows things down.

@nutomic@lemmy.ml that might be a good thing to note about the different instances? Or would that just tell the spammers where to spam?

path: 0 21518 22413 27277 27367, hotness: undefined, score: 3, children: 3

god 4 points 3 years ago

Saw some dude who had spent a week on and off trying every couple of days to join and all the servers rejected him and he had given up lol. Most users are lazy fucks. If you want quality users, it's okay to hide the goods and let the best people find them. If you want mass adoption you need to adapt to the lowest common demoninator.

path: 0 21518 22413 27277 27367 28889, hotness: undefined, score: 4, children: 2

Toebeans 2 points 3 years ago

Tbf this is the sh.it just.works instance - some of the others a more hit and miss

path: 0 21518 22413 27277 27692, hotness: undefined, score: 2, children: 0

god 6 points 3 years ago

Joined from recommended a bit less than an hour ago. Tried lemmy.ml, got rejected, went for this one cuz it was top recommended and I figured with the name and the user count the likelihood of rejection was smaller.

path: 0 21518 28898, hotness: undefined, score: 6, children: 0

pax 11 points 3 years ago

I am from reddit migration. it's 100000000000 times better, e.g. more accessible with my screenreader, on reddit, I needed to use special app, called dystopia which handled voiceover more gracefully.

path: 0 15442, hotness: undefined, score: 11, children: 1

sexy_peach 3 points 3 years ago

damn that's good to hear

path: 0 15442 21606, hotness: undefined, score: 3, children: 0

TheDude 8 points 3 years ago

A little update for all of you interested. I allocated the additional resources to the VM and will post some updated graphs once they update with the new configurations.

For those who are like me and like looking at graphs here are some prior to the upgrade.

CPU - 48 hours

Memory - 48 hours

Network - 48 hours

path: 0 19547, hotness: undefined, score: 8, children: 5

this 4 points 3 years ago

so if I'm reading this right, less than 10% CPU capacity and about 1/8th ram at peak times, before upgrades? gotta give you credit where it's due, that thing looks ready to take some abuse.

path: 0 19547 19893, hotness: undefined, score: 4, children: 3

TheDude 3 points 3 years ago

The only part I don't have graphed yet is the disk IOs. I'm going to need to invest a little more time to get that metric going captured.

Say you through 1000s of active users to this instance.. what would cause the bottle neck first? CPU, memory, network, disk? I'm thinking probably disk due to database optimization that need to be reworked on lemmy afterwards CPU and then memory.

path: 0 19547 19893 19917, hotness: undefined, score: 3, children: 1

Echolot 1 point 3 years ago

Yeah disk iops is probably is gonna be the biggest hurdle, but so far so good...

path: 0 19547 19893 19917 32032, hotness: undefined, score: 1, children: 0

sexy_peach 1 point 3 years ago

The server doesn't have any users atm though

path: 0 19547 19893 21604, hotness: undefined, score: 1, children: 0

Srootus 3 points 3 years ago

Hello Lemmy! I've just made an account to get my foot in the door incase the Reddit execs don't roll back after the blackout, let's sit back and watch the fireworks.

path: 0 19547 21328, hotness: undefined, score: 3, children: 0

carrot 6 points 3 years ago

shit just works

path: 0 37100, hotness: undefined, score: 6, children: 0

this 6 points 3 years ago

So the r/lemmymigration mod got his account unbanned from reddit, and it seems he might be willing to put together a currated and updated list of lemmy instances that are willing to host new users. Just in case that is something you would be interested in participating in, here is his comment on reddit: https://www.reddit.com/...

path: 0 13679, hotness: undefined, score: 6, children: 1

TheDude 7 points 3 years ago

Sure lets see how far we can get this server going!

path: 0 13679 19415, hotness: undefined, score: 7, children: 0

httpjames 6 points 3 years ago

Thanks for the transparency

path: 0 98420, hotness: undefined, score: 6, children: 0

smorks 6 points 3 years ago

oooh pretty colors!

path: 0 25422, hotness: undefined, score: 6, children: 0

Condodog 5 points 3 years ago

Just discovered this place. Nice to see you guys are getting ready for the influx of users.

path: 0 32783, hotness: undefined, score: 5, children: 0

manifex 5 points 3 years ago

Thanks @TheDude. I work in enterprise network/systems/cloud operations as a network/security engineer. Would love to contribute monetarily or with time.

path: 0 16131, hotness: undefined, score: 5, children: 0

phase_change 5 points 3 years ago

I’d love to see a few additional charts with the next update:

Disk space consumed
Subscribers
Communities federated

My guess is that’s not available vie SNMP for Librenms. :)

path: 0 57731, hotness: undefined, score: 5, children: 4

TheDude 9 points 3 years ago

I'm being a little private about the total amount of disk assigned to this instance for now. I do plan on sharing these details when I have implemented a viable solution. That being said I can tell you that with the amount of current users and activities this instance is growing by about 20GB per day in disk size.

path: 0 57731 73211, hotness: undefined, score: 9, children: 3

LetterboxPancake 5 points 3 years ago

Holy shit... You tell us when you'll accept donations, right? This doesn't have to be your financial responsibility.

Or do you and I just haven't found it yet?

path: 0 57731 73211 87913, hotness: undefined, score: 5, children: 2

TheDude 4 points 3 years ago

Not taking donations at the moment. Still need to figure out the best way to collect donations and haven't had the chance to think about it! I will eventually

path: 0 57731 73211 87913 88772, hotness: undefined, score: 4, children: 1

LetterboxPancake 1 point 3 years ago

No pressure :-)
And thank you for this <3

path: 0 57731 73211 87913 88772 88838, hotness: undefined, score: 1, children: 0

lightrush 5 points 3 years ago

Is this an actual bare metal machine?

path: 0 37646, hotness: undefined, score: 5, children: 4

TheDude 5 points 3 years ago

No its on a vm.

path: 0 37646 38545, hotness: undefined, score: 5, children: 3

lightrush 4 points 3 years ago

Which cloud?

path: 0 37646 38545 38617, hotness: undefined, score: 4, children: 2

TheDude 7 points 3 years ago

It's on colocated hardware that I have in a local datacenter.

path: 0 37646 38545 38617 41273, hotness: undefined, score: 7, children: 1

lightrush 4 points 3 years ago

Colo - very nice.

path: 0 37646 38545 38617 41273 42986, hotness: undefined, score: 4, children: 0

PCChipsM922U 3 points 3 years ago

Can this thing run on a Pi clone with 4GB RAM and 4 cores?

path: 0 17025, hotness: undefined, score: 3, children: 6

TheDude 4 points 3 years ago

It can definitely run on a Pi instance. The storage medium is going to be important as it will need to house a PostgreSQL database. Would be fine for a few users but not sure how many it would be able to handle concurrently.

path: 0 17025 19458, hotness: undefined, score: 4, children: 3

PCChipsM922U 2 points 3 years ago

It's got 4 cores at 2.16GHz (I think 🤔), 4GB of RAM. Regarding the storage, that can be arranged. Currently there's a 32GB SD card in it, it can be swapped for a 512GB one, no prolem.

So, how many users can an instance on a Pi like that serve?

path: 0 17025 19458 23372, hotness: undefined, score: 2, children: 2

TheDude 3 points 3 years ago

My biggest concern with your setup is the SD card. SD cards aren't typically known for their write durability. There is a way to use a SSD drive instead however its been a while since I checked the pi stats and might limited at speeds of usb 2. You'll be fine for a small amount of users but you're going to hit a bottleneck pretty quickly if you make your instance open to all.

path: 0 17025 19458 23372 23392, hotness: undefined, score: 3, children: 1

PCChipsM922U 1 point 3 years ago

This is actually a Pi clone, BannanaPi, it has more I/Os than a Pi, but it doesn't have SATA... so, I can't plug in an SSD, the best I can do is plug in a USB and save and load from that, but, as you said, it's limited to USB 2.0 transfer speed.

Sorry, I just don't have the funds to run a full blown server at home. A Pi is the best I can do 🤷.

Maybe ask someone from r/homelab to run an instance or two, those people have like a lot of extra hardware, not to mention they don't mind spending a lot of money on electricity bills.

path: 0 17025 19458 23372 23392 25090, hotness: undefined, score: 1, children: 0

boydster 2 points 3 years ago

I was looking at that earlier too. I found LemmyNet on GitHub but haven't had a chance to test it. But "can it run" is also a fundamentally different question than "can it run smoothly for end users as it gets hit with a huge influx of users," and given the larger context of a potential Reddit exodus, that scalability concern is probably not negligible.

path: 0 17025 17090, hotness: undefined, score: 2, children: 1

PCChipsM922U 1 point 3 years ago

My point was, those that can, make one, fill it till it can serve users, then just disable registrations, so it doesn't overload.

path: 0 17025 17090 23378, hotness: undefined, score: 1, children: 0

ericjmorey 3 points 3 years ago

You may want to check this out: https://sh.itjust.works/post/2654

They are working on fixes, but they will certainly not be ready by the 12th

path: 0 15685, hotness: undefined, score: 3, children: 0

pigeonholedpoetry 3 points 3 years ago

Any recommendations on VPS hosts. Thinking about creating my own.

path: 0 25173, hotness: undefined, score: 3, children: 6

TheDude 3 points 3 years ago

There are many to choose from. Where are you located?

path: 0 25173 25311, hotness: undefined, score: 3, children: 1

pigeonholedpoetry 1 point 3 years ago

USA, IN

path: 0 25173 25311 25433, hotness: undefined, score: 1, children: 0

donut 2 points 3 years ago

Check out Vultr. Super customizable. Have not gone down since I got mine (currently has 900 days uptime).

path: 0 25173 29017, hotness: undefined, score: 2, children: 0

god 1 point 3 years ago

I've never hosted anything big but cloud fanatic is cheaper than digital ocean. I migrated from the latter to the former earlier this year to save. And then someone told me AWS is basically free up to a certain usage point. $200 free credits/mo? I can't remember what the "free" part implied. As always, do your own research, but you could join one and switch servers later on I guess.

path: 0 25173 28866, hotness: undefined, score: 1, children: 1

DevoidWisdom 1 point 3 years ago

path: 0 25173 28866 55884, hotness: undefined, score: 1, children: 0

crius 1 point 3 years ago

I've recently migrated a forum community (vbulletin -> discourse) using a vpn by Contabo (germany hosting). Their price are incredible and while you don't get an official SLA, it's around 99.5%.

path: 0 25173 126436, hotness: undefined, score: 1, children: 0

borari 2 points 3 years ago

I’m starting to get random 500’s, I guess that’s a sign I need to go to bed and let the server breathe lol. Can’t wait for the updated graphs!

path: 0 82031, hotness: undefined, score: 2, children: 2

TheDude 5 points 3 years ago

Looks like it was related to a setting that needed to be tuned. I just updated the graphs. Server is not on fire yet!

path: 0 82031 88487, hotness: undefined, score: 5, children: 1

borari 1 point 3 years ago

Awesome, great to hear it!

path: 0 82031 88487 92790, hotness: undefined, score: 1, children: 0

Dhrun 2 points 3 years ago

Yeah I'm trying Lemmy anticipating that Relay Pro may stop working soon. Searched for my sub reddit subjects, seemed to work though some of them seem a bit empty so far.

path: 0 77435, hotness: undefined, score: 2, children: 0

Jumuta 2 points 3 years ago

wow, that's much more hardware than I thought.

path: 0 15619, hotness: undefined, score: 2, children: 9

TheDude 4 points 3 years ago

It probably doesn't need that much but I'm curious in seeing how much It can handle until it can't I have more resources available but it would make more sense after this size to start separating the different components of Lemmy into their own individual instances across multiple servers instead of trying to scale it vertically. Might require a little more time than I have but I'll see what I can manage!

path: 0 15619 18150, hotness: undefined, score: 4, children: 8

Jumuta 3 points 3 years ago

huh, interesting.

How much room is there right now for lemmy to scale horizontally?

path: 0 15619 18150 18575, hotness: undefined, score: 3, children: 6

TheDude 3 points 3 years ago

A couple hundred gigs worth of ram and around ~100 cores. I'm not sure I would want to bring it to that size without thinking first of a plan to properly maintain, manage and support.

path: 0 15619 18150 18575 19408, hotness: undefined, score: 3, children: 5

imperator 1 point 3 years ago

Just joined. I tried a couple of instances but couldn't get signed up. Yours worked!

Could Lemmy handle having the same insurance running on multiple servers with a load balancer and database syncing?

I don't know anything, just asking lol

path: 0 15619 18150 18575 19408 26653, hotness: undefined, score: 1, children: 4

FlatDuckFridays 1 point 3 years ago

I'm interested to see how this develops.

I don't have a lot of experience administrating rust apps, but usually single instances hit bottlenecks when trying to utilize resources on a large enough server.

Like others have said, I'm happy to help if there's anything you need to keep this going. I have many years of Linux and Windows admin experience, observability/instrumentation, and I can code to an extent.

path: 0 15619 18150 77448, hotness: undefined, score: 1, children: 0

poVoq 2 points 3 years ago

The monitoring is with Prometheus / Grafana? Can you share your setup?

path: 0 21961, hotness: undefined, score: 2, children: 1

TheDude 3 points 3 years ago

Prometheus /grafana would be great but I'm using an existing librenms setup via snmp probing

path: 0 21961 22214, hotness: undefined, score: 3, children: 0

main

@sh.itjust.works

login for more options

Home of the sh.itjust.works instance.

go to feed...

main

@sh.itjust.works

login for more options

Home of the sh.itjust.works instance.

go to feed...