How quickly will a lemmy instance eat up storage?

3 years ago by i_lost_my_bagel to c/selfhosted

Just started self hosting this instance. Nothing on the docs mentioned anything about storage considerations.

ruud 288 points 3 years ago

This is lemmy.world after 4 weeks:

58G	pictrs
34G	postgres

save

path: 0 784410, hotness: undefined, score: 288, children: 25

i_lost_my_bagel 82 points 3 years ago

Considering this is going to be around a 5 user instance at most I think I'll be good for awhile. Thanks!

save

path: 0 784410 784481, hotness: undefined, score: 82, children: 15

manitcor 59 points 3 years ago

im running 50 users right now, subbed to A LOT of communities, seeing db growth of about 100mb per day.

save

path: 0 784410 784481 784671, hotness: undefined, score: 59, children: 12

Pleonasm 17 points 3 years ago

That seems high when you extrapolate that to 10000 users, like a larger instance might have.

save

path: 0 784410 784481 784671 784772, hotness: undefined, score: 17, children: 11

terribleplan 70 points 3 years ago

It's all about how many communities your user(s) subscribe to since your instance basically acts as a mirror for those.

My instance has been running for 23 days, and I am pretty much the only active local user:

7.3G    pictrs
5.3G    postgres

edit: I may have a slight Reddit Lemmy problem

save

path: 0 784410 784481 784671 784772 784856, hotness: undefined, score: 70, children: 8

Stubborn9867 23 points 3 years ago

It won't scale linearly. A lot of those users will be subscribed to subs the instance is already replicating. It would only be new subs that would add to the growth.

save

path: 0 784410 784481 784671 784772 784894, hotness: undefined, score: 23, children: 1

Dran_Arcana 16 points 3 years ago

Question if you know: does a lemmy instance have to be publically accessable to work? Like, if I make an instance on my homelab can the instance "fetch" content and serve it faster locally? Could I reply to a post and have others see it? Etc

save

path: 0 784410 784481 805379, hotness: undefined, score: 16, children: 1

msinfo32 1 point 3 years ago

wondering this also! wouldnt it require a domain for your account though?

save

path: 0 784410 784481 805379 819268, hotness: undefined, score: 1, children: 0

HappyHam 19 points 3 years ago

Now I wonder how viable it would be to support video hosting. The answer is almost certainly "God no!"

save

path: 0 784410 819480, hotness: undefined, score: 19, children: 1

GatoB 1 point 3 years ago

It is viable through other hostings

save

path: 0 784410 819480 1030847, hotness: undefined, score: 1, children: 0

Grimr0c 17 points 3 years ago

Honestly, Less than I thought!

save

path: 0 784410 785178, hotness: undefined, score: 17, children: 0

mruczek 14 points 3 years ago

Interesting, I thought it would be waaayyy more

save

path: 0 784410 784870, hotness: undefined, score: 14, children: 1

BigWigglyStyle 16 points 3 years ago

At the end of the day the vast majority of what needs to be saved is text. If media content is embedded, the the server just has to save the path to the file not the file itself.

save

path: 0 784410 784870 785375, hotness: undefined, score: 16, children: 0

myersguy 11 points 3 years ago

Wow, that is surprisingly not bad given the size of the instance!

save

path: 0 784410 815507, hotness: undefined, score: 11, children: 0

bdonvr 11 points 3 years ago

Yeah lemmy seems to use just about nothing for data storage.

save

path: 0 784410 786785, hotness: undefined, score: 11, children: 0

lightrush 5 points 3 years ago

Feels like this will benefit from some sort of fuzzy deduplication in the pictrs storage. I bet there are a lot of similar pics in there. E.g. if one pic or a gif is very similar to another, say just different quality or size, or compression, it should keep only one copy. It might already do this for the same files uploaded by different people as those can be compared trivially via hashing, but I doubt it does similarity based deduplication.

save

path: 0 784410 819716, hotness: undefined, score: 5, children: 0

jukes 1 point 3 years ago

That's not super terrible given the size of lemmy.world.

save

path: 0 784410 1670017, hotness: undefined, score: 1, children: 0

NettoHikari 67 points 3 years ago

This is my small instance with way fewer users than lemmy.world.

11G	pictrs
5.2G	postgres

save

path: 0 784575, hotness: undefined, score: 67, children: 8

Molecular0079 9 points 3 years ago

Out of curiosity, how long has your instance been up? Just want to get a sense of how fast storage is increasing for you.

save

path: 0 784575 787292, hotness: undefined, score: 9, children: 1

NettoHikari 4 points 3 years ago

23 days.

save

path: 0 784575 787292 796645, hotness: undefined, score: 4, children: 0

Kushan 7 points 3 years ago

How has your Lemmy experience been on a self hosted instance? I'm currently using lemmy.world and it's very error prone, would self hosting reduce those errors at the expense of anything? Does federation take long or do you find you're getting federated content quickly enough?

save

path: 0 784575 790437, hotness: undefined, score: 7, children: 5

NettoHikari 16 points 3 years ago

The experience has been pretty good, to be honest. No instability, easy updates, etc. I find federated content quite quickly, because I use this script to populate the "All" feed.

save

path: 0 784575 790437 796915, hotness: undefined, score: 16, children: 3

Kushan 6 points 3 years ago

Thanks for the script!

save

path: 0 784575 790437 796915 802489, hotness: undefined, score: 6, children: 1

NettoHikari 5 points 3 years ago

I didn't make it! :) I think, @fmstrat@nowsci.com made it.

save

path: 0 784575 790437 796915 802489 805663, hotness: undefined, score: 5, children: 0

RedWizard 4 points 3 years ago

Oh this is very cool

save

path: 0 784575 790437 796915 805911, hotness: undefined, score: 4, children: 0

bdonvr 7 points 3 years ago

You won't get any old content, so that's a downside. You'll only get content after you start federating. Unless someone votes or comments on old content.

Other than that the only downside is spending time maintaining and updating it.

save

path: 0 784575 790437 790802, hotness: undefined, score: 7, children: 0

russjr08 30 points 3 years ago

My instance has 13 users, and has been up for 2 months now:

1.5G    ./pictrs
3.4G    ./postgres

save

path: 0 793643, hotness: undefined, score: 30, children: 0

useful_idiot 27 points 3 years ago

476M ./postgres 1.1G ./pictrs

After 3 weeks

save

path: 0 784548, hotness: undefined, score: 27, children: 1

baduhai 4 points 3 years ago

How many users?

save

path: 0 784548 791657, hotness: undefined, score: 4, children: 0

A10 25 points 3 years ago

Is there any way to purge old data?

save

path: 0 784842, hotness: undefined, score: 25, children: 4

holycrap 39 points 3 years ago

I really hope it doesn't get purged if lemmy is to be a Reddit replacement. A lot of the value Reddit had was obscure knowledge and making google searches actually usable.

save

path: 0 784842 786368, hotness: undefined, score: 39, children: 3

Molecular0079 24 points 3 years ago

I think as long as the original community the post is in doesn't purge the data, it's fine for other instances to purge if necessary.

save

path: 0 784842 786368 787321, hotness: undefined, score: 24, children: 1

punio7 3 points 3 years ago

Exactly, when dealing with big data, you need a strategy to archive old data. You can't just store everything in one DB. Smaller instances may not feel like keeping all the date from all the time. Even big instances should have a mechanism to move old data do different databases.

save

path: 0 784842 786368 787321 805390, hotness: undefined, score: 3, children: 0

HKayn 4 points 3 years ago

Are you planning on donating to instances that don't purge old data?

save

path: 0 784842 786368 806834, hotness: undefined, score: 4, children: 0

tugg 15 points 3 years ago

Small instance with about 3 users and myself online for about 2 weeks.

pictrs   930M
postgres 1.4G

save

path: 0 785186, hotness: undefined, score: 15, children: 0

key 14 points 3 years ago

Depends. If you have a lot of users posting a lot of pictures and you use pictrs out of the box config, then a lot. If you are just running a few users with finite communities being synced then a lot less. The number is going to vary a lot as lemmy grows and gets older so hard to document realistic expectations. But docker images are probably going to take up more disk space than actual contents unless you get quite big. I just threw my PG volume into a tgz to move servers and it's less than a gig.

save

path: 0 784547, hotness: undefined, score: 14, children: 1

bdonvr 6 points 3 years ago

The lemmy.world admin said above that their instance currently takes up less than 100GB

Though this will accrue over time I suppose.

But for self hosting? You should be good for a long long time. The only pictures stored are the ones you upload, the rest is just text.

save

path: 0 784547 786860, hotness: undefined, score: 6, children: 0

i_lost_my_bagel 10 points 3 years ago

lol lemmy died almost immediately after i posted this time to figure out what the hell caused that

save

path: 0 785221, hotness: undefined, score: 10, children: 4

i_lost_my_bagel 14 points 3 years ago

it was because i set a damn server icon

save

path: 0 785221 788760, hotness: undefined, score: 14, children: 3

b3nsn0w 5 points 3 years ago

lmao just how powerful is your server icon?

save

path: 0 785221 788760 806418, hotness: undefined, score: 5, children: 2

i_lost_my_bagel 3 points 3 years ago

this powerful

save

path: 0 785221 788760 806418 815120, hotness: undefined, score: 3, children: 1

b3nsn0w 3 points 3 years ago

lol, yeah, that would crash any instance

(jokes aside, you'll probably need to keep it somewhat low-res, and i'd also recommend cropping it to square. my instance uses a 128x128 icon)

save

path: 0 785221 788760 806418 815120 815225, hotness: undefined, score: 3, children: 0

holycrap 10 points 3 years ago

Unless they changed all of the comment and post ids to bigints that'll probably bring the site down before it runs out of storage. In defense of the lemmy developers they have been receptive to feedback, so I don't think it'll take long for that to be fixed if it hasn't already.

save

path: 0 785904, hotness: undefined, score: 10, children: 0

hitagi 9 points 3 years ago

My instance eats up almost 100MB everyday. It mostly depends on what your users subscribe to. It was barely growing on my first few days until I invited a couple of friends over to try it out.

save

path: 0 790718, hotness: undefined, score: 9, children: 0

HKayn 7 points 3 years ago

My instance dormi.zone has been running for around 3½ weeks now, has a 3-digit amount of users and hosts a community with little more than 1000 subscribers. Here's how much storage it currently takes up:

6.2 GiB postgres
4.9 GiB pictrs

In the default Ansible configuration, storage will mostly be accumulated by log files that are automatically generated by Docker and deleted whenever you restart the Docker containers.

save

path: 0 837818, hotness: undefined, score: 7, children: 0

lightrush 7 points 3 years ago

How many cans-of-beans.jpg can you store?

save

path: 0 824775, hotness: undefined, score: 7, children: 1

i_lost_my_bagel 3 points 3 years ago

At least 3. Maybe 4.

save

path: 0 824775 826964, hotness: undefined, score: 3, children: 0

lemmy 6 points 3 years ago

After hosting my own instance with just me for ca. 2 weeks:

1.99Gi pictrs

5.21Gi postgres

save

path: 0 785459, hotness: undefined, score: 6, children: 0

GustavoM 2 points 3 years ago

I haven't tried it out just yet, but I'd say... a whole lot. Depending of how popular your instance is, but... your PC will be hammered in any way.

save

path: 0 787417, hotness: undefined, score: 2, children: 0

jukes 1 point 3 years ago

It depends on how many communities you end up pulling in. Your instance will only sync with communities that a user on your instance is subscribed to.

save

path: 0 1670594, hotness: undefined, score: 1, children: 1

jukes 1 point 3 years ago

I've had my instance running for about 1 week and I'm the only user.

2.1G pictrs

2.5G postgres

save

path: 0 1670594 1680535, hotness: undefined, score: 1, children: 0

JoShmoe -20 points 3 years ago

Holding onto all that data is pointless if you’re not selling it to someone.

save

path: 0 786550, hotness: undefined, score: -20, children: 6

AFKBRBChocolate 9 points 3 years ago

I disagree. One big hunk of value of a place like this is being able to look back at old threads. How many times did people say they always put "Reddit" in front of their Google searches to get the information they were looking for? This could be the same.

save

path: 0 786550 787378, hotness: undefined, score: 9, children: 3

nan 3 points 3 years ago

That’s a good reason for an instance to put “lemmy” in its url too, I imagine. Search engines are already returning Lemmy results for things.

save

path: 0 786550 787378 789824, hotness: undefined, score: 3, children: 0

JoShmoe 1 point 3 years ago

That’s unsustainable. Why do you think the mainstream platforms are selling out?

save

path: 0 786550 787378 820742, hotness: undefined, score: 1, children: 1

AFKBRBChocolate 1 point 3 years ago

It's really not, at least for the text part. Text posts and comments take almost nothing and storage continues to get cheaper.

Mainstream platforms are selling out because they've always had others and shareholders who ultimately want to make money.

save

path: 0 786550 787378 820742 826844, hotness: undefined, score: 1, children: 0

Molecular0079 2 points 3 years ago

Info is still useful for people doing google searches. It would be nice to be able to find common troubleshooting tips on Lemmy, etc.

save

path: 0 786550 787352, hotness: undefined, score: 2, children: 1

JoShmoe 1 point 3 years ago

Not everything posted here holds any value.

save

path: 0 786550 787352 820691, hotness: undefined, score: 1, children: 0