That post seems to be missing.
tiptoes 52 points 4 months ago

The equivalent would be either zfs or btrfs compression. Transparent to applications, you don’t have to do anything special other than enable it.

path: 0 23423117, hotness: undefined, score: 52, children: 7
vortexal 10 points 4 months ago
path: 0 23423117 23423230, hotness: undefined, score: 10, children: 6
tiptoes 12 points 4 months ago

How small are these devices? I think the other problem is that neither BTRFS nor zfs really are suitable for removable devices, and definitely not for ones smaller than probably 8Gb at the very least.

Unlike NTFS which is just a file system, both BTRFS and ZFS do volume management too, so it’s not just a single partition thing; they prefer to take over an entire volume and manage everything.

So while they’re the closest filesystem with NTFS-like transparent compression…..they don’t match exactly.

I also hazard to guess if the devices you’re using are too small to accept a BTRFS formatted volume, no amount of compression is going to be enough to fit what you need.

If you just want to play with a bunch of small old devices…..maybe play with LVM and small RAID arrays and configurations instead. You can the build a bigger volume out of a bunch of those disks together and then put a BTRFS or zfs volume on them. Can be fun to experiment and learn with anyway.

path: 0 23423117 23423230 23423855, hotness: undefined, score: 12, children: 0
BananaTrifleViolin 7 points 4 months ago

How small is your smallest device? BTRFS doesn't have a minimum size, but practically probably 50-100mb is just about doable before even just setting things up get complex. Having said that though, it's copy-on-write and has overhead as a result, so may not function well below 1gb.

ZFS meanwhile really won't work well below probably 8gb. It's also copy-on-write but with a lot more overhead due to how it works. It really works best on big drives and filesystems.

If your old storage is in the mb range, then really neither will help you achieve what you want.

BTRFS and ZFS do offer the same benefits as NTFS with regard to compression and speeding up some slower devices (due to lowering the actual read/writes needed to achieve the same result as the data is compressed into a smaller space and decompressed rapidly by the PC in memory), but NTFS can go be used on much smaller disk sizes due to how it works. BTRFS and ZFS are designed and optimised with other benefits in mind. And NTFS compression isn't well supported in Linux.

path: 0 23423117 23423230 23423829, hotness: undefined, score: 7, children: 0
ivn 6 points 4 months ago path: 0 23423117 23423230 23426023, hotness: undefined, score: 6, children: 2
vortexal 2 points 4 months ago
path: 0 23423117 23423230 23426023 23426110, hotness: undefined, score: 2, children: 1
ivn 6 points 4 months ago

You'll need to read the documentation to understand its concepts first. You don't use mkfs but zpool to create a pool then use zfs to create the fs on it. ZFS is usually used with some sort of raid configuration so most doc will show this but you can create a pool with no replication using a single drive.

path: 0 23423117 23423230 23426023 23426110 23426379, hotness: undefined, score: 6, children: 0
daggermoon 6 points 4 months ago

PM me if you need help with ZFS. I've gotten quite good at setting it up.

path: 0 23423117 23423230 23425487, hotness: undefined, score: 6, children: 0
daggermoon 19 points 4 months ago

You could a filesystem that supports compression like ZFS or BTRFS.

path: 0 23425469, hotness: undefined, score: 19, children: 0
Shimitar 13 points 4 months ago

This is at file system level... Checkout btrfs and zfs, I am quite positive both can compress like you want.

Never used this feature myself, so cannot be more specific.

Also, there are some read-only compressed filesystems for Linux that you can also use, they offer best compression but data is read only.

path: 0 23422968, hotness: undefined, score: 13, children: 32
vortexal 2 points 4 months ago
path: 0 23422968 23423220, hotness: undefined, score: 2, children: 30
Shimitar 9 points 4 months ago

How small is that device??? Didn't even knew btrfs has a minimum size.

After for zfs, you probably have to install some software or kernel patches

path: 0 23422968 23423220 23423309, hotness: undefined, score: 9, children: 28
vortexal 5 points 4 months ago
path: 0 23422968 23423220 23423309 23423540, hotness: undefined, score: 5, children: 27
hendrik 6 points 4 months ago

Huh. My computer allows me to format a 128MB image file with brtfs. It won't do it at 64MB though.

path: 0 23422968 23423220 23423309 23423540 23423575, hotness: undefined, score: 6, children: 14
edinbruh 4 points 4 months ago

If it's a flash memory (sd card, usb stick, ssd, etc), you could try f2fs, it's very light, and it supports compression and is meant specifically for that kind of devices (well, more for ssds).

But judging your experience from your comments, I suggest you don't delve into niche filesystems until you have more experience with Linux, especially for something like 128MB. I especially suggest you avoid zfs for now.

path: 0 23422968 23423220 23423309 23423540 23424065, hotness: undefined, score: 4, children: 8
edinbruh 1 point 4 months ago
path: 0 23422968 23423220 23423309 23423540 23423994, hotness: undefined, score: 1, children: 0
xthexder 1 point 4 months ago

Damn, where did you even find such a small drive? The last time I used such a small device was the USB stick I brought to highschool in 2009. Even the free giveaway USB drives I have are at least 2GB. You probably have more RAM than storage?

path: 0 23422968 23423220 23423309 23423540 23437143, hotness: undefined, score: 1, children: 1
Shimitar 6 points 4 months ago

Xfs and zfs are two different filesystems.

path: 0 23422968 23423220 23423318, hotness: undefined, score: 6, children: 0
vortexal 1 point 4 months ago
path: 0 23422968 23422994, hotness: undefined, score: 1, children: 0
Diplomjodler3 6 points 4 months ago

Just download more RAM.

path: 0 23423428, hotness: undefined, score: 6, children: 0
undrwater 6 points 4 months ago path: 0 23442557, hotness: undefined, score: 6, children: 5
vortexal 2 points 4 months ago
path: 0 23442557 23442628, hotness: undefined, score: 2, children: 4
undrwater 2 points 4 months ago path: 0 23442557 23442628 23444469, hotness: undefined, score: 2, children: 3
vortexal 1 point 4 months ago
path: 0 23442557 23442628 23444469 23444823, hotness: undefined, score: 1, children: 2
undrwater 1 point 4 months ago

The Gentoo link I provided had instructions. Use the stack exchange for the read-write.

Good luck!

path: 0 23442557 23442628 23444469 23444823 23445264, hotness: undefined, score: 1, children: 1
wltr 5 points 4 months ago

Not sure that’s relevant, but I’m playing around with Linux on some obsolete Windows tablets with cheap eMMC disks (one of which is broken, so I replaced it with a no name microSD card, plus USB drive), and I format my disks to f2fs, which theoretically should help with both keeping the disks for longer, and accessing the data faster.

path: 0 23423626, hotness: undefined, score: 5, children: 7
vortexal 4 points 4 months ago
path: 0 23423626 23424474, hotness: undefined, score: 4, children: 6
Natanael 5 points 4 months ago

Make sure you're actually filling the volume, and also keep in mind reporting may be different (with or without filesystem index metadata, etc)

Also, you can simply use regular file systems and compressed files, and then use a RAM drive (assuming you have enough RAM free) and access the files that way instead

path: 0 23423626 23424474 23424690, hotness: undefined, score: 5, children: 5
vortexal 1 point 4 months ago
path: 0 23423626 23424474 23424690 23424780, hotness: undefined, score: 1, children: 4
Natanael 1 point 4 months ago

RAM drive is when you hold the entire file system in RAM, it's used for stuff like Linux Live CD boot with no writable storage but you can use it for anything.

path: 0 23423626 23424474 23424690 23424780 23426318, hotness: undefined, score: 1, children: 3
MonkderVierte 5 points 4 months ago

What do you mean, btrfs "takes too much space"? That's not a sentence that belongs to file system stuff in any way.
The disk is the space. You make it usable with a fs, for which you can partition portions of it.

And also, i've had bad experiences with f2fs long-term stability.

path: 0 23431196, hotness: undefined, score: 5, children: 8
vortexal 2 points 4 months ago
path: 0 23431196 23438762, hotness: undefined, score: 2, children: 7
Hupf 2 points 4 months ago path: 0 23431196 23438762 23445613, hotness: undefined, score: 2, children: 6
vortexal 1 point 4 months ago
path: 0 23431196 23438762 23445613 23445747, hotness: undefined, score: 1, children: 2
Hupf 1 point 4 months ago

You may have to clear it first.

Check dmesg output for usually more verbose mount errors.

Also I'd use compress-force and test out the other compression algorithms with your data set.

path: 0 23431196 23438762 23445613 23445747 23446132, hotness: undefined, score: 1, children: 1
vortexal 1 point 4 months ago
path: 0 23431196 23438762 23445613 23445932, hotness: undefined, score: 1, children: 2
Hupf 1 point 4 months ago

Mind you that "free space" figures with compression, with btrfs and with such tiny file systems are tricky at best. You'd have to test all options by actually copying your data set first.

path: 0 23431196 23438762 23445613 23445932 23446157, hotness: undefined, score: 1, children: 1
dabu 5 points 4 months ago

There's an interesting project called DwarFS. I have it on my todolist to check it out but as I understand it you basically create a compressed read-only archive that is mountable and readable just like any other disk - https://github.com/mhx/dwarfs . Maybe this is something for your use case?

path: 0 23423007, hotness: undefined, score: 5, children: 4
vortexal 1 point 4 months ago
path: 0 23423007 23423052, hotness: undefined, score: 1, children: 3
bjoern_tantau 3 points 4 months ago

Funnily enough it's often used in the pirate scene to distribute games. But only as archives.

path: 0 23423007 23423052 23424480, hotness: undefined, score: 3, children: 0
dabu 1 point 4 months ago

Unfortunately I haven't tested it so I cannot fully advise on that but I have seen some "repacks" on the high seas where a whole wine prefix with a game already installed was compressed to a single dwar which you can mount and play with configs and saves in your ~.

path: 0 23423007 23423052 23423170, hotness: undefined, score: 1, children: 1
vortexal 1 point 4 months ago
path: 0 23423007 23423052 23423170 23423265, hotness: undefined, score: 1, children: 0
Ftumch 5 points 4 months ago

Sounds like you're looking for something like archivemount.

path: 0 23424849, hotness: undefined, score: 5, children: 3
vortexal 3 points 4 months ago
path: 0 23424849 23424918, hotness: undefined, score: 3, children: 2
xthexder 1 point 4 months ago

Could that be due to a failing SD card or flash memory? The uncompressed data might be getting corrupted too, it just wouldn't return you an error depending on the file type (text documents for example would never error, they would just end up with garbled text).

Based on your other comments, if you're using an old 128mb SD card, this seems kind of likely. I've had many SD cards and USB sticks go bad after files have been copied on and off of them too many times.

path: 0 23424849 23424918 23437275, hotness: undefined, score: 1, children: 1
vortexal 1 point 4 months ago
path: 0 23424849 23424918 23437275 23438326, hotness: undefined, score: 1, children: 0
tychosmoose 4 points 4 months ago

What are you saving on that drive? Many data file formats already have compression of their own and don't benefit much from file system compression. So if this is for media files, for example, it's likely to add CPU overhead without a big benefit in transfer speed.

ZFS is not installed by default with most Linux distributions due to its license. It's something you install after the os. Btrfs should work, but I see some discussion online of 128 or 256MB minimum volume size.

path: 0 23423773, hotness: undefined, score: 4, children: 1
vortexal 3 points 4 months ago
path: 0 23423773 23424177, hotness: undefined, score: 3, children: 0
corsicanguppy 4 points 4 months ago

The Compress attribute has been in even ext3 since day 1. I've never tried it, though.

path: 0 23423983, hotness: undefined, score: 4, children: 1
vortexal 3 points 4 months ago
path: 0 23423983 23424235, hotness: undefined, score: 3, children: 0
ZkhqrD5o 2 points 4 months ago

sudo nano /etc/fstab

Then replace "defaults" with "compress=zstd" on your desired partitions.

IMO stick with btrfs. Also, the storage space between file systems is exactly the same, given the same hardware.

path: 0 23435935, hotness: undefined, score: 2, children: 3
vortexal 1 point 4 months ago
path: 0 23435935 23438526, hotness: undefined, score: 1, children: 2
ZkhqrD5o 1 point 4 months ago

The fstab file is used to define how disk partitions or remote file systems are mounted into your computer. Removable drives such as USB drives or SD cards are not shown there, because if they were, your system would complain at boot that it can't find the requested USB drive.

About F2FS having double the storage space of btrfs. In all honesty, it doesn't make sense to me. Do you mean it shows you potentially being able to store 80 megabytes if you can get compression working or does it just show 80 megabytes instead of 40, after formatting the file system?

path: 0 23435935 23438526 23441314, hotness: undefined, score: 1, children: 1
vortexal 1 point 4 months ago
path: 0 23435935 23438526 23441314 23441557, hotness: undefined, score: 1, children: 0
monovergent 2 points 4 months ago

F2FS seems to do what you want, it'll reserve the original size of your file but compress what's actually written. Performance numbers might be massively inflated if your writes don't saturate the cache in RAM.

I've used BTRFS on SD cards before and it's mostly fine, but it will struggle massively if over 90% full, or if you have a < 1 GB volume and are, say, frequently updating a handful of files that together take up more than half its capacity. Mostly due to the CoW mechanism, it needs some headroom to make a copy of whatever files are being modified.

path: 0 23424958, hotness: undefined, score: 2, children: 2
vortexal 2 points 4 months ago
path: 0 23424958 23425056, hotness: undefined, score: 2, children: 0
vortexal 2 points 4 months ago
path: 0 23424958 23425316, hotness: undefined, score: 2, children: 0
HubertManne 1 point 4 months ago

Never really had the need. My linux install already takes up way less space than my old windows one so I have much more space.

path: 0 23425995, hotness: undefined, score: 1, children: 6
vortexal 1 point 4 months ago
path: 0 23425995 23426047, hotness: undefined, score: 1, children: 5
greedytacothief 5 points 4 months ago

Probably been mentioned already, but ZFS and BTRFS have options for full disk compression. Though there is cpu overhead with higher compression settings.

path: 0 23425995 23426047 23426394, hotness: undefined, score: 5, children: 1
vortexal 1 point 4 months ago
path: 0 23425995 23426047 23426394 23426466, hotness: undefined, score: 1, children: 0
utopiah 0 points 4 months ago

running games on old storage devices that hold less than 32 GB and have terrible read and write speeds.

I'm confused. How about copying that to newer storage devices?

Also typically game assets are VERY well compressed so I would suggest doing a comparison with/without compression before a full on migration. Compression tools help but aren't magical. If your assets are e.g. .jpg or .mp4 or .mp3 or a combination of that (as typically game assets are, including 3D models with their textures) then you can test yourself to .zip them (or bzip2 or whatever you prefer) and you will seem some gains but they'll be nearly negligible, e.g. < 10% reduction.

path: 0 23425995 23426047 23434812, hotness: undefined, score: 0, children: 2
vortexal 1 point 4 months ago
path: 0 23425995 23426047 23434812 23438682, hotness: undefined, score: 1, children: 1
utopiah 1 point 4 months ago

You don't have to share your personal situation and I'm sorry to read that you are struggling. My point isn't to argue that you must do like everyone else or that consumerism is good, rather than in the typical case (not a lot of time, hardware getting cheaper of the year, game assets being compressed already) switching to newer hardware is a much much more convenient solution. That's why I warned about it.

path: 0 23425995 23426047 23434812 23438682 23445514, hotness: undefined, score: 1, children: 0
doodoo_wizard 1 point 4 months ago

My “I don’t need to save space, I want it for a different purpose” tee shirt is raising a lot of questions answered by my “I don’t need to save space, I want it for a different purpose” tee shirt.

path: 0 23428309, hotness: undefined, score: 1, children: 1
MonkderVierte 1 point 4 months ago
path: 0 23428309 23431131, hotness: undefined, score: 1, children: 0
non_burglar 1 point 4 months ago

It's not clear here if you mean block compression, file compression, or stream compression.

path: 0 23426941, hotness: undefined, score: 1, children: 3
vortexal 1 point 4 months ago
path: 0 23426941 23426996, hotness: undefined, score: 1, children: 2
non_burglar 1 point 4 months ago

BTRFS and ZFS both use block compression, ZFS by default. It's meant to increase both storage efficiency and access speed and has nearly zero impact on performance. The files aren't compressed from the filesystem point of view, which would satisfy your requirement not to need any other tools.

I'm confused about your statement that you don't want to save space with compression, but you indicated that you want to "make the most of your storage". Are you looking for long-term archiving?

path: 0 23426941 23426996 23427808, hotness: undefined, score: 1, children: 1
vortexal 1 point 4 months ago
path: 0 23426941 23426996 23427808 23427930, hotness: undefined, score: 1, children: 0
chilltrek97 0 points 4 months ago
path: 0 23431456, hotness: undefined, score: 0, children: 1
vortexal 1 point 4 months ago
path: 0 23431456 23438844, hotness: undefined, score: 1, children: 0
jobbies -1 points 4 months ago

I just want to say that if you are going to either resort to belittling me

That's like a red rag to a bull

you haven't fully read the post before commenting

You could've just tldr'd it. You used 4 paras to ask 'how do I compress files/my fs'

I'm not going to respond to your comment

Well that was a big fat lie, wasn't it.

path: 0 23446106, hotness: undefined, score: -1, children: 0
mina86 -9 points 4 months ago
path: 0 23423193, hotness: undefined, score: -9, children: 7
vortexal 5 points 4 months ago
path: 0 23423193 23423255, hotness: undefined, score: 5, children: 6
mina86 -6 points 4 months ago
path: 0 23423193 23423255 23423512, hotness: undefined, score: -6, children: 5
Aatube 1 point 4 months ago

the device I'm testing first is too small for btrfs

path: 0 23423193 23423255 23423512 23424863, hotness: undefined, score: 1, children: 4
mina86 0 points 4 months ago
path: 0 23423193 23423255 23423512 23424863 23425665, hotness: undefined, score: 0, children: 3

thanks for using Leebra!

go to feed...