introducing copyparty, the FOSS file server
a year ago by tripflag to c/selfhosted
I made a video about copyparty, the selfhosted fileserver I've been making for the past 5 years.
The main focus of the video is the features, but it also touches upon configuration. Was hoping it would be easier to follow than the readme on github... not sure how well that went, but hey :D
This video is also available to watch on the copyparty demo server, as a high-quality AV1 file and a lower-quality h264.
SFTP is not currently on the roadmap, but it's not entirely implausible.
FTPS is supported, but it requires an optional dependency to be installed (pyopenssl), so it's not available in the Windows EXE. And I just realized that the dependency is currently not present inside the docker images either, so I'll get that fixed right away.
That's a neat idea -- I've heard that a lot of stuff uses the subsonic API under the hood, so I'll see what it would take to become compatible with that. At first glance it looks like I'd have to mine and index way more information about audio files, but could still be doable :>
Hey fellow scener, cool project!
Just a few thoughts/questions:
BTRFS and ZFS support real deduplication via copy on write, and would eliminate all current disadvantages of symlink and hardlink deduplication. It just works.
yeah that's a good point, I'll add an option to take advantage of this if you know you're running on a filesystem where that works as intended.
Why have it be one huge python source file?
oh don't worry, it's all separate files during development -- there's a build-stage which bundles everything up into a single file for distribution. But thanks for the concern :D
Ah, so you have compiled it into one file? Didn't know that was possible for python, what tool do you use for this?
sooo this is one of the things that started with someone saying "wouldn't it be funny if..."
if you open copyparty-sfx.py in a text editor, you'll see how -- but please make sure to use an editor which is able to handle about 600 KiB of comments which contain invalid utf8 / binary garbage ๐
I ended up rolling my own packer since I wanted optimal encoding efficiency, and everything I could find would do stuff like base85 or ucs2 tricks, but it turns out python is perfectly happy with binary garbage in comments if you declare that the file is latin-1 so it realizes all hope is lost :D
the only drawback of the sfx.py is that it needs to extract to $TEMP before running, so that's the slight advantage of the zipapp (the .pyz alternative), but that suffers from some performance reduction in return, and is more hermetic (doesn't let you swap out the bundled dependencies with fresh versions as easily if necessary)
Ah, reminds me of the old self-extracting gzip executable trick. I used that once a very long time ago to make a 4k linux intro, before I realized to be competitive I should switch to windows to be able to use Crinkler, which is superior even though the decompressor is part of the executable.
Why not use zipapp?
What do you use to bundle into one file?
copyparty-sfx.py is a custom packer (see this reply) created by make-sfx.sh, and copyparty.pyz is a standard zipapp, created by make-pyz.sh. The zipapp has more disadvantages than the sfx.py, so that's the default/recommended build.
Just a remark from someone who runs ZFS since the beginning. Many people don't like the deduplication feature because of its memory footprint.
It's also nice to have this feature without relying on a certain filesystem.
I assume you mean automatic deduplication? I haven't used ZFS, but BTRFS does not have that. There are a variety of ways to perform deduplication, I have duperemove scheduled to run regularly.
If ZFS is still capable of being instructed to perform deduplication when automatic deduplication is turned off, which it really should be able to do, then this should work even with it turned off.
The fact you mention security features, without ever saying it's 'super secure' tells me you know a lot about what you're doing. I'm so sick of apps like this that start with "most secure app on the net" but you know they're delusional. Thank you, going to check this out.
so uhh, sorry for the late response to this -- was going to reply much earlier, but then suddenly it became more timely than ever...
the good news is, I'm fairly confident in how it handles the filesystem and permissions, preventing unauthorized access to files.
but the part I'm a bit less sure about is sanitizing user data; the kind of vulnerabilities where someone uploads a malicious file and bad stuff happens if you then open that file in a certain way, or someone sends you a malicious link and trick you into clicking it -- in other words, the kind of vulnerabilities which require the attacker to have a certain level of access already, or that requires tricking you into doing something.
...and with version 1.18.5 released just now, we got a prime example of exactly one of those. Really unfortunate timing, but it's a blessing to have so many new and curious eyes on it to spot these sooner rather than later. It is what it is.
Yup, I tried to run the docker image with the suggested docker command and it errored out for lack of a config file (though it did offer a fix in the logs for mounting the current directory as read/write)
Would also be willing to help translate the UI.
I agree but it's still in an early development state. Not really usable for everyday work let alone most people never heard about it ๐ But yeah still cool to mention it under "modern" browsers. I wish them good luck with the first alpha next year. I hope it'll be successful.
Holy shit this is so unbelievably full featured it's not even funny. This is going to rattle some big names. I wonder how long till they try to shut it down because it's better than anything commercially available.
I haven't even tried it yet, but just from the video you can tell it's going to be insanely good. I'm so impressed.
It's the first bit of software I've seen in a long time where I took one look and immediately thought "Fuck me, I need that!"
I use Unraid for my NAS server and just on the off-chance I checked the Unraid community 'app store' and someone's already created a Docker definition for it, published just today! The hype is real
I'll be giving this a shot
That wonโt work for me since I have multiple people using this Nextcloud instance, and I also use it to publicly share files. I also have a big network share (currently 5 TB of data) that I would like to better integrate somehow (right now itโs available from Nextcloud read-only in essentially โanonymous modeโ with no access to private user files).
I want something with a permission model that works the same when accessed over a network share (SMB, ideally NFS) and access over a web interface. Ideally it would have a Mac File Provider sync client and whatever the Windows equivalent is called as well.
Nextcloud is fine but itโs not that.
Nextcloud is like Windows 95, it works great when you install it then it just keeps getting slower as you fill it with content
They recently recieved a "Blauer Engel" certificate for nextcloud-server. Iirc that seal requires the user to be able to choose what he wants. So new features are OK but they have to be optional. That certificate is handed out by a german government Institution. And very recently they started handing it out to software too, with nextcloud beeing the first BIG reciever.
I bumped into copyparty the other month when looking for a software that could let me transfer files to a friend with the ability to pause/resume. Didn't bother with it, tried another software instead. Never really got it to work so I gave up on it.
Bumped into the YouTube video today, decided to give copyparty a shot, damn sir you've written a fine piece of software. It's so easy to get up and configure. The UI is a bit janky, but charming at the same time. Thanks for all the hard work!
"It's janky. It's charming. It works. It's Copyparty." ยฎ
This looks great but I really wish it had SFTP.
no, CP is copypasta, the thing you put into your food boxes.
As long as it's not causing any issues or drawbacks for modern python versions (and it isn't), I don't see any reason to do that -- on the contrary, I know people are running copyparty on retro equipment, so I'd very much prefer to keep it for as long as possible :>
Just because something is EOL doesn't make it useless.
So long as it's known and managed, it's fine.
I'm properly in awe at this project. Not only does it support a wide range of protocols and runs practically everywhere, but it can play audio, video, display images and has keyboard shortcut support ???
It's got everything I could think about and so much more.
I can't wait to try it out !
Thank you so much for your time and for making this open source.
Elderly raspberry pi B [โ]
Large portable drive gathering dust [โ]
Guess I'm setting up a locally hosted file server in the near future.
Ho... Ly... Shit... This is great! The UI is a bit confusing at first but doesn't take long to get what's going on. I might even be disappointed with a UI revamp ๐ I can't believe how much functionality this has. It's already replacing some processes I have for mounting drives and backing up files. Maybe I missed something, but my only complaint would be the lack of an automatic one-way folder sync in the Party UP! app.
I'm blown away, great job!
That "functional but charming" UI reminds me of audiobookshelf with the soundleaf app - same vibe where it dosn't look fancy but gets the job done better than most polished alternatives!
I have a question, and I want to emphasise thar this is not criticism but a request for dive into technicalities.
In the video you mentioned copyparty has an one-way sync tool. Is there a good reason why it's not two-way, or is this just something you weren't motivated to do?
No worries, good question :>
The problem with bidirectional filesync is that it's an absolutely massive can of worms, very easy to mess up, and the consequences of messing up are usually the worst kind (loss of data). There's an insane amount of edgecases to keep in mind, and you need to get every edgecase right every single time, otherwise you might wipe someone's vacation photos, or suddenly downgrade someone's keepass database to an older version... And stuff like syncing multiple devices to the same server makes it balloon further.
I've started becoming more confident in copyparty's filesystem-index database, but it's still just a hint/guideline, with the filesystem being the only source of truth -- it's still not something I'd trust with tracking sync-state against one or more clients.
The bigger guys who offer bidirectional sync (nextcloud, syncthing, etc.) have spent years perfecting their logic, so I'd like to leave this in their capable hands.
Thank you for your answer! Do you think copyparty would work together with Syncthing on the same backing directory, or would they compete for changes etc? Copyparty in this scenario would be for sharing content with friends and occasional remote upload
Great software!
Oh my gawd what a README!! I'm on my phone and I was trying to scroll back to the top of it from the bottom and I just kept on scrolling... Holy shit I'm going to put this on my kanban board give it proper attention
Well done! This is exactly what I've been looking for! And it's fun to use and has an aesthetic that reminds me of browsing ftp servers back in the day. And these themes?! I love the whimsy mixed with actual usability.
And its a damn good file server so far: fast, easy to use, makes reasonable security choices, and accessible in the ways I want (rclone mount)
Don't believe it. This is easily too good to be true.
Great job on something like this! I'll probably give it a whirl soon, I like Nextcloud but find it clunky sometimes because it's often a bit more than I need. Maybe breaking it up into Immich + this would help! Thank you for sharing your project!
One thing to note, your comparison against Nextcloud has a partially-incorrect point regarding file upload max size. The client does upload chunking, so is unaffected by the Cloudflare issue as well, but I believe the web client is still affected, just not the apps. https://docs.nextcloud.com/...
I suspect a few others may be as well, but I'm only familiar with the Nextcloud one because that's what I've been running, and discovered in making sure I could still upload video files recorded while out and about.
Also love that it looks like a simpler install!
Sometimes I feel so new to setting up my own digital ecosystem because I look at a thing and think "that's so cool" but struggle to imagine it at home. So could someone help me understand.
This would be a replacement for something like Google Drive or Proton Drive? The actions I would use this for would be:
So I would spin this up on my NAS or my main PC and replace those services and accomplish those actions using this software?
Are there other services or actions I'm missing? Am I misunderstanding the premise entirely?
Oh yea, copyparty could do that. I might just do that too. My issue is more how do I grant them access to my network to get the thing tough? I currently use wireguard profiles and lock down where they can reach with rules and shit on a firewalla on a per account basis but thatโs really complex and inelegant. It works and would working copypasta, but I kind of wish there was a simple webUI where I could define what a WireGuard user should be able to reach on my network with simple checkboxes by rules I have created over time. Probably wouldnโt tie into firewalla nicely though it could be more likely with OPNsense.
Hmm. Surely someone must have thought of that already. It would make adopting things like copypasta much simpler and less risky.
I think Copyparty would be great for that purpose. The only thing you're missing is a way to expose it to the internet, such as a public IP or some tunnel
Yep! Depending on what your home connection looks like, you have a few options:
if you are lucky enough to have your own private IP-address and are able to open ports, then you're almost done already -- you can put copyparty on some port (or keep the default 3923), and then anyone could connect to it by going to https://your.ip.address:3923/
(with this approach, you will want to create your own HTTPS certificate so the traffic is properly encrypted -- the best option here is to get a domain and get a certificate for the domain)
however, if you are behind CGNAT, meaning your internet provider has given you a shared IP-address, then people cannot connect directly to your home-PC. One way around that issue is by setting up a machine somewhere on the internet which bridges the gap back home to your PC. Cloudflare offers this as service, and this is explained in the copyparty readme -- see the "at home" section for one way to do that.
if you are against using Cloudflare for idealistic reasons (they are becoming quite powerful since they run a whole lot of the internet), then you can set up a cheap VPS which serves the same purpose. That's my setup, and how you are accessing the copyparty demo server right now -- I have the cheapest VPS you can get from Hetzner. The VPS is running nginx, and it forwards the traffic to my homeserver through an SSH tunnel. I haven't documented this approach in the copyparty readme, but I have a feeling a lot of other people have :>
I've been using copyparty for months, and it's just been an absolute gem. and the fact there are even iOS shortcuts to upload things you stumble across while doomscrolling is just icing on top.
This is very impressive and I'm highly likely to give it a whirl. My question is, though: would it be something that my very non-tech savvy wife could use?
Eg. I'm thinking setup the app on her phone with a default location and when she asks me for a file I can just tell her that I've "put it in the app", and she'll be able to easily retrieve it. Also same thing but vice versa, though the video seems to cover that via the Android share menu...
Again, super impressive. Good job!
I have a hunch that the true answer, to be honest, is "no" -- at least with the current UI as it is. I've come to terms with not being the best at making intuitive user interfaces, so I went all-in on making it poweruser-friendly and efficiency instead.
Yeah, there's the android app for sending files to the server, and it'll always send files to the same folder, so that part should be pretty solid. But actually grabbing files from the server, perhaps not so much. Not sure I'd risk it, but I'll leave the decision to you hehe
I'm not aware of any user-friendly android/iOS apps for connecting to a webdav / ftps / sftp server, but if those exist, then that would probably have been a good option!
Been using DAVx5 on Android with it (you don't set it up as a full account, just a WebDAV mount) and the full exported filesystem shows up in the android file browser (as a remote mount like google drive)
Haven't looked at the project yet, but that's just the greatest name for a fileserver...
I have a samba fileserver behind wireguard. Would copyparty be an improvement over samba?
If all you need is basic remote file storage such as a samba server, and especially if you need samba in particular, then your current solution is probably a better fit for you.
Copyparty's main selling points is the large number of features in one package, and being pretty good at receiving file uploads (usually faster than other alternatives), but it does not have good samba support. Instead of samba, copyparty has WebDAV support, so you can still connect to it from your file manager -- but the performance will be different; depending on your access pattern and the type of files, it could be faster or slower than samba.
I like samba because I can mount it as a folder in the file explorer of my Linux desktop. But Samba is a little slow.
Why use smba when there's NFS?
Thank you so very much. I needed to transfer some files locally yesterday and didn't have a NAS set up. So I thought I would quickly set up a samba share on my pop-os media server so my spouse (windows) and I (mint) could swap/store files on the fly in the future. It took me 25 minutes and I almost gave up because my config file was having issues.
This looks sooooo much nicer. May every road rise up to meet you
I would add PairDrop to your list to have bookmarked. It's completely web-based so no download required and thus fully cross-platform. It also works across different networks (i.e. over the internet) by pairing devices or creating a room. Basically Apple AirDrop, but universal and on steroids.
Bookmarked, thanks. I'll have to try them all out at some point and see what works quickest. Obviously the storage aspect of the poster is a different ball game, but out and about this sounds useful
That's for the recommendation. That could be useful as well!
Very sleek project. The language switcher bit was brilliant hahaha. Seriously, good job.
man... that's really unexpected, I went with h264+mp3 which should be the most conservative / broadly-supported combination you could possibly use, yet still (๏พ ๏พใฎ๏พ)๏พ ~โปโโป
what webbrowser / device / player are you using?
This is me browsing slrpnk.net using the voyager app,on iOS latest version on iPhone 14 Pro, it then opens it in whatever built in player the app uses. Other versions work fine though. This app has me excited as I consider starting self hosting!
Iโm in the market for a self hosted file server so I can use it as a destination for website backups. Absolutely going to give this a look next week. Thanks for posting about it!
Ok this is too damn cool. Will be checking it out soon, but moving to a new apartment this week so I probably shouldn't dive into new distractions just yet...
OMG I just posted this to lemmy and saw it had already been posted. Great work! Amazing video!!
Looks fantastic, I'll actually be trying this. Love how it doesn't lock my files into some obscure format like seafiles.
nice project! I'll check it out! I've also really enjoyed your replies here. it's obvious you really know your stuff. thanks!
Wo wo wow, is that my man JC Denton ??
"There are two editors in case you hate one of them"
You crack me up !!!
this looks amazing! (from the youtube video. also the controls/features seem to be well thought-through) i'll give it a shot tomorrow
I looked at the comparison for Seafile as that's the one I'm most familiar with. In my opinion Seafile's greatest strength is its encryption, but in your comparison you seem to see this as a negative as I assume this bullet refers to the encryption? "isolated on-disk file hierarchy, incompatible with other software. much worse than nextcloud in that regard"
the intention with that statement was that seafile, by default, places all the files inside its own proprietary file container thing, where the files are not easily accessible from the server's actual filesystem, using regular linux utilities. My knowledge of seafile is really minimal, so this could be wrong -- in which case I'll fix that right away! or, at the very least, try to clarify what I meant to avoid this confusion.
in case you happen to know -- are you aware if it's possible to use Seafile while having it just place all the files and folders on the disk like any other program would?
@tripflag @disobey2623 Your statement is correct; the way seafile stores files is in blocks (for de-duplication, apparently).
They offer a fuse extension that allows you to view stuff like a normal filesystem, though I've never tried it: https://manual.seafile.com/latest/extension/fuse/
And obviously, encrypted folders can't be accessed through the file system even with the fuse add on, because that would break the whole point of encryption.
To me, the one big advantage Seafile has is its e2e encryption and encrypted folders, as it allows me to host it externally without allowing access to my files to the server administrator.
I played around with copyparty and I have to say it is just awesome! The config is just fun to mess around with and everything feels snappy.
But I ran into an issue with FTP (probably just something I configured incorrectly) and could not find a discussion for that. I should probably start an issue on Github but as a non-developer I'm not sure how to do that in the correct way.
Edit: I managed to "solve" the problem. Apparently the user uploading to a subfolder over FTP needs the move permission for the folder above said subfolder.
Any way to run the server as a docker container?
Yep -- https://github.com/...
Hopefully that description makes sense (let me know if it doesn't)
You can run absolutely anything as a docker container that you have the binary (and other files if needed), or you can go fancy and compile from source in docker.
Just create a dockerfile.
From (some base image you want to use like Ubuntu or Alpine)
Copy necessary files
Run the binary
You can run it straight from command line, put it in a docker compose file, or even tag it and upload it to a repository (and then reference that in your docker compose)
This looks nighsome as blossom!
Does anyone know if you can manage multiple devices from a single interface? If I had it on a couple of old phones and a laptop, for example.
I'm not sure what you mean by this. Copyparty is a fileserver that I'm using for quick sharing of files and folders with others. "Managing multiple devices" is not what I would use it for, whatever you might mean by that. It does have one-way sync, if that's what you're looking for.
Hey! Just set this up today and it seems pretty good. Only issue I'm having is that is pretty slow to download (~2mb/s) what are some good ways to speed it up?
Hello!
Its been just about a year since someone forwarded this post to me, and I just want to say, I ABSOLUTELY LOVE COPYPARTY
I've kinda started using it for everything. It may have gotten out-of-hand...
A short list:
Thank you so much for making copyparty, and thank you for posting this <3
@cyclicircuit @tripflag I loved copyparty, but my non-technical users sadly found it confusing. Even with White-Gold-Theme-For-Copyparty.
I want to take the copyparty backend and glue on the frontend from the (now-archived) File Browser project..
sure! my implementation is really basic, just the stuff that's needed to make the clients i've tested happy, so there's probably still clients that won't be able to connect (And i'll fix those as soon as I hear about them!)
httpcli.py is the http methods handler, and the webdav-specific handlers are all next to eachother, propfind // proppatch // lock // unlock // mkcol // and there's also put for the uploads, but that's not entirely webdav-specific, just webdav-aware.
Thanks! So you put in the displayname prop even if itโs not set by the client. For the life of me, I canโt figure out what that prop is supposed to be in the spec. It calls it a live prop, but doesnโt give an explanation or an equivalent HTTP header.
I love how you named the error for 400 statuses, โPebkacโ! xD
@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
go to feed...
@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.
Resources:
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
go to feed...
You made this on your phone on the bus ride to and from work.
I cleaned the cat box yesterday and considered that an accomplishment.
Fuck.
save