How to share a folder in Linux?

7 months ago by ranzispa to c/sysadmin

I work on an HPC and often I have to share files with other users. The most approachable solution is to have an external cloud storage and recline back and forth. However there's some projects that are quite heavy (several TB) and that is unfeasible. We do not have a shared group. The following is the only solution I found which is not to just set al permissions to 777, and I still don't like it.

Create a directory and set ACL to give access to the selected users. This works fine if the users create new files in there, but it does not work if they copy from somewhere else as default umask is 022. Thus the only appropriate solution is to change default umask to 002, which however affects file creation system wide. The alternative is to change permissions every time you copy something, but you all know very well that is not going to happen.

Does it really have to be such a pain in the ass?

bjoern_tantau 18 points 7 months ago

Uh, why not create the shared group? That's more or less exactly the purpose of their existence.

path: 0 22509206, hotness: undefined, score: 18, children: 7

ranzispa -2 points 7 months ago

Lots of paperwork involved around our HPC. Politica in the middle and plausibly would have to sign confidentiality agreement with everyone who shares anything in there, which would have to go through a review process which is generally takes about 6 months.

path: 0 22509206 22509443, hotness: undefined, score: -2, children: 6

ag10n 14 points 7 months ago

There’s a reason for this governance and you’re putting your whole team at risk trying to do this yourself

path: 0 22509206 22509443 22510946, hotness: undefined, score: 14, children: 5

ranzispa 1 point 7 months ago

It's just default Linux permissions. People who get access to the HPC through the same institution are placed in the same group. As you may know scientific collaboration is quite important. Indeed when collaborating we sign all the necessary paperwork, but that does not translate to the HPC administrators who are part of a separate institution. To request a separate group you have to contact the HPC institution, they will have to contact the institutions involved. Those institutions will have to check that NDAs are already in place. If the NDAs are in place they will have to check that the data to be shared is actually covered by the project. I will have to fill a bunch of paperwork. This will be sent to an external auditor to check that everything is correct, and then everything goes back in that chain.

I already waste way too much of my time on paperwork. Worst case scenario is a collaborator leaks some data which will be published publicly in a few months anyway. And those collaborators will have access to such data anyway, just through other less comfortable means.

path: 0 22509206 22509443 22510946 22511195, hotness: undefined, score: 1, children: 4

ag10n 8 points 7 months ago

The fact that you’re sharing this internal policy stuff so openly is definitely a red flag.

path: 0 22509206 22509443 22510946 22511195 22516712, hotness: undefined, score: 8, children: 3

warmaster 14 points 7 months ago

I'm no sysadmin, I just run my homelab. Let me get this straight... You want to bypass system level access level restrictions with some form of control but not go through your company's standard method of doing so because of bureaucracy?

If that's the case: why not put something in front Like opencloud for example?

I mean, maybe OC is not what you need, but conceptually... would a middleman solution work for you? If so, you could go with a thousand different alternatives depending on your needs.

path: 0 22509740, hotness: undefined, score: 14, children: 17

ranzispa 1 point 7 months ago

A cloud solution is indeed an option, however not a very palatable one. The main problem with a cloud solution would be pricing. From what I can see, you can get 1TB for about 10€/month. We'd need substantially more than that. The cost is feasible and not excessive, but frankly it's a bit of a joke to have to use someone else's server when we have our own.

You want to bypass system level access level restrictions with some form of control but not go through your company's standard method of doing so because of bureaucracy?

Yes. Not a company but public research, which means asking for a group change may lead to several people in the capital discussing on whether that is appropriate or not. I'd like this to be a joke, but it is not. We'd surely get access eventually if we do that, but that would lead to the unfortunate side: if we work in that way every new person who has to get in has to wait all that paperwork.

path: 0 22509740 22509825, hotness: undefined, score: 1, children: 16

possiblylinux127 7 points 7 months ago

Don't bypass your organizational policies

path: 0 22509740 22509825 22518757, hotness: undefined, score: 7, children: 3

ranzispa 1 point 7 months ago

I am not bypassing any policy: the HPC Is there to collaborate on and data can be shared. Not having a shared group is not a policy, it's just that not all users are in the same group and users are added to just one group by default. We are indeed allowed to share files, hell most of the people I want to share stuff with are part of my own research group. ACL is allowed on the HPC. I'm asking how to properly use ACL.

If you have anything actually useful go ahead, otherwise don't worry that I know better than you do what I should or should not do.

path: 0 22509740 22509825 22518757 22527872, hotness: undefined, score: 1, children: 2

possiblylinux127 3 points 7 months ago

You are in way over your head

Stop now before you get yourself in hot water

path: 0 22509740 22509825 22518757 22527872 22532779, hotness: undefined, score: 3, children: 1

Luckyfriend222 4 points 7 months ago

I think he meant self-hosting Opencloud

path: 0 22509740 22509825 22514281, hotness: undefined, score: 4, children: 10

warmaster 3 points 7 months ago

Yes. That's what I recommended. Self-host whatever middleman software. Opencloud, WebDAV, S3, FTP, anything he puts in the middle can accomplish what he wants.

path: 0 22509740 22509825 22514281 22524490, hotness: undefined, score: 3, children: 9

ranzispa 1 point 7 months ago

I see! Well, I currently do not have another server that has so much storage that we could use for thi purpose. Maybe in the future and that will solve a bunch of problems, this is only one of them.

We do have a storage server, but that is local only and backup only: not going to open it to the internet.

It is indeed a solution. What is absurd to me is to have to consider such a solution that requires two servers.

path: 0 22509740 22509825 22514281 22524490 22527955, hotness: undefined, score: 1, children: 8

warmaster 1 point 7 months ago

I recommended Self-hosting whatever middleman software. Opencloud, WebDAV, S3, FTP, anything you put in the middle can accomplish what you want.

path: 0 22509740 22509825 22524494, hotness: undefined, score: 1, children: 0

chris 5 points 7 months ago

You can set acls on directories that get applied recursively. This makes ist possible to have all files be the correct permission. I am on the go right now but you should look into setfacl. It's been a while but I am pretty sure that worked. That way you should even be able to say which groups or users can do what with granularity.

path: 0 22528402, hotness: undefined, score: 5, children: 0

blackbirdbiryani 3 points 7 months ago

I'm in a similar position as you. Our lab has a partition on HPC but i need a way to quasi-administrate other lab members without truly having root access. What I found works is to have a shared bashrc script (which also contains useful common aliases and env variables) and get all your users to source it (in their own bashrc files). Set the umask within the shared bashrc file. Set certain folders to read only (for common references, e.g. genomes) if you don't want people messing with shares resources. However, I've found that it's only worth trying to admin shared resources and large datasets, otherwise let everyone junk their home folder with their own analyses. If the home folder is size limited, create a user's folder in the scratch partition and let people store their junk there however they want. Just routinely check that nobody is abusing your storage quota.

EDIT: absolutely under no circumstances give people write access to raw shared data on hpc. I guarantee some idiot will edit it and mess it up for everyone. If people need to rename files they can learn how to symlink them.

path: 0 22525713, hotness: undefined, score: 3, children: 6

biber 2 points 7 months ago

This is a pretty good idea!

In addition, I recommend having all data e.g. as a (private)datalad archive synchronized to Dataverse, osf, figshare or wherever - edits are versioned then

path: 0 22525713 22525905, hotness: undefined, score: 2, children: 2

ranzispa 1 point 7 months ago

I am generally using DVC to version data, are those better options?

path: 0 22525713 22525905 22528027, hotness: undefined, score: 1, children: 1

biber 1 point 7 months ago

I don't know, seems to be quite similar :)

path: 0 22525713 22525905 22528027 22530267, hotness: undefined, score: 1, children: 0

ranzispa 1 point 7 months ago

Thanks, this is a great idea! I can see you have been doing this for a long time and you're talking from experience. Regarding shared data: I use this more as a way to give raw data to other people and collect results from them. I use it mostly as a temporary directory used to transfer data, anything significant will get copied over to my share and backed up.

I can see how you could be worried about storage quota, luckily I don't have that many people to worry about. But it is funny you mention it as I could really see someone stashing a few conda environments in there just because they finished their inside quota...

path: 0 22525713 22528015, hotness: undefined, score: 1, children: 2

blackbirdbiryani 1 point 7 months ago

If you're not that worried about storage then you can just make copies if necessary, then you don't really have to worry about permissions (apart from read, which is typically default for the same group). But yea if there's any chance more than 1 person might work off the same copy of data on HPC, make it read only for the peace of mind. Regarding conda envs, yea I have a few common read only conda environments so that scripts can be used by multiple users without the hassle of ensuring everyone has the same env. Quite useful.

path: 0 22525713 22528015 22534749, hotness: undefined, score: 1, children: 1

ranzispa 1 point 7 months ago

The shared environment thing seems like a very cool idea! I'll try to set it up.

path: 0 22525713 22528015 22534749 22544180, hotness: undefined, score: 1, children: 0

twack 3 points 7 months ago

Here's someone that solved this by monitoring the directory using inotifywait, but based on the restrictions you already mentioned I'm assuming you can't install packages or set up root daemons, correct?

https://bbs.archlinux.org/viewtopic.php?id=280937

Edit: CallMeAI beat me with this exact same answer by 15 minutes.

path: 0 22510890, hotness: undefined, score: 3, children: 0

linuxguy 2 points 7 months ago

Maybe some sticky bit https://www.redhat.com/...

path: 0 22511131, hotness: undefined, score: 2, children: 6

ranzispa 1 point 7 months ago

I thought sticky bits were used to allow other users to edit files but not delete them. Do they also allow inheriting the parent directory permissions?

path: 0 22511131 22511264, hotness: undefined, score: 1, children: 5

linuxguy 2 points 7 months ago

I didn't intend and don't think the stick bit stuff will or could be a complete solution for you. You've got some oddly specific and kinda cruddy restrictions that you've got to workaround and when they get that nonsensical one ends up solidly in "cruddy hack" territory.

From the article:

group + s (pecial)

Commonly noted as SGID, this special permission has a couple of functions:

If set on a file, it allows the file to be executed as the group that owns the file (similar to SUID) If set on a directory, any files created in the directory will have their group ownership set to that of the directory owner

You could run something like https://pypi.org/project/uploadserver/ in screen or run a cron every minute that just recursively sets the correct permissions.

path: 0 22511131 22511264 22512103, hotness: undefined, score: 2, children: 4

ranzispa 3 points 7 months ago

Wow, that group +s seems exactly what I'm looking for! That actually looks like the clean solution I was looking for. I'll test it out and report back, I'll have to wait on Monday for the colleagues to be back in the server, but it seems very promising.

Thank you very much!

path: 0 22511131 22511264 22512103 22512278, hotness: undefined, score: 3, children: 3

sem 2 points 7 months ago

Can you check back in here and let us know if it worked?

path: 0 22511131 22511264 22512103 22512278 22528538, hotness: undefined, score: 2, children: 1

linuxguy 1 point 7 months ago

Wahoo! Best of luck!

path: 0 22511131 22511264 22512103 22512278 22512308, hotness: undefined, score: 1, children: 0

frongt 2 points 7 months ago

A dedicated file sharing application.

path: 0 22509657, hotness: undefined, score: 2, children: 14

ranzispa 0 points 7 months ago

What do you mean? Is there an application that allows easily sharing files on one Linux system? That would be nice!

If you mean going through an external server or peer to peer transfer, that is not too feasible. I do not have other storage places with tens of terabytes available, and transfering that much data through some P2P layer, while feasible, would probably be even less user friendly.

path: 0 22509657 22509731, hotness: undefined, score: 0, children: 13

WhyJiffie 2 points 7 months ago

Is there an application that allows easily sharing files on one Linux system? That would be nice!

this sentence made me remember this: https://github.com/9001/copyparty

path: 0 22509657 22509731 22533426, hotness: undefined, score: 2, children: 1

ranzispa 1 point 7 months ago

Not what I need, but it looks very cool!

path: 0 22509657 22509731 22533426 22544141, hotness: undefined, score: 1, children: 0

IceFoxX 0 points 7 months ago

path: 0 22509657 22509731 22510467, hotness: undefined, score: 0, children: 10

ranzispa 0 points 7 months ago

This is a large computing cluster, there are no such mountpoints available and I'm definitely not allowed to go there and plug a few disks into the racks.

path: 0 22509657 22509731 22510467 22510572, hotness: undefined, score: 0, children: 8

IceFoxX 2 points 7 months ago

path: 0 22509657 22509731 22510467 22510572 22511970, hotness: undefined, score: 2, children: 7

possiblylinux127 0 points 7 months ago

Skip NFS and ftp as they cause more problems than they solve

path: 0 22509657 22509731 22510467 22518748, hotness: undefined, score: 0, children: 0

CallMeAl 2 points 7 months ago

I'm pretty sure you can do this by adding default user entries to the directory acl which will then be set on files added to that dir.

path: 0 22509386, hotness: undefined, score: 2, children: 5

ranzispa 1 point 7 months ago

Default user entries are in there and do work, however when copying existing files those get masked with the existing group permissions. As such, the only solution I found is to have everyone set their umask to 002 as otherwise we would not get write access to files which are copied and not created in place.

path: 0 22509386 22509463, hotness: undefined, score: 1, children: 4

CallMeAl 5 points 7 months ago

Ah, I see. Well its ugly, but you could inotify to trigger a tiny script to update the perms when files are added or copied to the share dir.

path: 0 22509386 22509463 22510403, hotness: undefined, score: 5, children: 3

ranzispa 1 point 7 months ago

That is a possibility, but how would the setup look like? Only the owner can update the permissions. This would mean that all users need an inotify daemon on that folder for whenever they copy something in there. Not to mention, this is an HPC and we mostly live in login nodes; our sessions are limited to 8 hours which makes setting up such a daemon a bit tricky. Could probably set up somewhere else a cronjob to connect and start it, but it feels a bit cumbersome.

path: 0 22509386 22509463 22510403 22510542, hotness: undefined, score: 1, children: 2

CallMeAl 2 points 7 months ago

Running the inotify script as a service as root would require only one instance. You could trigger it on close_write and then run setfacl to add ACL entries to the new file for all the share users.

If you can't add a daemon or service to the system then you can skip inotify and just slam a cron job at it every minute to find new files and update their perms if needed. Ugly but effective.

Another option to consider: You could write a little script that changes umask, copies files, and changes it back. Tell people they must use that "share_cp" script to put files into the share dir.

path: 0 22509386 22509463 22510403 22510542 22510715, hotness: undefined, score: 2, children: 1

poinck 1 point 7 months ago

path: 0 22527563, hotness: undefined, score: 1, children: 1

ranzispa 2 points 7 months ago

This is a good idea and something I may setup once we setup our own compute server. However at that point wouldn't a synced directory be a better fit for the purpose? Such as you define a directory on the external server to be used to share data and every user syncs it to their own share on the main server to get all the shared data through rsync or unison.

Just throwing it out there, I'm not sure if that fits your use case.

path: 0 22527563 22528108, hotness: undefined, score: 2, children: 0

Zozano 1 point 7 months ago

path: 0 22509714, hotness: undefined, score: 1, children: 0

knobbysideup 0 points 7 months ago

path: 0 22509839, hotness: undefined, score: 0, children: 0

possiblylinux127 0 points 7 months ago

I would hire someone who knows what they are doing. It sounds like you are out of your element here which is risky.

To answer your question, you have a few options:

Samba
- Samba is just a SMB server. If you have a local active directory setup use this.
SCP
- this just copies files over a ssh connection
Rsync
- this performs a sync of one directory to the other. It can run over SCP
Unison
- like rsync but two directional. For it to work it needs to track state
Syncthing
- never actually used it but it might be close to what you want.

path: 0 22518719, hotness: undefined, score: 0, children: 1

ranzispa 2 points 7 months ago

Thank you for the reply and for trying to help out. You may have misunderstood the question: don't worry it is ok.

It sounds like you are out of your element here which is risky.

I am ok, it is not risky do not worry. Judging from your answer I'd say I have a bit more experience than you do.

path: 0 22518719 22527828, hotness: undefined, score: 2, children: 0

sysadmin

@lemmy.world

login for more options

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

go to feed...

sysadmin

@lemmy.world

login for more options

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

go to feed...