Kevin Korb
2023-Sep-22 00:08 UTC
rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
I had intended to come back to this but because I didn't really think I had anything to add to the discussion I put it at a low enough priority that I forgot about it. But I saw your bug report and was surprised to see that I was already unhelpful on this topic but because that original poster didn't have access to do an rm while you are trying to get a performance boost. I have heard in the past that rsyncing an empty dir over a tree to delete the tree is faster than an rm -rf but I can't say I have ever benchmarked it to get any actual numbers. But now that I am hearing that rsync actually adds a bunch of pointless chmods to the process. Is it still faster given this problem? If so maybe we should be trying to investigate why rm is so slow. Otherwise, I would agree that an optimization to not do a bunch of pointless chmods is a good idea. I would suspect that they are only there to prevent permission denied errors perhaps on some obscure filesystem (I tested on vfat since it is so non-compatible but read only files unlink fine there). On 9/18/23 19:42, rsync--- via rsync wrote:> Context > ------- > > I am one of the active developers of the open source application "Back in Time" > which uses "rsync" as backend and I want to fix an open issue: > > "Back in Time"-Bug: > https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507 > > "Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking > unchanged files in the backups/snapshots and tries to keep the permissions > by using "--perms --group --owner" by default. > > Older snapshots are then deleted according to a schedule. > > > > Issue > ----- > > When deleting a complete snapshot folder with rsync using an empty source folder > (which is a best practice for faster deletion than "rm -f") the permissions of hardlinked > files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted files > with existing hardlinks: > > mkdir empty > rsync -a --delete -s empty/ snapshot1/ > > This distorts the backup history. > > There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the issue is still unfixed: > > https://bugzilla.samba.org/show_bug.cgi?id=12806 > > > > Questions > --------- > > 1. Is there a reliable workaround ("--super" is proposed in the issue but may probably not always work)? > 2. If a rsync developer is reading this: Is there any chance to fix this? > > > > Steps to reproduce > ------------------ > > I have attached a bash script "setup.sh" as txt file which makes the issue fully reproducible (on Ubuntu 22.04). > >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
rsync at altfeld-im.de
2023-Sep-22 08:14 UTC
rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote:> I have heard in the past that rsyncing an empty dir over a tree to > delete the tree is faster than an rm -rf but I can't say I have ever > benchmarked it to get any actual numbers.This **may** indeed be a myth (for a long time now) re-cited again and again and - could no longer be valid today - could apply only when deleting explicitly named files but not deleting the complete folder (as we need to do in "Back in Time") At least I could not find a holistic benchmark with many files and different scenarios (file systems, rsync'ing locally vs. over network, snapshot sizes, number of files, file sizes, rsync and rm versions...) Q: Does `rsync` provide a test case that I could use as basis to prepare such a holistic benchmark?> But now that I am hearing > that rsync actually adds a bunch of pointless chmods to the process.? Is > it still faster given this problem?? If so maybe we should be trying to > investigate why rm is so slow.Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not.
Possibly Parallel Threads
- rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
- rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
- rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
- rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
- [Bug 12806] New: Deleting in a row of hardlinked snapshots resets file permissions.