Hello. I'm a long-time rsync user, but I've encountered a case today and I just can't seem to get rsync to do the right thing. I want to mirror a directory A on server www1.example.com to a directory B on www2.example.com. The directory A contains very large but essentially read-only files. A server process on www1.example.com periodically creates new large files and deletes old ones. I want to synchronize the directories roughly every ten minutes. It's important that a file in B only becomes "visible" (that is, has the same name as a file in A) when the two files contain the exact same content. Files that no longer exist in A should be deleted from B. An initial pass: www1$ rsync -avz --delete-after /A/ www2.example.com:/B/ This works, but the transfers are not resumable for some reason. If I ^C the rsync process and then run it again, I see partial files left in B, and rsync ignores these the next time I run the command. I've seen this: https://unix.stackexchange.com/questions/48298/can-rsync-resume-after-being-interrupted But, quite frankly, I feel like I've tried every combination of flags and can't get the behaviour I'm looking for. Either rsync copies the files in-place (violating the invariant that a file in B should should not be visible by the name it had in A before the contents match), or it places partial files in the same directory (or in a separate .rsync-partial directory) but then ignores them next time the transfer runs. Can someone spell it out for me? Which flags do I need? -- Mark Raynsford | http://www.io7m.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20190726/600ee992/attachment.sig>
Hi On Fri, 26 Jul 2019 20:15:27 +0100 Mark Raynsford via rsync wrote:> I want to mirror a directory A on server www1.example.com to a > directory B on www2.example.com. The directory A contains very > large but essentially read-only files. A server process on > www1.example.com periodically creates new large files and deletes old > ones. I want to synchronize the directories roughly every ten minutes. > It's important that a file in B only becomes "visible" (that is, has > the same name as a file in A) when the two files contain the exact > same content. Files that no longer exist in A should be deleted from B.> An initial pass:> www1$ rsync -avz --delete-after /A/ www2.example.com:/B/Try: rsync -avz --delete-after --delay-updates --partial /A/ www2.example.com:/B/ The man indicate also for --delay-updates: See also the "atomic-rsync" perl script in the "support" subdir for an update algorithm that is even more atomic (it uses --link-dest and a parallel hierarchy of files). PS: You may prefer also: - using -i (--itemize-changes) instead of -v - do not compress (no -z)> This works, but the transfers are not resumable for some reason. If I > ^C the rsync process and then run it again, I see partial files left in > B, and rsync ignores these the next time I run the command.Weird: never seen that. -- francis
On 2019-07-27T07:59:56 +0200 Francis.Montagnac at inria.fr wrote:> > Try: > > rsync -avz --delete-after --delay-updates --partial /A/ www2.example.com:/B/ > > The man indicate also for --delay-updates: > > See also the "atomic-rsync" perl script in the "support" subdir > for an update algorithm that is even more atomic (it uses > --link-dest and a parallel hierarchy of files). > > PS: You may prefer also: > > - using -i (--itemize-changes) instead of -v > - do not compress (no -z)Hello! Same effect on ^C, unfortunately. The exact command I'm running (with names changed to protect the innocent :): $ /usr/local/bin/rsync \ -a -L -i '--chmod=ugo-rwx,Dugo+x,ugo+r,u+w' \ --delete-after \ --delay-updates \ --partial \ /A/ www2.example.com:/B/ If I ^C this command, I see temporary files left in /B, and the next time I run this same command, new temporary files are created and the old ones are ignored. This is on FreeBSD 12. $ rsync --version rsync version 3.1.3 protocol version 31 Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes, no prealloc, file-flags -- Mark Raynsford | http://www.io7m.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20190727/02bfde3d/attachment.sig>