I maintain a directory structure containing dirs and files that I regularly push
to ~50 hosts, which are divided into 3 groups that have slightly different needs
(minor mods in a couple of files).
So ideally I would have 4 directories:
/path/to/sync/common/ <- common files
/path/to/sync/group1/ <- group1 specific only
/path/to/sync/group2/ <- group2 specific only
/path/to/sync/group3/ <- group3 specific only
Then I'd run an rsync like:
rsync -av --overlay /path/to/sync/groupN \
/path/to/sync/common remotehost:
Thinking in terms of a list of files to be transferred, I would like:
- Anything present in common/ added to the file list; then
- Anything present in groupN/ added to the list, clobbering if applicable
(regardless of mtime)
- The destination directory to show no sign of the common / overlay structure
I suspect I could populate the list myself and use
'--files-from=<LIST>' but I would rather have rsync work it out if
possible. If all else fails, I will do the merge locally first.
TEMPDIR=$(mktemp -d)
cp -r /path/to/sync/common/* $TEMPDIR
cp -r /path/to/sync/groupN/* $TEMPDIR
rsync -av $TEMPDIR/* remotehost:
rm -r $TEMPDIR
Thanks in advance,
tom
On 03/31/2016 07:40 AM, tomr wrote:> I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files). > > So ideally I would have 4 directories: > /path/to/sync/common/ <- common files > /path/to/sync/group1/ <- group1 specific only > /path/to/sync/group2/ <- group2 specific only > /path/to/sync/group3/ <- group3 specific only > > Then I'd run an rsync like: > rsync -av --overlay /path/to/sync/groupN \ > /path/to/sync/common remotehost: > > Thinking in terms of a list of files to be transferred, I would like: > - Anything present in common/ added to the file list; then > - Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime) > - The destination directory to show no sign of the common / overlay structureI really like the idea, however I have one question. How is rsync going to decide which files to be present in the final list? I mean, should rsync always prefer the files from the overlay dir instead of the files in the common dir, no matter is they differ or not?> > I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible. If all else fails, I will do the merge locally first. > > TEMPDIR=$(mktemp -d) > cp -r /path/to/sync/common/* $TEMPDIR > cp -r /path/to/sync/groupN/* $TEMPDIR > rsync -av $TEMPDIR/* remotehost: > rm -r $TEMPDIR > > Thanks in advance, > tom > > >-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20160331/0341c3c1/signature.sig>
> On 31 Mar 2016, at 16:22, Marian Marinov <mm at yuhu.biz> wrote: > > On 03/31/2016 07:40 AM, tomr wrote: >> I maintain a directory structure containing dirs and files that I regularly push to ~50 hosts, which are divided into 3 groups that have slightly different needs (minor mods in a couple of files). >> >> So ideally I would have 4 directories: >> /path/to/sync/common/ <- common files >> /path/to/sync/group1/ <- group1 specific only >> /path/to/sync/group2/ <- group2 specific only >> /path/to/sync/group3/ <- group3 specific only >> >> Then I'd run an rsync like: >> rsync -av --overlay /path/to/sync/groupN \ >> /path/to/sync/common remotehost: >> >> Thinking in terms of a list of files to be transferred, I would like: >> - Anything present in common/ added to the file list; then >> - Anything present in groupN/ added to the list, clobbering if applicable (regardless of mtime) >> - The destination directory to show no sign of the common / overlay structure > > I really like the idea, however I have one question. How is rsync going to decide which files to be present in the final list? > I mean, should rsync always prefer the files from the overlay dir instead of the files in the common dir, no matter is they differ or not?Yes :-) In my thinking rsync should never know or even check whether they differ: if a filename (or directory?) is present in the overlay path, it should clobber any matching filename (or directory?) in the common path. An example for clarity.... I start with: common/config/settings.conf common/config/include.d/names.conf common/config/include.d/numbers.conf common/config/include.d/plugins.conf common/bin/someapp common/bin/someplugin common/lib/somelib common/someapp-release Almost all of the config of someapp has remained unchanged in the latest major release. I don't want to fork all that config, and maintain in two places until we're fully upgraded, and I want to be able to provision servers with the old and the new config easily in the mean time. So I create (minimally): group1/config/include.d/plugins.conf group1/config/include.d/newfeature.conf group1/bin/someplugin group1/lib/somelib group1/someapp-release An rsync with eg '--overlay-path group1/' would sync plugins.conf, newfeature.conf, someplugin, somelib and someapp-release from under group1 regardless of any fact other than that they exist. The automation that produces names.conf and numbers.conf can continue running without having to know about the arrangement. If I eventually want to collapse group1 into common I can just run 'cp -r group1/* common && rm -r group1/*' I think ;) best, tom>> I suspect I could populate the list myself and use '--files-from=<LIST>' but I would rather have rsync work it out if possible. If all else fails, I will do the merge locally first. >> >> TEMPDIR=$(mktemp -d) >> cp -r /path/to/sync/common/* $TEMPDIR >> cp -r /path/to/sync/groupN/* $TEMPDIR >> rsync -av $TEMPDIR/* remotehost: >> rm -r $TEMPDIR >> >> Thanks in advance, >> tom >> >> >> >-------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: <http://lists.samba.org/pipermail/rsync/attachments/20160331/ae6447a5/signature.sig>
In <B4E1EC5D-5E46-4A98-9018-C5E4C4441712 at equalit.ie>, on 03/31/16 at 03:40 PM, tomr <tom at equalit.ie> said: Hi,>Then I'd run an rsync like: > rsync -av --overlay /path/to/sync/groupN \ > /path/to/sync/common remotehost:>Thinking in terms of a list of files to be transferred, I would like: - >Anything present in common/ added to the file list; then - Anything >present in groupN/ added to the list, clobbering if applicable >(regardless of mtime) - The destination directory to show no sign of the >common / overlay structureThe best solution will depend on the number of files in the special groups. Since you say the number is small, I suspect that something simple like: rsync -hia /path/to/sync/common remotehost: rsync -hia --ignore-times /path/to/sync/groupN remotehost: will be close to optimal. Another option is to build an --exclude-from list for the files in the special group and apply it to the first rsync. Then you can run the second rsync without the --ignore-times, which may result in faster overall throughput. FWIW, I think you will find the -hi output more useful than the -v output. Steven -- ---------------------------------------------------------------------- "Steven Levine" <steve53 at earthlink.net> Warp/DIY/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com ----------------------------------------------------------------------
>Since you say the number is small, I suspect that something simple like: > > rsync -hia /path/to/sync/common remotehost: > rsync -hia --ignore-times /path/to/sync/groupN remotehost: > >will be close to optimal. > >Another option is to build an --exclude-from list for the files in the >special group and apply it to the first rsync. Then you can run the >second rsync without the --ignore-times, which may result in faster >overall throughput.Another way could be to build the specific directories and in there create links to the common files if there aren't too many. Then on rsyncing you need to dereference the links. That way you only have one directory (structure) which rsync can already handle. bye Fabi