rsync with overlay tree

tomr
Thu, 31 Mar 2016 05:46:28 -0000

> On 31 Mar 2016, at 16:22, Marian Marinov <mm at yuhu.biz> wrote:
>
> On 03/31/2016 07:40 AM, tomr wrote:
>> I maintain a directory structure containing dirs and files that I
regularly push to ~50 hosts, which are divided into 3 groups that have slightly
different needs (minor mods in a couple of files).
>>
>> So ideally I would have 4 directories:
>>    /path/to/sync/common/   <- common files
>>    /path/to/sync/group1/   <- group1 specific only
>>    /path/to/sync/group2/   <- group2 specific only
>>    /path/to/sync/group3/   <- group3 specific only
>>
>> Then I'd run an rsync like:
>>    rsync -av --overlay /path/to/sync/groupN \
>>     /path/to/sync/common remotehost:
>>
>> Thinking in terms of a list of files to be transferred, I would like:
>> - Anything present in common/ added to the file list; then
>> - Anything present in groupN/ added to the list, clobbering if
applicable (regardless of mtime)
>> - The destination directory to show no sign of the common / overlay
structure
>
> I really like the idea, however I have one question. How is rsync going to
decide which files to be present in the final list?
> I mean, should rsync always prefer the files from the overlay dir instead
of the files in the common dir, no matter is they differ or not?

Yes :-)

In my thinking rsync should never know or even check whether they differ: if a
filename (or directory?) is present in the overlay path, it should clobber any
matching filename (or directory?) in the common path.

An example for clarity.... I start with:

    common/config/settings.conf
    common/config/include.d/names.conf
    common/config/include.d/numbers.conf
    common/config/include.d/plugins.conf
    common/bin/someapp
    common/bin/someplugin
    common/lib/somelib
    common/someapp-release

Almost all of the config of someapp has remained unchanged in the latest major
release.  I don't want to fork all that config, and maintain in two places
until we're fully upgraded, and I want to be able to provision servers with
the old and the new config easily in the mean time.  So I create (minimally):

    group1/config/include.d/plugins.conf
    group1/config/include.d/newfeature.conf
    group1/bin/someplugin
    group1/lib/somelib
    group1/someapp-release

An rsync with eg '--overlay-path group1/' would sync plugins.conf,
newfeature.conf, someplugin, somelib and someapp-release from under group1
regardless of any fact other than that they exist.  The automation that produces
names.conf and numbers.conf can continue running without having to know about
the arrangement.

If I eventually want to collapse group1 into common I can just run 'cp -r
group1/* common && rm -r group1/*'

I think ;)

best,
tom



>> I suspect I could populate the list myself and use
'--files-from=<LIST>' but I would rather have rsync work it out if
possible.  If all else fails, I will do the merge locally first.
>>
>>    TEMPDIR=$(mktemp -d)
>>    cp -r /path/to/sync/common/* $TEMPDIR
>>    cp -r /path/to/sync/groupN/* $TEMPDIR
>>    rsync -av $TEMPDIR/* remotehost:
>>    rm -r $TEMPDIR
>>
>> Thanks in advance,
>> tom
>>
>>
>>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20160331/ae6447a5/signature.sig>