rsync - Apr 2012 - rsync files from subfolders on source to root of a folder on destination

I wish to sync a bunch of flac files that reside in various subfolders
to the root of a folder on a destination.

An example of the directory structure on the source is:

source> tree Music/
Music/
??? R
?   ??? Radiohead
?   ?   ??? OK Computer
?   ?       ??? 01 - Radiohead - Airbag.flac
?   ?       ??? 02 - Radiohead - Paranoid Android.flac
?   ??? Red Hot Chilli Peppers
?       ??? Greatest Hits
?           ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
?           ??? 02 - Red Hot Chili Peppers - Give It Away.flac

I am using this command which syncs fine but includes the directory
paths on the destination:

rsync -rltDzvh --delete -e ssh Music/ user at 192.168.1.1:/Destination/
--include="*/" --include="*.flac" --exclude="*"

So on the destination the structure is:

destination> tree /Destination/
/Destination/
??? R
?   ??? Radiohead
?   ?   ??? OK Computer
?   ?       ??? 01 - Radiohead - Airbag.flac
?   ?       ??? 02 - Radiohead - Paranoid Android.flac
?   ??? Red Hot Chilli Peppers
?       ??? Greatest Hits
?           ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
?           ??? 02 - Red Hot Chili Peppers - Give It Away.flac

I want to prune all directories so only the files are placed into the
root of /Destination/ e.g.

destination> tree /Destination/
/Destination/
??? 01 - Radiohead - Airbag.flac
??? 02 - Radiohead - Paranoid Android.flac
??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
??? 02 - Red Hot Chili Peppers - Give It Away.flac

All the files have different names so that's ok and I have reviewed
the various options in rsync such as --no-relative but have been
unable to get it working as desired.

I also came up with:

find ./Music/ -name "*.flac" -exec rsync -ltDzvh {} -e ssh
user at 192.168.1.1:/Destination/ \;

But this means using ssh keys and is probably inefficient and if I
want to use the --delete it would likely break everything.

So how would I achieve this whilst still being able to use --delete or
--delete-excluded for times when I add, remove or rename items on my
Music library on the source and have those changes sync to the
destination?

If I were you, I would start by creating a unique folder, the image of
Destination/, and fill it with symlinks to your flac files, using -exec,
*then* calling rsync only once with the -L option (--copy-links). Thus you
should be able to benefit from the --delete option and keep an up-to-date
destination tree... but take care of the corrupted symlinks (another -exec
loop?).
Greg

On Mon, Apr 23, 2012 at 8:50 AM, James Robertson <j at mesrobertson.com>
wrote:
> I wish to sync a bunch of flac files that reside in various subfolders
> to the root of a folder on a destination.
>
> An example of the directory structure on the source is:
>
> source> tree Music/
> Music/
> ??? R
> ?   ??? Radiohead
> ?   ?   ??? OK Computer
> ?   ?       ??? 01 - Radiohead - Airbag.flac
> ?   ?       ??? 02 - Radiohead - Paranoid Android.flac
> ?   ??? Red Hot Chilli Peppers
> ?       ??? Greatest Hits
> ?           ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
> ?           ??? 02 - Red Hot Chili Peppers - Give It Away.flac
>
> I am using this command which syncs fine but includes the directory
> paths on the destination:
>
> rsync -rltDzvh --delete -e ssh Music/ user at 192.168.1.1:/Destination/
> --include="*/" --include="*.flac"
--exclude="*"
>
> So on the destination the structure is:
>
> destination> tree /Destination/
> /Destination/
> ??? R
> ?   ??? Radiohead
> ?   ?   ??? OK Computer
> ?   ?       ??? 01 - Radiohead - Airbag.flac
> ?   ?       ??? 02 - Radiohead - Paranoid Android.flac
> ?   ??? Red Hot Chilli Peppers
> ?       ??? Greatest Hits
> ?           ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
> ?           ??? 02 - Red Hot Chili Peppers - Give It Away.flac
>
> I want to prune all directories so only the files are placed into the
> root of /Destination/ e.g.
>
> destination> tree /Destination/
> /Destination/
> ??? 01 - Radiohead - Airbag.flac
> ??? 02 - Radiohead - Paranoid Android.flac
> ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
> ??? 02 - Red Hot Chili Peppers - Give It Away.flac
>
> All the files have different names so that's ok and I have reviewed
> the various options in rsync such as --no-relative but have been
> unable to get it working as desired.
>
> I also came up with:
>
> find ./Music/ -name "*.flac" -exec rsync -ltDzvh {} -e ssh
> user at 192.168.1.1:/Destination/ \;
>
> But this means using ssh keys and is probably inefficient and if I
> want to use the --delete it would likely break everything.
>
> So how would I achieve this whilst still being able to use --delete or
> --delete-excluded for times when I add, remove or rename items on my
> Music library on the source and have those changes sync to the
> destination?
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20120423/615bc078/attachment.html>

On Mon 23 Apr 2012, James Robertson wrote:
> 
> So on the destination the structure is:
> 
> destination> tree /Destination/
> /Destination/
> ??? R
> ?   ??? Radiohead
> ?   ?   ??? OK Computer
> ?   ?       ??? 01 - Radiohead - Airbag.flac
> ?   ??? Red Hot Chilli Peppers
> ?       ??? Greatest Hits
> ?           ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
> 
> I want to prune all directories so only the files are placed into the
> root of /Destination/ e.g.
> 
> destination> tree /Destination/
> /Destination/
> ??? 01 - Radiohead - Airbag.flac
> ??? 02 - Radiohead - Paranoid Android.flac
> ??? 01 - Red Hot Chili Peppers - Under the Bridge.flac
> ??? 02 - Red Hot Chili Peppers - Give It Away.flac
How about:
    cd Music
    rsync -ai --relative */*/*/./ /Destination/

Assuming the argument list doesn't get too large, I think it should
work as desired.


Paul

On 23.04.2012 16:50, James Robertson wrote:
> I wish to sync a bunch of flac files that reside in various subfolders
> to the root of a folder on a destination.
> 
> An example of the directory structure on the source is:
...

I think hard-links are the easiest solution.

mkdir hardlinks
find Music -type f -print0 | xargs -0 cp -al -t hardlinks
(You get a warning from cp about doubles, if any)

Then you can rsync the 'hardlinks'-directory as usual.

When you want to update you should just 'rm -rf hardlinks' then execute 
the find again. That way even deletions work as expected (Or just put it 
in a script).




Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.