Hello, I have a problem while rsyncing a directory with perhaps 1.000.000 files. Before this I generate some files locally and add them to temp files. So this files should be excluded from syncing. My rsync call looks like this: rsync --exclude='tmp/' \ --exclude-from=/tmp/tmp.GF7SsFPnS3 \ --exclude-from=/tmp/tmp.8SjJNCHyaI \ --exclude-from=/tmp/tmp.CxZXEoPjgV \ --exclude-from=/tmp/tmp.G3g2iMo4bs \ --exclude-from=/tmp/tmp.H9KJYPMfMS \ --exclude-from=/tmp/tmp.PNi7cJaREP \ --exclude-from=/tmp/tmp.S4N9H4lsU7 \ --exclude-from=/tmp/tmp.a5Zlgh6pUK \ --exclude-from=/tmp/tmp.eiUlMluAe8 \ --exclude-from=/tmp/tmp.ma0S1YSewc \ --exclude-from=/tmp/tmp.sLR95oVbVD \ --exclude-from=/tmp/tmp.zbfeLpezMX \ -ax --info=progress2 \ -e 'ssh -x -T -o Compression=no' \ '/srv/project/shared/uploads/' \ 'project at host:/srv/project/shared/uploads/' In every temp file are 1000s of lines with files like this: orders/scan/file/2234480/scan.pdf When I started rsync it only should transfer perhaps 1000 files and delete 20 ones. But it takes 15 minutes to do that. I think the big count of excluded files tear down the compare speed of files. What can I do to speed up this process? Thanks for your advise! Georg
Georg Limbach via rsync wrote:> Hello, > > I have a problem while rsyncing a directory with perhaps 1.000.000 > files. Before this I generate some files locally and add them to temp > files. So this files should be excluded from syncing. > > My rsync call looks like this: > > rsync --exclude='tmp/' \ > --exclude-from=/tmp/tmp.GF7SsFPnS3 \ > --exclude-from=/tmp/tmp.8SjJNCHyaI \ > --exclude-from=/tmp/tmp.CxZXEoPjgV \ > --exclude-from=/tmp/tmp.G3g2iMo4bs \ > --exclude-from=/tmp/tmp.H9KJYPMfMS \ > --exclude-from=/tmp/tmp.PNi7cJaREP \ > --exclude-from=/tmp/tmp.S4N9H4lsU7 \ > --exclude-from=/tmp/tmp.a5Zlgh6pUK \ > --exclude-from=/tmp/tmp.eiUlMluAe8 \ > --exclude-from=/tmp/tmp.ma0S1YSewc \ > --exclude-from=/tmp/tmp.sLR95oVbVD \ > --exclude-from=/tmp/tmp.zbfeLpezMX \ > -ax --info=progress2 \ > -e 'ssh -x -T -o Compression=no' \ > '/srv/project/shared/uploads/' \ > 'project at host:/srv/project/shared/uploads/' > > In every temp file are 1000s of lines with files like this: > > orders/scan/file/2234480/scan.pdf > > When I started rsync it only should transfer perhaps 1000 files and > delete 20 ones. But it takes 15 minutes to do that. I think the big > count of excluded files tear down the compare speed of files. > > What can I do to speed up this process? > > Thanks for your advise! > > GeorgI found a big speed up when I starting providing a list of candidate files to rsync with the --files-from option. If you have some way of identifying which files might need to be rsynced that's quicker than rsync itself checking everything, it can make a big difference. That's not why I did it, but it was a nice bonus. cheers, raf
raf via rsync wrote:> Georg Limbach via rsync wrote: > > > Hello, > > > > I have a problem while rsyncing a directory with perhaps 1.000.000 > > files. Before this I generate some files locally and add them to temp > > files. So this files should be excluded from syncing. > > > > My rsync call looks like this: > > > > rsync --exclude='tmp/' \ > > --exclude-from=/tmp/tmp.GF7SsFPnS3 \ > > --exclude-from=/tmp/tmp.8SjJNCHyaI \ > > --exclude-from=/tmp/tmp.CxZXEoPjgV \ > > --exclude-from=/tmp/tmp.G3g2iMo4bs \ > > --exclude-from=/tmp/tmp.H9KJYPMfMS \ > > --exclude-from=/tmp/tmp.PNi7cJaREP \ > > --exclude-from=/tmp/tmp.S4N9H4lsU7 \ > > --exclude-from=/tmp/tmp.a5Zlgh6pUK \ > > --exclude-from=/tmp/tmp.eiUlMluAe8 \ > > --exclude-from=/tmp/tmp.ma0S1YSewc \ > > --exclude-from=/tmp/tmp.sLR95oVbVD \ > > --exclude-from=/tmp/tmp.zbfeLpezMX \ > > -ax --info=progress2 \ > > -e 'ssh -x -T -o Compression=no' \ > > '/srv/project/shared/uploads/' \ > > 'project at host:/srv/project/shared/uploads/' > > > > In every temp file are 1000s of lines with files like this: > > > > orders/scan/file/2234480/scan.pdf > > > > When I started rsync it only should transfer perhaps 1000 files and > > delete 20 ones. But it takes 15 minutes to do that. I think the big > > count of excluded files tear down the compare speed of files. > > > > What can I do to speed up this process? > > > > Thanks for your advise! > > > > Georg > > I found a big speed up when I starting providing a list of > candidate files to rsync with the --files-from option. > If you have some way of identifying which files might need > to be rsynced that's quicker than rsync itself checking > everything, it can make a big difference. That's not why > I did it, but it was a nice bonus.But it might not help if doing --delete at the same time. Can the --delete be done occasionally instead of every time? Unless the files to be deleted can be added to the list of files handed to rsync. I don't know if that'll do what I think but it might. cheers, raf