Rsync takes long time to finish

Matthias Schniedermeyer
Thu, 12 Apr 2012 22:17:03 -0000

On 12.04.2012 23:59, vijay patel wrote:
>
> Hi Friends,
>
> I am using rsync to copy data from Production File Server to Disaster
Recovery file server. I have 100Mbps link setup between these two servers.
Folder structure is very deep. It is having path like
/reports/folder1/date/folder2/file.tx, where we have 1600 directories like
'folder1',  daily folders since last year in date folder and 2 folders
for each date folder like folder2  which ultimately will contain the file. Files
are not too big but just design of folder structure is complex. Folder structure
design is done by application and we can't change it at the moment. I am
using following command in cron to run rsync.
>
> rsync -avh --delete --exclude-from 'ex_file.txt' /reports/
10.10.10.100:/reports/ | tee /tmp/rsync_report.out >>
/tmp/rsync_report.out.$today
>
> Initially we were running it every 5 mins then we increased it to every 30
mins since one instance was not getting finished in 5 mins. Now we have made it
to run every 8 hours because of lots of folders. Is there a way i can improve
performance of my rsync??

You description and the ones in the other mails, read like something
else is more appropriate: lsyncd
http://code.google.com/p/lsyncd/

It uses inotify to to catch the events of files beeing
created/changed/.. and then syncs those files/directories (using rsync).





Bis denn

--
Real Programmers consider "what you see is what you get" to be just as
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated,
cryptic, powerful, unforgiving, dangerous.