Hi: I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows. rsync server config: ------------------------------------------------------------------------------------------------------ strict modes = false hosts allow = 10.200.121.104 hosts allow = 10.171.209.87 hosts deny = * log file = /cygdrive/d/log/rsyncd.log port=8730 uid = 0 gid = 0 # Module definitions # Remember cygwin naming conventions : c:\work becomes /cygwin/c/work # [d_hst] path = /cygdrive/d/hst read only = false transfer logging = yes auth users = admin secrets file = etc/user.pass list = no ignore errors ------------------------------------------------------------------------------------------------------ client bat script: ------------------------------------------------------------------------------------------------------ @echo off set RSYNC_HOME="c:\Program Files (x86)\cwRsync\bin\" cd /d %RSYNC_HOME% :: Source root directory set src_dir=rsync://admin at 10.160.27.48:8730/d_hst :: Destination root directory set dst_dir=/cygdrive/e/filetest :: Make sure path without space set pwd_file="/cygdrive/c/user.pass" :: Log file set log_dir=/cygdrive/d/log :: Synchronize Files, with 16 connections according to 16 directories(0~f) for %%i in (0 1 2 3 4 5 6) do ( setlocal enabledelayedexpansion set log_file=%log_dir%/%date:~3,4%%date:~8,2%%date:~11,2%%time:~0,2%%time:~3,2%%time:~6,2%_%%i.txt :: Remove space in log_file set "log_file=!log_file: =!" start cmd /c rsync -avvu --delete %src_dir%/Files/%%i %dst_dir%/Files --password-file=%pwd_file% --log-file=!log_file! --log-file-format="" ) ------------------------------------------------------------------------------------------------------ In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours. It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow? Does anybody encounted cases like this? Any help will be thankful. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20151124/40ebcf55/attachment.html>
At 14:22 24.11.2015, =?gb18030?B?t+fQzg==?= wrote:>Hi: > I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows. > >In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. >When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours. >It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow? >Does anybody encounted cases like this? Any help will be thankful.If this is just one directory on one hard disk then the disk head has to move constantly from one place to another to another to another for each process. It's possible that it will run even faster if you have just one process. On the other hand Windows will cache file info in RAM so a second run could be faster then the first one. This depends one your RAM, how much have you got? Have you tried with just one process? bye Fabi
>>> I have a large directory with about 0 Milillion files in it to backup by network. Server and client both windows. >>> >>>In order to make backup faster, I created many rsync intances with "-avvu" options. It seems worked, and only take about 5 hours to complete backup job. >>>When I restarted the client bat script, and expect very little time to complete backup, but it still cost about 3 hours. >>>It seems that concurrence makes no effect, I can not figure out what happend, the file checking is so slow? >>>Does anybody encounted cases like this? Any help will be thankful. >> >>If this is just one directory on one hard disk then the disk head has to >>move constantly from one place to another to another to another for >>each process. It's possible that it will run even faster if you have >>just one process. On the other hand Windows will cache file info >>in RAM so a second run could be faster then the first one. This >>depends one your RAM, how much have you got? Have you tried >>with just one process? > >Thanks for your reply. I have tried one process, but have no improvement in essence. So the question is >why the file comparison consumes so much time? In my mind, building file list may consumes some time, >but comparison should very fast, what is the problem?Please keep it on the mailing list. I'm not an expert, others might know more than me. rsync has to stat each file and directory. I don't know what exactly takes much time. But maybe you can give more information about your setup: What OS, what file system, how many files and dirs, small/big files, connection speed, full command line with all rsync options, RAM, CPU etc. Are you using cwRsync? With the CygWin emulation? This is known to be slower. But even then 5h seems slow. Just as an example: Here's a log from a backup running on a slow NAS to another NAS in the local LAN not doing much as the files are already the same. But both are running Linux or actually BusyBox: Number of files: 3445179 Number of files transferred: 0 Total file size: 349083766943 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 71253503 File list generation time: 0.016 seconds File list transfer time: 0.000 seconds Total bytes sent: 71953972 Total bytes received: 700464 sent 71953972 bytes received 700464 bytes 66079.52 bytes/sec total size is 349083766943 speedup is 4804.71 real 18m18.760s user 0m55.883s sys 2m10.016s So over 3 million files in 20 minutes. What are your --stats? bye Fabi