Kevin
The link-dest parameter is a single directory (the previous day's
directory), the destination is today's directory.
I haven't tried deleting a backup, there's no particular need in space
terms, at the current rate there's enough space for several years of daily
backups.
I've reverted to daily backups on a small subset of the total; the full
backup now takes around 30 hours. Clearly not practical.
As the small subset takes only a few minutes to complete I can't yet see if
this time is incrementing too.
John On 3 Jan 2019 17:06, Kevin Korb via rsync <rsync at
lists.samba.org> wrote:>
> It does normally take some time to analyze large trees of files. It has
> to call stat() on each file to get the size and timestamp.
>
> However, 15 hours seems a bit excessive even though I have never tried
> to do this on Windows or a NAS system. Just to be clear, is your
> --link-dest parameter a single directory or are you trying to tell it to
> use all of the previous backups?
>
> Also, have you deleted a backup yet? In my experience that takes a lot
> longer than running one so if you need 15 hours to run a backup I would
> expect deleting one to take about a week.
>
> On 1/3/19 4:23 AM, John Simpson via rsync wrote:
> >
> >
> > I've been running rsync as a cygwin task on Windows Server 2008
for about two months now. I'm using the --link-dest option to do a daily
'snapshot' of the contents of a server containing about 10TB of data,
about 13 million files, to a Linux based NAS server. Things started out great
but I soon noticed that the time take to complete was slowly incrementing. It
started at around three hours, but is now around fifteen.
> >
> > The command is as follows...
> >
> > rsync -rlptDhPR \
> > --password-file=password \
> > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
> > --Stats \
> > --delete \
> > --log-file=logfilename \
> > --link-dest=linkdestpath \
> > sourceDirectory \
> > rsync at 192.168.1.2::destinationDirectory
> >
> > I'm not using the full -a option as the differences between the
Windows and Linux ownership stuff messed things up.
> >
> > The first log file looked like this...
> >
> > 2018/10/01 23:00:14 [2164] building file list
> > ...transfer file list here
> > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg:
12,260,176, dir: 1,499,821, link: 1)
> > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291,
dir: 11)
> > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
> > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
> > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
> > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
> > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
> > 2018/10/02 02:11:30 [2164] File list size: 10.42M
> > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
> > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
> > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
> > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
> > 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes
21.17K bytes/sec
> > 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79
> >
> > the most recent looks like this...
> >
> > 2018/11/24 23:00:15 [2924] building file list
> > 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/
> > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg:
12,274,642, dir: 1,501,780, link: 1)
> > 2018/11/25 13:21:16 [2924] Number of created files: 0
> > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
> > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
> > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
> > 2018/11/25 13:21:16 [2924] Literal data: 0 bytes
> > 2018/11/25 13:21:16 [2924] Matched data: 0 bytes
> > 2018/11/25 13:21:16 [2924] File list size: 10.35M
> > 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
> > 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
> > 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
> > 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
> > 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes
4.72K bytes/sec
> > 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96
> >
> > As you can see the start time is 11:00PM (23:00) in both cases. The
first log shows that identifying the files to transfer took about three hours
(I've omitted the file list - it's quite long), the second log takes
fourteen hours to do the same job (in this case this was done at the weekend and
I've include the whole log file which correctly identifies that no files
have changed)
> >
> > The number of files is as might be expected as is everything else.
It's just the time taken "building file list" is significantly
larger.
> >
> >
>
> --
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: https://sanitarium.net/
> PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
