Hey folks, I did some googling on this but did not come up with much. I'm using rsnapshot which uses rsync, and I notice some pretty long pauses in the xfers as you can see on this graph from "munin". THe machine in question right at the moment is doing nothing but rsyncing ( rsnapshoting ) some 12T of NAS storage to local disk, so there is nothing else going on at all. https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250 You can see a couple of spots in the graph where it just drops to nothing ( checking more closely you do see a very small bit of activity ) Here is another graph from MRTG which records the traffic from the switch port side. You can see around 2am it drops off pretty low, then from about 4am til about 7am it drops even lower. https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322 Is that normal? Or is this something I should ask on the rsnapshot list (if there is one)? I did find something in google about depending on the types of files it can change xfer rate but not much detail. thanks, -Alan
Hi, What is the state of your rsync process ? I'm pretty you enter the uninterruptible sleep state (D in ps) Regards, -----Message d'origine----- De?: rsync-bounces at lists.samba.org [mailto:rsync-bounces at lists.samba.org] De la part de Alan McKay Envoy??: mardi 12 juin 2012 15:25 ??: rsync at lists.samba.org Objet?: rsync takes long pauses in xfer ? Hey folks, I did some googling on this but did not come up with much. I'm using rsnapshot which uses rsync, and I notice some pretty long pauses in the xfers as you can see on this graph from "munin". THe machine in question right at the moment is doing nothing but rsyncing ( rsnapshoting ) some 12T of NAS storage to local disk, so there is nothing else going on at all. https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250 You can see a couple of spots in the graph where it just drops to nothing ( checking more closely you do see a very small bit of activity ) Here is another graph from MRTG which records the traffic from the switch port side. You can see around 2am it drops off pretty low, then from about 4am til about 7am it drops even lower. https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322 Is that normal? Or is this something I should ask on the rsnapshot list (if there is one)? I did find something in google about depending on the types of files it can change xfer rate but not much detail. thanks, -Alan -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Alan,
What is the rsync command used? If it hits a large group of files
with no differences (easy to believe across 12TB) but has to compare the
checksums anyways (forced by you or failing the size+time compare) you'll
be I/O bound with almost no corresponding network traffic as it just has
to send the checksums.
Eric Bambach | Discover
Senior Assoc. Programmer, Warehouse Infrastructure and Tools
2500 Lake Cook Road, Riverwoods IL 60015
P: 224.405.2896 ericbambach1 at discover.com
From: Alan McKay <alan.mckay+rsync at gmail.com>
To: <rsync at lists.samba.org>
Date: 06/12/2012 08:30 AM
Subject: rsync takes long pauses in xfer ?
Sent by: <rsync-bounces at lists.samba.org>
Hey folks,
I did some googling on this but did not come up with much. I'm using
rsnapshot which uses rsync, and I notice some pretty long pauses in
the xfers as you can see on this graph from "munin". THe machine in
question right at the moment is doing nothing but rsyncing (
rsnapshoting ) some 12T of NAS storage to local disk, so there is
nothing else going on at all.
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250
You can see a couple of spots in the graph where it just drops to
nothing ( checking more closely you do see a very small bit of
activity )
Here is another graph from MRTG which records the traffic from the
switch port side. You can see around 2am it drops off pretty low,
then from about 4am til about 7am it drops even lower.
https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753140504922328322
Is that normal? Or is this something I should ask on the rsnapshot
list (if there is one)?
I did find something in google about depending on the types of files
it can change xfer rate but not much detail.
thanks,
-Alan
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Please consider the environment before printing this email.
On Tue, Jun 12, 2012 at 9:39 AM, <ericbambach1 at discover.com> wrote:> Alan, > > ? ? ? ?What is the rsync command used? If it hits a large group of files > with no differences (easy to believe across 12TB) but has to compare the > checksums anyways (forced by you or failing the size+time compare) you'll > be I/O bound with almost no corresponding network traffic as it just has > to send the checksums.Here is what shows up in the process list. BTW this is running Ubuntu 12.04 Live USB stick. But I saw the same behavior on the same machine last week on RHEL5. 0 R root 20991 20986 15 80 0 - 51725 - Jun11 pts/2 03:05:18 /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --include-from=/usr/local/etc/rsnapshot.d/include --exclude-from=/usr/local/etc/rsnapshot.d/exclude /data /mnt/1J4400/rsnapshot/daily.0/ZFS/ 5 S root 20992 20991 0 80 0 - 48034 poll_s Jun11 pts/2 00:00:34 /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --include-from=/usr/local/etc/rsnapshot.d/include --exclude-from=/usr/local/etc/rsnapshot.d/exclude /data /mnt/1J4400/rsnapshot/daily.0/ZFS/ 5 R root 20993 20992 16 80 0 - 59897 - Jun11 pts/2 03:14:28 /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded --include-from=/usr/local/etc/rsnapshot.d/include --exclude-from=/usr/local/etc/rsnapshot.d/exclude /data /mnt/1J4400/rsnapshot/daily.0/ZFS/
On Tue, Jun 12, 2012 at 9:25 AM, Alan McKay <alan.mckay+rsync at gmail.com> wrote:> https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753139625064539250 > > You can see a couple of spots in the graph where it just drops to > nothing ( checking more closely you do see a very small bit of > activity )For anyone interested ... lots of quiet time overnight last night https://plus.google.com/u/0/photos/113518077816006248932/albums/5732421463318682369/5753537939542074994 I'm only at about 4.2T of my 12T ... after 48 hours.