Hi!
I do a rsync between 2 machines. The throughput is only 2 MByte/Sec.
Each machine is a Supermicro server with
2 x 8 Core Opteron 6128
64 GByte of ECC RAM
1 LSI MegaRAID SAS 9280-24i4e
24 x 2TByte SATA Disks as a RAID6
2 Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network-cards
Both run Ubuntu 11.04 64Bit.
Both use rsync version 3.0.7 protocol version 30
There are no errors reported.
The local read and write rate to the disks is 1,1 GB/s.
A network test shows 800MByte/s over the direct coupled network-cards.
The CPU-usage is 5 to 7% of one core.
Memory-usage is 1703 out of 64557MB.
There are no other jobs running on the two machines.
The LSI controllers both say: State: Optimal
machine A has a rsync-daemon with the following configuration:
----------
read only = false
use chroot = false
hosts allow = 10.99.11.0/24
[Daten1]
path = /var/daten1
use chroot = yes
read only = yes
list = yes
uid = root
gid = root
strict modes = yes
ignore errors = no
ignore nonreadable = yes
transfer logging = no
timeout = 600
refuse options = checksum dry-run
dont compress = *
----------
machine B runs:
/usr/bin/rsync -ax --numeric-ids --delete --delete-excluded \
--exclude-from=/Backup/RB_None.excl \
--link-dest=/Backup/Base/xxxxx \
rsync://root at 10.99.11.11/Daten1/ \
/Backup/Base/yyyyyyy \
--no-c --no-z
The files are all about 8 MB in size.
8302592 100% 1.98MB/s 0:00:04 (xfer#861, to-check=51173/94679)
The same - slow - transfer happens at 1GB and 5GB files.
Any ideas what can cause this slow transmission?
--
Mit freundlichen Gr??en / best regards
Ing. Rainer Pietsch
----------------------------------------------------------
PCS - Pichler Computer Systeme
Inh. Claudia Pichler-Pietsch
Hauptplatz 10
A-2751 Steinabr?ckl
?sterreich / Austria
----------------------------------------------------------
mail: r.pietsch at pcs-at.com
web: http://www.pcs-at.com
tel.: +43 (2622) 420 19 / 15
mobil: +43 (676) 31 242 69
fax: +43 (2622) 420 19 / 20
----------------------------------------------------------
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Are these files new files being initially copied or are they modified files being updated? Does --whole-file or --inplace make a difference? On 01/03/12 08:42, R. Pietsch wrote:> > Hi! > > I do a rsync between 2 machines. The throughput is only 2 > MByte/Sec. > > Each machine is a Supermicro server with 2 x 8 Core Opteron 6128 64 > GByte of ECC RAM 1 LSI MegaRAID SAS 9280-24i4e 24 x 2TByte SATA > Disks as a RAID6 2 Intel Corporation 82599EB 10-Gigabit SFI/SFP+ > Network-cards > > Both run Ubuntu 11.04 64Bit. Both use rsync version 3.0.7 protocol > version 30 > > There are no errors reported. The local read and write rate to the > disks is 1,1 GB/s. A network test shows 800MByte/s over the direct > coupled network-cards. The CPU-usage is 5 to 7% of one core. > Memory-usage is 1703 out of 64557MB. There are no other jobs > running on the two machines. The LSI controllers both say: State: > Optimal > > > machine A has a rsync-daemon with the following configuration: > ---------- read only = false use chroot = false hosts allow > 10.99.11.0/24 [Daten1] path = /var/daten1 use chroot = yes read > only = yes list = yes uid = root gid = root strict modes = yes > ignore errors = no ignore nonreadable = yes transfer logging = no > timeout = 600 refuse options = checksum dry-run dont compress = * > ---------- > > machine B runs: > > /usr/bin/rsync -ax --numeric-ids --delete --delete-excluded \ > --exclude-from=/Backup/RB_None.excl \ > --link-dest=/Backup/Base/xxxxx \ rsync://root at 10.99.11.11/Daten1/ > \ /Backup/Base/yyyyyyy \ --no-c --no-z > > The files are all about 8 MB in size. > > 8302592 100% 1.98MB/s 0:00:04 (xfer#861, > to-check=51173/94679) > > The same - slow - transfer happens at 1GB and 5GB files. > > > Any ideas what can cause this slow transmission? > >- -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8DUYYACgkQVKC1jlbQAQePuwCeN8FyFNmWlXYOBx5CrBd8VHod hq8AmQFSHAMdOIV0SHLAb3iEEHWLnxTd =932T -----END PGP SIGNATURE-----
On Tue, Jan 3, 2012 at 3:42 PM, R. Pietsch <rp.rsync at pcs-at.com> wrote:> > Hi! > > I do a rsync between 2 machines. The throughput is only 2 MByte/Sec. > > Each machine is a Supermicro server with > ? ?2 x 8 Core Opteron 6128 > ? ?64 GByte of ECC RAM > ? ?1 LSI MegaRAID SAS 9280-24i4e > ? ?24 x 2TByte SATA Disks as a RAID6 > ? ?2 Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network-cards > > Both run Ubuntu 11.04 64Bit. > Both use rsync version 3.0.7 ?protocol version 30 > > There are no errors reported. > The local read and write rate to the disks is 1,1 GB/s. > A network test shows 800MByte/s over the direct coupled network-cards. > The CPU-usage is ?5 to 7% of one core. > Memory-usage is 1703 out of 64557MB. > There are no other jobs running on the two machines. > The LSI controllers both say: State: Optimal > > > machine A has a rsync-daemon with the following configuration: > ---------- > read only = false > use chroot = false > hosts allow = 10.99.11.0/24 > [Daten1] > ? ? ? ?path = /var/daten1 > ? ? ? ?use chroot = yes > ? ? ? ?read only = yes > ? ? ? ?list = yes > ? ? ? ?uid = root > ? ? ? ?gid = root > ? ? ? ?strict modes = yes > ? ? ? ?ignore errors = no > ? ? ? ?ignore nonreadable = yes > ? ? ? ?transfer logging = no > ? ? ? ?timeout = 600 > ? ? ? ?refuse options = checksum dry-run > ? ? ? ?dont compress = * > ---------- > > machine B runs: > > /usr/bin/rsync -ax --numeric-ids --delete --delete-excluded \ > ? ?--exclude-from=/Backup/RB_None.excl \ > ? ?--link-dest=/Backup/Base/xxxxx \ > ? ?rsync://root at 10.99.11.11/Daten1/ \ > ? ?/Backup/Base/yyyyyyy \ > ? ?--no-c --no-z > > The files are all about 8 MB in size. > > ? ?8302592 100% ? ?1.98MB/s ? ?0:00:04 (xfer#861, to-check=51173/94679) > > The same - slow - transfer happens at 1GB and 5GB files. > > > Any ideas what can cause this slow transmission?You do have jumbo frames enabled on both servers and the network equipment in between? Do check the TCP/IP stack parameters, as most bandwidth testing is done with UDP.
Am 04.01.2012 10:46, schrieb Hendrik Visage:> On Tue, Jan 3, 2012 at 3:42 PM, R. Pietsch <rp.rsync at pcs-at.com> wrote: >> Hi! >> >> I do a rsync between 2 machines. The throughput is only 2 MByte/Sec. >> >> Each machine is a Supermicro server with >> 2 x 8 Core Opteron 6128 >> 64 GByte of ECC RAM >> 1 LSI MegaRAID SAS 9280-24i4e >> 24 x 2TByte SATA Disks as a RAID6 >> 2 Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network-cards >> >> Both run Ubuntu 11.04 64Bit. >> Both use rsync version 3.0.7 protocol version 30 >> >> There are no errors reported. >> The local read and write rate to the disks is 1,1 GB/s. >> A network test shows 800MByte/s over the direct coupled network-cards. >> The CPU-usage is 5 to 7% of one core. >> Memory-usage is 1703 out of 64557MB. >> There are no other jobs running on the two machines. >> The LSI controllers both say: State: Optimal >> >> >> machine A has a rsync-daemon with the following configuration: >> ---------- >> read only = false >> use chroot = false >> hosts allow = 10.99.11.0/24 >> [Daten1] >> path = /var/daten1 >> use chroot = yes >> read only = yes >> list = yes >> uid = root >> gid = root >> strict modes = yes >> ignore errors = no >> ignore nonreadable = yes >> transfer logging = no >> timeout = 600 >> refuse options = checksum dry-run >> dont compress = * >> ---------- >> >> machine B runs: >> >> /usr/bin/rsync -ax --numeric-ids --delete --delete-excluded \ >> --exclude-from=/Backup/RB_None.excl \ >> --link-dest=/Backup/Base/xxxxx \ >> rsync://root at 10.99.11.11/Daten1/ \ >> /Backup/Base/yyyyyyy \ >> --no-c --no-z >> >> The files are all about 8 MB in size. >> >> 8302592 100% 1.98MB/s 0:00:04 (xfer#861, to-check=51173/94679) >> >> The same - slow - transfer happens at 1GB and 5GB files. >> >> >> Any ideas what can cause this slow transmission? > > You do have jumbo frames enabled on both servers and the network > equipment in between? > > Do check the TCP/IP stack parameters, as most bandwidth testing is > done with UDP.Jumbo frames are NOT enabled - (they cause trouble with samba) and the test is done with netperf and tcp. The result is ----------------------- TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.99.11.11 (10.99.11.11) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 7300.70 ----------------------- So we get 7.3 GBit with a tcp-stream - rawly 800MByte/sec -- Mit freundlichen Gr??en / best regards Ing. Rainer Pietsch ---------------------------------------------------------- PCS - Pichler Computer Systeme Inh. Claudia Pichler-Pietsch Hauptplatz 10 A-2751 Steinabr?ckl ?sterreich / Austria ---------------------------------------------------------- mail: r.pietsch at pcs-at.com web: http://www.pcs-at.com tel.: +43 (2622) 420 19 / 15 mobil: +43 (676) 31 242 69 fax: +43 (2622) 420 19 / 20 ----------------------------------------------------------
did you try scp (although that could be CPU-bound due to crypto), ftp or wget - ie see how other TCP apps do the same job? If they all show the same speed - it's not an rsync problem -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Am 04.01.2012 16:24, schrieb Donald Pearson:> Perhaps it's time to look at a pcap of the rsync transfer. Throughput > is in the aggregate over time, I've seen instances where there are > excessively long delays between packets causing a low overall throughput.Ok, verry good idea! I try this and post the results.> > On Wed, Jan 4, 2012 at 5:06 AM, Rainer Pietsch <rp.rsync at pcs-at.com > <mailto:rp.rsync at pcs-at.com>> wrote: > > > > Am 04.01.2012 10:46, schrieb Hendrik Visage: > > On Tue, Jan 3, 2012 at 3:42 PM, R. Pietsch <rp.rsync at pcs-at.com > <mailto:rp.rsync at pcs-at.com>> wrote: > >> Hi! > >> > >> I do a rsync between 2 machines. The throughput is only 2 > MByte/Sec. > >> > >> Each machine is a Supermicro server with > >> 2 x 8 Core Opteron 6128 > >> 64 GByte of ECC RAM > >> 1 LSI MegaRAID SAS 9280-24i4e > >> 24 x 2TByte SATA Disks as a RAID6 > >> 2 Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network-cards > >> > >> Both run Ubuntu 11.04 64Bit. > >> Both use rsync version 3.0.7 protocol version 30 > >> > >> There are no errors reported. > >> The local read and write rate to the disks is 1,1 GB/s. > >> A network test shows 800MByte/s over the direct coupled > network-cards. > >> The CPU-usage is 5 to 7% of one core. > >> Memory-usage is 1703 out of 64557MB. > >> There are no other jobs running on the two machines. > >> The LSI controllers both say: State: Optimal > >> > >> > >> machine A has a rsync-daemon with the following configuration: > >> ---------- > >> read only = false > >> use chroot = false > >> hosts allow = 10.99.11.0/24 <http://10.99.11.0/24> > >> [Daten1] > >> path = /var/daten1 > >> use chroot = yes > >> read only = yes > >> list = yes > >> uid = root > >> gid = root > >> strict modes = yes > >> ignore errors = no > >> ignore nonreadable = yes > >> transfer logging = no > >> timeout = 600 > >> refuse options = checksum dry-run > >> dont compress = * > >> ---------- > >> > >> machine B runs: > >> > >> /usr/bin/rsync -ax --numeric-ids --delete --delete-excluded \ > >> --exclude-from=/Backup/RB_None.excl \ > >> --link-dest=/Backup/Base/xxxxx \ > >> rsync://root at 10.99.11.11/Daten1/ > <http://root at 10.99.11.11/Daten1/> \ > >> /Backup/Base/yyyyyyy \ > >> --no-c --no-z > >> > >> The files are all about 8 MB in size. > >> > >> 8302592 100% 1.98MB/s 0:00:04 (xfer#861, > to-check=51173/94679) > >> > >> The same - slow - transfer happens at 1GB and 5GB files. > >> > >> > >> Any ideas what can cause this slow transmission? > > > > You do have jumbo frames enabled on both servers and the network > > equipment in between? > > > > Do check the TCP/IP stack parameters, as most bandwidth testing is > > done with UDP. > Jumbo frames are NOT enabled - (they cause trouble with samba) and the > test is done with netperf and tcp. > The result is > ----------------------- > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.99.11.11 > (10.99.11.11) port 0 AF_INET : demo > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 87380 16384 16384 10.00 7300.70 > > ----------------------- > So we get 7.3 GBit with a tcp-stream - rawly 800MByte/sec > > > -- > Mit freundlichen Gr??en / best regards > > Ing. Rainer Pietsch > ---------------------------------------------------------- > PCS - Pichler Computer Systeme > Inh. Claudia Pichler-Pietsch > Hauptplatz 10 > A-2751 Steinabr?ckl > ?sterreich / Austria > ---------------------------------------------------------- > mail: r.pietsch at pcs-at.com <mailto:r.pietsch at pcs-at.com> > web: http://www.pcs-at.com > tel.: +43 (2622) 420 19 / 15 > <tel:%2B43%20%282622%29%20420%2019%20%2F%2015> > mobil: +43 (676) 31 242 69 <tel:%2B43%20%28676%29%2031%20242%2069> > fax: +43 (2622) 420 19 / 20 > <tel:%2B43%20%282622%29%20420%2019%20%2F%2020> > ---------------------------------------------------------- > > -- > Please use reply-all for most replies to avoid omitting the > mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html > <http://www.catb.org/%7Eesr/faqs/smart-questions.html> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20120106/14cd5c92/attachment.html>
I had network data transfer issue some time ago where transfers in one (A to B) direction were at full network speed.. Transfers in opposite directions (B to A) where going at 1 to 2 Kbit/sec. Eventually the cause was tracked down and it turned out to be a duplex mismatch caused by auto-negotiation protocols between two switches, and also between one of the switches and one of the hosts Tomasz On Mon, January 9, 2012 20:33, Hendrik Visage wrote:> On Sat, Jan 7, 2012 at 10:32 PM, R. Pietsch <rp.rsync at pcs-at.com> wrote: >> I try to capture such a rsync-stream with wireshark. > > Hmmm... receiving or transmitting side? > I *assume* the transmitting side given the time difference between the > packets and their ACKs. > >> the times are for example: >> 10.510155 for the rsync-packet (Len=27512) >> 10.550024 for the Ack >> 10.550181 for the next rsync (len=27512) >> 10.589985 for the Ack > > What would be interesting, would be to compare the absolute times of > both sides during this to see the actual network latency, and the > processing time on the receiving side, as the .03-.04 difference could > be anything. > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >-- Tomasz M. Ciolek ******************************************************************************* tmc at vandradlabs dot com dot au ******************************************************************************* GPG Key ID: 0x41C4C2F0 GPG Key Fingerprint: 3883 B308 8256 2246 D3ED A1FF 3A1D 0EAD 41C4 C2F0 Key available on good key-servers *******************************************************************************
Am 2012-01-09 11:37, schrieb tmc at vandradlabs.com.au:> I had network data transfer issue some time ago where transfers in one (A > to B) direction were at full network speed.. Transfers in opposite > directions (B to A) where going at 1 to 2 Kbit/sec. > > Eventually the cause was tracked down and it turned out to be a duplex > mismatch caused by auto-negotiation protocols between two switches, and > also between one of the switches and one of the hosts > > TomaszHere are the infos about the links: Receiver to Transmitter: root at Marc:~# ethtool eth3 Settings for eth3: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full 10000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full 10000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: on Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes root at Marc:~# netperf -H 10.99.11.11 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.99.11.11 (10.99.11.11) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 6132.09 root at Marc:~# iperf -c 10.99.11.11 ---------------------------------------------------------- Client connecting to 10.99.11.11, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.99.11.6 port 48260 connected with 10.99.11.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 5.83 GBytes 5.01 Gbits/sec root at Marc:~# dd if=/dev/zero of=/var/samba/zerotest bs=1M count=10K 10240+0 Datens?tze ein 10240+0 Datens?tze aus 10737418240 Bytes (11 GB) kopiert, 11,6261 s, 924 MB/s root at Marc:~# dd if=/var/samba/zerotest of=/dev/null 20971520+0 Datens?tze ein 20971520+0 Datens?tze aus 10737418240 Bytes (11 GB) kopiert, 17,8593 s, 601 MB/s root at Marc:~# dd if=/var/samba/zerotest of=/dev/null bs=1M 10240+0 Datens?tze ein 10240+0 Datens?tze aus 10737418240 Bytes (11 GB) kopiert, 4,62835 s, 2,3 GB/s root at Marc:~# date ; nc 10.99.11.6 1234 </var/samba/zerotest ; date Mo 9. Jan 17:03:39 CET 2012 Mo 9. Jan 17:04:29 CET 2012 #################### Transmitter to receiver: root at titus:~# ethtool eth2 Settings for eth2: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full 10000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full 10000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: on Supports Wake-on: d Wake-on: d Current message level: 0x00000007 (7) drv probe link Link detected: yes root at titus:~# netperf -H 10.99.11.6 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.99.11.6 (10.99.11.6) port 0 AF_INET : demo Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.00 7617.07 root at titus:~# iperf -c 10.99.11.6 ------------------------------------------------------------ Client connecting to 10.99.11.6, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.99.11.248 port 48717 connected with 10.99.11.6 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 8.65 GBytes 7.43 Gbits/sec root at titus:~# dd if=/dev/zero of=/Backup/Base/zerotest bs=1M count=10K 10240+0 Datens?tze ein 10240+0 Datens?tze aus 10737418240 Bytes (11 GB) kopiert, 11,7954 s, 910 MB/s ##################> > On Mon, January 9, 2012 20:33, Hendrik Visage wrote: >> On Sat, Jan 7, 2012 at 10:32 PM, R. Pietsch <rp.rsync at pcs-at.com> wrote: >>> I try to capture such a rsync-stream with wireshark. >> Hmmm... receiving or transmitting side? >> I *assume* the transmitting side given the time difference between the >> packets and their ACKs. >> >>> the times are for example: >>> 10.510155 for the rsync-packet (Len=27512) >>> 10.550024 for the Ack >>> 10.550181 for the next rsync (len=27512) >>> 10.589985 for the Ack >> What would be interesting, would be to compare the absolute times of >> both sides during this to see the actual network latency, and the >> processing time on the receiving side, as the .03-.04 difference could >> be anything. >> -- >> Please use reply-all for most replies to avoid omitting the mailing list. >> To unsubscribe or change options: >> https://lists.samba.org/mailman/listinfo/rsync >> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html >> >-- Mit freundlichen Gr??en / best regards Ing. Rainer Pietsch ---------------------------------------------------------- PCS - Pichler Computer Systeme Inh. Claudia Pichler-Pietsch Hauptplatz 10 A-2751 Steinabr?ckl ?sterreich / Austria ---------------------------------------------------------- mail: r.pietsch at pcs-at.com web: http://www.pcs-at.com tel.: +43 (2622) 420 19 / 15 mobil: +43 (676) 31 242 69 fax: +43 (2622) 420 19 / 20 ----------------------------------------------------------