zfs discuss - Jul 2010 - zfs send to remote any ideas for a faster way than ssh?

I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before the 10g link is.

I''vw also tried mbuffer, but I get broken pipe errors part way through
the transfer.

I''m open to ideas for faster ways to to either zfs send directly or
through a compressed file of the zfs send output.

For the moment I;

zfs send > pigz
scp arcfour the file gz file to the remote host
gunzip < to zfs receive

This takes a very long time for 3 TB of data, and barely makes use the 10g
connection between the machines due to the CPU limiting on the scp and gunzip
processes.

Thank you for your thoughts

Richard J.
-- 
This message posted from opensolaris.org

On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com> wrote:
> I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before the 10g link is.
>
> I''vw also tried mbuffer, but I get broken pipe errors part way
through the transfer.
>
> I''m open to ideas for faster ways to to either zfs send directly
or through a compressed file of the zfs send output.
If this is across a trusted link, have a look at the HPN patches to
ZFS.  There are three main benefits to these patches:
  - increased (and dynamic) buffers internal to SSH
  - adds a multi-threaded aes cipher
  - adds the NONE cipher for non-encrypted data transfers
(authentication is still encrypted)

If one end of the SSH connection is HPN-enabled, you can increase your
bulk data transfer around 10-15%, just by adjusting the buffer size in
ssh_config or sshd_config (depending on which side has HPN).

If both ends of the SSH connection are HPN-enabled, you can increase
your bulk data transfer rate around 30% just by adjusting the buffer
in the sshd_config.

Enabling the -mtr versions of the cipher will use multiple CPU cores
for encrypting/decrypting, improving throughput.

If you trust the link completely (private link), you can enable the
NONE cipher on the ssh commandline and via sshd_config, and the data
transfer will happen unencrypted, thus maxing out the bandwidth.

We can saturate a gigabit fibre link between two ZFS storage servers
using rsync.  You should be able to saturate a 10G link using zfs
send/recv, so long as both the systems can read/write that fast.

http://www.psc.edu/networking/projects/hpn-ssh/

-- 
Freddie Cash
fjwcash at gmail.com

Richard Jahnel wrote:
> I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before the 10g link is.
>
> I''vw also tried mbuffer, but I get broken pipe errors part way
through the transfer.
>   
Any idea why? Does the zfs send or zfs receive bomb out part way through?

Might be worth trying it over rsh if security isn''t an issue, and then 
you lose the encryption overhead. Trouble is that then you''ve got
almost
no buffering, which can do bad things to the performance, which is why 
mbuffer would be ideal if it worked for you.
> I''m open to ideas for faster ways to to either zfs send directly
or through a compressed file of the zfs send output.
>
> For the moment I;
>
> zfs send > pigz
> scp arcfour the file gz file to the remote host
> gunzip < to zfs receive
>
> This takes a very long time for 3 TB of data, and barely makes use the 10g
connection between the machines due to the CPU limiting on the scp and gunzip
processes.
>   
Also, if you have multiple datasets to send, might be worth seeing if 
sending them in parallel helps.

-- 
Andrew Gabriel

>Any idea why? Does the zfs send or zfs receive bomb out part way through?
I have no idea why mbuffer fails. Changing the -s from 128 to 1536 made it take
longer to occur and slowed it down bu about 20% but didn''t resolve the
issue. It just ment I might get as far as 2.5gb before mbuffer bombed with
broken pipe. Trying -r and -R with various values had no effect.
>Might be worth trying it over rsh if security isn''t an issue, and
then
>you lose the encryption overhead. Trouble is that then you''ve got
almost
>no buffering, which can do bad things to the performance, which is why 
>mbuffer would be ideal if it worked for you.
I seem to remember reading that rsh was remapped to ssh in Solaris.

I heard of some folks using netcat.
I haven''t figured out where to get netcat nor the syntax for using it
yet.
-- 
This message posted from opensolaris.org

>If this is across a trusted link, have a look at the HPN patches to
>ZFS. There are three main benefits to these patches:
>- increased (and dynamic) buffers internal to SSH
>- adds a multi-threaded aes cipher
>- adds the NONE cipher for non-encrypted data transfers
>(authentication is still encrypted)
Yes I''ve looked at that site before. While I''m comfortable
with the zfs storage aspects of osol, I''m still quite leary of
compiling from source code. I really really don''t want to break these
machines. If it''s a package I can figure out how to get and install it,
but code patches scare me.
-- 
This message posted from opensolaris.org

On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com> wrote:
> I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before the 10g link is.
>
> I''vw also tried mbuffer, but I get broken pipe errors part way
through the transfer.
>
> I''m open to ideas for faster ways to to either zfs send directly
or through a compressed file of the zfs send output.
>
> For the moment I;
>
> zfs send > pigz
> scp arcfour the file gz file to the remote host
> gunzip < to zfs receive
>
> This takes a very long time for 3 TB of data, and barely makes use the 10g
connection between the machines due to the CPU limiting on the scp and gunzip
processes.
>
> Thank you for your thoughts
>
> Richard J.
> --
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
I found builds <130 had issues with TCP. I could reproduce TCP
timeouts/socket errors up until I got on 132. I have stayed on 132 so
far since I haven''t found any other show stoppers.
Mbuffer is probably your best bet, I rolled mbuffer into my
replication scripts, which I could share if anyone''s interested.
Older versions of my script are on www.brentrjones.com but I have a
new one which uses mbuffer



-- 
Brent Jones
brent at servuhome.net

Using SunOS XXXXX 5.11 snv_133 i86pc i386 i86pc. So the network thing that
was fixed in 129 shouldn''t be the issue.

-----Original Message-----
From: Brent Jones [mailto:brent at servuhome.net] 
Sent: Monday, July 19, 2010 1:02 PM
To: Richard Jahnel
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] zfs send to remote any ideas for a faster way
than ssh?

On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com>
wrote:
> I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before
the 10g link is.
>
> I''vw also tried mbuffer, but I get broken pipe errors part way
through the
transfer.
>
> I''m open to ideas for faster ways to to either zfs send directly
or
through a compressed file of the zfs send output.
>
> For the moment I;
>
> zfs send > pigz
> scp arcfour the file gz file to the remote host gunzip < to zfs 
> receive
>
> This takes a very long time for 3 TB of data, and barely makes use the 10g
connection between the machines due to the CPU limiting on the scp and
gunzip processes.
>
> Thank you for your thoughts
>
> Richard J.
> --
> This message posted from opensolaris.org 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
I found builds <130 had issues with TCP. I could reproduce TCP
timeouts/socket errors up until I got on 132. I have stayed on 132 so far
since I haven''t found any other show stoppers.
Mbuffer is probably your best bet, I rolled mbuffer into my replication
scripts, which I could share if anyone''s interested.
Older versions of my script are on www.brentrjones.com but I have a new one
which uses mbuffer



--
Brent Jones
brent at servuhome.net

Hi,

If you can share those scripts that make use of mbuffer, please feel
free to do so ;)


Bruno
On 19-7-2010 20:02, Brent Jones wrote:
> On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com> wrote:
>   
>> I''ve tried ssh blowfish and scp arcfour. both are CPU limited
long before the 10g link is.
>>
>> I''vw also tried mbuffer, but I get broken pipe errors part way
through the transfer.
>>
>> I''m open to ideas for faster ways to to either zfs send
directly or through a compressed file of the zfs send output.
>>
>> For the moment I;
>>
>> zfs send > pigz
>> scp arcfour the file gz file to the remote host
>> gunzip < to zfs receive
>>
>> This takes a very long time for 3 TB of data, and barely makes use the
10g connection between the machines due to the CPU limiting on the scp and
gunzip processes.
>>
>> Thank you for your thoughts
>>
>> Richard J.
>> --
>> This message posted from opensolaris.org
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
>>     
> I found builds <130 had issues with TCP. I could reproduce TCP
> timeouts/socket errors up until I got on 132. I have stayed on 132 so
> far since I haven''t found any other show stoppers.
> Mbuffer is probably your best bet, I rolled mbuffer into my
> replication scripts, which I could share if anyone''s interested.
> Older versions of my script are on www.brentrjones.com but I have a
> new one which uses mbuffer
>
>
>
>   

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Richard,

On 19 Jul 2010, at 18:49, Richard Jahnel wrote:
>
> I heard of some folks using netcat.
> I haven''t figured out where to get netcat nor the syntax for using
> it yet.

I also did a bit of research into using netcat and found this...

http://www.mail-archive.com/storage-discuss at opensolaris.org/msg05920.html
> sourcehost:
> zfs send | netcat $remotehost $remoteport
>
> desthost:
> netcat -l -p $myport | zfs receive
Hope that helps,

- Daniel

Richard Jahnel wrote:
>> Any idea why? Does the zfs send or zfs receive bomb out part way
through?
>>     
>
> I have no idea why mbuffer fails. Changing the -s from 128 to 1536 made it
take longer to occur and slowed it down bu about 20% but didn''t resolve
the issue. It just ment I might get as far as 2.5gb before mbuffer bombed with
broken pipe. Trying -r and -R with various values had no effect.
>   
I found that where the network bandwidth and the disks'' throughput are 
similar (which requires a pool with many top level vdevs in the case of 
a 10Gb link), you ideally want a buffer on the receive side which will 
hold about 5 seconds worth of data. A large buffer on the transmit side 
didn''t help. The aim is to be able to continue steaming data across the
network whilst a transaction commit happens at the receive end and zfs 
receive isn''t reading, but to have the data ready locally for zfs 
receive when it starts reading again. Then the network will stream, in 
spite of the bursty read nature of zfs receive.

I recorded this in bugid 
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6729347
However, I haven''t verified the extent to which this still happens on 
more recent builds.
>> Might be worth trying it over rsh if security isn''t an issue,
and then
>> you lose the encryption overhead. Trouble is that then you''ve
got almost
>> no buffering, which can do bad things to the performance, which is why 
>> mbuffer would be ideal if it worked for you.
>>     
>
> I seem to remember reading that rsh was remapped to ssh in Solaris.
>   
No.
On the system you''re rsh''ing to, you will have to "svcadm
enable
svc:/network/shell:default", and set up appropriate authorisation in 
~/.rhosts
> I heard of some folks using netcat.
> I haven''t figured out where to get netcat nor the syntax for using
it yet.
>   
I used a buffering program of my own, but I presume mbuffer would work too.

-- 
Andrew Gabriel

On Mon, Jul 19, 2010 at 11:14 AM, Bruno Sousa <bsousa at epinfante.com>
wrote:
> Hi,
>
> If you can share those scripts that make use of mbuffer, please feel
> free to do so ;)
>
>
> Bruno
> On 19-7-2010 20:02, Brent Jones wrote:
>> On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com> wrote:
>>
>>> I''ve tried ssh blowfish and scp arcfour. both are CPU
limited long before the 10g link is.
>>>
>>> I''vw also tried mbuffer, but I get broken pipe errors part
way through the transfer.
>>>
>>> I''m open to ideas for faster ways to to either zfs send
directly or through a compressed file of the zfs send output.
>>>
>>> For the moment I;
>>>
>>> zfs send > pigz
>>> scp arcfour the file gz file to the remote host
>>> gunzip < to zfs receive
>>>
>>> This takes a very long time for 3 TB of data, and barely makes use
the 10g connection between the machines due to the CPU limiting on the scp and
gunzip processes.
>>>
>>> Thank you for your thoughts
>>>
>>> Richard J.
>>> --
>>> This message posted from opensolaris.org
>>> _______________________________________________
>>> zfs-discuss mailing list
>>> zfs-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>>
>>>
>> I found builds <130 had issues with TCP. I could reproduce TCP
>> timeouts/socket errors up until I got on 132. I have stayed on 132 so
>> far since I haven''t found any other show stoppers.
>> Mbuffer is probably your best bet, I rolled mbuffer into my
>> replication scripts, which I could share if anyone''s
interested.
>> Older versions of my script are on www.brentrjones.com but I have a
>> new one which uses mbuffer
>>
I can''t seem to upload files to my Wordpress site any longer, so I put
it up on Pastebin for now:

http://pastebin.com/2feviTCy

Hope it helps others


-- 
Brent Jones
brent at servuhome.net

On 19-7-2010 20:36, Brent Jones wrote:
> On Mon, Jul 19, 2010 at 11:14 AM, Bruno Sousa <bsousa at
epinfante.com> wrote:
>   
>> Hi,
>>
>> If you can share those scripts that make use of mbuffer, please feel
>> free to do so ;)
>>
>>
>> Bruno
>> On 19-7-2010 20:02, Brent Jones wrote:
>>     
>>> On Mon, Jul 19, 2010 at 9:06 AM, Richard Jahnel <richard at
ellipseinc.com> wrote:
>>>
>>>       
>>>> I''ve tried ssh blowfish and scp arcfour. both are CPU
limited long before the 10g link is.
>>>>
>>>> I''vw also tried mbuffer, but I get broken pipe errors
part way through the transfer.
>>>>
>>>> I''m open to ideas for faster ways to to either zfs
send directly or through a compressed file of the zfs send output.
>>>>
>>>> For the moment I;
>>>>
>>>> zfs send > pigz
>>>> scp arcfour the file gz file to the remote host
>>>> gunzip < to zfs receive
>>>>
>>>> This takes a very long time for 3 TB of data, and barely makes
use the 10g connection between the machines due to the CPU limiting on the scp
and gunzip processes.
>>>>
>>>> Thank you for your thoughts
>>>>
>>>> Richard J.
>>>> --
>>>> This message posted from opensolaris.org
>>>> _______________________________________________
>>>> zfs-discuss mailing list
>>>> zfs-discuss at opensolaris.org
>>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>>>
>>>>
>>>>         
>>> I found builds <130 had issues with TCP. I could reproduce TCP
>>> timeouts/socket errors up until I got on 132. I have stayed on 132
so
>>> far since I haven''t found any other show stoppers.
>>> Mbuffer is probably your best bet, I rolled mbuffer into my
>>> replication scripts, which I could share if anyone''s
interested.
>>> Older versions of my script are on www.brentrjones.com but I have a
>>> new one which uses mbuffer
>>>
>>>       
> I can''t seem to upload files to my Wordpress site any longer, so I
put
> it up on Pastebin for now:
>
> http://pastebin.com/2feviTCy
>
> Hope it helps others
>
>
>   
Thanks mate!
Bruno

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

On Jul 19, 2010, at 10:49 AM, Richard Jahnel wrote:
>> Any idea why? Does the zfs send or zfs receive bomb out part way
through?
> 
> I have no idea why mbuffer fails. Changing the -s from 128 to 1536 made it
take longer to occur and slowed it down bu about 20% but didn''t resolve
the issue. It just ment I might get as far as 2.5gb before mbuffer bombed with
broken pipe. Trying -r and -R with various values had no effect.
I''ve used mbuffer to transfer hundreds of TB without a problem in
mbuffer
itself. You will get disconnected if the send or receive prematurely ends,
though.
>> Might be worth trying it over rsh if security isn''t an issue,
and then
>> you lose the encryption overhead. Trouble is that then you''ve
got almost
>> no buffering, which can do bad things to the performance, which is why 
>> mbuffer would be ideal if it worked for you.
> 
> I seem to remember reading that rsh was remapped to ssh in Solaris.
> 
> I heard of some folks using netcat.
> I haven''t figured out where to get netcat nor the syntax for using
it yet.
On Nexenta systems, netcat (and nc) are in /usr/bin and the man pages
are in the default MANPATH.
 -- richard

-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
Enterprise class storage for everyone
www.nexenta.com

>I''ve used mbuffer to transfer hundreds of TB without a problem in
mbuffer
>itself. You will get disconnected if the send or receive prematurely ends,
>though.

mbuffer itself very specifically ends with a broken pipe error. Very quickly
with s set to 128 or after sometime with s set over 1024.

My only thought at this point is it maybe something with either the myri10ge
driver or the fact the machines are directly connected with out the use of an
intervening switch.

Perhaps someone over in networking my have a thought in that direction.

Richard J.
-- 
This message posted from opensolaris.org

FWIW I found netcat over at CSW.

http://www.opencsw.org/packages/CSWnetcat/
-- 
This message posted from opensolaris.org

On 07/20/10 08:20 AM, Richard Jahnel wrote:
>> I''ve used mbuffer to transfer hundreds of TB without a problem
in mbuffer
>> itself. You will get disconnected if the send or receive prematurely
ends,
>> though.
>>      
>
> mbuffer itself very specifically ends with a broken pipe error. Very
quickly with s set to 128 or after sometime with s set over 1024.
>
> My only thought at this point is it maybe something with either the
myri10ge driver or the fact the machines are directly connected with out the use
of an intervening switch.
>
>    
I assume the error comes from the sending side?  That may mean the 
receiving process has died.  Are there any errors (do you use -v for 
send and or receive?) reported?

I don''t use mbuffer, but do use large buffers to couple zfs sends to 
receives and I''ve never seen a broken pipe.

-- 
Ian.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Richard Jahnel
> 
> I''vw also tried mbuffer, but I get broken pipe errors part way
through
> the transfer.
The standard answer is mbuffer.  I think you should ask yourself what''s
going wrong with mbuffer.  You''re not, by any chance, sending across a
LACP
aggregated link, are you?  Most people don''t notice it, but I sure do,
that
usually LACP introduces packet errors.  Just watch your error counter, and
start cramming data through there.  So far I''ve never seen a single
implementation that passed this test...  Although I''m sure
I''ve just had bad
luck.

If you''re having some packet loss, that might explain the poor
performance
of ssh too.  Although ssh is known to slow things down in the best of cases
... I don''t know if the speed you''re seeing is reasonable
considering.

> zfs send > pigz
> scp arcfour the file gz file to the remote host
> gunzip < to zfs receive
I would recommend "xz -1" instead of pigz.  It''s roughly the
same speed
compression, with approx 2x smaller result.  I know xz does multi-threading,
but you might just want to "top" it or "prstat" it to be
sure.  I have a
hunch that it can only multi-thread actual files, and maybe can''t
multi-thread compression of stdin.  Maybe.

> This takes a very long time for 3 TB of data, and barely makes use the
> 10g connection between the machines due to the CPU limiting on the scp
> and gunzip processes.
That''s something ... I know pigz, pbzip2, and xz all compress
multi-threaded.  But what about uncompress?  You say your gunzip is
cpu-bound...  

If the above don''t decompress multi-threaded, I know threadzip does
multi-thread both compression & decompression.  For speed, I would recommend
using the "--fast" option.

If you consider threadzip, just ignore the comment about "this project is
considered dead, in favor of pigz, pbzip2, xz."  I''m the guy
behind
threadzip, and if there''s a reason for it to exist, I''ll just
put that on
the page instead.

On Mon, Jul 19, 2010 at 11:06 PM, Richard Jahnel <richard at
ellipseinc.com> wrote:
> I''ve tried ssh blowfish and scp arcfour. both are CPU limited long
before the 10g link is.
>
> I''vw also tried mbuffer, but I get broken pipe errors part way
through the transfer.
>
> I''m open to ideas for faster ways to to either zfs send directly
or through a compressed file of the zfs send output.
>
> For the moment I;
>
> zfs send > pigz
> scp arcfour the file gz file to the remote host
> gunzip < to zfs receive
>
> This takes a very long time for 3 TB of data, and barely makes use the 10g
connection between the machines due to the CPU limiting on the scp and gunzip
processes.
I usually do

zfs send | lzop -c | ssh -c blowfish ip_of_remote_server "lzop -dc |
zfs receive"

lzop is much faster than gzip. I''d also check how fast your disks are,
make sure it''s not the bottleneck.

-- 
Fajar

On Tue, Jul 20, 2010 at 12:59 AM, Edward Ned Harvey <shill at
nedharvey.com> wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Richard Jahnel
>>
>> I''vw also tried mbuffer, but I get broken pipe errors part way
through
>> the transfer.
>
> The standard answer is mbuffer. ?I think you should ask yourself
what''s
> going wrong with mbuffer. ?You''re not, by any chance, sending
across a LACP
> aggregated link, are you? ?Most people don''t notice it, but I sure
do, that
> usually LACP introduces packet errors. ?Just watch your error counter, and
> start cramming data through there. ?So far I''ve never seen a
single
> implementation that passed this test... ?Although I''m sure
I''ve just had bad
> luck.
>
> If you''re having some packet loss, that might explain the poor
performance
> of ssh too. ?Although ssh is known to slow things down in the best of cases
> ... I don''t know if the speed you''re seeing is reasonable
considering.
We have hundreds of servers using LACP and so far have not noticed any
increase in the error rate.

Could you share what implementations (OS, switch) have you tested and
how it was done ? I would like to try to simulate these issues.

-- 
Giovanni Tirloni
gtirloni at sysdroid.com

On the receiver

/opt/csw/bin/mbuffer -m 1G -I Ostor-1:8000 | zfs recv -F es1 at sunday
in @  0.0 kB/s, out @  0.0 kB/s, 43.7 GB total, buffer 100% fullcannot receive
new filesystem stream: invalid backup stream
mbuffer: error: outputThread: error writing to <stdout> at offset
0xaedf6a000: Broken pipe

summary: 43.7 GByte in  3 min 25.5 sec - average of  218 MB/s
mbuffer: warning: error during output to <stdout>: Broken pipe

On the sender

/sbin/zfs send es1 at sunday | /opt/csw/bin/mbuffer -m 1G -O Ostor-2:8000
in @  0.0 kB/s, out @  0.0 kB/s, 44.7 GB total, buffer 100% fullmbuffer: error:
outputThread: error writing to Ostor-2:8000 at offset 0xb2e0e6000: Broken pipe

summary: 44.7 GByte in  3 min 25.6 sec - average of  223 MB/s
mbuffer: warning: error during output to Ostor-2:8000: Broken pipe
-- 
This message posted from opensolaris.org

I''ll try an export/import and scrub of the receiving pool and see what
that does. I can''t take the sending pool offline to try that stuff
though.
-- 
This message posted from opensolaris.org

On 07/21/10 03:12 AM, Richard Jahnel wrote:
> On the receiver
>
> /opt/csw/bin/mbuffer -m 1G -I Ostor-1:8000 | zfs recv -F es1 at sunday
> in @  0.0 kB/s, out @  0.0 kB/s, 43.7 GB total, buffer 100% fullcannot
receive new filesystem stream: invalid backup stream
> mbuffer: error: outputThread: error writing to<stdout>  at offset
0xaedf6a000: Broken pipe
>
>    
It looks like zfs receive found an error in the received stream and 
broke the connection, hence the SIGPIPE on the sender.

Try streaming the receive data to a file rather than zfs receive and see 
if the full stream is received.  You could do the same on the sander and 
compare (md5sum) the two files.

-- 
Ian.

> From: Giovanni Tirloni [mailto:gtirloni at sysdroid.com]
> 
> We have hundreds of servers using LACP and so far have not noticed any
> increase in the error rate.
If the error rate is not zero, you have an increased error rate.

In linux, you just do this:
sudo /sbin/ifconfig -a | grep errors | grep -v ''errors:0 ''

All of my servers right now have error counter = 0 cumulative since last
reboot.

In solaris, I had to figure out the command once before, but I forget it
now.  I remember being misled ... Seeing one error counter stay at zero,
while another error counter increased.  While we were trying to enable LACP,
we''d get one error packet per ... I guess every 1Gb or so that
traversed the
LACP link.  This was fatal to ssh connections.  So unless you see a nonzero
error counter, I''m not sure how to tell you which error counter to
check.
Sorry I couldn''t be more informative on that ...

> Could you share what implementations (OS, switch) have you tested and
> how it was done ? I would like to try to simulate these issues.
I''ve done this on Dell switches, almost exclusively.  I''ve
done it with
Linux, Solaris 10u6 x86, and I think one Netapp.  So no, I cannot exclude
Dell switches as the possible cause.  I revisit and reattempt once every few
years, always with the same result.  I simply do not use LACP anywhere.