[crossbow-discuss] Networking problem between virtual hosts on a single machine...

James Carlson
Thu, 15 Jul 2010 15:42:15 -0000

> I have a observed a networking problem occurring largely between virtual
hosts running on the same machine. I have observed the problem with both
VirtualBox and xVM guests and it appears to affect multiple protocols (at least
ssh and NFS). It appears that an application network connection gets clogged
with transfers of data as small as 2K or so. For the purposes of testing,
I''m using the command "ssh HOST cat /etc/passwd ... | wc"
(where ... are additional copies of /etc/passwd).  The /etc/passwd file is a
little over 1K in size. I consider a test successful if a cat over 6000 bytes
succeeds.  Typically, the test fails with two copies of /etc/passwd (about 2400
bytes).  I am not an expert in networking so I''m describing the problem
at the application level.

That sounds a lot like a bad MTU configuration.  A fundamental
requirement for communicating on a shared medium like Ethernet (whether
real or virtual) is that all of the attached nodes have the same MTU
configured.  If they don''t, then the sorts of hangs you''re
seeing are
the result as large packets fail to get through.

(More precisely: the MTU of any node connected to a shared medium cannot
be larger than the MRU of any other node on that same medium without
special provision to make sure that the largest frame sent to the
restricted nodes is below their MRU.  But since Ethernet implementations
rarely allow separate configuration of the MRU and MTU, and since
special mechanisms to restrict MRU to individual hosts are both rare and
hard to manage, it devolves to simply "use same MTU everywhere.")

If I''m right on that, then I would expect that the "fail"
cases also
fail if you do "ping -sn HOST 2400".

What does "netstat -ni" say on each of these hosts?

(It''s also possible, but less likly, that some sort of LSO-type MTU
fudging is causing the problem.  I would expect a case like that to
leave ''ping'' intact but cause TCP to fail.)