[crossbow-discuss] Networking problem between virtual hosts on a single machine...

Rand S. Huntzinger
Thu, 15 Jul 2010 17:55:18 -0000

> That sounds a lot like a bad MTU configuration.  A
> fundamental
> requirement for communicating on a shared medium like
> Ethernet (whether
> real or virtual) is that all of the attached nodes
> have the same MTU
> configured.  If they don''t, then the sorts of hangs
> you''re seeing are
> the result as large packets fail to get through.
>
> (More precisely: the MTU of any node connected to a
> shared medium cannot
> be larger than the MRU of any other node on that same
> medium without
> special provision to make sure that the largest frame
> sent to the
> restricted nodes is below their MRU.  But since
> Ethernet implementations
> rarely allow separate configuration of the MRU and
> MTU, and since
> special mechanisms to restrict MRU to individual
> hosts are both rare and
> hard to manage, it devolves to simply "use same MTU
> everywhere.")
>
> If I''m right on that, then I would expect that the
> "fail" cases also
> fail if you do "ping -sn HOST 2400".

Thank you so much for the help. You are certainly on the right track. On the
Fail connections the above command does hang. I titered the packet size and
found that the largest ping size that would work is 1468 which I assume is
ethernet MTU (1500) minus a few overhead bytes.

The netstat -ni  on the two VirtualBox hosts show:

indy$ netstat -ni
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 127.0.0.0     127.0.0.1      20160  0     20160  0     0      0
e1000g0 1500 10.10.102.0   10.10.102.105  422269 0     113615 0     0      0

Name  Mtu  Net/Dest                    Address                     Ipkts  Ierrs
Opkts  Oerrs Collis
lo0   8252 ::1                         ::1                         20160  0
20160  0     0
e1000g0 1500 fe80::8:20ff:fef5:70d0/10   fe80::8:20ff:fef5:70d0      422271 0
113616 0     0
indy$

-- and --

nevada$ netstat -ni
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 127.0.0.0     127.0.0.1      20026  0     20026  0     0      0
e1000g0 1500 10.10.102.0   10.10.102.103  417630 0     119392 0     0      0

Name  Mtu  Net/Dest                    Address                     Ipkts  Ierrs
Opkts  Oerrs Collis
lo0   8252 ::1                         ::1                         20026  0
20026  0     0
e1000g0 1500 fe80::8:20ff:fea3:c813/10   fe80::8:20ff:fea3:c813      417632 0
119395 0     0
nevada$


-- Now for something you didn''t request.  The underlying machine has
the following netstat -ni.

hobbit$ netstat -ni
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 127.0.0.0     127.0.0.1      556534060 0     556534060 0     0      0
bnx0  1496 10.10.102.0   10.10.102.130  1177212916 0     3754428493 0     0
0
vboxnet0 1500 192.168.56.0  192.168.56.1   0      0     4952   0     0      0

Name  Mtu  Net/Dest                    Address                     Ipkts  Ierrs
Opkts  Oerrs Collis
lo0   8252 ::1                         ::1                         556534060 0
556534060 0     0
hobbit$

---   Notice that the MTU on bnx0 is 1496 (not 1500). The same thing is true for
xVM hosts. The underlying bnx0 interface has a 4 byte smaller MTU than the xnf0
interfaces in the domU''s. Therefore, I wouldn''t be a bit
surprised if this is what is clogging the plumbing.  Let''s see.  Yes,
if I use ifconfig to lower the MTU inside the VB guests the ping starts working
again.

   This would appear to be a bug but it isn''t clear to me
who''s bug it is. Is this something which should be fixed by the
virtualization tools or in the underlying network (crossbow?). Should the
virtual switch between the crossbow interfaces be resolving this?  Again,
I''m no networking guru.

   What would be the best way to persistently set a reduce MTU in the virtual
hosts as a workaround?  I''m assuming the MTU on bnx0 is set down 4
bytes from the normal 1500 for some reason.

-- Rand Huntzinger