[crossbow-discuss] Networking problem between virtual hosts on a single machine...

James Carlson
Thu, 15 Jul 2010 18:20:45 -0000

> Thank you so much for the help. You are certainly on the right track. On
the Fail connections the above command does hang. I titered the packet size and
found that the largest ping size that would work is 1468 which I assume is
ethernet MTU (1500) minus a few overhead bytes.

1468 + ICMP header 8 + IP 20 = 1496 octets in the IP datagram that will
fit on the wire.

> -- Now for something you didn''t request.  The underlying machine
has the following netstat -ni.
>
> hobbit$ netstat -ni
> Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis
Queue
> lo0   8232 127.0.0.0     127.0.0.1      556534060 0     556534060 0     0
0
> bnx0  1496 10.10.102.0   10.10.102.130  1177212916 0     3754428493 0     0
0
> vboxnet0 1500 192.168.56.0  192.168.56.1   0      0     4952   0     0
0

That seems consistent to me.

> ---   Notice that the MTU on bnx0 is 1496 (not 1500). The same thing is
true for xVM hosts. The underlying bnx0 interface has a 4 byte smaller MTU than
the xnf0 interfaces in the domU''s. Therefore, I wouldn''t be a
bit surprised if this is what is clogging the plumbing.  Let''s see.
Yes, if I use ifconfig to lower the MTU inside the VB guests the ping starts
working again.

I''m not too familiar with "bnx," but why would it be set like
that?

>    This would appear to be a bug but it isn''t clear to me
who''s bug it is. Is this something which should be fixed by the
virtualization tools or in the underlying network (crossbow?). Should the
virtual switch between the crossbow interfaces be resolving this?  Again,
I''m no networking guru.

I would call it the system administrator''s fault.  The system is doing
what it''s being told to do, though it''s clearly not what you
want.

If this were a real physical network comprising multiple independent
nodes rather than virtual ones on a single box, there would be no good
way (other than something exotic like LLDP) to detect this sort of
misconfiguration.  It would fail just like you''re seeing now.

But, yeah, it''d be nice if these virtual networking devices had
enhanced
abilities to detect configuration errors -- at least the ones that could
in theory be detected locally -- and it''d also be nice if some of those
drivers were more liberal in what they accept (i.e., having your MTU
forced down to 1496 doesn''t mean that you need to discard a 1500 octet
packet if you receive one).

I''m just not sure I''d call it necessarily a bug.

>    What would be the best way to persistently set a reduce MTU in the
virtual hosts as a workaround?  I''m assuming the MTU on bnx0 is set
down 4 bytes from the normal 1500 for some reason.

I dunno what''s up with bnx0 (quasi VLANish hackery?).  That''s
the one
thing here that does look like a "bug."

Assuming that interface can''t just be fixed, I would expect that you
should be able to set the ''mtu'' property on those other
interfaces using
dladm ... though I''m not an expert in xVM or VirtualBox issues.