xen discuss - Sep 2009 - Using an aggregation as a physical bridge for a CentOS 5.3 vnic?

Hello all!


It seems that I am unable to use an aggregation as a physical bridge
for a CentOS 5.3 client.


The dom0 is running SXCE 122 and has the following interfaces defined:

  root@glaurung# dladm show-link
  LINK        CLASS    MTU    STATE    OVER
  nge0        phys     1500   up       --
  nge1        phys     1500   up       --
  bge0        phys     1500   up       --
  bge1        phys     1500   up       --
  public0     aggr     1500   up       bge0 nge0

  root@glaurung# dladm show-aggr
  LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
  public0         L4       auto                 off           short       -----


The three physical interfaces bge0, nge0, and bge1 are all connected
to the same gigabit ethernet switch.


The domU is running a PVM CentOS 5.3:

  root@cent-01# uname -a
  Linux cent-01 2.6.18-128.el5xen #1 SMP Wed Jan 21 11:12:42 EST 2009 x86_64
x86_64 x86_64 GNU/Linux


Initially I defined the bridge for the xVM vnic to be public0:

    <interface type=''bridge''>
      <mac address=''00:bb:c0:00:40:01''/>
      <source bridge=''public0''/>
      <script path=''/usr/lib/xen/scripts/vif-vnic''/>
    </interface>

[Sorry to the forum readers for the broken XML display.]


This seriously confuses the CentOS domU.  While booting, it will complain
about a duplicate address:

  Sep 22 23:12:01 cent-01 avahi-daemon[1689]: Joining mDNS multicast group on
interface eth0.IPv4 with address 192.168.222.157.
  Sep 22 23:12:01 cent-01 avahi-daemon[1689]: Registering new address record for
192.168.222.157 on eth0.
  Sep 22 23:12:02 cent-01 kernel: eth0: duplicate address detected!

When the domU is up, TCP traffic does not work.  ICMP is the only thing
that seems to get through.  Specifically, NFS is unusable.


So I redefine the domU, having changed the vnic to go across bge1:

    <interface type=''bridge''>
      <mac address=''00:bb:c0:00:40:01''/>
      <source bridge=''bge1/>
      <script path=''/usr/lib/xen/scripts/vif-vnic''/>
    </interface>

And instantly, everything starts working.  Using snoop on the dom0,
I see that cent-01 traffic indeed goes across bge1.


Is this an inherent limitation of Solaris aggregations?  If so, any
pointer to the documentation of that fact would be appreciated.  In
that case, it''s strange that only CentOS is affected.  WinXP and
OpenSolaris show no problems at all.

Is there a bug in the Linux ''avahi'' daemon that might trigger
the problem?
Maybe a bug in Solaris?  Simply stopping the ''avahi'' stuff
again does not
solve anything; just moving the bridge to be bge1 does.

Or have I missed some other setting within the CentOS domU?


Thanks -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Hello all!


Last week, I wrote a lengthy email about the following
problem:
> It seems that I am unable to use an aggregation as a physical bridge
> for a CentOS 5.3 client.
Is anyone out there running a Centos 5.3 domU under SXCE 122 or 123 and
uses an aggregation as physical bridge?  If so, could you please tell
me so that I know that _someone_ has got it working?  :-)


Thanks -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Am 28.09.2009 09:06, schrieb Volker A. Brandt:
> Hello all!
> 
> 
> Last week, I wrote a lengthy email about the following problem:
>> It seems that I am unable to use an aggregation as a physical bridge
>> for a CentOS 5.3 client.
> 
> Is anyone out there running a Centos 5.3 domU under SXCE 122 or 123 and
> uses an aggregation as physical bridge?  If so, could you please tell
> me so that I know that _someone_ has got it working?  :-)
> 
> 
> Thanks -- Volker
I haven''t used Centos ever, but I have used for a long time a
port-aggregation on x4150 SunFires with xvm. Running hvm Windows (with
and without PV-IO) and OpenSolaris guests and also paravirtual Linux
(Gentoo x64) and OpenSolaris guests. Never have had trouble with
networking. Even not when the OpenSolaris Dom0 has crashed totally, the
guests where running fine.
So maybe it is a issue with crossbow support for your nics?
My X4150s are equipped with intel''s e1000e dualTwin nic.

Florian

Thank you for your reply, Florian.
> I haven''t used Centos ever, but I have used for a long time a
> port-aggregation on x4150 SunFires with xvm. Running hvm Windows (with
> and without PV-IO) and OpenSolaris guests and also paravirtual Linux
> (Gentoo x64) and OpenSolaris guests. Never have had trouble with
> networking. Even not when the OpenSolaris Dom0 has crashed totally, the
> guests where running fine.
> So maybe it is a issue with crossbow support for your nics?
> My X4150s are equipped with intel''s e1000e dualTwin nic.
In my second email, I did not provide any context.  Maybe I should
have. :-)  I am running WXP and OpenSolaris just fine across the
aggregation.  All NICs are crossbow-ready.

You mention that you can run Gentoo.  So to me it seems to be a
CentOS-only problem right now.


Thanks again -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Am 28.09.2009 12:25, schrieb Volker A. Brandt:
> Thank you for your reply, Florian.
> 
>> I haven''t used Centos ever, but I have used for a long time a
>> port-aggregation on x4150 SunFires with xvm. Running hvm Windows (with
>> and without PV-IO) and OpenSolaris guests and also paravirtual Linux
>> (Gentoo x64) and OpenSolaris guests. Never have had trouble with
>> networking. Even not when the OpenSolaris Dom0 has crashed totally, the
>> guests where running fine.
>> So maybe it is a issue with crossbow support for your nics?
>> My X4150s are equipped with intel''s e1000e dualTwin nic.
> 
> In my second email, I did not provide any context.  Maybe I should
> have. :-)  I am running WXP and OpenSolaris just fine across the
> aggregation.  All NICs are crossbow-ready.
> 
> You mention that you can run Gentoo.  So to me it seems to be a
> CentOS-only problem right now.
> 
> 
> Thanks again -- Volker
Ok, thats interesting. Currently I use a Xvm from 3.4 gate based on
build 122.
Currently no port aggregation due to other reasons.
Futureplan includes the idea to use vlan trunking (thats afaik the term)
maybe in combination with port aggregation if possible, need to check
this and start testing the idea.

Florian

> Ok, thats interesting. Currently I use a Xvm from 3.4 gate based on
> build 122.
So you are newer, since I use the xVM that comes with SXCE 122:

v3.3.2-xvm chgset ''Tue Aug 18 03:21:41 2009 -0700
18433:7e735e9e9bf6''
> Currently no port aggregation due to other reasons.
> Futureplan includes the idea to use vlan trunking (thats afaik the term)
> maybe in combination with port aggregation if possible, need to check
> this and start testing the idea.
When you have done some testing I would like to hear the results! :-)


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Am 28.09.2009 12:53, schrieb Volker A. Brandt:
>> Ok, thats interesting. Currently I use a Xvm from 3.4 gate based on
>> build 122.
> 
> So you are newer, since I use the xVM that comes with SXCE 122:
> 
> v3.3.2-xvm chgset ''Tue Aug 18 03:21:41 2009 -0700
18433:7e735e9e9bf6''
Uhm maybe that was misleading. I were using port aggregation with SXCE
b90 - b109 or so. I stopped using it when gambling around with
Xvm-Server EarlyAccess. After that I started testing and using self
build xvm from gate (runs rock stable so I would prefer a putback as
soon as possible).

Florian
> 
>> Currently no port aggregation due to other reasons.
>> Futureplan includes the idea to use vlan trunking (thats afaik the
term)
>> maybe in combination with port aggregation if possible, need to check
>> this and start testing the idea.
> 
> When you have done some testing I would like to hear the results! :-)
> 
> 
> Regards -- Volker

* Volker A. Brandt (vab@bb-c.de) wrote:
> > Ok, thats interesting. Currently I use a Xvm from 3.4 gate based on
> > build 122.
> 
> So you are newer, since I use the xVM that comes with SXCE 122:
> 
> v3.3.2-xvm chgset ''Tue Aug 18 03:21:41 2009 -0700
18433:7e735e9e9bf6''
I''m using the stock xvm from 121.
> > Currently no port aggregation due to other reasons.
> > Futureplan includes the idea to use vlan trunking (thats afaik the
term)
> > maybe in combination with port aggregation if possible, need to check
> > this and start testing the idea.
> 
> When you have done some testing I would like to hear the results! :-)
It''s been a couple of weeks since I played around with this, so my
mental notes might be a little out of whack.

I''m using LACP on 2 Intel NIC''s 
# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
e1000g1      Ethernet             up         1000   full      e1000g1
e1000g0      Ethernet             up         1000   full      e1000g0
# dladm show-aggr
LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
aggr1           L4       auto                 active        short       -----

On top of that I''m using VLAN''s (vnic''s with vlan
option) to do security
separation between network segments.

# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE    VID
xvm3_0       aggr1        1000   0:16:36:72:ff:cf  fixed          11
xvm4_0       aggr1        1000   0:16:36:72:ff:d1  fixed          11
wifi0        aggr1        1000   2:8:20:49:a6:9e   random         20
wan0         aggr1        1000   2:8:20:cc:43:e1   random         99
public0      aggr1        1000   2:8:20:a5:39:fa   random         100
server0      aggr1        1000   2:8:20:16:ae:a2   random         11
xvm27_0      aggr1        1000   0:16:36:72:ff:df  fixed          11

As I recall I was unable to get network traffic running for HVM guests 
(Windows, FreeBSD, Debian) when using the vlan option for vif-vnic. 
Using tcpdump in the domU I verified that the domU was receiving traffic
and sending traffic (arp requests and responses) but I never saw them in
dom0. Creating a static arp on the domU for another host on the same
vlan enabled me to ping that host with a RTT of 0.000ms.
Anyway I finally gave up and started using etherstub''s and routing on 
dom0 instead. Not optimal but at least it works.

BTW It works perfectly for debian-lenny pv domU''s. Hopefully it will
work for FreeBSD''s XEN port when/if that ever happens....
-- 
Bjørn Connolly
Mostly RFC3514 compliant.

> I''m using LACP on 2 Intel NIC''s 
> # dladm show-phys
> LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
> e1000g1      Ethernet             up         1000   full      e1000g1
> e1000g0      Ethernet             up         1000   full      e1000g0
> # dladm show-aggr
> LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER  
FLAGS
> aggr1           L4       auto                 active        short      
-----
> 
> On top of that I''m using VLAN''s (vnic''s with
vlan option) to do security
> separation between network segments.
> 
> # dladm show-vnic
> LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE    VID
> xvm3_0       aggr1        1000   0:16:36:72:ff:cf  fixed          11
> xvm4_0       aggr1        1000   0:16:36:72:ff:d1  fixed          11
> wifi0        aggr1        1000   2:8:20:49:a6:9e   random         20
> wan0         aggr1        1000   2:8:20:cc:43:e1   random         99
> public0      aggr1        1000   2:8:20:a5:39:fa   random         100
> server0      aggr1        1000   2:8:20:16:ae:a2   random         11
> xvm27_0      aggr1        1000   0:16:36:72:ff:df  fixed          11
> 
> As I recall I was unable to get network traffic running for HVM guests 
> (Windows, FreeBSD, Debian) when using the vlan option for vif-vnic. 
> Using tcpdump in the domU I verified that the domU was receiving traffic
> and sending traffic (arp requests and responses) but I never saw them in
> dom0.
These symptoms seem to be different from mine: my HVM guests work OK, 
the PVM guest NetBSD does so, too (more or less :-), just the PVM domU
CentOS 5.3 consistently does not.  Over a single physical link, everything
is fine.
>  Creating a static arp on the domU for another host on the same
> vlan enabled me to ping that host with a RTT of 0.000ms.
Will try that RSN.
> Anyway I finally gave up and started using etherstub''s and routing
on
> dom0 instead. Not optimal but at least it works.
This sounds a bit more complicated, so I''ll leave that for
later.
> 
> BTW It works perfectly for debian-lenny pv domU''s. Hopefully it
will
> work for FreeBSD''s XEN port when/if that ever happens....
NetBSD is pretty good in that regard. I wish they had a ZFS port
and an NFSv4 port. :-)


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Am 29.09.2009 21:48, schrieb Volker A. Brandt:
>> I''m using LACP on 2 Intel NIC''s 
>> # dladm show-phys
>> LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
>> e1000g1      Ethernet             up         1000   full      e1000g1
>> e1000g0      Ethernet             up         1000   full      e1000g0
>> # dladm show-aggr
>> LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER  
FLAGS
>> aggr1           L4       auto                 active        short      
-----
>>
>> On top of that I''m using VLAN''s (vnic''s with
vlan option) to do security
>> separation between network segments.
>>
>> # dladm show-vnic
>> LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE    VID
>> xvm3_0       aggr1        1000   0:16:36:72:ff:cf  fixed          11
>> xvm4_0       aggr1        1000   0:16:36:72:ff:d1  fixed          11
>> wifi0        aggr1        1000   2:8:20:49:a6:9e   random         20
>> wan0         aggr1        1000   2:8:20:cc:43:e1   random         99
>> public0      aggr1        1000   2:8:20:a5:39:fa   random         100
>> server0      aggr1        1000   2:8:20:16:ae:a2   random         11
>> xvm27_0      aggr1        1000   0:16:36:72:ff:df  fixed          11
>>
>> As I recall I was unable to get network traffic running for HVM guests 
>> (Windows, FreeBSD, Debian) when using the vlan option for vif-vnic. 
>> Using tcpdump in the domU I verified that the domU was receiving
traffic
>> and sending traffic (arp requests and responses) but I never saw them
in
>> dom0.
> 
> These symptoms seem to be different from mine: my HVM guests work OK, 
> the PVM guest NetBSD does so, too (more or less :-), just the PVM domU
> CentOS 5.3 consistently does not.  Over a single physical link, everything
> is fine.
> 
>>  Creating a static arp on the domU for another host on the same
>> vlan enabled me to ping that host with a RTT of 0.000ms.
> 
> Will try that RSN.
> 
>> Anyway I finally gave up and started using etherstub''s and
routing on
>> dom0 instead. Not optimal but at least it works.
> 
> This sounds a bit more complicated, so I''ll leave that for later.
>>
>> BTW It works perfectly for debian-lenny pv domU''s. Hopefully
it will
>> work for FreeBSD''s XEN port when/if that ever happens....
> 
> NetBSD is pretty good in that regard. I wish they had a ZFS port
> and an NFSv4 port. :-)
Uhm, worth to give opensolaris a try?

Florian
> 
> 
> Regards -- Volker

Some 2½ weeks ago I wrote:
> It seems that I am unable to use an aggregation as a physical bridge
> for a CentOS 5.3 client.
>
> The dom0 is running SXCE 122 and has the following interfaces defined:
[...]

I have upgraded the machine to SXCE 124, and the problem has disappeared.
Checking the changelogs from 122 to 124, I haven''t spotted any relevant
putbacks.   So I don''t really know why it works now, but it does work.
:-)


Regards -- Volker
-- 
------------------------------------------------------------------------
Volker A. Brandt                  Consulting and Support for Sun Solaris
Brandt & Brandt Computer GmbH                   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim                     Email: vab@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513              Schuhgröße: 45
Geschäftsführer: Rainer J. H. Brandt und Volker A. Brandt

Volker A. Brandt wrote:
> Some 2½ weeks ago I wrote:
> 
>> It seems that I am unable to use an aggregation as a physical bridge
>> for a CentOS 5.3 client.
>>
>> The dom0 is running SXCE 122 and has the following interfaces defined:
> [...]
> 
> I have upgraded the machine to SXCE 124, and the problem has disappeared.
> Checking the changelogs from 122 to 124, I haven''t spotted any
relevant
> putbacks.   So I don''t really know why it works now, but it does
work.
> :-)
> 
   http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855136



MRJ

On 12 Oct 2009, at 2:52pm, Mark Johnson wrote:
> Volker A. Brandt wrote:
>> Some 2½ weeks ago I wrote:
>>> It seems that I am unable to use an aggregation as a physical
bridge
>>> for a CentOS 5.3 client.
>>>
>>> The dom0 is running SXCE 122 and has the following interfaces  
>>> defined:
>> [...]
>> I have upgraded the machine to SXCE 124, and the problem has  
>> disappeared.
>> Checking the changelogs from 122 to 124, I haven''t spotted any
>> relevant
>> putbacks.   So I don''t really know why it works now, but it
does
>> work.
>> :-)
>
>  http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6855136
That wouldn''t fit - Volker is running a CentOS guest and that fix is  
for a Solaris guest.