Fedora xen - Dec 2007 - Re: Fedora Core 8 + Xenbr0 + network bridging?

Hi Dale:
 
I work with David who posted the original question to the mailing list.  I think
we need to give a bit more background info on what we are trying to do.  We are
running a mixed environment of mostly CentOS 3, 4and 5, we do have a few windows
servers and XP systems as well.  We are looking to virtualize all these
platforms.  Normally we have a bonded pair of NICs for the physical hosts, we
were able to get this running using CentOS 5 x86_64 with no problems, the guest
machines use the bonded pair in bridged mode as expected after a bit of
tweaking.  The biggest issue we found with EL5 is that windows guest performace
is dismal at best, hence our decision to have a look at Fedora Core 8 x86_64.  I
am happy to report that performance for all of our guest platforms is *very*
good with FC8, but it seems that libvirt changed the way networking is setup for
Xen.  The default NAT configuration is pretty useless for production server
environment.  Thanks to the mailing list we are now able to bridge a single NIC
on FC8 (like eth0 for example), but we cannot figure out how to get a bridge for
bond0 (comprised of eth0 and eth1) defined and available to Xen.  All the tweaks
that worked find on EL5 have not worked so far on FC8.  I am going to review
your document tomorrow and give it a try, but any idea on whether your
methodology will work on FC8 and libvirt?  I am willing to blow a Sunday to get
this worked out once and for all :)
 
Basically we are after good performance on both para and fully virtualized
guests using a bonded pair of GB NICs for speed and redundancy.  If this can be
achieved with enterprise linux then that would be preferable, but we will go FC8
if the bonding thing can be sorted out.  By the way Xensource 4.x looks to be a
respin of RHEL5 and has pretty good performance but their free version is
limited to 32bit (and hence 4GB ram).  Adding the clustering failover is the
next step of course :)
 
Thanks again for the help so far.
 
/Christian
 
 
 
>>>>>>>>>>>
just FYI for the list, I have a how-to for a bonded and VLAN tagged network.

http://www.certifried.com

ODT and PDF formats available.

It might not be the best way, but I''ve sent it out to my colleagues
several times and have never received any negative feedback.
Mark



Dale Bewley wrote:

	I haven''t done bonding, but you should be able to bond them and then
compose a bridge on top of this bonded device I would think.
	
	--
	Dale Bewley - Unix Administrator - Shields Library - UC Davis
	GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3
	
	--
	Fedora-xen mailing list
	Fedora-xen redhat com
	https://www.redhat.com/mailman/listinfo/fedora-xen
	



****************************************************************************
            Checked by MailWasher server (www.Firetrust.com)
                WARNING. No FirstAlert account found.
             To reduce spam further activate FirstAlert.
    This message can be removed by purchasing a FirstAlert Account.
****************************************************************************

Hi Mark:
 
Thank you very much for your response, I did indeed read the original poster as
Dale by mistake :)  So what you are saying makes perfect sense to me and sounds
like exactly what we are after, I will have 3 vlans to bridge myself ultimately.
My next question is the relative merits of RHEL5.1 as compared to Fedora 8. 
Obviously I would prefer the stable enterprise release rather than bleeding edge
Fedora, but has fully virtualized windows performance been fixed in this
release?  At any rate I am looking forward to getting this up and running
tomorrow!
 
/Christian
 

________________________________

From: Mark Nielsen [mailto:mnielsen@redhat.com]
Sent: Sat 12/1/2007 3:19 PM
To: Christian Lahti
Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?



hmm, did you mean "Hi Mark" ??

I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
copper (eth0 and eth1) and call it bond0. bond0 carries my "private"
IP
for cluster suite communications on the dom0 (physical) cluster.

Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
network for the dom0 cluster on bond1.100 (for example, that would be
VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
(bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
of these bond/VLAN devices. This allows me to put any particular VM on
any particular (or combination up to 3) of these xen bridged bonded VLAN
device.

My document explains, in detail, how to do all of this :) The only added
step is that I have to "undefine" (virsh net-undefine default) the
default network that the new libvirt creates (virbr0). Even with this
new NAT thing they added, I''ve been told (by our devs) that the
preferred way to do static network configurations is with the method I
lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi,
etc).

I''m pretty sure there weren''t any significant changes in
Fedora 8 (we''ve
dropped the word "core" now, btw) that don''t exist in RHEL
5.1 with
respects to the network. 5.0 -> 5.1 is when that NAT change came down
the pipe.

Mark

p.s. I''m happy to answer any other questions you may have about my
document. I''m quite certain that, if you follow it, you''ll
have what
you''re looking for.

Christian Lahti wrote:
> Hi Dale:
> 
> I work with David who posted the original question to the mailing
> list.  I think we need to give a bit more background info on what we
> are trying to do.  We are running a mixed environment of mostly CentOS
> 3, 4and 5, we do have a few windows servers and XP systems as well. 
> We are looking to virtualize all these platforms.  Normally we have a
> bonded pair of NICs for the physical hosts, we were able to get this
> running using CentOS 5 x86_64 with no problems, the guest machines use
> the bonded pair in bridged mode as expected after a bit of tweaking. 
> The biggest issue we found with EL5 is that windows guest performace
> is dismal at best, hence our decision to have a look at Fedora Core 8
> x86_64.  I am happy to report that performance for all of our guest
> platforms is *very* good with FC8, but it seems that libvirt changed
> the way networking is setup for Xen.  The default NAT configuration is
> pretty useless for production server environment.  Thanks to the
> mailing list we are now able to bridge a single NIC on FC8 (like eth0
> for example), but we cannot figure out how to get a bridge for bond0
> (comprised of eth0 and eth1) defined and available to Xen.  All the
> tweaks that worked find on EL5 have not worked so far on FC8.  I am
> going to review your document tomorrow and give it a try, but any idea
> on whether your methodology will work on FC8 and libvirt?  I am
> willing to blow a Sunday to get this worked out once and for all :)
> 
> Basically we are after good performance on both para and fully
> virtualized guests using a bonded pair of GB NICs for speed and
> redundancy.  If this can be achieved with enterprise linux then that
> would be preferable, but we will go FC8 if the bonding thing can be
> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
> and has pretty good performance but their free version is limited to
> 32bit (and hence 4GB ram).  Adding the clustering failover is the next
> step of course :)
> 
> Thanks again for the help so far.
> 
> /Christian
> 
> 
> 
> >>>>>>>>>>>
> just FYI for the list, I have a how-to for a bonded and VLAN tagged
network.
>
> http://www.certifried.com
>
> ODT and PDF formats available.
>
>  
> It might not be the best way, but I''ve sent it out to my
colleagues
> several times and have never received any negative feedback.
> Mark
>
>
>
> Dale Bewley wrote:
>  
>
>     I haven''t done bonding, but you should be able to bond them
and then compose a bridge on top of this bonded device I would think.
>
>     --
>     Dale Bewley - Unix Administrator - Shields Library - UC Davis
>     GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3
>
>     --
>     Fedora-xen mailing list
>     Fedora-xen redhat com
>     https://www.redhat.com/mailman/listinfo/fedora-xen
>        
>
>
****************************************************************************
> Checked by MailWasher server (www.Firetrust.com)
> WARNING. No FirstAlert account found.
> To reduce spam further activate FirstAlert.
> This message can be removed by purchasing a FirstAlert Account.
>
****************************************************************************
>
> ------------------------------------------------------------------------
>
> --
> Fedora-xen mailing list
> Fedora-xen@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-xen
>

Christian Lahti wrote:
> Hi Mark:
>  
> Thank you very much for your response, I did indeed read the original
poster as Dale by mistake :)  So what you are saying makes perfect sense to me
and sounds like exactly what we are after, I will have 3 vlans to bridge myself
ultimately.  My next question is the relative merits of RHEL5.1 as compared to
Fedora 8.  Obviously I would prefer the stable enterprise release rather than
bleeding edge Fedora, but has fully virtualized windows performance been fixed
in this release?  At any rate I am looking forward to getting this up and
running tomorrow!
>  
> /Christian
>  
> 
> ________________________________
> 
> From: Mark Nielsen [mailto:mnielsen@redhat.com]
> Sent: Sat 12/1/2007 3:19 PM
> To: Christian Lahti
> Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
> 
> 
> 
> hmm, did you mean "Hi Mark" ??
> 
> I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
> they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
> copper (eth0 and eth1) and call it bond0. bond0 carries my
"private" IP
> for cluster suite communications on the dom0 (physical) cluster.
> 
> Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
> network for the dom0 cluster on bond1.100 (for example, that would be
> VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
> (bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
> of these bond/VLAN devices. This allows me to put any particular VM on
> any particular (or combination up to 3) of these xen bridged bonded VLAN
> device.
> 
> My document explains, in detail, how to do all of this :) The only added
> step is that I have to "undefine" (virsh net-undefine default)
the
> default network that the new libvirt creates (virbr0). Even with this
> new NAT thing they added, I''ve been told (by our devs) that the
> preferred way to do static network configurations is with the method I
> lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi,
> etc).
> 
> I''m pretty sure there weren''t any significant changes in
Fedora 8 (we''ve
> dropped the word "core" now, btw) that don''t exist in
RHEL 5.1 with
> respects to the network. 5.0 -> 5.1 is when that NAT change came down
> the pipe.
> 
> Mark
> 
> p.s. I''m happy to answer any other questions you may have about my
> document. I''m quite certain that, if you follow it,
you''ll have what
> you''re looking for.
> 
> Christian Lahti wrote:
>> Hi Dale:
>>
>> I work with David who posted the original question to the mailing
>> list.  I think we need to give a bit more background info on what we
>> are trying to do.  We are running a mixed environment of mostly CentOS
>> 3, 4and 5, we do have a few windows servers and XP systems as well. 
>> We are looking to virtualize all these platforms.  Normally we have a
>> bonded pair of NICs for the physical hosts, we were able to get this
>> running using CentOS 5 x86_64 with no problems, the guest machines use
>> the bonded pair in bridged mode as expected after a bit of tweaking. 
>> The biggest issue we found with EL5 is that windows guest performace
>> is dismal at best, hence our decision to have a look at Fedora Core 8
>> x86_64.  I am happy to report that performance for all of our guest
>> platforms is *very* good with FC8, but it seems that libvirt changed
>> the way networking is setup for Xen.  The default NAT configuration is
>> pretty useless for production server environment.  Thanks to the
>> mailing list we are now able to bridge a single NIC on FC8 (like eth0
>> for example), but we cannot figure out how to get a bridge for bond0
>> (comprised of eth0 and eth1) defined and available to Xen.  All the
>> tweaks that worked find on EL5 have not worked so far on FC8.  I am
>> going to review your document tomorrow and give it a try, but any idea
>> on whether your methodology will work on FC8 and libvirt?  I am
>> willing to blow a Sunday to get this worked out once and for all :)
>>
>> Basically we are after good performance on both para and fully
>> virtualized guests using a bonded pair of GB NICs for speed and
>> redundancy.  If this can be achieved with enterprise linux then that
>> would be preferable, but we will go FC8 if the bonding thing can be
>> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
>> and has pretty good performance but their free version is limited to
>> 32bit (and hence 4GB ram).  Adding the clustering failover is the next
>> step of course :)
>>
>> Thanks again for the help so far.
In your position, I might consider another Sunday to see whether the f8 
tools run on C5, and not, then what''s needed.

The -xen kernel''s probably needed along with the most obvious *virt*. 
There might not be a lot of building to do, and the odds are good that a 
Fedora kernel will "just work," depending on whether you need extra
drivers.





-- 

Cheers
John

-- spambait
1aaaaaaa@coco.merseine.nu  Z1aaaaaaa@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

Unfortunately, I took interest in this discussion and decided to
mess around with it (primarily to see if there really were noticeable
performance gains between xen''s built-in bridge script and this manual
method) even though I don''t currently have a test box.  I am running
F7, and
prior to trying to do this, I had xenbr0 working fine (perhaps from
modifying xend-config.sxp, I don''t remember exactly) alongside virbr0
(which
I didn''t want, but couldn''t get rid of).  I thought undonig
changes would
surely get me back to where I started, so I didn''t bother with backups
(though, admittedly, backups really equate to undoing changes, so I
don''t
know what good additional copies of the files I might have backed up would
have done).  My experience went something like this:

	1)  I created the ifcfg-peth0 file as described in "xen-lke
bridging" here:
http://watzmann.net/blog/index.php/2007/04/27/networking_with_kvm_and_libvir
t
	2)  I created a ifcfg-eth0 and an ifcfg-br0 in hopes of making an
eth0 separate from the bridge as xen does.  When that didn''t work (by
simply
restarting the network) I got rid of the ifcfg-eth0 and used only the
ifcfg-br0 (bridge=br0 was the setting in my peth0 all along), note that I
had changed network-script to /bin/false and restarted xend as well by this
point.
	3)  I added the iptables command as in the page from the link above
	4)  I went ahead and enabled forwarding in procfs via sysctl.conf
thinking that xen was probably manually setting and unsetting this with its
script
	5)  I rebooted and still had xenbr0 and virbr0, along with eth0 and
br0 (both bridges) and peth0 (the real nic)
	6)  I started one of my HVM domains using bridge=br0, and it worked,
as peth0 was bridged on br0 (the other three bridges weren''t being used
at
all)
	7)  I ran "virsh net-autostart --disable default" thinking it might
get rid of xenbr0 and virbr0, it didn''t, they come back on subsequent
reboots
	8)  I decided to change back so there would be less unnecessary
bridges (shown as interfaces in ifconfig) since xenbr0 was still there, I
thought that would be simple enough
	9)  I got rid of ifcfg-br0 and ifcfg-peth0 and rebuilt my original
ifcfg-eth0 (though probably in a different order and without the presumably
unnecessary ipv6 lines)
	10)  I changed network script back to network-bridge and rebooted
	11)  The system came up and still had all four bridges, brctl show
now shows peth0 to be a member of bridge eth0, if I disable the network, all
interfaces left up (ifconfig br0 down, etc), and all bridges ([brctl delif
eth0 peth0] brctl delbr br0, etc), then when I start the network back up,
eth0 is real eth0 (though it takes some time to start up), but when I start
xend back up again, all of that mess comes back.
	12)  If I reboot into the standard kernel, eth0 is eth0 and all is
well, but upon rebooting back into xen, I have the same mess (4 bridges with
peth0 on the eth0 bridge), but I got my domUs working by setting bridge to
eth0 in their config files (when that was set to xenbr0, I couldn''t
contact
them from dom0 like I could before this fiasco, and I don''t think they
could
get out to the network either, though I didn''t test that exhaustively).

	While it is working and that is fine, I would like to get rid of the
virbr0 (and at this point, also xenbr0 and br0 I guess), but I am not sure
how to go about that.  Brctl certainly isn''t deleting them permanently,
and
virsh doesn''t seem to have anything to do with them.  I did stumble
upon a
gui bridge controller at one point while messing with all of this, but
haven''t found it again and don''t know if it would have caused
this stuff to
stick.  I may have caused all of this confusion when messing with
system-config-network, as I thought a second device in there was a NIC I had
removed, so I deleted it, and that may have been used by xen (as opposed to
the NIC sitting in a drawer that may or may not have been installed and
removed on this particular system), but the network-scripts folder in
sysconfig is right (according to my inspection and the happy boot into
non-xen f7).  That said, can anyone tell me where settings might be hiding
that would somehow be creating my br0 along with the other unnecessary (in
my current situation) bridges?  Thanks,

	Dustin


-----Original Message-----
From: fedora-xen-bounces@redhat.com [mailto:fedora-xen-bounces@redhat.com]
On Behalf Of John Summerfield
Sent: Saturday, December 01, 2007 19:34
To: fedora-xen@redhat.com
Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?

Christian Lahti wrote:
> Hi Mark:
>  
> Thank you very much for your response, I did indeed read the original
poster as Dale by mistake :)  So what you are saying makes perfect sense to
me and sounds like exactly what we are after, I will have 3 vlans to bridge
myself ultimately.  My next question is the relative merits of RHEL5.1 as
compared to Fedora 8.  Obviously I would prefer the stable enterprise
release rather than bleeding edge Fedora, but has fully virtualized windows
performance been fixed in this release?  At any rate I am looking forward to
getting this up and running tomorrow!
>  
> /Christian
>  
> 
> ________________________________
> 
> From: Mark Nielsen [mailto:mnielsen@redhat.com]
> Sent: Sat 12/1/2007 3:19 PM
> To: Christian Lahti
> Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
> 
> 
> 
> hmm, did you mean "Hi Mark" ??
> 
> I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
> they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
> copper (eth0 and eth1) and call it bond0. bond0 carries my
"private" IP
> for cluster suite communications on the dom0 (physical) cluster.
> 
> Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
> network for the dom0 cluster on bond1.100 (for example, that would be
> VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
> (bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
> of these bond/VLAN devices. This allows me to put any particular VM on
> any particular (or combination up to 3) of these xen bridged bonded VLAN
> device.
> 
> My document explains, in detail, how to do all of this :) The only added
> step is that I have to "undefine" (virsh net-undefine default)
the
> default network that the new libvirt creates (virbr0). Even with this
> new NAT thing they added, I''ve been told (by our devs) that the
> preferred way to do static network configurations is with the method I
> lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi,
> etc).
> 
> I''m pretty sure there weren''t any significant changes in
Fedora 8 (we''ve
> dropped the word "core" now, btw) that don''t exist in
RHEL 5.1 with
> respects to the network. 5.0 -> 5.1 is when that NAT change came down
> the pipe.
> 
> Mark
> 
> p.s. I''m happy to answer any other questions you may have about my
> document. I''m quite certain that, if you follow it,
you''ll have what
> you''re looking for.
> 
> Christian Lahti wrote:
>> Hi Dale:
>>
>> I work with David who posted the original question to the mailing
>> list.  I think we need to give a bit more background info on what we
>> are trying to do.  We are running a mixed environment of mostly CentOS
>> 3, 4and 5, we do have a few windows servers and XP systems as well. 
>> We are looking to virtualize all these platforms.  Normally we have a
>> bonded pair of NICs for the physical hosts, we were able to get this
>> running using CentOS 5 x86_64 with no problems, the guest machines use
>> the bonded pair in bridged mode as expected after a bit of tweaking. 
>> The biggest issue we found with EL5 is that windows guest performace
>> is dismal at best, hence our decision to have a look at Fedora Core 8
>> x86_64.  I am happy to report that performance for all of our guest
>> platforms is *very* good with FC8, but it seems that libvirt changed
>> the way networking is setup for Xen.  The default NAT configuration is
>> pretty useless for production server environment.  Thanks to the
>> mailing list we are now able to bridge a single NIC on FC8 (like eth0
>> for example), but we cannot figure out how to get a bridge for bond0
>> (comprised of eth0 and eth1) defined and available to Xen.  All the
>> tweaks that worked find on EL5 have not worked so far on FC8.  I am
>> going to review your document tomorrow and give it a try, but any idea
>> on whether your methodology will work on FC8 and libvirt?  I am
>> willing to blow a Sunday to get this worked out once and for all :)
>>
>> Basically we are after good performance on both para and fully
>> virtualized guests using a bonded pair of GB NICs for speed and
>> redundancy.  If this can be achieved with enterprise linux then that
>> would be preferable, but we will go FC8 if the bonding thing can be
>> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
>> and has pretty good performance but their free version is limited to
>> 32bit (and hence 4GB ram).  Adding the clustering failover is the next
>> step of course :)
>>
>> Thanks again for the help so far.
In your position, I might consider another Sunday to see whether the f8 
tools run on C5, and not, then what''s needed.

The -xen kernel''s probably needed along with the most obvious *virt*. 
There might not be a lot of building to do, and the odds are good that a 
Fedora kernel will "just work," depending on whether you need extra
drivers.





-- 

Cheers
John

-- spambait
1aaaaaaa@coco.merseine.nu  Z1aaaaaaa@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
Fedora-xen mailing list
Fedora-xen@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-xen

Dustin Henning wrote:
> 	Unfortunately, I took interest in this discussion and decided to
> mess around with it (primarily to see if there really were noticeable
> performance gains between xen''s built-in bridge script and this
manual
> method) even though I don''t currently have a test box.  I am
running F7, and
> prior to trying to do this, I had xenbr0 working fine (perhaps from
> modifying xend-config.sxp, I don''t remember exactly) alongside
virbr0 (which
> I didn''t want, but couldn''t get rid of).  I thought
undonig changes would
> surely get me back to where I started, so I didn''t bother with
backups
> (though, admittedly, backups really equate to undoing changes, so I
don''t
> know what good additional copies of the files I might have backed up would
> have done).  My experience went something like this:
I don''t feel like reading the rest when I can point you at a backup:-)

cd <some sandpit>

rpm2cpio <whatever.rpm | cpio --extract --make-directories

Find and copy the files you want.




-- 

Cheers
John

-- spambait
1aaaaaaa@coco.merseine.nu  Z1aaaaaaa@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

Fair enough; I certainly can''t blame you for not wanting to read
that big long thing. However, while I thank you for this useful piece of
advice, the actual problem is that I don''t know what files are causing
my
issue, but I am pretty sure they aren''t files I modified by hand.  I
think
maybe one of the GUI tools changed some file I am not familiar with (or F7
did on account of a change I made in a file I am familiar with).  So  I am
hoping someone could point me toward such a file.  I need to know what could
cause my situation, which I believe is fully described starting at
"11)".
Everything prior to that is part of the cause and/or attempts to get rid of
extra bridges.
	Dustin

-----Original Message-----
From: fedora-xen-bounces@redhat.com [mailto:fedora-xen-bounces@redhat.com]
On Behalf Of John Summerfield
Sent: Tuesday, December 04, 2007 04:08
To: Fedora Xen
Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?

Dustin Henning wrote:
> 	Unfortunately, I took interest in this discussion and decided to
> mess around with it (primarily to see if there really were noticeable
> performance gains between xen''s built-in bridge script and this
manual
> method) even though I don''t currently have a test box.  I am
running F7,
and
> prior to trying to do this, I had xenbr0 working fine (perhaps from
> modifying xend-config.sxp, I don''t remember exactly) alongside
virbr0
(which
> I didn''t want, but couldn''t get rid of).  I thought
undonig changes would
> surely get me back to where I started, so I didn''t bother with
backups
> (though, admittedly, backups really equate to undoing changes, so I
don''t
> know what good additional copies of the files I might have backed up would
> have done).  My experience went something like this:
I don''t feel like reading the rest when I can point you at a backup:-)

cd <some sandpit>

rpm2cpio <whatever.rpm | cpio --extract --make-directories

Find and copy the files you want.




-- 

Cheers
John

-- spambait
1aaaaaaa@coco.merseine.nu  Z1aaaaaaa@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

--
Fedora-xen mailing list
Fedora-xen@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-xen

Dustin Henning wrote:
> 	Fair enough; I certainly can''t blame you for not wanting to read
> that big long thing. However, while I thank you for this useful piece of
:-)
> advice, the actual problem is that I don''t know what files are
causing my
> issue, but I am pretty sure they aren''t files I modified by hand. 
I think
to see what you really changed, rpm -V

Then you can use the other trick to get the original files, and hence 
their differences with, er, diff.

> maybe one of the GUI tools changed some file I am not familiar with (or F7
> did on account of a change I made in a file I am familiar with).  So  I am
> hoping someone could point me toward such a file.  I need to know what
could
> cause my situation, which I believe is fully described starting at
"11)".
> Everything prior to that is part of the cause and/or attempts to get rid of
> extra bridges.
> 	Dustin
> 
> -----Original Message-----
> From: fedora-xen-bounces@redhat.com [mailto:fedora-xen-bounces@redhat.com]
> On Behalf Of John Summerfield
> Sent: Tuesday, December 04, 2007 04:08
> To: Fedora Xen
> Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
> 
> Dustin Henning wrote:
>> 	Unfortunately, I took interest in this discussion and decided to
>> mess around with it (primarily to see if there really were noticeable
>> performance gains between xen''s built-in bridge script and
this manual
>> method) even though I don''t currently have a test box.  I am
running F7,
> and
>> prior to trying to do this, I had xenbr0 working fine (perhaps from
>> modifying xend-config.sxp, I don''t remember exactly) alongside
virbr0
> (which
>> I didn''t want, but couldn''t get rid of).  I thought
undonig changes would
>> surely get me back to where I started, so I didn''t bother with
backups
>> (though, admittedly, backups really equate to undoing changes, so I
don''t
>> know what good additional copies of the files I might have backed up
would
>> have done).  My experience went something like this:
> 
> I don''t feel like reading the rest when I can point you at a
backup:-)
> 
> cd <some sandpit>
> 
> rpm2cpio <whatever.rpm | cpio --extract --make-directories
> 
> Find and copy the files you want.
> 
> 
> 
> 

-- 

Cheers
John

-- spambait
1aaaaaaa@coco.merseine.nu  Z1aaaaaaa@coco.merseine.nu
-- Advice
http://webfoot.com/advice/email.top.php
http://www.catb.org/~esr/faqs/smart-questions.html
http://support.microsoft.com/kb/555375

You cannot reply off-list:-)

Hi Mark:

 

Well, 

 

I carefully went through your PDF document for setting up a bonded pair
on RHEL 5.1 and at the end of the day this does not work for me.  This
is the procedure more or less I have tried:

 

*	Make sure eth0 and eth1 work as normal eth devices without LACP
*	Edit /etc/modprobe.conf to add the bonding kernel module
*	Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 to
be slave devices with the master as bond0
*	Create /etc/sysconfig/network-scripts/ifcfg-bond0 with static IP
info
*	Plug into LACP enabled ports on switch and reboot the box
*	At this point bond0 works fine, as expected, we have been doing
this all day long on non-xen kernels for years
*	NOW...following your document
*	Remove the /etc/libvirt/qemu/networks/autostart/* so that the
pointless 192.168.122.x NAT network does not get created
*	Grab the network-scripts-bridge-bond-vlan script from the link
in the message and put into /etc/xen/scripts
*	Edit /etc/xen/xend-config.sxp and replace (network-script
network-bridge) with (network-script network-bridge-bond-vlan)
*	Restart networking
*	At this point networking for the physical host in invariably
completely hosed, and there is no xen networking at all
*	It is noteworthy to add that I did not make any modifications to
the network-scripts-bridge-bond-vlan script as you suggest since I am on
a single vlan (so far) and the default setup in the script seems sane
for a single NON-VLAN bond0 configuration

 

So in a nutshell the it seems that libvirt interface just does not seem
to use the xen scripts...rather it''s own (undocumented?) crap.  After a
week of head-banging on this we really have no choice but to abandon
Redhat in favor of Suse 10.3, which incidentally works right out of the
box with bridge, NAT, bond, you name it.  I suggest RH staff install a
copy of this to see how they did it, you don''t even have to jump
through
hoops to configure a bond from the command line, you can do this with
their stock GUI tools if you so choose.  I am not knocking RedHat, but
we need something that "just works" and so far for xen + bond + RH has
been a non-starter for us after much effort.  By the way, to get
networking un-foobar''ed on the physical machine I have to run
system-config-network and remove all Ethernet devices, then reboot, then
redefine eth0 and eth1 as as normal DHCP devices.  I can then go
re-setup the bond0 device from the command line.

 

If anyone has step-by-step directions on how to do this with RHEL 5.1 I
am happy to try again at some future point.  The goals are simple:

 

*	Step one: create a bond0 from eth0 and eth1 that can be used as
a bridged device by xen on a single VLAN
*	Step two: create a bond0 (aliases?) that can be assigned
multiple VLANs, AND have these VLAN devices available to be used by xen

 

I would settle for item #1 really.

 

/Christian

 

 

 

 

 

________________________________

From: Christian Lahti 
Sent: Saturday, December 01, 2007 3:56 PM
To: mnielsen@redhat.com; fedora-xen@redhat.com
Subject: RE: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?

 

Hi Mark:

 

Thank you very much for your response, I did indeed read the original
poster as Dale by mistake :)  So what you are saying makes perfect sense
to me and sounds like exactly what we are after, I will have 3 vlans to
bridge myself ultimately.  My next question is the relative merits of
RHEL5.1 as compared to Fedora 8.  Obviously I would prefer the stable
enterprise release rather than bleeding edge Fedora, but has fully
virtualized windows performance been fixed in this release?  At any rate
I am looking forward to getting this up and running tomorrow!

 

/Christian

 

 

________________________________

From: Mark Nielsen [mailto:mnielsen@redhat.com]
Sent: Sat 12/1/2007 3:19 PM
To: Christian Lahti
Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?

hmm, did you mean "Hi Mark" ??

I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
copper (eth0 and eth1) and call it bond0. bond0 carries my "private"
IP
for cluster suite communications on the dom0 (physical) cluster.

Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
network for the dom0 cluster on bond1.100 (for example, that would be
VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
(bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
of these bond/VLAN devices. This allows me to put any particular VM on
any particular (or combination up to 3) of these xen bridged bonded VLAN
device.

My document explains, in detail, how to do all of this :) The only added
step is that I have to "undefine" (virsh net-undefine default) the
default network that the new libvirt creates (virbr0). Even with this
new NAT thing they added, I''ve been told (by our devs) that the
preferred way to do static network configurations is with the method I
lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi,
etc).

I''m pretty sure there weren''t any significant changes in
Fedora 8 (we''ve
dropped the word "core" now, btw) that don''t exist in RHEL
5.1 with
respects to the network. 5.0 -> 5.1 is when that NAT change came down
the pipe.

Mark

p.s. I''m happy to answer any other questions you may have about my
document. I''m quite certain that, if you follow it, you''ll
have what
you''re looking for.

Christian Lahti wrote:
> Hi Dale:
> 
> I work with David who posted the original question to the mailing
> list.  I think we need to give a bit more background info on what we
> are trying to do.  We are running a mixed environment of mostly CentOS
> 3, 4and 5, we do have a few windows servers and XP systems as well. 
> We are looking to virtualize all these platforms.  Normally we have a
> bonded pair of NICs for the physical hosts, we were able to get this
> running using CentOS 5 x86_64 with no problems, the guest machines use
> the bonded pair in bridged mode as expected after a bit of tweaking. 
> The biggest issue we found with EL5 is that windows guest performace
> is dismal at best, hence our decision to have a look at Fedora Core 8
> x86_64.  I am happy to report that performance for all of our guest
> platforms is *very* good with FC8, but it seems that libvirt changed
> the way networking is setup for Xen.  The default NAT configuration is
> pretty useless for production server environment.  Thanks to the
> mailing list we are now able to bridge a single NIC on FC8 (like eth0
> for example), but we cannot figure out how to get a bridge for bond0
> (comprised of eth0 and eth1) defined and available to Xen.  All the
> tweaks that worked find on EL5 have not worked so far on FC8.  I am
> going to review your document tomorrow and give it a try, but any idea
> on whether your methodology will work on FC8 and libvirt?  I am
> willing to blow a Sunday to get this worked out once and for all :)
> 
> Basically we are after good performance on both para and fully
> virtualized guests using a bonded pair of GB NICs for speed and
> redundancy.  If this can be achieved with enterprise linux then that
> would be preferable, but we will go FC8 if the bonding thing can be
> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
> and has pretty good performance but their free version is limited to
> 32bit (and hence 4GB ram).  Adding the clustering failover is the next
> step of course :)
> 
> Thanks again for the help so far.
> 
> /Christian
> 
> 
> 
> >>>>>>>>>>>
> just FYI for the list, I have a how-to for a bonded and VLAN tagged
network.
>
> http://www.certifried.com
>
> ODT and PDF formats available.
>
>  
> It might not be the best way, but I''ve sent it out to my
colleagues
> several times and have never received any negative feedback.
> Mark
>
>
>
> Dale Bewley wrote:
>  
>
>     I haven''t done bonding, but you should be able to bond them
and
then compose a bridge on top of this bonded device I would
think.
>
>     --
>     Dale Bewley - Unix Administrator - Shields Library - UC Davis
>     GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3
>
>     --
>     Fedora-xen mailing list
>     Fedora-xen redhat com
>     https://www.redhat.com/mailman/listinfo/fedora-xen
>        
>
>
************************************************************************
****
> Checked by MailWasher server (www.Firetrust.com)
> WARNING. No FirstAlert account found.
> To reduce spam further activate FirstAlert.
> This message can be removed by purchasing a FirstAlert Account.
>
************************************************************************
****
>
>
------------------------------------------------------------------------
>
> --
> Fedora-xen mailing list
> Fedora-xen@redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-xen
>

Christian,

Sorry to hear it didn''t work out for you. It''s not a
straight-forward or
simple configuration. The problem is that there are so many possible 
combinations of VLANs, bonds, networks that most of the configuration is 
left up to the ability and knowledge of the administrator. It sounds 
like you might''ve missed a few steps somewhere along the way. I admit 
that my document isn''t 100% step-by-step, it takes a pretty good 
understanding of everything that''s going on. Your problem could be in 
your step where you "Restart networking". This stuff all needs to
happen
in a particular order, and that might not have happened. Unfortunately 
it''s nearly impossible to help troubleshoot that in e-mails. I can say 
that I have the configuration I describe (and it sounds like what you 
want) working perfectly, so I know it''s possible.

Mark

Christian Lahti wrote:
> Hi Mark:
>
>  
>
> Well, 
>
>  
>
> I carefully went through your PDF document for setting up a bonded pair
> on RHEL 5.1 and at the end of the day this does not work for me.  This
> is the procedure more or less I have tried:
>
>  
>
> *	Make sure eth0 and eth1 work as normal eth devices without LACP
> *	Edit /etc/modprobe.conf to add the bonding kernel module
> *	Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 to
> be slave devices with the master as bond0
> *	Create /etc/sysconfig/network-scripts/ifcfg-bond0 with static IP
> info
> *	Plug into LACP enabled ports on switch and reboot the box
> *	At this point bond0 works fine, as expected, we have been doing
> this all day long on non-xen kernels for years
> *	NOW...following your document
> *	Remove the /etc/libvirt/qemu/networks/autostart/* so that the
> pointless 192.168.122.x NAT network does not get created
> *	Grab the network-scripts-bridge-bond-vlan script from the link
> in the message and put into /etc/xen/scripts
> *	Edit /etc/xen/xend-config.sxp and replace (network-script
> network-bridge) with (network-script network-bridge-bond-vlan)
> *	Restart networking
> *	At this point networking for the physical host in invariably
> completely hosed, and there is no xen networking at all
> *	It is noteworthy to add that I did not make any modifications to
> the network-scripts-bridge-bond-vlan script as you suggest since I am on
> a single vlan (so far) and the default setup in the script seems sane
> for a single NON-VLAN bond0 configuration
>
>  
>
> So in a nutshell the it seems that libvirt interface just does not seem
> to use the xen scripts...rather it''s own (undocumented?) crap. 
After a
> week of head-banging on this we really have no choice but to abandon
> Redhat in favor of Suse 10.3, which incidentally works right out of the
> box with bridge, NAT, bond, you name it.  I suggest RH staff install a
> copy of this to see how they did it, you don''t even have to jump
through
> hoops to configure a bond from the command line, you can do this with
> their stock GUI tools if you so choose.  I am not knocking RedHat, but
> we need something that "just works" and so far for xen + bond +
RH has
> been a non-starter for us after much effort.  By the way, to get
> networking un-foobar''ed on the physical machine I have to run
> system-config-network and remove all Ethernet devices, then reboot, then
> redefine eth0 and eth1 as as normal DHCP devices.  I can then go
> re-setup the bond0 device from the command line.
>
>  
>
> If anyone has step-by-step directions on how to do this with RHEL 5.1 I
> am happy to try again at some future point.  The goals are simple:
>
>  
>
> *	Step one: create a bond0 from eth0 and eth1 that can be used as
> a bridged device by xen on a single VLAN
> *	Step two: create a bond0 (aliases?) that can be assigned
> multiple VLANs, AND have these VLAN devices available to be used by xen
>
>  
>
> I would settle for item #1 really.
>
>  
>
> /Christian
>
>  
>
>  
>
>  
>
>  
>
>  
>
> ________________________________
>
> From: Christian Lahti 
> Sent: Saturday, December 01, 2007 3:56 PM
> To: mnielsen@redhat.com; fedora-xen@redhat.com
> Subject: RE: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
>
>  
>
> Hi Mark:
>
>  
>
> Thank you very much for your response, I did indeed read the original
> poster as Dale by mistake :)  So what you are saying makes perfect sense
> to me and sounds like exactly what we are after, I will have 3 vlans to
> bridge myself ultimately.  My next question is the relative merits of
> RHEL5.1 as compared to Fedora 8.  Obviously I would prefer the stable
> enterprise release rather than bleeding edge Fedora, but has fully
> virtualized windows performance been fixed in this release?  At any rate
> I am looking forward to getting this up and running tomorrow!
>
>  
>
> /Christian
>
>  
>
>  
>
> ________________________________
>
> From: Mark Nielsen [mailto:mnielsen@redhat.com]
> Sent: Sat 12/1/2007 3:19 PM
> To: Christian Lahti
> Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
>
> hmm, did you mean "Hi Mark" ??
>
> I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
> they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
> copper (eth0 and eth1) and call it bond0. bond0 carries my
"private" IP
> for cluster suite communications on the dom0 (physical) cluster.
>
> Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
> network for the dom0 cluster on bond1.100 (for example, that would be
> VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
> (bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
> of these bond/VLAN devices. This allows me to put any particular VM on
> any particular (or combination up to 3) of these xen bridged bonded VLAN
> device.
>
> My document explains, in detail, how to do all of this :) The only added
> step is that I have to "undefine" (virsh net-undefine default)
the
> default network that the new libvirt creates (virbr0). Even with this
> new NAT thing they added, I''ve been told (by our devs) that the
> preferred way to do static network configurations is with the method I
> lay out. NAT is more for dynamic networks (cable modems, dial-up, wifi,
> etc).
>
> I''m pretty sure there weren''t any significant changes in
Fedora 8 (we''ve
> dropped the word "core" now, btw) that don''t exist in
RHEL 5.1 with
> respects to the network. 5.0 -> 5.1 is when that NAT change came down
> the pipe.
>
> Mark
>
> p.s. I''m happy to answer any other questions you may have about my
> document. I''m quite certain that, if you follow it,
you''ll have what
> you''re looking for.
>
> Christian Lahti wrote:
>   
>> Hi Dale:
>>
>> I work with David who posted the original question to the mailing
>> list.  I think we need to give a bit more background info on what we
>> are trying to do.  We are running a mixed environment of mostly CentOS
>> 3, 4and 5, we do have a few windows servers and XP systems as well. 
>> We are looking to virtualize all these platforms.  Normally we have a
>> bonded pair of NICs for the physical hosts, we were able to get this
>> running using CentOS 5 x86_64 with no problems, the guest machines use
>> the bonded pair in bridged mode as expected after a bit of tweaking. 
>> The biggest issue we found with EL5 is that windows guest performace
>> is dismal at best, hence our decision to have a look at Fedora Core 8
>> x86_64.  I am happy to report that performance for all of our guest
>> platforms is *very* good with FC8, but it seems that libvirt changed
>> the way networking is setup for Xen.  The default NAT configuration is
>> pretty useless for production server environment.  Thanks to the
>> mailing list we are now able to bridge a single NIC on FC8 (like eth0
>> for example), but we cannot figure out how to get a bridge for bond0
>> (comprised of eth0 and eth1) defined and available to Xen.  All the
>> tweaks that worked find on EL5 have not worked so far on FC8.  I am
>> going to review your document tomorrow and give it a try, but any idea
>> on whether your methodology will work on FC8 and libvirt?  I am
>> willing to blow a Sunday to get this worked out once and for all :)
>>
>> Basically we are after good performance on both para and fully
>> virtualized guests using a bonded pair of GB NICs for speed and
>> redundancy.  If this can be achieved with enterprise linux then that
>> would be preferable, but we will go FC8 if the bonding thing can be
>> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
>> and has pretty good performance but their free version is limited to
>> 32bit (and hence 4GB ram).  Adding the clustering failover is the next
>> step of course :)
>>
>> Thanks again for the help so far.
>>
>> /Christian
>>
>>
>>
>>     
>> just FYI for the list, I have a how-to for a bonded and VLAN tagged
>>     
> network.
>   
>> http://www.certifried.com
>>
>> ODT and PDF formats available.
>>
>>  
>> It might not be the best way, but I''ve sent it out to my
colleagues
>> several times and have never received any negative feedback.
>> Mark
>>
>>
>>
>> Dale Bewley wrote:
>>  
>>
>>     I haven''t done bonding, but you should be able to bond
them and
>>     
> then compose a bridge on top of this bonded device I would think.
>   
>>     --
>>     Dale Bewley - Unix Administrator - Shields Library - UC Davis
>>     GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098 A0F3
>>
>>     --
>>     Fedora-xen mailing list
>>     Fedora-xen redhat com
>>     https://www.redhat.com/mailman/listinfo/fedora-xen
>>        
>>
>>
>>     
> ************************************************************************
> ****
>   
>> Checked by MailWasher server (www.Firetrust.com)
>> WARNING. No FirstAlert account found.
>> To reduce spam further activate FirstAlert.
>> This message can be removed by purchasing a FirstAlert Account.
>>
>>     
> ************************************************************************
> ****
>   
>>     
> ------------------------------------------------------------------------
>   
>> --
>> Fedora-xen mailing list
>> Fedora-xen@redhat.com
>> https://www.redhat.com/mailman/listinfo/fedora-xen
>>  
>>     
>
>

When xend starts on my F7 box, it creates virbr0 and xenbr0 as
expected, but it also creates br0, which is a bridge I had set up by
creating an ifcfg-br0 in network-scripts, and that file is long since gone
(the box has gone through two kernel updates and at least as many boots
since then).  I didn''t use xend tools to create this stuff, so I have
no
idea why xend has taken to creating the bridge (which was never even used
for a domU).  Additionally, when I was messing around trying to get rid of
it, xend stopped creating peth0.  I created some network-scripts and added a
line to rc.local to get the system working again and make all of this stuff
to act on my terms, but I would like to find the config files responsible
and get this set up properly.  In the discussion labeled "Fedora Core 8 +
Xenbr0 + network bridging?" John Summerfield suggested that I use rpm -V to
find any changed files that might be responsible.  I ran rpm- V for the
following packages:
kvm
qemu
kernel-xen
xen
xen-libs
libvirt-python
libvirt
virt-manager
python-virtinst
	The only files that aren''t in original condition are xendomains and
xen-config.sxp.  A diff of extracted originals and my copies shows nothing
different in either of these files that would cause my problem.  I have done
enough testing to determine that ''xend start'' (in the
appropriate bin
folder, not even the init script [which notably runs ''xend
start'' causing
the same end result]) creates br0 and no longer creates peth0 (these results
did not appear in tandem).  Can anyone point me toward any standard files
that might (when using xen cli utils but f7 runlevel 5) somehow get
automatically updated and effect how xend sets up the network (system boots
normal in non-xen kernels or when xend is prevented from starting) that
wouldn''t be associated with any of the above rpms (pointers toward any
other
rpms I might need to check would be appreciated too).  Thank you,
	Dustin

Hey all,

I''ve been meaning to respond to this for quite awhile but work is
crazy,
then the holidays etc etc. We were finally able to get this working on
Centos 5.1, though I''d hazard the same would work for Fedora 8 as I
have
that running on my desktop and was able to reproduce what I found on
that system.

We now have this going on several bonded NIC machines with a whole bunch
of Xen Virtual machines going and man oh man it''s great!

The heart of the issue seemed to be that we were working with a Bonded
nic and for some reason as soon as we bonded them virt-manager no longer
recognized anything.

What I was able to figure out was that as soon as you remove the
"HWADDR=00:00:00:00:00:00" line from your ifcfg-eth0 file virt-manager
no longer recognizes it as a valid bridge device. HOWEVER, if you use
virt-install from the command line you can STILL point it to the bridge
and it will work for the install and function from that point on NP. But
you cannot install through virt-manager.

I''m pretty sure that this is a bug, but it''s possible that
I''m missing
something so I wanted to pass it onto all you folks that were so helpful
when I was struggling through this the first time.

On my Fedora 8 desktop if I remove that HWADDR line from ifcfg-eth0 then
I can no longer select my bridge, but as soon as I add it back I can
once again select it.

The process that I went through to get this successfully function went
as follows (It was awhile ago, so if anyone tries this and runs into
anything not listed please let me know):

1. Install OS with both nics connected to non-bonded pair of connections
in the switch, both setup to be DHCP to make sure that both pull an
address.

2. After install I virsh net-undefine default and in
/etc/xen/xend-config.sxp comment out the line (network-script
network-bridge) and add the line (network-script /bin/true)

3. In /etc/xen/scripts/network-bridge I comment out the line
netdev=${netdev:-eth${vifnum}} and add the line netdev="bond0"

4. I create the bond normally, reboot and move the network connection to
the bonded pair on the switch and test to make sure the bond is fully
functional.

5. Once the bond is proved working I go back to
/etc/xen/xend-config.sxp, and uncomment (network-script network-bridge)
while commenting out (network-script /bin/true) Then I reboot so that it
can get the Xen bridge configured to use the bond.

6. After reboot you should see bond0, eth0, eth1, pbond0 and xenbr0
letting you know that the bridge is configured and ready to go. If you
try and create through virt-manager you will have NO option for bridged
networking, BUT if you run: virt-install --bridge=xenbr0 you can use the
bridge to setup machine NP!

Thanks for all of your help, there is no way I''d have been able to
figure out everything I did without insights from each of you. I''m
exceptionally satisfied with the end product and Long Live Xen!

David



-----Original Message-----
From: fedora-xen-bounces@redhat.com
[mailto:fedora-xen-bounces@redhat.com] On Behalf Of Mark Nielsen
Sent: Wednesday, December 05, 2007 11:59 AM
To: Christian Lahti
Cc: fedora-xen@redhat.com
Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?

Christian,

Sorry to hear it didn''t work out for you. It''s not a
straight-forward or

simple configuration. The problem is that there are so many possible 
combinations of VLANs, bonds, networks that most of the configuration is

left up to the ability and knowledge of the administrator. It sounds 
like you might''ve missed a few steps somewhere along the way. I admit 
that my document isn''t 100% step-by-step, it takes a pretty good 
understanding of everything that''s going on. Your problem could be in 
your step where you "Restart networking". This stuff all needs to
happen

in a particular order, and that might not have happened. Unfortunately 
it''s nearly impossible to help troubleshoot that in e-mails. I can say 
that I have the configuration I describe (and it sounds like what you 
want) working perfectly, so I know it''s possible.

Mark

Christian Lahti wrote:
> Hi Mark:
>
>  
>
> Well, 
>
>  
>
> I carefully went through your PDF document for setting up a bonded
pair
> on RHEL 5.1 and at the end of the day this does not work for me.  This
> is the procedure more or less I have tried:
>
>  
>
> *	Make sure eth0 and eth1 work as normal eth devices without LACP
> *	Edit /etc/modprobe.conf to add the bonding kernel module
> *	Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 to
> be slave devices with the master as bond0
> *	Create /etc/sysconfig/network-scripts/ifcfg-bond0 with static IP
> info
> *	Plug into LACP enabled ports on switch and reboot the box
> *	At this point bond0 works fine, as expected, we have been doing
> this all day long on non-xen kernels for years
> *	NOW...following your document
> *	Remove the /etc/libvirt/qemu/networks/autostart/* so that the
> pointless 192.168.122.x NAT network does not get created
> *	Grab the network-scripts-bridge-bond-vlan script from the link
> in the message and put into /etc/xen/scripts
> *	Edit /etc/xen/xend-config.sxp and replace (network-script
> network-bridge) with (network-script network-bridge-bond-vlan)
> *	Restart networking
> *	At this point networking for the physical host in invariably
> completely hosed, and there is no xen networking at all
> *	It is noteworthy to add that I did not make any modifications to
> the network-scripts-bridge-bond-vlan script as you suggest since I am
on
> a single vlan (so far) and the default setup in the script seems sane
> for a single NON-VLAN bond0 configuration
>
>  
>
> So in a nutshell the it seems that libvirt interface just does not
seem
> to use the xen scripts...rather it''s own (undocumented?) crap. 
After
a
> week of head-banging on this we really have no choice but to abandon
> Redhat in favor of Suse 10.3, which incidentally works right out of
the
> box with bridge, NAT, bond, you name it.  I suggest RH staff install a
> copy of this to see how they did it, you don''t even have to jump
through
> hoops to configure a bond from the command line, you can do this with
> their stock GUI tools if you so choose.  I am not knocking RedHat, but
> we need something that "just works" and so far for xen + bond +
RH has
> been a non-starter for us after much effort.  By the way, to get
> networking un-foobar''ed on the physical machine I have to run
> system-config-network and remove all Ethernet devices, then reboot,
then
> redefine eth0 and eth1 as as normal DHCP devices.  I can then go
> re-setup the bond0 device from the command line.
>
>  
>
> If anyone has step-by-step directions on how to do this with RHEL 5.1
I
> am happy to try again at some future point.  The goals are simple:
>
>  
>
> *	Step one: create a bond0 from eth0 and eth1 that can be used as
> a bridged device by xen on a single VLAN
> *	Step two: create a bond0 (aliases?) that can be assigned
> multiple VLANs, AND have these VLAN devices available to be used by
xen
>
>  
>
> I would settle for item #1 really.
>
>  
>
> /Christian
>
>  
>
>  
>
>  
>
>  
>
>  
>
> ________________________________
>
> From: Christian Lahti 
> Sent: Saturday, December 01, 2007 3:56 PM
> To: mnielsen@redhat.com; fedora-xen@redhat.com
> Subject: RE: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
>
>  
>
> Hi Mark:
>
>  
>
> Thank you very much for your response, I did indeed read the original
> poster as Dale by mistake :)  So what you are saying makes perfect
sense
> to me and sounds like exactly what we are after, I will have 3 vlans
to
> bridge myself ultimately.  My next question is the relative merits of
> RHEL5.1 as compared to Fedora 8.  Obviously I would prefer the stable
> enterprise release rather than bleeding edge Fedora, but has fully
> virtualized windows performance been fixed in this release?  At any
rate
> I am looking forward to getting this up and running tomorrow!
>
>  
>
> /Christian
>
>  
>
>  
>
> ________________________________
>
> From: Mark Nielsen [mailto:mnielsen@redhat.com]
> Sent: Sat 12/1/2007 3:19 PM
> To: Christian Lahti
> Subject: Re: [Fedora-xen] Fedora Core 8 + Xenbr0 + network bridging?
>
> hmm, did you mean "Hi Mark" ??
>
> I have 8 Dell 2950s running RHEL 5.1 (new libvirt with that funky NAT
> they added). I have 4 NICs in each; 2 copper, 2 fiber. I bond the 2
> copper (eth0 and eth1) and call it bond0. bond0 carries my
"private"
IP
> for cluster suite communications on the dom0 (physical) cluster.
>
> Then I bond eth2 and eth3 (fiber) in to bond1. I lay down the public
> network for the dom0 cluster on bond1.100 (for example, that would be
> VLAN 100). I also add many (up to 10 or so now) VLANs on bond1
> (bond1.20, bond1.21, bond1.22, etc). Then I create xen bridges to each
> of these bond/VLAN devices. This allows me to put any particular VM on
> any particular (or combination up to 3) of these xen bridged bonded
VLAN
> device.
>
> My document explains, in detail, how to do all of this :) The only
added
> step is that I have to "undefine" (virsh net-undefine default)
the
> default network that the new libvirt creates (virbr0). Even with this
> new NAT thing they added, I''ve been told (by our devs) that the
> preferred way to do static network configurations is with the method I
> lay out. NAT is more for dynamic networks (cable modems, dial-up,
wifi,
> etc).
>
> I''m pretty sure there weren''t any significant changes in
Fedora 8
(we''ve
> dropped the word "core" now, btw) that don''t exist in
RHEL 5.1 with
> respects to the network. 5.0 -> 5.1 is when that NAT change came down
> the pipe.
>
> Mark
>
> p.s. I''m happy to answer any other questions you may have about my
> document. I''m quite certain that, if you follow it,
you''ll have what
> you''re looking for.
>
> Christian Lahti wrote:
>   
>> Hi Dale:
>>
>> I work with David who posted the original question to the mailing
>> list.  I think we need to give a bit more background info on what we
>> are trying to do.  We are running a mixed environment of mostly
CentOS
>> 3, 4and 5, we do have a few windows servers and XP systems as well. 
>> We are looking to virtualize all these platforms.  Normally we have a
>> bonded pair of NICs for the physical hosts, we were able to get this
>> running using CentOS 5 x86_64 with no problems, the guest machines
use
>> the bonded pair in bridged mode as expected after a bit of tweaking. 
>> The biggest issue we found with EL5 is that windows guest performace
>> is dismal at best, hence our decision to have a look at Fedora Core 8
>> x86_64.  I am happy to report that performance for all of our guest
>> platforms is *very* good with FC8, but it seems that libvirt changed
>> the way networking is setup for Xen.  The default NAT configuration
is
>> pretty useless for production server environment.  Thanks to the
>> mailing list we are now able to bridge a single NIC on FC8 (like eth0
>> for example), but we cannot figure out how to get a bridge for bond0
>> (comprised of eth0 and eth1) defined and available to Xen.  All the
>> tweaks that worked find on EL5 have not worked so far on FC8.  I am
>> going to review your document tomorrow and give it a try, but any
idea
>> on whether your methodology will work on FC8 and libvirt?  I am
>> willing to blow a Sunday to get this worked out once and for all :)
>>
>> Basically we are after good performance on both para and fully
>> virtualized guests using a bonded pair of GB NICs for speed and
>> redundancy.  If this can be achieved with enterprise linux then that
>> would be preferable, but we will go FC8 if the bonding thing can be
>> sorted out.  By the way Xensource 4.x looks to be a respin of RHEL5
>> and has pretty good performance but their free version is limited to
>> 32bit (and hence 4GB ram).  Adding the clustering failover is the
next
>> step of course :)
>>
>> Thanks again for the help so far.
>>
>> /Christian
>>
>>
>>
>>     
>> just FYI for the list, I have a how-to for a bonded and VLAN tagged
>>     
> network.
>   
>> http://www.certifried.com
>>
>> ODT and PDF formats available.
>>
>>  
>> It might not be the best way, but I''ve sent it out to my
colleagues
>> several times and have never received any negative feedback.
>> Mark
>>
>>
>>
>> Dale Bewley wrote:
>>  
>>
>>     I haven''t done bonding, but you should be able to bond
them and
>>     
> then compose a bridge on top of this bonded device I would think.
>   
>>     --
>>     Dale Bewley - Unix Administrator - Shields Library - UC Davis
>>     GPG: 0xB098A0F3 0D5A 9AEB 43F4 F84C 7EFD  1753 064D 2583 B098
A0F3
>>
>>     --
>>     Fedora-xen mailing list
>>     Fedora-xen redhat com
>>     https://www.redhat.com/mailman/listinfo/fedora-xen
>>        
>>
>>
>>     
>
************************************************************************
> ****
>   
>> Checked by MailWasher server (www.Firetrust.com)
>> WARNING. No FirstAlert account found.
>> To reduce spam further activate FirstAlert.
>> This message can be removed by purchasing a FirstAlert Account.
>>
>>     
>
************************************************************************
> ****
>   
>>     
>
------------------------------------------------------------------------
>   
>> --
>> Fedora-xen mailing list
>> Fedora-xen@redhat.com
>> https://www.redhat.com/mailman/listinfo/fedora-xen
>>  
>>     
>
>   
--
Fedora-xen mailing list
Fedora-xen@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-xen