Alexander Pirsig
2009-Aug-30 22:30 UTC
[Xen-users] Multiple Network Cards + Multiple Bridges on debian lenny
Hi all, I''m searching for a tutorial how to setup multiple network cards with xen network bridge setup in debian lenny. My problem is, bridges seams only to work if i put an ip adress in dom0 to them. Thanks, Alex _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Maximilian Wilhelm
2009-Aug-30 23:14 UTC
Re: [Xen-users] Multiple Network Cards + Multiple Bridges on debian lenny
Anno domini 2009 Alexander Pirsig scripsit: Hi!> I''m searching for a tutorial how to setup multiple network cards with > xen network bridge setup in debian lenny. My problem is, bridges seams > only to work if i put an ip adress in dom0 to them.Can you be a bit more specific about your problem? When you have install the bridge-utils package, which is required for configuring bridges on Debian, you can put stanzas like this into the /etc/network/interfaces file: auto br123 iface br123 inet manual bridge_ports eth42 bridge_stp no bridge_fd 2 This will setup the bridge ''br123'' and attach the physical NIC ''eth42'' to it, which practically means you have a virtual switch inside your Dom0 where you can plug DomUs in. Dom0 will not be able to directly communicate with them, though. If you want your Dom0 to be part of this network, you have to configure an IP address on the bridge which can be done like this: auto br123 iface br123 inet static addresss 10.0.123.42 netmask 255.255.255.0 bridge_ports eth42 bridge_stp no bridge_fd 2 Beware of the ''static'' vs. ''manual'' thing. If you are using VLANs you can go one step further an do something like this (package vlan required) auto eth0 iface eth0 inet manual post-up ifconfig eth0 0.0.0.0 up pre-down ifconfig eth0 0.0.0.0 down auto eth0.123 iface eth0.123 inet manual vlan_raw_device eth0 post-up ifconfig $IFACE 0.0.0.0 up pre-down ifconfig $IFACE 0.0.0.0 down auto br123 iface br123 inet static addresss 10.0.123.42 netmask 255.255.255.0 bridge_ports eth0.123 bridge_stp no bridge_fd 2 HTH Ciao Max -- "Ich habe eher spontan mitbestellt, ohne genau zu wissen, was ich da gerade kaufe. Immer noch besser, als Schuhe zu kaufen." -- uschebit zum Thema SheevaPlug _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexander Pirsig
2009-Aug-31 06:23 UTC
Re: [Xen-users] Multiple Network Cards + Multiple Bridges on debian lenny
Hi Max,
thank you for you answer. Okay maybe I have to give more information
about my problem with bridge setup. Let me describe my server setup first.
I have a server with 4 network cards eth0-eth3. Network interfaces
eth1-eth3 should be bridged and connected with xen domUs. At the moment
my /etc/network/interfaces on Dom0 looks like this:
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.158
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
gateway 192.168.0.2
dns-nameservers 192.168.0.88
# eth1 -> intern 1
auto eth1
iface eth1 inet static
up ifconfig eth1 0.0.0.0 up
# eth1 -> xenbr1
auto xenbr1
iface xenbr1 inet manual
bridge_ports eth1
down ifconfig xenbr1 down
post-down brctl delbr xenbr1
# eth2 -> intern 2
auto eth1
iface eth1 inet static
up ifconfig eth1 0.0.0.0 up
# eth2 -> xenbr2
auto xenbr2
iface xenbr2 inet manual
bridge_ports eth2
down ifconfig xenbr2 down
post-down brctl delbr xenbr2
and so on.... in my /etc/xen/xend-config.sxp I have the following entries
(network-script network-dummy)
# (vif-script vif-bridge)
Then I configured networking for my DomU''s like this:
#
# Networking
#
vif = [ ''ip=192.168.0.173,bridge=xenbr1'' ]
Bridge config on Dom0 looks like this:
xenbr1 8000.001b213b8a69 no eth1
vif6.0
xenbr2 8000.001b213b8a6c no eth2
vif1.0
vif7.0
xenbr3 8000.001b213b8a6d no eth3
vif2.0
Yesterday I discovered that sometimes during Highload on DomU''s part of
the network is hard rechable with ssh or connections stalls even if I
try to connect to dom0 with has seperate uplink.
checking arp sometimes shows strange information (during highload) like:
arp -na
? (192.168.0.158) at 00:1b:21:3b:8a:68 [ether] on eth0
? (192.168.0.158) at 00:1b:21:3b:8a:68[ether] on xenbr1
So my question are:
1. Is it normal that bridge xenbr1 claims ip mac address of eth1?
2. Do I have to reset mac address on the bridge (xenbr1) or network
interface (eth1) to FF:FF:FF:FF bevor starting it.
3. What could cause the problem with the arp table?
4. Do I have to turn of arp/multicast on eth1 or xenbr1 like this ->
ip link set xenbr1 arp off
ip link set xenbr1 multicast off
Regards,
Alex
Maximilian Wilhelm wrote:> Anno domini 2009 Alexander Pirsig scripsit:
>
> Hi!
>
>
>> I''m searching for a tutorial how to setup multiple network
cards with
>> xen network bridge setup in debian lenny. My problem is, bridges seams
>> only to work if i put an ip adress in dom0 to them.
>>
>
> Can you be a bit more specific about your problem?
>
> When you have install the bridge-utils package, which is required for
> configuring bridges on Debian, you can put stanzas like this into the
> /etc/network/interfaces file:
>
>
> auto br123
> iface br123 inet manual
> bridge_ports eth42
> bridge_stp no
> bridge_fd 2
>
> This will setup the bridge ''br123'' and attach the
physical NIC ''eth42''
> to it, which practically means you have a virtual switch inside your
> Dom0 where you can plug DomUs in. Dom0 will not be able to directly
> communicate with them, though.
>
> If you want your Dom0 to be part of this network, you have to
> configure an IP address on the bridge which can be done like this:
>
> auto br123
> iface br123 inet static
> addresss 10.0.123.42
> netmask 255.255.255.0
> bridge_ports eth42
> bridge_stp no
> bridge_fd 2
>
> Beware of the ''static'' vs. ''manual''
thing.
>
>
> If you are using VLANs you can go one step further an do something
> like this (package vlan required)
>
> auto eth0
> iface eth0 inet manual
> post-up ifconfig eth0 0.0.0.0 up
> pre-down ifconfig eth0 0.0.0.0 down
>
> auto eth0.123
> iface eth0.123 inet manual
> vlan_raw_device eth0
> post-up ifconfig $IFACE 0.0.0.0 up
> pre-down ifconfig $IFACE 0.0.0.0 down
>
> auto br123
> iface br123 inet static
> addresss 10.0.123.42
> netmask 255.255.255.0
> bridge_ports eth0.123
> bridge_stp no
> bridge_fd 2
>
>
> HTH
> Ciao
> Max
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Aug-31 06:54 UTC
Re: [Xen-users] Multiple Network Cards + Multiple Bridges on debian lenny
On Mon, Aug 31, 2009 at 1:23 PM, Alexander Pirsig<alex@pirsig.net> wrote:> Bridge config on Dom0 looks like this: > > xenbr1 8000.001b213b8a69 no eth1 > vif6.0> Yesterday I discovered that sometimes during Highload on DomU''s part of the > network is hard rechable with ssh or connections stalls even if I try to > connect to dom0 with has seperate uplink.There was a thread on this list sometime ago, regarding iptables. Do you have /proc/net/ip_conntrack on dom0? Does the number of lines close to /proc/sys/net/ipv4/netfilter/ip_conntrack_max?> > checking arp sometimes shows strange information (during highload) like: > > arp -na > ? (192.168.0.158) at 00:1b:21:3b:8a:68 [ether] on eth0 > ? (192.168.0.158) at 00:1b:21:3b:8a:68[ether] on xenbr1 > > So my question are: > > 1. Is it normal that bridge xenbr1 claims ip mac address of eth1?Yes.> 2. Do I have to reset mac address on the bridge (xenbr1) or network > interface (eth1) to FF:FF:FF:FF bevor starting it.Not really. Your standard bridge setup using OS init scripts should work. That''s what I use, with RHEL.> 3. What could cause the problem with the arp table? > 4. Do I have to turn of arp/multicast on eth1 or xenbr1 like this -> > ip link set xenbr1 arp off > ip link set xenbr1 multicast offI''m not sure about these two. I don''t arp and multicast use their default settings on RHEL, YMMV. Since you mention "high load" I suspect it has something to do with iptables connection tracking. I wonder about your vif-script though. Why do you have it commeted out? Which vif-script did you use? # (vif-script vif-bridge) -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexander Pirsig
2009-Aug-31 08:25 UTC
Re: [Xen-users] Multiple Network Cards + Multiple Bridges on debian lenny
Hi Fajar, I think it could be the conntrack tables but at the moment its hard tell because the system load is back to normal. At the moment my conntrack table and settings look like this: cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max 4097152 cat /proc/net/ip_conntrack | wc -l 17186>I wonder about your vif-script though. Why do you have it commeted >out? Which vif-script did you use? ># (vif-script vif-bridge)The answer to this question is: cat /etc/xen/scripts/vif-bridge | head #!/bin/bash #===========================================================================# /etc/xen/vif-bridge # # Script for configuring a vif in bridged mode. # The hotplugging system will call this script if it is specified either in # the device configuration given to Xend, or the default Xend configuration # in /etc/xen/xend-config.sxp. If the script is specified in neither of those # places, then this script is the default. So that means if you comment out vif-bridge vif-bridge is used via default in Debian. I just wonder because in our application we''re getting a large amount of different Ip''s. A normal client connect takes 0.3ms and then they are gone :) probably I''ll should raise conntrack table settings. Regards, Alex Fajar A. Nugraha wrote:> On Mon, Aug 31, 2009 at 1:23 PM, Alexander Pirsig<alex@pirsig.net> wrote: > > >> Bridge config on Dom0 looks like this: >> >> xenbr1 8000.001b213b8a69 no eth1 >> vif6.0 >> > > >> Yesterday I discovered that sometimes during Highload on DomU''s part of the >> network is hard rechable with ssh or connections stalls even if I try to >> connect to dom0 with has seperate uplink. >> > > There was a thread on this list sometime ago, regarding iptables. Do > you have /proc/net/ip_conntrack on dom0? Does the number of lines > close to /proc/sys/net/ipv4/netfilter/ip_conntrack_max? > > >> checking arp sometimes shows strange information (during highload) like: >> >> arp -na >> ? (192.168.0.158) at 00:1b:21:3b:8a:68 [ether] on eth0 >> ? (192.168.0.158) at 00:1b:21:3b:8a:68[ether] on xenbr1 >> >> So my question are: >> >> 1. Is it normal that bridge xenbr1 claims ip mac address of eth1? >> > > Yes. > > >> 2. Do I have to reset mac address on the bridge (xenbr1) or network >> interface (eth1) to FF:FF:FF:FF bevor starting it. >> > > Not really. Your standard bridge setup using OS init scripts should > work. That''s what I use, with RHEL. > > >> 3. What could cause the problem with the arp table? >> 4. Do I have to turn of arp/multicast on eth1 or xenbr1 like this -> >> ip link set xenbr1 arp off >> ip link set xenbr1 multicast off >> > > I''m not sure about these two. I don''t arp and multicast use their > default settings on RHEL, YMMV. > Since you mention "high load" I suspect it has something to do with > iptables connection tracking. > > I wonder about your vif-script though. Why do you have it commeted > out? Which vif-script did you use? > # (vif-script vif-bridge) > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users