Xen users - Apr 2009 - network-bridge breaks networking when eth0:1 is added

I added a virtual interface eth0:1 on one of my machines and suddenly 
networking breaks on startup of the system (eth0 is gone) when xend starts 
and runs the network-bridge script. This is supposed to be fixed according 
to: https://bugzilla.redhat.com/show_bug.cgi

I checked and the network-bridge script contains this fix. It works fine 
on one of my PCs when eth0:1 is added, but not on this one. The one where 
it works runs Xen 3.2.1, the one where it doesn''t work runs Xen 3.3.1. 
Both are CentOS 5.3. I compared the scripts and both are absolutely 
identical in this area. I also copied the 3.2.1 script over to the problem 
machine and get the same problem. So, it''s probably not Xen version 
related.
Another difference is that the problem machine has also eth1 and eth2 and 
I first assumed the problem might be associated to that. But it happens 
even with eth1 and eth2 disabled.

I can reproduce the problem when I boot with xend disabled. When I then 
start xend I get the error message as described in the bug report and eth0 
is killed. I can then restart the network and all is well. Also, after I 
started xend *once* it seems I can restart it and the problem does *not* 
reappear. It seems to happen only on the first start of xend and not 
thereafter. Which is rather weird, isn''t it?

Anyone knows how to fix this or find the cause?

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Kai Schaetzl wrote on Mon, 27 Apr 2009 01:47:07 +0200:
> Which is rather weird, isn''t it?
Ok, the reason for this is simple. It seems that once the bridge is up a 
xend restart does not restart the network-bridge script. If I run the 
network-bridge script manually it breaks eth0 each time, not just the 
first time. It even breaks it with "network-bridge stop".

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Apr 27, 2009 at 7:27 AM, Kai Schaetzl <maillists@conactive.com>
wrote:
> Kai Schaetzl wrote on Mon, 27 Apr 2009 01:47:07 +0200:
>
>> Which is rather weird, isn''t it?
>
> Ok, the reason for this is simple. It seems that once the bridge is up a
> xend restart does not restart the network-bridge script. If I run the
> network-bridge script manually it breaks eth0 each time, not just the
> first time. It even breaks it with "network-bridge stop".
For non-standard networking setup (like the one where you have eth0:1)
it''d be much easier if you let the OS handle the bridge and have
something like (network-script /bin/true) on xend.sxp. So for example
you can have these files

/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-xenbr0
/etc/sysconfig/network-scripts/ifcfg-xenbr0:1

which would create the bridge xenbr0 with eth0 as physical device and
create a virtual interface on xenbr0:1 where you can assign a second
IP address.

An advantage of such approach is that you don''t have to depend on xend
for network setup. You can restart xend anytime you want and
networking would still be OK.

Let me know if you need more detail on this setup (like what those
file''s contents look like).

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha wrote on Mon, 27 Apr 2009 08:11:00 +0700:
> For non-standard networking setup (like the one where you have eth0:1)
> it''d be much easier if you let the OS handle the bridge and have
> something like (network-script /bin/true) on xend.sxp.
The point is that this setup works fine on the second machine I tested it 
on, but not on the first one. But the network setup regarding eth0 is the 
same except for the IP numbers. I can''t see a reason why it would work
on
one machine but not on the other.

So for example
> you can have these files
> 
> /etc/sysconfig/network-scripts/ifcfg-eth0
> /etc/sysconfig/network-scripts/ifcfg-xenbr0
This looks like the old way it was done in Xen 3.0. I was quite satisfied 
with the results when they swapped eth0 and peth0 and made eth0 the 
bridge. It has worked for me since then. I would rather like to continue 
with that I think.
> An advantage of such approach is that you don''t have to depend on
xend
> for network setup. You can restart xend anytime you want and
> networking would still be OK.
Ok, I see that this might be an advantage, although I usually never 
restart xend. The current problem manifests only when I boot. That''s
also
how I found out that something is wrong. When I first added eth0:1 
everything seemed to be fine and as expected.
> Let me know if you need more detail on this setup (like what those
> file''s contents look like).
Yes, I''m not experienced in bridging. If you could tell me a sample
config
I could try that.
I would also need to tell xen to use xenbr0 for the vifs then. I remember 
that from the early days, but I removed that now, of course. Looking 
around in old files I assume I have to use the following for my config 
files?
vif = [ ''mac=...., bridge=xenbr0'' ]

Maybe there is another, better way for my purpose, anyway? The reason why 
I want to add another IP address to eth0 is that I want to be able to 
access the domUs on eth0 from eth1 and eth2 which are in a different non-
routable subnet.
I had already tried by attaching eth1 and eth2 to the bridge but this 
killed the networking completely. (as I said, I''m not experienced in 
bridging ;-)

Thanks,

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Apr 27, 2009 at 4:17 PM, Kai Schaetzl <maillists@conactive.com>
wrote:
> Fajar A. Nugraha wrote on Mon, 27 Apr 2009 08:11:00 +0700:
>
>> For non-standard networking setup (like the one where you have eth0:1)
>> it''d be much easier if you let the OS handle the bridge and
have
>> something like (network-script /bin/true) on xend.sxp.
>
> The point is that this setup works fine on the second machine I tested it
> on, but not on the first one.
Didn''t you say Xen version is different?
I''m not saying that''s the root cause of problem (cause I
don''t know
whether or not it''s true by only looking at your info). My point is
that you said yourself the setup were different.
> So for example
>> you can have these files
>>
>> /etc/sysconfig/network-scripts/ifcfg-eth0
>> /etc/sysconfig/network-scripts/ifcfg-xenbr0
>
> This looks like the old way it was done in Xen 3.0.
RHEL5/Centos still uses the same thing. So does Xen 3.3.1 port from
Gitco repository.
You are correct however, if you say that Xen 3.3.1 (the vanilla one
from xen.org) behavior have changed.

>> Let me know if you need more detail on this setup (like what those
>> file''s contents look like).
>
> Yes, I''m not experienced in bridging. If you could tell me a
sample config
> I could try that.
Something like this should work
/etc/sysconfig/network-scripts/ifcfg-eth0 :
DEVICE=eth0
BOOTPROTO=static
ONBOOT=yes
BRIDGE=xenbr0

/etc/sysconfig/network-scripts/ifcfg-xenbr0 :
DEVICE=xenbr0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=static
IPADDR=192.168.1.10
NETMASK=255.255.255.0
GATEWAY=192.168.1.1

/etc/sysconfig/network-scripts/ifcfg-xenbr0:1 :
DEVICE=xenbr0:1
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.1.11
NETMASK=255.255.255.0

You could alyas change the bridge names if you want. I usually use
brXXX where XXX is the vlan number.
> I would also need to tell xen to use xenbr0 for the vifs then.
If I remember correctly Xen would use the first bridge it found by
default. Since I have lots of bridges (for different vlans), I like to
specify the bridge explicitly.
> The reason why
> I want to add another IP address to eth0 is that I want to be able to
> access the domUs on eth0 from eth1 and eth2 which are in a different non-
> routable subnet.
I''m not following you here. If you mean you want to create a private,
non-routeable network between dom0 and domU, you could simply create
another bridge.

/etc/sysconfig/network-scripts/ifcfg-brdummy0 :
DEVICE=brdummy0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=static
IPADDR=10.2.1.10
NETMASK=255.255.255.0

But if you want to create another bridge with eth1 or eth2 as the
physical device (thus domU would have two or more NICs, each connected
to a different bridge), you can copy the eth0/xenbr0 config above and
modify them accordingly.
> I had already tried by attaching eth1 and eth2 to the bridge but this
> killed the networking completely. (as I said, I''m not experienced
in
> bridging ;-)
If you mean you attached dom0''s eth0, eth1, and eth2 to the same
bridge when they''re on different network (a.k.a different vlan) and
connected to the same switch, then that is only expected :)

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

r A. Nugraha wrote on Mon, 27 Apr 2009 19:51:55 +0700:
> Didn''t you say Xen version is different?
But the network-bridge script is the same. I even copied over the one from 
3.2.1 to the 3.3.1 setup, just to be sure. But comparing them didn''t
reveal
any big difference, anyway. The main difference is some extra code for 
detecting NFS.
> > So for example
> >> you can have these files
> >>
> >> /etc/sysconfig/network-scripts/ifcfg-eth0
> >> /etc/sysconfig/network-scripts/ifcfg-xenbr0
> >
> > This looks like the old way it was done in Xen 3.0.
> 
> RHEL5/Centos still uses the same thing. So does Xen 3.3.1 port from
> Gitco repository.
No, not here. On this machine I installed the rpms from gitco right away. 
There weren''t any self-made or xen.org supplied rpms and I think I even
didn''t
install the CentOS xen. I applied the gitco stuff right-away.
The network-bridge script is dated 5. 1.2009.
Are you having xenbr on a gitco setup? By simply installing the gitco rpms and 
starting xend?
> >> Let me know if you need more detail on this setup (like what those
> >> file''s contents look like).
I''m reluctant to post long scripts where most people might not be
interested
in. As I said, the network-bridge script (in /etc/xen/scripts) is identical in 
the relevant areas to the older script and if I copy the older network-bridge 
script and also the xen-network-common.sh script over I get the exact same 
behavior.
> >
> > Yes, I''m not experienced in bridging. If you could tell me a
sample config
> > I could try that.
> 
> Something like this should work
Ok, thanks, that looks easy, I will check that out as an alternative!
Still, I would like to know why the normal ways fails on that machine :-(
It''s not so easy to capture the output from the various commands in the
script, but it sure gets the same initial input for the inet interfaces from
ip addr show dev eth0.
> 
> > I would also need to tell xen to use xenbr0 for the vifs then.
> 
> If I remember correctly Xen would use the first bridge it found by
> default. Since I have lots of bridges (for different vlans), I like to
> specify the bridge explicitly.
Aha, ok, let''s try that first :-)
> 
> > The reason why
> > I want to add another IP address to eth0 is that I want to be able to
> > access the domUs on eth0 from eth1 and eth2 which are in a different
non-
> > routable subnet.
> 
> I''m not following you here. If you mean you want to create a
private,
> non-routeable network between dom0 and domU, you could simply create
> another bridge.
> 
> /etc/sysconfig/network-scripts/ifcfg-brdummy0 :
> DEVICE=brdummy0
> ONBOOT=yes
> TYPE=Bridge
> BOOTPROTO=static
> IPADDR=10.2.1.10
> NETMASK=255.255.255.0
But how do the domU''s recognize that? I''m just using eth0 in
there and as I
understand that independant of any bridge names on the dom0, anyway?
> 
> But if you want to create another bridge with eth1 or eth2 as the
> physical device (thus domU would have two or more NICs, each connected
> to a different bridge), you can copy the eth0/xenbr0 config above and
> modify them accordingly.
Ok. That actually sounds like the "real thing". I''m going to
check out the
simple, one-bridge thing from above and then build upon it once that works.

The situation is as follows.
Three machines. All in the same rack to the same switch, 100 MBit links, in 
the same datacenter. All eth0 are on the same routable subnet. Two of the 
machines are cross-over-cabled to the third machine via the additional ports. 
These ports are all on a non-routable subnet of their own, no gateway set. I 
want to access the domUs via these extra 1 Gig links for instance for backup 
purposes. Going thru the direct cable link would be much faster. So, I need 
something to "bridge" from eth1 to eth0 on the source machines. If I
add an IP
address from the same subnet as eth1 to eth0:1 and to each of the running 
domUs I can access them (I guess by way of broadcasting).



Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha wrote on Mon, 27 Apr 2009 19:51:55 +0700:

Ok, the bridge setup with eth0/xenbr0 seems to work, but xenbr0:1
doesn''t
give me access to the domUs as I had before with eth0:1.

I used the following config:

ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:1E:C9:FE:FB:AB
#IPADDR=192.168.1.24
#NETMASK=255.255.255.0
ONBOOT=yes
BRIDGE=xenbr0

ifcfg-xenbr0:
DEVICE=xenbr0
TYPE=Bridge
BOOTPROTO=static
#HWADDR=00:1E:C9:FE:FB:AB
IPADDR=192.168.1.24
NETMASK=255.255.255.0
ONBOOT=yes

This IP address will actually get changed later to a public address.

DEVICE=xenbr0:1
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.2.5
NETMASK=255.255.255.224

This is the network that eth1 and eth2 are on. But I can''t access a
domU
now that has 192.168.2.n on eth0:1

However, I can with this setup and network-bridge doing it''s work with 
eth0 and peth0.

ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
DHCPCLASSHWADDR=00:19:66:57:7F:86
IPADDR=192.168.1.232
NETMASK=255.255.255.0
ONBOOT=yes

ifcfg-eth0:1
DEVICE=eth0:1
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.2.15
NETMASK=255.255.255.224

> But if you want to create another bridge with eth1 or eth2 as the
> physical device (thus domU would have two or more NICs, each connected
> to a different bridge), you can copy the eth0/xenbr0 config above and
> modify them accordingly.
It seems if I don''t use the network-bridge script I will have to go
that
route and create a second bridge. If I understand correctly I do the same 
as before but with eth1 and xenbr1 for instance and then add another vif 
device to the config file?

If I do that I could go without the first bridge as xenbr0:1 doesn''t
help
me, anyway :-) But all in all it looks cleaner.

When I reloaded the network first after changing the configuration to 
eth0/xenbr0 my eth0 device was completely gone. I couldn''t revive it
other
than reboot. I hope that was a single-time glitch.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Kai Schaetzl wrote on Mon, 27 Apr 2009 18:02:46 +0200:
> It seems if I don''t use the network-bridge script I will have to
go that
> route and create a second bridge. If I understand correctly I do the same 
> as before but with eth1 and xenbr1 for instance and then add another vif 
> device to the config file?
Done that. It''s not as nice as my original config, but it works. I have
to
explicitely state the xenbr1 interface when I want to ping the domU from the 
dom0. But I can''t ping back from the domU to the dom0, not even if I
specify
to use eth1 on it. Both ways worked without a problem with my original 
setup.
I suppose I should be able to come thru eth1 from outside without specifying 
a device and access the domUs. That''s the main thing I want to do. I
can''t
test this case at the moment, though, as I don''t have a spare box that
I
could attach here with ease.

Thanks for the help!

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, Apr 27, 2009 at 9:37 PM, Kai Schaetzl <maillists@conactive.com>
wrote:
> The situation is as follows.
> Three machines. All in the same rack to the same switch, 100 MBit links, in
> the same datacenter. All eth0 are on the same routable subnet. Two of the
> machines are cross-over-cabled to the third machine via the additional
ports.
> These ports are all on a non-routable subnet of their own, no gateway set.
I
> want to access the domUs via these extra 1 Gig links for instance for
backup
> purposes. Going thru the direct cable link would be much faster. So, I need
> something to "bridge" from eth1 to eth0 on the source machines.
If I add an IP
> address from the same subnet as eth1 to eth0:1 and to each of the running
> domUs I can access them (I guess by way of broadcasting).
No, that won''t work. Are you famliar with the difference between
bridge and route?
I believe you have two alternatives :

(1) Setup multiple bridges
For example, br0 for eth0 and br1 for eth1. Then you assign two NICs
to domU, each NIC on different bridge. Think of it like having two
switches: one switch for eth0, another for eth1. In this scenario domU
will be like another dom0 in that it have a "private connection" to
third machine via second NIC.

(2) setup static routing on dom0 and domU.
This way traffic from domU to thrid machine can go something like this:
domU eth0 -> dom0 xenbr0 -> dom0 eth1 -> third machine eth1.
Note that this does not involve adding extra bridge or another IP
address. You just setup static routes and enable ipv4 forwarding on
dom0.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Im trying to use iptables on one of the guests.

my chain policy is drop and my rules are

iptables -A INPUT -p icmp -s 0/0 -d 0/0 -j ACCEPT

My default output policy is ACCEPT



Fajar A. Nugraha wrote:
> On Mon, Apr 27, 2009 at 9:37 PM, Kai Schaetzl
<maillists@conactive.com> wrote:
>   
>> The situation is as follows.
>> Three machines. All in the same rack to the same switch, 100 MBit
links, in
>> the same datacenter. All eth0 are on the same routable subnet. Two of
the
>> machines are cross-over-cabled to the third machine via the additional
ports.
>> These ports are all on a non-routable subnet of their own, no gateway
set. I
>> want to access the domUs via these extra 1 Gig links for instance for
backup
>> purposes. Going thru the direct cable link would be much faster. So, I
need
>> something to "bridge" from eth1 to eth0 on the source
machines. If I add an IP
>> address from the same subnet as eth1 to eth0:1 and to each of the
running
>> domUs I can access them (I guess by way of broadcasting).
>>     
>
> No, that won''t work. Are you famliar with the difference between
> bridge and route?
> I believe you have two alternatives :
>
> (1) Setup multiple bridges
> For example, br0 for eth0 and br1 for eth1. Then you assign two NICs
> to domU, each NIC on different bridge. Think of it like having two
> switches: one switch for eth0, another for eth1. In this scenario domU
> will be like another dom0 in that it have a "private connection"
to
> third machine via second NIC.
>
> (2) setup static routing on dom0 and domU.
> This way traffic from domU to thrid machine can go something like this:
> domU eth0 -> dom0 xenbr0 -> dom0 eth1 -> third machine eth1.
> Note that this does not involve adding extra bridge or another IP
> address. You just setup static routes and enable ipv4 forwarding on
> dom0.
>
> Regards,
>
> Fajar
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
>   

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

First, pelase don''t hijack threads. If you want to post a new question,
then use "new message", please.
Second, it''s not clear what you actually want or what the problem is.

Please repost "new" and add that information.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha wrote on Tue, 28 Apr 2009 04:55:55 +0700:
> No, that won''t work.
It works fine with my original setup and on my other machine. I think even 
without forwarding switched on. That''s why I think it works via 
broadcasting.

Are you famliar with the difference between
> bridge and route?
> I believe you have two alternatives :
> 
> (1) Setup multiple bridges
> For example, br0 for eth0 and br1 for eth1. Then you assign two NICs
> to domU, each NIC on different bridge. 
That''s what I did and mentioned in my posting your replied to. That
works
in one direction (dom0 eth1 -> domU), but not from domU eth1 -> dom0. I 
don''t need that second variant, but it''s somehow
"disturbing" that it
doesn''t work.

Think of it like having two
> switches: one switch for eth0, another for eth1. In this scenario domU
> will be like another dom0 in that it have a "private connection"
to
> third machine via second NIC.
> 
> (2) setup static routing on dom0 and domU.
> This way traffic from domU to thrid machine can go something like this:
> domU eth0 -> dom0 xenbr0 -> dom0 eth1 -> third machine eth1.
I thought about that earlier, but it''s not clear to me what I should
set
as a gateway then. A static route for the subnet on third machine eth1 
with gateway IP dom0 eth1?


Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Apr 28, 2009 at 6:31 PM, Kai Schaetzl <maillists@conactive.com>
wrote:
> Fajar A. Nugraha wrote on Tue, 28 Apr 2009 04:55:55 +0700:
>
>> No, that won''t work.
>
> It works fine with my original setup and on my other machine. I think even
> without forwarding switched on. That''s why I think it works via
> broadcasting.
Seriously?
It shouldn''t work. Unless of course I misunderstood your explanantion.
Please post the output of these commands on the working dom0, the
non-working dom0, and on domU :
- brctl show
- ip addr list | grep "inet "

plus the ip address of the third machine (the one that''s connected to
dom0''s eth1).
>
> Are you famliar with the difference between
>> bridge and route?
>> I believe you have two alternatives :
>>
>> (1) Setup multiple bridges
>> For example, br0 for eth0 and br1 for eth1. Then you assign two NICs
>> to domU, each NIC on different bridge.
>
> That''s what I did and mentioned in my posting your replied to.
That works
> in one direction (dom0 eth1 -> domU), but not from domU eth1 -> dom0.
I
> don''t need that second variant, but it''s somehow
"disturbing" that it
> doesn''t work.
Are you sure this is what you did? As far as I can tell from your
previous explataion you only have one bridge. Then again I could''ve
misunderstood your explanation. Either way, the output of commands
above should give enough information.
>> (2) setup static routing on dom0 and domU.
>> This way traffic from domU to thrid machine can go something like this:
>> domU eth0 -> dom0 xenbr0 -> dom0 eth1 -> third machine eth1.
>
> I thought about that earlier, but it''s not clear to me what I
should set
> as a gateway then. A static route for the subnet on third machine eth1
> with gateway IP dom0 eth1?
It''d be easier once I know what your setup is like. At this point
I''m
at loss as to what your setup is like.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nugraha wrote on Tue, 28 Apr 2009 19:58:26 +0700:
> Seriously?
Yes. I would love to give you all the details. The problem is that what 
I''m working with is not the final setup but some "semblance".
I had to
improvise to get something that *should* behave similar than the final 
setup.
The final setup will be machine a4 with three NICs attached to two other 
machines (say a1 and a2, both with more than one NIC) via crossover cable 
and a4 pulling from the domUs on a1 and a2 data. I want to go thru the 
crossover link and not the switch, via a private network where all 
additional NICs and the domUs are on.
The problem is that a1 and a2 are not here in the office, but in 
production in the datacenter. So, I have a4 for testing and maybe can 
attach another box if need be. But most boxes I have here are in use and 
have only one NIC. So I can use them only for a short time and have to 
reconfigure them each time. I did this once and tested from another box 
(name it a3) to domUs on a4 with the network setup I mentioned in my first 
posting, so everythign done by the standard network-bridge script. I was 
happy with that as long as I didn''t reboot and noticed the initial
running
of network-bridge breaks the eth0/peth0 bridge.
Now, as I cannot always attach another machine for testing (and I don''t
have one with two NICs anyway) my base assumption is that what works from 
a3 -> a4 -> domU should also work without a3. e.g. I don''t ping
from a3
but from a4 dom0 -> a4 domUs, but using the interface/IP number that is on 
the private subnet ("ping 192.168.2.10" for instance, or "ping -I
xenbr1
192.168.2.10" which would ping from the xenbr1 interface which is bridging 
to eth1, where the external machine comes in).
I''m using this simplified test scenario for testing. ping the private
IP
on domU from dom0. If it works, fine. If it doesn''t, use -I xenbr1 for
the
ping. If that works, fine, but not so comfortable. If that doesn''t
work:
bad :-(

Does this make sense so far?

Now I created two bridges as you suggested on a4 and two NICs on the dom0. 
That works but only in one way (dom0 -> domU) and only if I attach the 
ping to xenbr1. I just played around with routes and have found that 
adding a static route (on dom0) solves this so I can ping from either side 
and without specifying an interface. This solution is much more 
comfortable. The private subnet in question is 192.168.2.0/27.
The output of this setup is now as follows:

dom0:
brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.001ec9fefbab       no              eth0
                                                        vif14.0
xenbr1          8000.001ec9fefbac       no              eth1
                                                        vif14.1

ip addr list | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 192.168.2.4/27 brd 192.168.2.31 scope global eth2
    inet 192.168.1.24/24 brd 192.168.1.255 scope global xenbr0
    inet 192.168.2.3/27 brd 192.168.2.31 scope global xenbr1

ip route
192.168.2.10 via 192.168.2.3 dev xenbr1  scope link
192.168.2.0/27 dev eth2  proto kernel  scope link  src 192.168.2.4
192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
192.168.1.0/24 dev xenbr0  proto kernel  scope link  src 192.168.1.24
default via 192.168.1.1 dev xenbr0

domU is straight forward:
no bridge

ip addr list | grep "inet "
    inet 127.0.0.1/8 scope host lo
    inet 212.202.99.237/28 brd 212.202.99.239 scope global eth0
    inet 192.168.1.237/24 brd 192.168.1.255 scope global eth0:1
    inet 192.168.2.10/27 brd 192.168.2.31 scope global eth1

ip route
212.202.99.224/28 dev eth0  proto kernel  scope link  src 212.202.99.237
192.168.2.0/27 dev eth1  proto kernel  scope link  src 192.168.2.10
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.237
default via 192.168.1.1 dev eth0  src 192.168.1.237
default via 212.202.99.225 dev eth0

I had forwarding enabled in the meantime, but it works without it as well, 
so I disabled that again.

So, just to make it clear, *this* setup with the additional route is 
working now in all directions. I now realize that my best option is 
probably to use a different subnet each for eth1 and eth2 and different 
subnets on the two machines it goes out to (remember a4 will not be the 
target, but the source). If I do that I can use a route for each subnet, 
otherwise I have to use a route per single IP address.

Oh, and I just find that using different nets for eth1 and eth2 solves the 
problem, anyway, without a static route. Like so:
ip route
192.168.3.0/27 dev eth2  proto kernel  scope link  src 192.168.3.1
192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
192.168.1.0/24 dev xenbr0  proto kernel  scope link  src 192.168.1.24
default via 192.168.1.1 dev xenbr0

The big problem obviously was eth2 being on the same subnet as eth1/xenbr1 
and thus catching the packets to nowhere.

Good. I had just tried to add the static route with sysconfig/route-xenbr1 
and all variations I tried failed with obscure errors. I''m going to 
follow-up on this on the CentOS list.

Thanks for your answers, they were all really very helpful!

I think I have found a solution now, although I still would like to know 
why the original method works on some machines but not this one. 
Interestingly, one of the errors I got when trying to use route-xenbr1 was 
the same as when using the original setup and the network-bridge script.

If you are still interested in the original data I can revert this machine 
to the original setup and grab the data from there or from the other 
machine I mentioned where it works fine.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Im trying to apply rules on the guest and they dont work.

Craig Sparks wrote:
> Im trying to use iptables on one of the guests.
>
> my chain policy is drop and my rules are
>
> iptables -A INPUT -p icmp -s 0/0 -d 0/0 -j ACCEPT
>
> My default output policy is ACCEPT
>
>
>
> Fajar A. Nugraha wrote:
>> On Mon, Apr 27, 2009 at 9:37 PM, Kai Schaetzl 
>> <maillists@conactive.com> wrote:
>>  
>>> The situation is as follows.
>>> Three machines. All in the same rack to the same switch, 100 MBit 
>>> links, in
>>> the same datacenter. All eth0 are on the same routable subnet. Two 
>>> of the
>>> machines are cross-over-cabled to the third machine via the 
>>> additional ports.
>>> These ports are all on a non-routable subnet of their own, no 
>>> gateway set. I
>>> want to access the domUs via these extra 1 Gig links for instance 
>>> for backup
>>> purposes. Going thru the direct cable link would be much faster.
So,
>>> I need
>>> something to "bridge" from eth1 to eth0 on the source
machines. If I
>>> add an IP
>>> address from the same subnet as eth1 to eth0:1 and to each of the 
>>> running
>>> domUs I can access them (I guess by way of broadcasting).
>>>     
>>
>> No, that won''t work. Are you famliar with the difference
between
>> bridge and route?
>> I believe you have two alternatives :
>>
>> (1) Setup multiple bridges
>> For example, br0 for eth0 and br1 for eth1. Then you assign two NICs
>> to domU, each NIC on different bridge. Think of it like having two
>> switches: one switch for eth0, another for eth1. In this scenario domU
>> will be like another dom0 in that it have a "private
connection" to
>> third machine via second NIC.
>>
>> (2) setup static routing on dom0 and domU.
>> This way traffic from domU to thrid machine can go something like this:
>> domU eth0 -> dom0 xenbr0 -> dom0 eth1 -> third machine eth1.
>> Note that this does not involve adding extra bridge or another IP
>> address. You just setup static routes and enable ipv4 forwarding on
>> dom0.
>>
>> Regards,
>>
>> Fajar
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>>
>>   
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Apr 29, 2009 at 5:41 AM, Craig Sparks <csparks@iguananet.com>
wrote:
> Im trying to apply rules on the guest and they dont work.
And what does your setup looks like? Which distro and xen version,
packaged or self-compiled?

How was domU setup, PV or HVM? Does it boot from dom0 kernel, or does
it use pygrub?

Where did you apply the rules, on domU or on dom0 (yes, this is also possible)?

How do you know it didn''t work? Did it give out errors?

Does your kernel have iptables support? have you load the necessary modules?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Apr 29, 2009 at 1:37 AM, Kai Schaetzl <maillists@conactive.com>
wrote:
> dom0:
> brctl show
> bridge name     bridge id               STP enabled     interfaces
> xenbr0          8000.001ec9fefbab       no              eth0
>                                                        vif14.0
> xenbr1          8000.001ec9fefbac       no              eth1
>                                                        vif14.1
>
> ip addr list | grep "inet "
>    inet 127.0.0.1/8 scope host lo
>    inet 192.168.2.4/27 brd 192.168.2.31 scope global eth2
>    inet 192.168.1.24/24 brd 192.168.1.255 scope global xenbr0
>    inet 192.168.2.3/27 brd 192.168.2.31 scope global xenbr1
You have an error here.
you assign eth2 and xenbr1 IPs from the same IP subnet (192.168.2.4
and 192.168.2.3 are both on 192.168.2.0/27), yet they''re not on the
same physical network (not on the same bridge, and not connected to
the same switch/vlan).

If the interfaces are on different physical network, you need to
assign different IP subnet.  I believe this is the root cause of
problem on your setup : basic TCP/IP networking.
>
> ip route
> 192.168.2.10 via 192.168.2.3 dev xenbr1  scope link
This one should not be necessary. When you setup the ip addresses
correctly you don''t need to assign this route.
> 192.168.2.0/27 dev eth2  proto kernel  scope link  src 192.168.2.4
> 192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
The correct way to fix the problem depends on how you want things to work.

First possibilty : If you want eth2 and xenbr1 to be on the same
network, you need to REMOVE eth2''s IP address and assign it as a
bridge to xenbr1, so "brctl show" should look like this

xenbr1          8000.001ec9fefbac       no              eth1
                                                        eth2
                                                        vif14.1

Note that this means you put eth1 and eth2 on the same physical
network. Meaning :
- If both eth1 and eth2 are connected to the same switch, this setup
has the same effect as plugging a crossover cable on two switch ports.
- If eth1 is connected to a switch and eth2 is connected to another
server via a crossover cabel, this setup has the same effect as
connecting the second server directly to the switch.


Second possibilty : if eth2 and xenbr1 is intended to be a separate
network, you need to change the IP address so they''re on a different
subnet.
> domU is straight forward:
> no bridge
>
> ip addr list | grep "inet "
>    inet 127.0.0.1/8 scope host lo
>    inet 212.202.99.237/28 brd 212.202.99.239 scope global eth0
>    inet 192.168.1.237/24 brd 192.168.1.255 scope global eth0:1
>    inet 192.168.2.10/27 brd 192.168.2.31 scope global eth1
Again, the same question. Does domU''s eth1 supposed to be on the same
network as dom0''s eth1, or is it supposed to be on the same network as
dom0''s eth2? Or both?
>
> So, just to make it clear, *this* setup with the additional route is
> working now in all directions.
Bad idea.
> I now realize that my best option is
> probably to use a different subnet each for eth1 and eth2
Correct.
> Oh, and I just find that using different nets for eth1 and eth2 solves the
> problem, anyway, without a static route. Like so:
> ip route
> 192.168.3.0/27 dev eth2  proto kernel  scope link  src 192.168.3.1
> 192.168.2.0/27 dev xenbr1  proto kernel  scope link  src 192.168.2.3
> 192.168.1.0/24 dev xenbr0  proto kernel  scope link  src 192.168.1.24
> default via 192.168.1.1 dev xenbr0
exactly :)
So I gather you chose the second possibilty?
This would mean that dom0 and domU can communicate (through bridging
on xenbr1). But this also mean domU will NOT be able to communicate to
whatever machine connected to dom0''s eth2 (a4 machine?). If you want
it you need to :
- Setup dom0 as router, OR
- create another bridge for eth2 (say xenbr2) and add another vif on
domU connected to that bridge.

I suggest you enlist the help of network admins with your setup. To
make it simpler, think of bridge as another dumb switch without vlan
support.

Hope that helps. Good luck with your setup.

Regards,

Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Again: Please repost "new" and add that information.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha wrote on Wed, 29 Apr 2009 10:45:22 +0700:
> You have an error here.
> you assign eth2 and xenbr1 IPs from the same IP subnet (192.168.2.4
> and 192.168.2.3 are both on 192.168.2.0/27), yet they''re not on
the
> same physical network (not on the same bridge, and not connected to
> the same switch/vlan).
> 
> If the interfaces are on different physical network, you need to
> assign different IP subnet.  I believe this is the root cause of
> problem on your setup : basic TCP/IP networking.
Do I? I think it''s not disallowed, but it creates extra problems.
Certainly.
In the end it isn''t worth it. If I had experience with three NICs I had
known
that right-away. But the other machines are 2 NIC and I
"automatically"
assigned different nets. Just not this time, because it didn''t seem
necessary.
> > I now realize that my best option is
> > probably to use a different subnet each for eth1 and eth2
> 
> Correct.
;-)
> This would mean that dom0 and domU can communicate (through bridging
> on xenbr1). But this also mean domU will NOT be able to communicate to
> whatever machine connected to dom0''s eth2 (a4 machine?). If you
want
> it you need to :
> - Setup dom0 as router, OR
> - create another bridge for eth2 (say xenbr2) and add another vif on
> domU connected to that bridge.
I have done the latter. As you may remember this machine will eventually not 
be the target (as in these tests), but the machine pulling the data from domUs 
on other machines. So, I go out via xenbr1 to machine a1 and via xenbr2 to 
machine a2. I don''t need to go in the other direction, at least not to
domUs.
So, there is no need for bridging on this machine in theory. Just for the case 
that I may want it in the future nevertheless I have it now changed to have 
three bridges, though. Once you know how this works it''s really easy. I
have
also reconfigured one other machine to use the bridge setup instead of relying 
on Xen''s network-bridge. One problem I encountered there (and I think
also
with c4) is that I had to reboot to get the complete new network setup 
working. eth0/peth0 didn''t want to go away and thus the new bridge
could not
be created. I couldn''t brctl delbr nor ifdown the devices.

Again, thanks for the help, I''m now set to continue on my own with the
two
other machines.

Kai

-- 
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users