Hong @ gmail
2006-Jan-26 16:28 UTC
[Xen-users] How to setup Xen for 3 bridges environment?
Hello, I am new to Xen and is trying a confugration that simulates a firewall environment. My target environment is to setup three bridges: one for Wan (xen-br0), one for DMZ (xbrdmz) and one for Lan (xbrlan) The three subnet are as follows: Wan: 192.168.21.0/24 Dmz: 192.168.22.0/24 Lan: 192.168.23.0/24 I use xen3.0 and FC4 for both dom0 and domU. My machine currently has one Nic, eth0, and I ensalve it into the Wan bridge xen-br0. Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP 192.168.22.15, 192.168.22.16). I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> 192.168.22.16 so that the PC from Wan can access the PC. Most of the things work fine. I can ping dom0 and the two domU and vice versa. I can ssh from dom0 and domU and vice versa and I can ssh from PC on Wan to dom0. The only problem is that I cannot ssh from PC on Wan to domU. I have tried another setup. If I don''t use the Wan bridge ( xen-br0 ) and just use the eth0 and the Dmz bridge (xbrdmz), everything works perfectly. (I can ssh from PC on Wan to domU also). However, I still want to have the Wan bridge cause I can add some domU in Wan subnet (so that I can say, add some IDS domU to Wan bridge). I have searched the mailing list and find a similar case is: http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html I have tried the NOTRACK option but still can help in my case. Just wonder anyone has setup similar environment? Thanks alot. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fernando Maior
2006-Jan-27 11:28 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
Hong, Did you have a look at Xen wiki documentation? You may compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse to your experience and try something like that. On 1/26/06, Hong @ gmail <whtsang22@gmail.com> wrote:> > Hello, > > I am new to Xen and is trying a confugration that simulates a firewall > environment. > My target environment is to setup three bridges: one for Wan (xen-br0), one > for DMZ (xbrdmz) and one for Lan (xbrlan) > The three subnet are as follows: > Wan: 192.168.21.0/24 > Dmz: 192.168.22.0/24 > Lan: 192.168.23.0/24 > > I use xen3.0 and FC4 for both dom0 and domU. > My machine currently has one Nic, eth0, and I ensalve it into the Wan bridge > xen-br0. > > Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP > 192.168.22.15, 192.168.22.16). > I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> 192.168.22.16 > so that the PC from Wan can access the PC. > Most of the things work fine. I can ping dom0 and the two domU and vice > versa. I can ssh from dom0 and domU and vice versa and I can ssh from PC on > Wan to dom0. > The only problem is that I cannot ssh from PC on Wan to domU. > > I have tried another setup. If I don''t use the Wan bridge ( xen-br0 ) and > just use the eth0 and the Dmz bridge (xbrdmz), everything works perfectly. > (I can ssh from PC on Wan to domU also). > > However, I still want to have the Wan bridge cause I can add some domU in > Wan subnet (so that I can say, add some IDS domU to Wan bridge). > I have searched the mailing list and find a similar case is: > http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html > > I have tried the NOTRACK option but still can help in my case. > > Just wonder anyone has setup similar environment? > Thanks alot. > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >-- Bye, Fernando Maior LPIC/1(31908) LinuxCounter(391325) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hong @ gmail
2006-Jan-27 16:16 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
Hi Fernando Maior,
Thanks for your information. I have a quick look on the Wiki document, but
find that it is a bit different from my current setup.
In my setup, I create 3 bridges in dom0 and dom0 also act as the firewall.
Following is the summary of my configuration:
1) I by-pass the Xen bridge.
- comment out the ''(network-script network-bridge)'' and
''(vif-script
vif-bridge)'' in ''xend-config.sxp''
- comment out the ''handle_iptable'' in vif-bridge.
2) Create my own 3 bridges as follows:
#
# xbrwan0 (Wan bridge)
#
ip link set eth0 down
ip addr flush eth0
brctl addbr xbrwan0
brctl stp xbrwan0 off
brctl setfd xbrwan0 0
ip addr add 192.168.21.11/24 dev xbrwan0 brd + (the xbrwan0 bridge ip)
ip addr add 192.168.21.15/24 dev xbrwan0 brd + (NAT ip to 192.168.22.15)
ip addr add 192.168.21.16/24 dev xbrwan0 brd + (NAT ip to 192.168.22.16)
sleep 2
brctl addif xbrwan0 eth0
ifconfig eth0 0.0.0.0 up
ip link set xbrwan0 up
#
# xbrdmz0 (Dmz bridge)
#
brctl addbr xbrdmz0
brctl stp xbrdmz0 off
brctl setfd xbrdmz0 0
ifconfig xbrdmz0 192.168.22.11 netmask 255.255.255.0 up (the Dmz bridge
ip)
#
# xbrlan0 (Lan bridge)
#
brctl addbr xbrlan0
brctl stp xbrlan0 off
brctl setfd xbrlan0 0
ifconfig xbrlan0 192.168.23.11 netmask 255.255.255.0 up (the Lan bridge
ip)
3) Start two domU with ip 15 and 16
xm create domU01 (ip 192.168.21.15)
xm create domU02 (ip 192.168.21.16)
4) Create the NAT tables.
#
iptables -X
iptables -F
iptables -X -t nat
iptables -F -t nat
iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.15 -j DNAT --to
192.168.22.15
iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.15 -j SNAT --to
192.168.21.15
iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.16 -j DNAT --to
192.168.22.16
iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.16 -j SNAT --to
192.168.21.16
sysctl -w net.ipv4.ip_forward=1
In above setup, I ssh from Pc in Wan to domU01 and domU02.
If I didn''t create the Wan bridge, instead using the eth0, I can ssh
from PC
in Wan to domU01 and domU02.
========================================================================Following
is the ifconfig and brctl show result.
[root@jssrv01 conf]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4710 errors:0 dropped:0 overruns:0 frame:0
TX packets:4760 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:717399 (700.5 KiB) TX bytes:469146 (458.1 KiB)
Interrupt:5 Base address:0xe800
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:33 errors:0 dropped:0 overruns:0 frame:0
TX packets:451 errors:0 dropped:22 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2418 (2.3 KiB) TX bytes:27310 (26.6 KiB)
vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:53 errors:0 dropped:0 overruns:0 frame:0
TX packets:434 errors:0 dropped:27 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3740 (3.6 KiB) TX bytes:25916 (25.3 KiB)
xbrdmz0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:192.168.22.11 Bcast:192.168.22.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:86 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4954 (4.8 KiB) TX bytes:2208 (2.1 KiB)
xbrlan0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet addr:192.168.23.11 Bcast:192.168.23.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
xbrwan0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88
inet addr:192.168.21.11 Bcast:192.168.21.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4709 errors:0 dropped:0 overruns:0 frame:0
TX packets:4049 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:646686 (631.5 KiB) TX bytes:416226 (406.4 KiB)
and brctl show result:
[root@jssrv01 conf]# brctl show
bridge name bridge id STP enabled interfaces
xbrdmz0 8000.feffffffffff no vif1.0
vif2.0
xbrlan0 8000.feffffffffff no vif3.0
xbrwan0 8000.000d0ba72f88 no eth0
Thanks
----- Original Message -----
From: "Fernando Maior" <fernando.souto.maior@gmail.com>
To: "Hong @ gmail" <whtsang22@gmail.com>; "Xen-users"
<xen-users@lists.xensource.com>
Sent: Friday, January 27, 2006 7:28 PM
Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment?
Hong,
Did you have a look at Xen wiki documentation? You may
compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse
to your experience and try something like that.
On 1/26/06, Hong @ gmail <whtsang22@gmail.com>
wrote:>
> Hello,
>
> I am new to Xen and is trying a confugration that simulates a firewall
> environment.
> My target environment is to setup three bridges: one for Wan (xen-br0),
> one
> for DMZ (xbrdmz) and one for Lan (xbrlan)
> The three subnet are as follows:
> Wan: 192.168.21.0/24
> Dmz: 192.168.22.0/24
> Lan: 192.168.23.0/24
>
> I use xen3.0 and FC4 for both dom0 and domU.
> My machine currently has one Nic, eth0, and I ensalve it into the Wan
> bridge
> xen-br0.
>
> Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP
> 192.168.22.15, 192.168.22.16).
> I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 ->
> 192.168.22.16
> so that the PC from Wan can access the PC.
> Most of the things work fine. I can ping dom0 and the two domU and vice
> versa. I can ssh from dom0 and domU and vice versa and I can ssh from PC
> on
> Wan to dom0.
> The only problem is that I cannot ssh from PC on Wan to domU.
>
> I have tried another setup. If I don''t use the Wan bridge (
xen-br0 ) and
> just use the eth0 and the Dmz bridge (xbrdmz), everything works perfectly.
> (I can ssh from PC on Wan to domU also).
>
> However, I still want to have the Wan bridge cause I can add some domU in
> Wan subnet (so that I can say, add some IDS domU to Wan bridge).
> I have searched the mailing list and find a similar case is:
> http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html
>
> I have tried the NOTRACK option but still can help in my case.
>
> Just wonder anyone has setup similar environment?
> Thanks alot.
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>
--
Bye,
Fernando Maior
LPIC/1(31908)
LinuxCounter(391325)
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Fernando Maior
2006-Jan-27 19:34 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
Dear Hong, I am sorry I cannot help. I even did not understand why you are using a so different setup. I believe that it is not the way Xen was designed to run, and I would not do that way, but it is your system and I do not know your reasons. Hope you have better luck with the List. On 1/27/06, Hong @ gmail <whtsang22@gmail.com> wrote:> Hi Fernando Maior, > > Thanks for your information. I have a quick look on the Wiki document, but > find that it is a bit different from my current setup. > In my setup, I create 3 bridges in dom0 and dom0 also act as the firewall. > Following is the summary of my configuration: > > 1) I by-pass the Xen bridge. > - comment out the ''(network-script network-bridge)'' and ''(vif-script > vif-bridge)'' in ''xend-config.sxp'' > - comment out the ''handle_iptable'' in vif-bridge. > > 2) Create my own 3 bridges as follows: > # > # xbrwan0 (Wan bridge) > # > ip link set eth0 down > ip addr flush eth0 > brctl addbr xbrwan0 > brctl stp xbrwan0 off > brctl setfd xbrwan0 0 > ip addr add 192.168.21.11/24 dev xbrwan0 brd + (the xbrwan0 bridge ip) > ip addr add 192.168.21.15/24 dev xbrwan0 brd + (NAT ip to 192.168.22.15) > ip addr add 192.168.21.16/24 dev xbrwan0 brd + (NAT ip to 192.168.22.16) > sleep 2 > brctl addif xbrwan0 eth0 > ifconfig eth0 0.0.0.0 up > ip link set xbrwan0 up > # > # xbrdmz0 (Dmz bridge) > # > brctl addbr xbrdmz0 > brctl stp xbrdmz0 off > brctl setfd xbrdmz0 0 > ifconfig xbrdmz0 192.168.22.11 netmask 255.255.255.0 up (the Dmz bridge > ip) > # > # xbrlan0 (Lan bridge) > # > brctl addbr xbrlan0 > brctl stp xbrlan0 off > brctl setfd xbrlan0 0 > ifconfig xbrlan0 192.168.23.11 netmask 255.255.255.0 up (the Lan bridge > ip) > > 3) Start two domU with ip 15 and 16 > xm create domU01 (ip 192.168.21.15) > xm create domU02 (ip 192.168.21.16) > > 4) Create the NAT tables. > # > iptables -X > iptables -F > iptables -X -t nat > iptables -F -t nat > iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.15 -j DNAT --to > 192.168.22.15 > iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.15 -j SNAT --to > 192.168.21.15 > iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.16 -j DNAT --to > 192.168.22.16 > iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.16 -j SNAT --to > 192.168.21.16 > sysctl -w net.ipv4.ip_forward=1 > > In above setup, I ssh from Pc in Wan to domU01 and domU02. > If I didn''t create the Wan bridge, instead using the eth0, I can ssh from PC > in Wan to domU01 and domU02. > > ========================================================================> Following is the ifconfig and brctl show result. > [root@jssrv01 conf]# ifconfig > eth0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4710 errors:0 dropped:0 overruns:0 frame:0 > TX packets:4760 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:717399 (700.5 KiB) TX bytes:469146 (458.1 KiB) > Interrupt:5 Base address:0xe800 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:33 errors:0 dropped:0 overruns:0 frame:0 > TX packets:451 errors:0 dropped:22 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:2418 (2.3 KiB) TX bytes:27310 (26.6 KiB) > > vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:53 errors:0 dropped:0 overruns:0 frame:0 > TX packets:434 errors:0 dropped:27 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:3740 (3.6 KiB) TX bytes:25916 (25.3 KiB) > > xbrdmz0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > inet addr:192.168.22.11 Bcast:192.168.22.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:86 errors:0 dropped:0 overruns:0 frame:0 > TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:4954 (4.8 KiB) TX bytes:2208 (2.1 KiB) > > xbrlan0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 > inet addr:192.168.23.11 Bcast:192.168.23.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > xbrwan0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 > inet addr:192.168.21.11 Bcast:192.168.21.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4709 errors:0 dropped:0 overruns:0 frame:0 > TX packets:4049 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:646686 (631.5 KiB) TX bytes:416226 (406.4 KiB) > > and brctl show result: > [root@jssrv01 conf]# brctl show > bridge name bridge id STP enabled interfaces > xbrdmz0 8000.feffffffffff no vif1.0 > > vif2.0 > xbrlan0 8000.feffffffffff no vif3.0 > xbrwan0 8000.000d0ba72f88 no eth0 > > Thanks > > > ----- Original Message ----- > From: "Fernando Maior" <fernando.souto.maior@gmail.com> > To: "Hong @ gmail" <whtsang22@gmail.com>; "Xen-users" > <xen-users@lists.xensource.com> > Sent: Friday, January 27, 2006 7:28 PM > Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment? > > > Hong, > > Did you have a look at Xen wiki documentation? You may > compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse > to your experience and try something like that. > > On 1/26/06, Hong @ gmail <whtsang22@gmail.com> wrote: > > > > Hello, > > > > I am new to Xen and is trying a confugration that simulates a firewall > > environment. > > My target environment is to setup three bridges: one for Wan (xen-br0), > > one > > for DMZ (xbrdmz) and one for Lan (xbrlan) > > The three subnet are as follows: > > Wan: 192.168.21.0/24 > > Dmz: 192.168.22.0/24 > > Lan: 192.168.23.0/24 > > > > I use xen3.0 and FC4 for both dom0 and domU. > > My machine currently has one Nic, eth0, and I ensalve it into the Wan > > bridge > > xen-br0. > > > > Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP > > 192.168.22.15, 192.168.22.16). > > I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> > > 192.168.22.16 > > so that the PC from Wan can access the PC. > > Most of the things work fine. I can ping dom0 and the two domU and vice > > versa. I can ssh from dom0 and domU and vice versa and I can ssh from PC > > on > > Wan to dom0. > > The only problem is that I cannot ssh from PC on Wan to domU. > > > > I have tried another setup. If I don''t use the Wan bridge ( xen-br0 ) and > > just use the eth0 and the Dmz bridge (xbrdmz), everything works perfectly. > > (I can ssh from PC on Wan to domU also). > > > > However, I still want to have the Wan bridge cause I can add some domU in > > Wan subnet (so that I can say, add some IDS domU to Wan bridge). > > I have searched the mailing list and find a similar case is: > > http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html > > > > I have tried the NOTRACK option but still can help in my case. > > > > Just wonder anyone has setup similar environment? > > Thanks alot. > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > > > > -- > Bye, > Fernando Maior > LPIC/1(31908) > LinuxCounter(391325) > >-- Bye, Fernando Maior LPIC/1(31908) LinuxCounter(391325) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Daniel Goertzen
2006-Jan-28 14:59 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
Hong, I don''t grasp the full complexity of what you''re trying to do, but I note that you are bypassing the vif0.x/veth0.x interfaces in dom0 which bypasses some special xen networking magic. Instead of assigning ip addresses directly to xbrwan0, add vif0.0 to xbrwan0 and assign the addresses to veth0. Likewise for the other bridges. Search the list tcp checksum errors for more info on this. Cheers, Dan. Hong @ gmail wrote:> Hi Fernando Maior, > > Thanks for your information. I have a quick look on the Wiki document, > but find that it is a bit different from my current setup. > In my setup, I create 3 bridges in dom0 and dom0 also act as the > firewall. > Following is the summary of my configuration: > > 1) I by-pass the Xen bridge. > - comment out the ''(network-script network-bridge)'' and ''(vif-script > vif-bridge)'' in ''xend-config.sxp'' > - comment out the ''handle_iptable'' in vif-bridge. > > 2) Create my own 3 bridges as follows: > # > # xbrwan0 (Wan bridge) > # > ip link set eth0 down > ip addr flush eth0 > brctl addbr xbrwan0 > brctl stp xbrwan0 off > brctl setfd xbrwan0 0 > ip addr add 192.168.21.11/24 dev xbrwan0 brd + (the xbrwan0 bridge > ip) > ip addr add 192.168.21.15/24 dev xbrwan0 brd + (NAT ip to > 192.168.22.15) > ip addr add 192.168.21.16/24 dev xbrwan0 brd + (NAT ip to > 192.168.22.16) > sleep 2 > brctl addif xbrwan0 eth0 > ifconfig eth0 0.0.0.0 up > ip link set xbrwan0 up > # > # xbrdmz0 (Dmz bridge) > # > brctl addbr xbrdmz0 > brctl stp xbrdmz0 off > brctl setfd xbrdmz0 0 > ifconfig xbrdmz0 192.168.22.11 netmask 255.255.255.0 up (the Dmz > bridge ip) > # > # xbrlan0 (Lan bridge) > # > brctl addbr xbrlan0 > brctl stp xbrlan0 off > brctl setfd xbrlan0 0 > ifconfig xbrlan0 192.168.23.11 netmask 255.255.255.0 up (the Lan > bridge ip) > > 3) Start two domU with ip 15 and 16 > xm create domU01 (ip 192.168.21.15) > xm create domU02 (ip 192.168.21.16) > > 4) Create the NAT tables. > # > iptables -X > iptables -F > iptables -X -t nat > iptables -F -t nat > iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.15 -j DNAT > --to 192.168.22.15 > iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.15 -j SNAT > --to 192.168.21.15 > iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.16 -j DNAT > --to 192.168.22.16 > iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.16 -j SNAT > --to 192.168.21.16 > sysctl -w net.ipv4.ip_forward=1 > > In above setup, I ssh from Pc in Wan to domU01 and domU02. > If I didn''t create the Wan bridge, instead using the eth0, I can ssh > from PC in Wan to domU01 and domU02. > > ========================================================================> Following is the ifconfig and brctl show result. > [root@jssrv01 conf]# ifconfig > eth0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4710 errors:0 dropped:0 overruns:0 frame:0 > TX packets:4760 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:717399 (700.5 KiB) TX bytes:469146 (458.1 KiB) > Interrupt:5 Base address:0xe800 > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > UP LOOPBACK RUNNING MTU:16436 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:33 errors:0 dropped:0 overruns:0 frame:0 > TX packets:451 errors:0 dropped:22 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:2418 (2.3 KiB) TX bytes:27310 (26.6 KiB) > > vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:53 errors:0 dropped:0 overruns:0 frame:0 > TX packets:434 errors:0 dropped:27 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:3740 (3.6 KiB) TX bytes:25916 (25.3 KiB) > > xbrdmz0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > inet addr:192.168.22.11 Bcast:192.168.22.255 > Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:86 errors:0 dropped:0 overruns:0 frame:0 > TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:4954 (4.8 KiB) TX bytes:2208 (2.1 KiB) > > xbrlan0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 > inet addr:192.168.23.11 Bcast:192.168.23.255 > Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) > > xbrwan0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 > inet addr:192.168.21.11 Bcast:192.168.21.255 > Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:4709 errors:0 dropped:0 overruns:0 frame:0 > TX packets:4049 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:646686 (631.5 KiB) TX bytes:416226 (406.4 KiB) > > and brctl show result: > [root@jssrv01 conf]# brctl show > bridge name bridge id STP enabled interfaces > xbrdmz0 8000.feffffffffff no vif1.0 > > vif2.0 > xbrlan0 8000.feffffffffff no vif3.0 > xbrwan0 8000.000d0ba72f88 no eth0 > > Thanks > > > ----- Original Message ----- From: "Fernando Maior" > <fernando.souto.maior@gmail.com> > To: "Hong @ gmail" <whtsang22@gmail.com>; "Xen-users" > <xen-users@lists.xensource.com> > Sent: Friday, January 27, 2006 7:28 PM > Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment? > > > Hong, > > Did you have a look at Xen wiki documentation? You may > compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse > to your experience and try something like that. > > On 1/26/06, Hong @ gmail <whtsang22@gmail.com> wrote: > >> >> Hello, >> >> I am new to Xen and is trying a confugration that simulates a firewall >> environment. >> My target environment is to setup three bridges: one for Wan >> (xen-br0), one >> for DMZ (xbrdmz) and one for Lan (xbrlan) >> The three subnet are as follows: >> Wan: 192.168.21.0/24 >> Dmz: 192.168.22.0/24 >> Lan: 192.168.23.0/24 >> >> I use xen3.0 and FC4 for both dom0 and domU. >> My machine currently has one Nic, eth0, and I ensalve it into the Wan >> bridge >> xen-br0. >> >> Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP >> 192.168.22.15, 192.168.22.16). >> I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> >> 192.168.22.16 >> so that the PC from Wan can access the PC. >> Most of the things work fine. I can ping dom0 and the two domU and vice >> versa. I can ssh from dom0 and domU and vice versa and I can ssh from >> PC on >> Wan to dom0. >> The only problem is that I cannot ssh from PC on Wan to domU. >> >> I have tried another setup. If I don''t use the Wan bridge ( xen-br0 ) >> and >> just use the eth0 and the Dmz bridge (xbrdmz), everything works >> perfectly. >> (I can ssh from PC on Wan to domU also). >> >> However, I still want to have the Wan bridge cause I can add some >> domU in >> Wan subnet (so that I can say, add some IDS domU to Wan bridge). >> I have searched the mailing list and find a similar case is: >> http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html >> >> I have tried the NOTRACK option but still can help in my case. >> >> Just wonder anyone has setup similar environment? >> Thanks alot. >> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> > > > -- > Bye, > Fernando Maior > LPIC/1(31908) > LinuxCounter(391325) > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hong @ gmail
2006-Jan-28 16:44 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
Hi Daniel, Really thanks for your hints and it is really the checksum errors. When I ssh from PC ( 192.168.21.100) in Wan to domU in xbrdmz0, it has checksum errors. It causes me not able to use ssh. ------------------------------------------------------------- [root@jssrv01 conf]# tcpdump -v -n host 192.168.21.15 tcpdump: WARNING: eth0: no IPv4 address assigned tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 00:13:01.797379 IP (tos 0x0, ttl 128, id 25066, offset 0, flags [DF], proto 6, length: 48) 192.168.21.100.1521 > 192.168.21.15.ssh: S [tcp s um ok] 4291786245:4291786245(0) win 65535 <mss 1460,nop,nop,sackOK> 00:13:01.798189 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto 6, length: 48) 192.168.21.15.ssh > 192.168.21.100.1521: S [tcp sum o k] 2137667989:2137667989(0) ack 4291786246 win 5840 <mss 1460,nop,nop,sackOK> 00:13:01.798364 IP (tos 0x0, ttl 128, id 25067, offset 0, flags [DF], proto 6, length: 40) 192.168.21.100.1521 > 192.168.21.15.ssh: . [tcp s um ok] ack 1 win 65535 00:13:01.823132 IP (tos 0x0, ttl 63, id 48214, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 00:13:04.816419 IP (tos 0x0, ttl 63, id 48216, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 00:13:08.706572 IP (tos 0x0, ttl 63, id 27133, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1520: P [bad t cp cksum b307 (->b507)!] 2129139235:2129139255(20) ack 731365070 win 5840 00:13:10.816417 IP (tos 0x0, ttl 63, id 48218, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 00:13:22.816419 IP (tos 0x0, ttl 63, id 48220, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad tcp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 00:13:32.706490 IP (tos 0x0, ttl 63, id 27135, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1520: P [bad tcp cksum b307 (->b507)!] 0:20(20) ack 1 win 5840 00:13:46.816434 IP (tos 0x0, ttl 63, id 48222, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad tcp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 However, as said before, if I don''t create the xbrwan0 and use eth0 instead. I can ssh from PC in Wan to domU. The tcpdump shows as follows: (PC and domU talks perfectly to each other). ------------------------------------------ [root@jssrv01 conf]# tcpdump -v -n host 192.168.21.15 tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 00:24:39.154125 IP (tos 0x0, ttl 128, id 29657, offset 0, flags [DF], proto 6, length: 48) 192.168.21.100.support > 192.168.21.15.ssh: S [tcp sum ok] 3072641878:3072641878(0) win 65535 <mss 1460,nop,nop,sackOK> 00:24:39.154804 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto 6, length: 48) 192.168.21.15.ssh > 192.168.21.100.support: S [tcp sum ok] 2802605415:2802605415(0) ack 3072641879 win 5840 <mss 1460,nop,nop,sackOK> 00:24:39.155017 IP (tos 0x0, ttl 128, id 29658, offset 0, flags [DF], proto 6, length: 40) 192.168.21.100.support > 192.168.21.15.ssh: . [tcp sum ok] ack 1 win 65535 00:24:39.195572 IP (tos 0x0, ttl 63, id 57410, offset 0, flags [DF], proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.support: P [bad tcp cksum abf2 (->ad7e)!] 1:21(20) ack 1 win 5840 00:24:39.200501 IP (tos 0x0, ttl 128, id 29661, offset 0, flags [DF], proto 6, length: 68) 192.168.21.100.support > 192.168.21.15.ssh: P [tcp sum ok] 1:29(28) ack 21 win 65515 00:24:39.201187 IP (tos 0x0, ttl 63, id 57412, offset 0, flags [DF], proto 6, length: 40) 192.168.21.15.ssh > 192.168.21.100.support: . [tcp sum ok] ack 29 win 5840 00:24:39.205163 IP (tos 0x0, ttl 63, id 57414, offset 0, flags [DF], proto 6, length: 680) 192.168.21.15.ssh > 192.168.21.100.support: P 21:661(640) ack 29 win 5840 00:24:39.210997 IP (tos 0x0, ttl 128, id 29665, offset 0, flags [DF], proto 6, length: 528) 192.168.21.100.support > 192.168.21.15.ssh: P 29:517(488) ack 661 win 64875 00:24:39.211064 IP (tos 0x0, ttl 128, id 29666, offset 0, flags [DF], proto 6, length: 56) 192.168.21.100.support > 192.168.21.15.ssh: P [tcp sum ok] 517:533(16) ack 661 win 64875 00:24:39.249606 IP (tos 0x0, ttl 63, id 57416, offset 0, flags [DF], proto 6, length: 40) 192.168.21.15.ssh > 192.168.21.100.support: . [tcp sum ok] ack 533 win 5840 00:24:39.258471 IP (tos 0x0, ttl 63, id 57418, offset 0, flags [DF], proto 6, length: 320) 192.168.21.15.ssh > 192.168.21.100.support: P 661:941(280) ack 533 win 5840 00:24:39.372868 IP (tos 0x0, ttl 128, id 29670, offset 0, flags [DF], proto 6, length: 40) 192.168.21.100.support > 192.168.21.15.ssh: . [tcp sum ok] ack 941 win 64595 00:24:39.551000 IP (tos 0x0, ttl 128, id 29672, offset 0, flags [DF], proto 6, length: 312) 192.168.21.100.support > 192.168.21.15.ssh: P 533:805(272) ack 941 win 64595 Though the problem is located, I still don''t know how to fix it...( I will search the list for any findings..) But....Just wonder Daniel has any further hints for me.... ;-> As I said before, I am new to Xen. This configuration is just an outcome of trial and error. I remember I initially use the default Xen configuration and has some problems (I cannot remember is it exactly the same as this one or not). Then, I change the configuration bit by bit and finally down this situation. I am happy to stick with Xen default. As Fernando gives me some reference, I will also check if it suit my target configuration. Hong... ----- Original Message ----- From: "Daniel Goertzen" <goertzen@ertw.com> To: "Hong @ gmail" <whtsang22@gmail.com> Cc: "Xen-users" <xen-users@lists.xensource.com> Sent: Saturday, January 28, 2006 10:59 PM Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment?> Hong, I don''t grasp the full complexity of what you''re trying to do, but I > note that you are bypassing the vif0.x/veth0.x interfaces in dom0 which > bypasses some special xen networking magic. Instead of assigning ip > addresses directly to xbrwan0, add vif0.0 to xbrwan0 and assign the > addresses to veth0. Likewise for the other bridges. > > Search the list tcp checksum errors for more info on this. > > Cheers, > Dan. > > Hong @ gmail wrote: > >> Hi Fernando Maior, >> >> Thanks for your information. I have a quick look on the Wiki document, >> but find that it is a bit different from my current setup. >> In my setup, I create 3 bridges in dom0 and dom0 also act as the >> firewall. >> Following is the summary of my configuration: >> >> 1) I by-pass the Xen bridge. >> - comment out the ''(network-script network-bridge)'' and ''(vif-script >> vif-bridge)'' in ''xend-config.sxp'' >> - comment out the ''handle_iptable'' in vif-bridge. >> >> 2) Create my own 3 bridges as follows: >> # >> # xbrwan0 (Wan bridge) >> # >> ip link set eth0 down >> ip addr flush eth0 >> brctl addbr xbrwan0 >> brctl stp xbrwan0 off >> brctl setfd xbrwan0 0 >> ip addr add 192.168.21.11/24 dev xbrwan0 brd + (the xbrwan0 bridge >> ip) >> ip addr add 192.168.21.15/24 dev xbrwan0 brd + (NAT ip to >> 192.168.22.15) >> ip addr add 192.168.21.16/24 dev xbrwan0 brd + (NAT ip to >> 192.168.22.16) >> sleep 2 >> brctl addif xbrwan0 eth0 >> ifconfig eth0 0.0.0.0 up >> ip link set xbrwan0 up >> # >> # xbrdmz0 (Dmz bridge) >> # >> brctl addbr xbrdmz0 >> brctl stp xbrdmz0 off >> brctl setfd xbrdmz0 0 >> ifconfig xbrdmz0 192.168.22.11 netmask 255.255.255.0 up (the Dmz bridge >> ip) >> # >> # xbrlan0 (Lan bridge) >> # >> brctl addbr xbrlan0 >> brctl stp xbrlan0 off >> brctl setfd xbrlan0 0 >> ifconfig xbrlan0 192.168.23.11 netmask 255.255.255.0 up (the Lan bridge >> ip) >> >> 3) Start two domU with ip 15 and 16 >> xm create domU01 (ip 192.168.21.15) >> xm create domU02 (ip 192.168.21.16) >> >> 4) Create the NAT tables. >> # >> iptables -X >> iptables -F >> iptables -X -t nat >> iptables -F -t nat >> iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.15 -j DNAT --to >> 192.168.22.15 >> iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.15 -j SNAT --to >> 192.168.21.15 >> iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.16 -j DNAT --to >> 192.168.22.16 >> iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.16 -j SNAT --to >> 192.168.21.16 >> sysctl -w net.ipv4.ip_forward=1 >> >> In above setup, I ssh from Pc in Wan to domU01 and domU02. >> If I didn''t create the Wan bridge, instead using the eth0, I can ssh from >> PC in Wan to domU01 and domU02. >> >> ========================================================================>> Following is the ifconfig and brctl show result. >> [root@jssrv01 conf]# ifconfig >> eth0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:4710 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:4760 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:1000 >> RX bytes:717399 (700.5 KiB) TX bytes:469146 (458.1 KiB) >> Interrupt:5 Base address:0xe800 >> >> lo Link encap:Local Loopback >> inet addr:127.0.0.1 Mask:255.0.0.0 >> UP LOOPBACK RUNNING MTU:16436 Metric:1 >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >> >> vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:33 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:451 errors:0 dropped:22 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:2418 (2.3 KiB) TX bytes:27310 (26.6 KiB) >> >> vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:53 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:434 errors:0 dropped:27 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:3740 (3.6 KiB) TX bytes:25916 (25.3 KiB) >> >> xbrdmz0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >> inet addr:192.168.22.11 Bcast:192.168.22.255 >> Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:86 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:4954 (4.8 KiB) TX bytes:2208 (2.1 KiB) >> >> xbrlan0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 >> inet addr:192.168.23.11 Bcast:192.168.23.255 >> Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >> >> xbrwan0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 >> inet addr:192.168.21.11 Bcast:192.168.21.255 >> Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> RX packets:4709 errors:0 dropped:0 overruns:0 frame:0 >> TX packets:4049 errors:0 dropped:0 overruns:0 carrier:0 >> collisions:0 txqueuelen:0 >> RX bytes:646686 (631.5 KiB) TX bytes:416226 (406.4 KiB) >> >> and brctl show result: >> [root@jssrv01 conf]# brctl show >> bridge name bridge id STP enabled interfaces >> xbrdmz0 8000.feffffffffff no vif1.0 >> >> vif2.0 >> xbrlan0 8000.feffffffffff no vif3.0 >> xbrwan0 8000.000d0ba72f88 no eth0 >> >> Thanks >> >> >> ----- Original Message ----- From: "Fernando Maior" >> <fernando.souto.maior@gmail.com> >> To: "Hong @ gmail" <whtsang22@gmail.com>; "Xen-users" >> <xen-users@lists.xensource.com> >> Sent: Friday, January 27, 2006 7:28 PM >> Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment? >> >> >> Hong, >> >> Did you have a look at Xen wiki documentation? You may >> compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse >> to your experience and try something like that. >> >> On 1/26/06, Hong @ gmail <whtsang22@gmail.com> wrote: >> >>> >>> Hello, >>> >>> I am new to Xen and is trying a confugration that simulates a firewall >>> environment. >>> My target environment is to setup three bridges: one for Wan (xen-br0), >>> one >>> for DMZ (xbrdmz) and one for Lan (xbrlan) >>> The three subnet are as follows: >>> Wan: 192.168.21.0/24 >>> Dmz: 192.168.22.0/24 >>> Lan: 192.168.23.0/24 >>> >>> I use xen3.0 and FC4 for both dom0 and domU. >>> My machine currently has one Nic, eth0, and I ensalve it into the Wan >>> bridge >>> xen-br0. >>> >>> Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP >>> 192.168.22.15, 192.168.22.16). >>> I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> >>> 192.168.22.16 >>> so that the PC from Wan can access the PC. >>> Most of the things work fine. I can ping dom0 and the two domU and vice >>> versa. I can ssh from dom0 and domU and vice versa and I can ssh from PC >>> on >>> Wan to dom0. >>> The only problem is that I cannot ssh from PC on Wan to domU. >>> >>> I have tried another setup. If I don''t use the Wan bridge ( xen-br0 ) >>> and >>> just use the eth0 and the Dmz bridge (xbrdmz), everything works >>> perfectly. >>> (I can ssh from PC on Wan to domU also). >>> >>> However, I still want to have the Wan bridge cause I can add some domU >>> in >>> Wan subnet (so that I can say, add some IDS domU to Wan bridge). >>> I have searched the mailing list and find a similar case is: >>> http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html >>> >>> I have tried the NOTRACK option but still can help in my case. >>> >>> Just wonder anyone has setup similar environment? >>> Thanks alot. >>> >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >>> >> >> >> -- >> Bye, >> Fernando Maior >> LPIC/1(31908) >> LinuxCounter(391325) >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> >> >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Daniel Goertzen
2006-Jan-29 19:17 UTC
Re: [Xen-users] How to setup Xen for 3 bridges environment?
If you fixed the vif/veth usage as I told you and it''s still broken, I can''t help you without more info. You did provide a lot of info, but there were some holes. If you''d like to provide more info, I suggest the following: 1. A script that sets up the whole thing 2. The outputs of the following after running the above script ip addr list ip route list brctl show iptables -L iptables -t nat -L xm list Cheers, Dan. Hong @ gmail wrote:> Hi Daniel, > > Really thanks for your hints and it is really the checksum errors. > When I ssh from PC ( 192.168.21.100) in Wan to domU in xbrdmz0, it has > checksum errors. It causes me not able to use ssh. > ------------------------------------------------------------- > [root@jssrv01 conf]# tcpdump -v -n host 192.168.21.15 > tcpdump: WARNING: eth0: no IPv4 address assigned > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size > 96 bytes > 00:13:01.797379 IP (tos 0x0, ttl 128, id 25066, offset 0, flags [DF], > proto 6, length: 48) 192.168.21.100.1521 > 192.168.21.15.ssh: S [tcp s > um ok] 4291786245:4291786245(0) win 65535 <mss 1460,nop,nop,sackOK> > 00:13:01.798189 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], > proto 6, length: 48) 192.168.21.15.ssh > 192.168.21.100.1521: S [tcp > sum o > k] 2137667989:2137667989(0) ack 4291786246 win 5840 <mss > 1460,nop,nop,sackOK> > 00:13:01.798364 IP (tos 0x0, ttl 128, id 25067, offset 0, flags [DF], > proto 6, length: 40) 192.168.21.100.1521 > 192.168.21.15.ssh: . [tcp s > um ok] ack 1 win 65535 > 00:13:01.823132 IP (tos 0x0, ttl 63, id 48214, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t > cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 > 00:13:04.816419 IP (tos 0x0, ttl 63, id 48216, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t > cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 > 00:13:08.706572 IP (tos 0x0, ttl 63, id 27133, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1520: P [bad t > cp cksum b307 (->b507)!] 2129139235:2129139255(20) ack 731365070 win 5840 > 00:13:10.816417 IP (tos 0x0, ttl 63, id 48218, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad t > cp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 > 00:13:22.816419 IP (tos 0x0, ttl 63, id 48220, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad > tcp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 > 00:13:32.706490 IP (tos 0x0, ttl 63, id 27135, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1520: P [bad > tcp cksum b307 (->b507)!] 0:20(20) ack 1 win 5840 > 00:13:46.816434 IP (tos 0x0, ttl 63, id 48222, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.1521: P [bad > tcp cksum 3a1 (->5a1)!] 1:21(20) ack 1 win 5840 > > However, as said before, if I don''t create the xbrwan0 and use eth0 > instead. I can ssh from PC in Wan to domU. > The tcpdump shows as follows: (PC and domU talks perfectly to each > other). > ------------------------------------------ > [root@jssrv01 conf]# tcpdump -v -n host 192.168.21.15 > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size > 96 bytes > 00:24:39.154125 IP (tos 0x0, ttl 128, id 29657, offset 0, flags [DF], > proto 6, length: 48) 192.168.21.100.support > 192.168.21.15.ssh: S > [tcp sum ok] 3072641878:3072641878(0) win 65535 <mss 1460,nop,nop,sackOK> > 00:24:39.154804 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], > proto 6, length: 48) 192.168.21.15.ssh > 192.168.21.100.support: S > [tcp sum ok] 2802605415:2802605415(0) ack 3072641879 win 5840 <mss > 1460,nop,nop,sackOK> > 00:24:39.155017 IP (tos 0x0, ttl 128, id 29658, offset 0, flags [DF], > proto 6, length: 40) 192.168.21.100.support > 192.168.21.15.ssh: . > [tcp sum ok] ack 1 win 65535 > 00:24:39.195572 IP (tos 0x0, ttl 63, id 57410, offset 0, flags [DF], > proto 6, length: 60) 192.168.21.15.ssh > 192.168.21.100.support: P > [bad tcp cksum abf2 (->ad7e)!] 1:21(20) ack 1 win 5840 > 00:24:39.200501 IP (tos 0x0, ttl 128, id 29661, offset 0, flags [DF], > proto 6, length: 68) 192.168.21.100.support > 192.168.21.15.ssh: P > [tcp sum ok] 1:29(28) ack 21 win 65515 > 00:24:39.201187 IP (tos 0x0, ttl 63, id 57412, offset 0, flags [DF], > proto 6, length: 40) 192.168.21.15.ssh > 192.168.21.100.support: . > [tcp sum ok] ack 29 win 5840 > 00:24:39.205163 IP (tos 0x0, ttl 63, id 57414, offset 0, flags [DF], > proto 6, length: 680) 192.168.21.15.ssh > 192.168.21.100.support: P > 21:661(640) ack 29 win 5840 > 00:24:39.210997 IP (tos 0x0, ttl 128, id 29665, offset 0, flags [DF], > proto 6, length: 528) 192.168.21.100.support > 192.168.21.15.ssh: P > 29:517(488) ack 661 win 64875 > 00:24:39.211064 IP (tos 0x0, ttl 128, id 29666, offset 0, flags [DF], > proto 6, length: 56) 192.168.21.100.support > 192.168.21.15.ssh: P > [tcp sum ok] 517:533(16) ack 661 win 64875 > 00:24:39.249606 IP (tos 0x0, ttl 63, id 57416, offset 0, flags [DF], > proto 6, length: 40) 192.168.21.15.ssh > 192.168.21.100.support: . > [tcp sum ok] ack 533 win 5840 > 00:24:39.258471 IP (tos 0x0, ttl 63, id 57418, offset 0, flags [DF], > proto 6, length: 320) 192.168.21.15.ssh > 192.168.21.100.support: P > 661:941(280) ack 533 win 5840 > 00:24:39.372868 IP (tos 0x0, ttl 128, id 29670, offset 0, flags [DF], > proto 6, length: 40) 192.168.21.100.support > 192.168.21.15.ssh: . > [tcp sum ok] ack 941 win 64595 > 00:24:39.551000 IP (tos 0x0, ttl 128, id 29672, offset 0, flags [DF], > proto 6, length: 312) 192.168.21.100.support > 192.168.21.15.ssh: P > 533:805(272) ack 941 win 64595 > > Though the problem is located, I still don''t know how to fix it...( I > will search the list for any findings..) > But....Just wonder Daniel has any further hints for me.... ;-> > > As I said before, I am new to Xen. This configuration is just an > outcome of trial and error. > I remember I initially use the default Xen configuration and has some > problems (I cannot remember is it exactly the same as this one or > not). Then, I change the configuration bit by bit and finally down > this situation. > I am happy to stick with Xen default. As Fernando gives me some > reference, I will also check if it suit my target configuration. > > Hong... > > > > > ----- Original Message ----- From: "Daniel Goertzen" <goertzen@ertw.com> > To: "Hong @ gmail" <whtsang22@gmail.com> > Cc: "Xen-users" <xen-users@lists.xensource.com> > Sent: Saturday, January 28, 2006 10:59 PM > Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment? > > >> Hong, I don''t grasp the full complexity of what you''re trying to do, >> but I note that you are bypassing the vif0.x/veth0.x interfaces in >> dom0 which bypasses some special xen networking magic. Instead of >> assigning ip addresses directly to xbrwan0, add vif0.0 to xbrwan0 and >> assign the addresses to veth0. Likewise for the other bridges. >> >> Search the list tcp checksum errors for more info on this. >> >> Cheers, >> Dan. >> >> Hong @ gmail wrote: >> >>> Hi Fernando Maior, >>> >>> Thanks for your information. I have a quick look on the Wiki >>> document, but find that it is a bit different from my current setup. >>> In my setup, I create 3 bridges in dom0 and dom0 also act as the >>> firewall. >>> Following is the summary of my configuration: >>> >>> 1) I by-pass the Xen bridge. >>> - comment out the ''(network-script network-bridge)'' and >>> ''(vif-script vif-bridge)'' in ''xend-config.sxp'' >>> - comment out the ''handle_iptable'' in vif-bridge. >>> >>> 2) Create my own 3 bridges as follows: >>> # >>> # xbrwan0 (Wan bridge) >>> # >>> ip link set eth0 down >>> ip addr flush eth0 >>> brctl addbr xbrwan0 >>> brctl stp xbrwan0 off >>> brctl setfd xbrwan0 0 >>> ip addr add 192.168.21.11/24 dev xbrwan0 brd + (the xbrwan0 >>> bridge ip) >>> ip addr add 192.168.21.15/24 dev xbrwan0 brd + (NAT ip to >>> 192.168.22.15) >>> ip addr add 192.168.21.16/24 dev xbrwan0 brd + (NAT ip to >>> 192.168.22.16) >>> sleep 2 >>> brctl addif xbrwan0 eth0 >>> ifconfig eth0 0.0.0.0 up >>> ip link set xbrwan0 up >>> # >>> # xbrdmz0 (Dmz bridge) >>> # >>> brctl addbr xbrdmz0 >>> brctl stp xbrdmz0 off >>> brctl setfd xbrdmz0 0 >>> ifconfig xbrdmz0 192.168.22.11 netmask 255.255.255.0 up (the Dmz >>> bridge ip) >>> # >>> # xbrlan0 (Lan bridge) >>> # >>> brctl addbr xbrlan0 >>> brctl stp xbrlan0 off >>> brctl setfd xbrlan0 0 >>> ifconfig xbrlan0 192.168.23.11 netmask 255.255.255.0 up (the Lan >>> bridge ip) >>> >>> 3) Start two domU with ip 15 and 16 >>> xm create domU01 (ip 192.168.21.15) >>> xm create domU02 (ip 192.168.21.16) >>> >>> 4) Create the NAT tables. >>> # >>> iptables -X >>> iptables -F >>> iptables -X -t nat >>> iptables -F -t nat >>> iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.15 -j DNAT >>> --to 192.168.22.15 >>> iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.15 -j SNAT >>> --to 192.168.21.15 >>> iptables -t nat -A PREROUTING -i xbrwan0 -d 192.168.21.16 -j DNAT >>> --to 192.168.22.16 >>> iptables -t nat -A POSTROUTING -o xbrwan0 -s 192.168.22.16 -j SNAT >>> --to 192.168.21.16 >>> sysctl -w net.ipv4.ip_forward=1 >>> >>> In above setup, I ssh from Pc in Wan to domU01 and domU02. >>> If I didn''t create the Wan bridge, instead using the eth0, I can ssh >>> from PC in Wan to domU01 and domU02. >>> >>> ========================================================================= >>> >>> Following is the ifconfig and brctl show result. >>> [root@jssrv01 conf]# ifconfig >>> eth0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:4710 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:4760 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:1000 >>> RX bytes:717399 (700.5 KiB) TX bytes:469146 (458.1 KiB) >>> Interrupt:5 Base address:0xe800 >>> >>> lo Link encap:Local Loopback >>> inet addr:127.0.0.1 Mask:255.0.0.0 >>> UP LOOPBACK RUNNING MTU:16436 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >>> >>> vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:33 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:451 errors:0 dropped:22 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:2418 (2.3 KiB) TX bytes:27310 (26.6 KiB) >>> >>> vif2.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:53 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:434 errors:0 dropped:27 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:3740 (3.6 KiB) TX bytes:25916 (25.3 KiB) >>> >>> xbrdmz0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF >>> inet addr:192.168.22.11 Bcast:192.168.22.255 >>> Mask:255.255.255.0 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:86 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:32 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:4954 (4.8 KiB) TX bytes:2208 (2.1 KiB) >>> >>> xbrlan0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 >>> inet addr:192.168.23.11 Bcast:192.168.23.255 >>> Mask:255.255.255.0 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:0 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) >>> >>> xbrwan0 Link encap:Ethernet HWaddr 00:0D:0B:A7:2F:88 >>> inet addr:192.168.21.11 Bcast:192.168.21.255 >>> Mask:255.255.255.0 >>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>> RX packets:4709 errors:0 dropped:0 overruns:0 frame:0 >>> TX packets:4049 errors:0 dropped:0 overruns:0 carrier:0 >>> collisions:0 txqueuelen:0 >>> RX bytes:646686 (631.5 KiB) TX bytes:416226 (406.4 KiB) >>> >>> and brctl show result: >>> [root@jssrv01 conf]# brctl show >>> bridge name bridge id STP enabled interfaces >>> xbrdmz0 8000.feffffffffff no vif1.0 >>> >>> vif2.0 >>> xbrlan0 8000.feffffffffff no vif3.0 >>> xbrwan0 8000.000d0ba72f88 no eth0 >>> >>> Thanks >>> >>> >>> ----- Original Message ----- From: "Fernando Maior" >>> <fernando.souto.maior@gmail.com> >>> To: "Hong @ gmail" <whtsang22@gmail.com>; "Xen-users" >>> <xen-users@lists.xensource.com> >>> Sent: Friday, January 27, 2006 7:28 PM >>> Subject: Re: [Xen-users] How to setup Xen for 3 bridges environment? >>> >>> >>> Hong, >>> >>> Did you have a look at Xen wiki documentation? You may >>> compare http://wiki.xensource.com/xenwiki/XenNetworkingSuse >>> to your experience and try something like that. >>> >>> On 1/26/06, Hong @ gmail <whtsang22@gmail.com> wrote: >>> >>>> >>>> Hello, >>>> >>>> I am new to Xen and is trying a confugration that simulates a firewall >>>> environment. >>>> My target environment is to setup three bridges: one for Wan >>>> (xen-br0), one >>>> for DMZ (xbrdmz) and one for Lan (xbrlan) >>>> The three subnet are as follows: >>>> Wan: 192.168.21.0/24 >>>> Dmz: 192.168.22.0/24 >>>> Lan: 192.168.23.0/24 >>>> >>>> I use xen3.0 and FC4 for both dom0 and domU. >>>> My machine currently has one Nic, eth0, and I ensalve it into the >>>> Wan bridge >>>> xen-br0. >>>> >>>> Dom0 has the IP address 192.168.21.11. I have two domU in DMZ with IP >>>> 192.168.22.15, 192.168.22.16). >>>> I use NAT 192.168.21.15 -> 192.168.22.15 and 192.168.21.16 -> >>>> 192.168.22.16 >>>> so that the PC from Wan can access the PC. >>>> Most of the things work fine. I can ping dom0 and the two domU and >>>> vice >>>> versa. I can ssh from dom0 and domU and vice versa and I can ssh >>>> from PC on >>>> Wan to dom0. >>>> The only problem is that I cannot ssh from PC on Wan to domU. >>>> >>>> I have tried another setup. If I don''t use the Wan bridge ( xen-br0 >>>> ) and >>>> just use the eth0 and the Dmz bridge (xbrdmz), everything works >>>> perfectly. >>>> (I can ssh from PC on Wan to domU also). >>>> >>>> However, I still want to have the Wan bridge cause I can add some >>>> domU in >>>> Wan subnet (so that I can say, add some IDS domU to Wan bridge). >>>> I have searched the mailing list and find a similar case is: >>>> http://lists.xensource.com/archives/html/xen-users/2005-06/msg00669.html >>>> >>>> >>>> I have tried the NOTRACK option but still can help in my case. >>>> >>>> Just wonder anyone has setup similar environment? >>>> Thanks alot. >>>> >>>> >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >>>> >>>> >>> >>> >>> -- >>> Bye, >>> Fernando Maior >>> LPIC/1(31908) >>> LinuxCounter(391325) >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >>> >> > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users