DBadmin Rottmann
2008-Mar-03 13:37 UTC
[Xen-users] combination of bridged and routed networking problem
Hi,
I have strange challenge regarding routing/networking on a xen host
(called tachyon).
We have setup the host using ubuntu 7.10 (gutsy), because this
is Debian-based and has xen3.1 included.
The vms should use either bridged network or routed network.
Therefore, we used
(network-script network-bridge)
(vif-script vif-bridge)
in xend-config.sxp.
Two vms (called etch32 and etch64) are connected with vif2.0 and vif3.0
to xenbr0. They have got IP addresses via DHCP and ip works
well to and from these vms.
The third vm (called lenny32) should be connected using a routed config.
Therefore, we have deleted vif4.0 from the bridge and activated
ARP, NAT, ... using the following commands on Dom0 (tachyon):
ip link set vif4.0 arp on
ip link set vif4.0 multicast on
ip addr flush vif4.0
ip link set vif4.0 addr 00:1E:0B:70:F6:9a
ifconfig vif4.0 192.168.0.1
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A PREROUTING -i xenbr0 -p tcp -m tcp --dport 222 -j DNAT
--to-destination 192.168.0.2:22
iptables -t nat -A PREROUTING -i xenbr0 -p tcp -m tcp --dport 443 -j DNAT
--to-destination 192.168.0.2:443
echo 1 > /proc/sys/net/ipv4/ip_forward
On lenny32, we have set the IP of eth0 to 192.168.0.2 statically.
IP connections initiated from lenny32 work well but inbound connections from
other hosts trying to connect to either port 222 or 443 on tachyon do not. These
connections should be directed via DNAT to lenny32. The TCP/IP handshake works
but the connection could not be established.
A TCP-Reset was send by the client.
Output of "tcpdump -i eth0 -s 0" at lenny32:
13:19:14.196801 arp who-has lenny32 tell 192.168.0.1
13:19:14.196827 arp reply lenny32 is-at 00:16:3e:54:f1:11 (oui Unknown)
13:19:14.196859 IP pc63002738.1054 > lenny32.ssh: S 1817694704:1817694704(0)
win 65535 <mss 1460,nop,nop,sackOK>
13:19:14.196890 IP lenny32.ssh > pc63002738.wl.1054: S
1304397835:1304397835(0) ack 1817694705 win 5840 <mss 1460,nop,nop,sackOK>
13:19:14.197074 IP pc63002738.1054 > lenny32.ssh: . ack 1 win 65535
13:19:14.203841 IP lenny32.ssh > pc63002738.wl.1054: P 1:32(31) ack 1 win
5840 (SSH: Server Protocol: SSH-2.0-OpenSSH_4.7p1 Debian-2)
13:19:14.204048 IP pc63002738.1054 > lenny32.ssh: R 1817694705:1817694705(0)
win 0 (Cients sends RESET !!???)
A direct ssh connection from tachyon to lenny32 works!
At lenny32 neither /etc/hosts.allow nor /etc/hosts.deny contains any entries.
A similiar behaviour could be observed when connecting to 443 on tachyon.
The command "iptables -Lvn" in the nat table shows that the DNAT rules
match, the policy of all
chains in the filter table is ACCEPT)
Any help is appreciated !!
Thanks in advance,
Valentin Rottmann
------------------------------------------------
# etch64: fake eth0 -> vif2.0 -+
# |
# etch32: fake eth0 -> vif3.0 -+
# |
# xenbr0 -> peth0 -> the network
# |
#tachyon: fake eth0 -> vif0.0 -+
# |
# (routing, MASQUERADING, DNAT)
# |
# vif4.0 <-> fake eth0 (lenny32)
root@tachyon:~# uname -a
Linux tachyon 2.6.22-14-xen #1 SMP Tue Feb 12 04:26:15 UTC 2008 x86_64 GNU/Linux
root@tachyon:~# dpkg -l | grep xen
ii libxen3.1 3.1.0-0ubuntu18 library
interface for Xen, a Virtual Machine
ii linux-headers-2.6.22-14-xen 2.6.22-14.52 Linux
kernel headers for version 2.6.22 on T
ii linux-image-2.6.22-14-xen 2.6.22-14.52 Linux
kernel image for version 2.6.22 on Thi
ii linux-image-xen 2.6.22.14.21 Linux
kernel image on Xen
ii linux-restricted-modules-2.6.22-14-xen 2.6.22.4-14.10 Non-free
Linux 2.6.22 modules on Xen
ii linux-restricted-modules-xen 2.6.22.14.21
Restricted Linux modules on Xen
ii linux-ubuntu-modules-2.6.22-14-xen 2.6.22-14.37 Ubuntu
supplied Linux modules for version 2.
ii linux-xen 2.6.22.14.21 Complete
Linux kernel on Xen
ii python-xen-3.1 3.1.0-0ubuntu18 python
bindings for Xen, a Virtual Machine M
ii xen-hypervisor-3.1 3.1.0-0ubuntu18 The Xen
Hypervisor for i386, amd64 amd lpia
ii xen-ioemu-3.1 3.1.0-0ubuntu18 XEN
administrative tools
ii xen-utils-3.1 3.1.0-0ubuntu18 XEN
administrative tools
root@tachyon:~# ip route show
192.168.0.0/24 dev vif4.0 proto kernel scope link src 192.168.0.1
10.35.18.0/24 dev eth0 proto kernel scope link src 10.35.18.38
default via 10.32.18.1 dev eth0 metric 100
root@tachyon:~# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0
peth0
vif2.0
vif3.0
root@tachyon:~# iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 1595 packets, 112K bytes)
pkts bytes target prot opt in out source destination
1112 75928 LOG 0 -- * * 0.0.0.0/0 0.0.0.0/0
LOG flags 0 level 4
15 860 DNAT tcp -- xenbr0 * 0.0.0.0/0 0.0.0.0/0
tcp dpt:222 to:192.168.0.2:22
32 1536 DNAT tcp -- xenbr0 * 0.0.0.0/0 0.0.0.0/0
tcp dpt:443 to:192.168.0.2:443
Chain POSTROUTING (policy ACCEPT 6785 packets, 418K bytes)
pkts bytes target prot opt in out source destination
1058 75471 MASQUERADE 0 -- * eth0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 5342 packets, 313K bytes)
pkts bytes target prot opt in out source destination
root@tachyon:~# iptables -L -v -n
Chain INPUT (policy ACCEPT 97072 packets, 7289K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 1649K packets, 100M bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0
PHYSDEV match --physdev-in vif2.0
7 1489 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0
PHYSDEV match --physdev-in vif3.0
90 6972 ACCEPT 0 -- * * 192.168.0.2 0.0.0.0/0
PHYSDEV match --physdev-in vif4.0
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0
PHYSDEV match --physdev-in vif4.0 udp spt:68 dpt:67
Chain OUTPUT (policy ACCEPT 81259 packets, 21M bytes)
pkts bytes target prot opt in out source destination
--
GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen!
Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
b52@entrap.de
2008-Mar-04 10:37 UTC
Re: [Xen-users] combination of bridged and routed networking problem
> Hi, > > I have strange challenge regarding routing/networking on a xen host > (called tachyon). > We have setup the host using ubuntu 7.10 (gutsy), because this > is Debian-based and has xen3.1 included. > > The vms should use either bridged network or routed network. > Therefore, we used > (network-script network-bridge) > (vif-script vif-bridge) > in xend-config.sxp. > > Two vms (called etch32 and etch64) are connected with vif2.0 and vif3.0 > to xenbr0. They have got IP addresses via DHCP and ip works > well to and from these vms. > > The third vm (called lenny32) should be connected using a routed config. > Therefore, we have deleted vif4.0 from the bridge and activated > ARP, NAT, ... using the following commands on Dom0 (tachyon): > > ip link set vif4.0 arp on > ip link set vif4.0 multicast on > ip addr flush vif4.0 > ip link set vif4.0 addr 00:1E:0B:70:F6:9a > ifconfig vif4.0 192.168.0.1 > > iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE > iptables -t nat -A PREROUTING -i xenbr0 -p tcp -m tcp --dport 222 -j DNAT > --to-destination 192.168.0.2:22 > iptables -t nat -A PREROUTING -i xenbr0 -p tcp -m tcp --dport 443 -j DNAT > --to-destination 192.168.0.2:443 > > echo 1 > /proc/sys/net/ipv4/ip_forward > > On lenny32, we have set the IP of eth0 to 192.168.0.2 statically. > > IP connections initiated from lenny32 work well but inbound connections > from other hosts trying to connect to either port 222 or 443 on tachyon do > not. These connections should be directed via DNAT to lenny32. The TCP/IP > handshake works but the connection could not be established. > A TCP-Reset was send by the client. > > Output of "tcpdump -i eth0 -s 0" at lenny32: > 13:19:14.196801 arp who-has lenny32 tell 192.168.0.1 > 13:19:14.196827 arp reply lenny32 is-at 00:16:3e:54:f1:11 (oui Unknown) > 13:19:14.196859 IP pc63002738.1054 > lenny32.ssh: S > 1817694704:1817694704(0) win 65535 <mss 1460,nop,nop,sackOK> > 13:19:14.196890 IP lenny32.ssh > pc63002738.wl.1054: S > 1304397835:1304397835(0) ack 1817694705 win 5840 <mss 1460,nop,nop,sackOK> > 13:19:14.197074 IP pc63002738.1054 > lenny32.ssh: . ack 1 win 65535 > 13:19:14.203841 IP lenny32.ssh > pc63002738.wl.1054: P 1:32(31) ack 1 win > 5840 (SSH: Server Protocol: SSH-2.0-OpenSSH_4.7p1 Debian-2) > 13:19:14.204048 IP pc63002738.1054 > lenny32.ssh: R > 1817694705:1817694705(0) win 0 (Cients sends RESET !!???) > > A direct ssh connection from tachyon to lenny32 works! > At lenny32 neither /etc/hosts.allow nor /etc/hosts.deny contains any > entries. > > A similiar behaviour could be observed when connecting to 443 on tachyon. > The command "iptables -Lvn" in the nat table shows that the DNAT rules > match, the policy of all > chains in the filter table is ACCEPT) > > Any help is appreciated !! > > Thanks in advance, > > Valentin Rottmann > > ------------------------------------------------ > # etch64: fake eth0 -> vif2.0 -+ > # | > # etch32: fake eth0 -> vif3.0 -+ > # | > # xenbr0 -> peth0 -> the network > # | > #tachyon: fake eth0 -> vif0.0 -+ > # | > # (routing, MASQUERADING, DNAT) > # | > # vif4.0 <-> fake eth0 (lenny32) > > > root@tachyon:~# uname -a > Linux tachyon 2.6.22-14-xen #1 SMP Tue Feb 12 04:26:15 UTC 2008 x86_64 > GNU/Linux > > root@tachyon:~# dpkg -l | grep xen > ii libxen3.1 3.1.0-0ubuntu18 > library interface for Xen, a Virtual Machine > ii linux-headers-2.6.22-14-xen 2.6.22-14.52 > Linux kernel headers for version 2.6.22 on T > ii linux-image-2.6.22-14-xen 2.6.22-14.52 > Linux kernel image for version 2.6.22 on Thi > ii linux-image-xen 2.6.22.14.21 > Linux kernel image on Xen > ii linux-restricted-modules-2.6.22-14-xen 2.6.22.4-14.10 > Non-free Linux 2.6.22 modules on Xen > ii linux-restricted-modules-xen 2.6.22.14.21 > Restricted Linux modules on Xen > ii linux-ubuntu-modules-2.6.22-14-xen 2.6.22-14.37 > Ubuntu supplied Linux modules for version 2. > ii linux-xen 2.6.22.14.21 > Complete Linux kernel on Xen > ii python-xen-3.1 3.1.0-0ubuntu18 > python bindings for Xen, a Virtual Machine M > ii xen-hypervisor-3.1 3.1.0-0ubuntu18 The > Xen Hypervisor for i386, amd64 amd lpia > ii xen-ioemu-3.1 3.1.0-0ubuntu18 XEN > administrative tools > ii xen-utils-3.1 3.1.0-0ubuntu18 XEN > administrative tools > > root@tachyon:~# ip route show > 192.168.0.0/24 dev vif4.0 proto kernel scope link src 192.168.0.1 > 10.35.18.0/24 dev eth0 proto kernel scope link src 10.35.18.38 > default via 10.32.18.1 dev eth0 metric 100 > > root@tachyon:~# brctl show > bridge name bridge id STP enabled interfaces > xenbr0 8000.feffffffffff no vif0.0 > peth0 > vif2.0 > vif3.0 > root@tachyon:~# iptables -t nat -L -v -n > Chain PREROUTING (policy ACCEPT 1595 packets, 112K bytes) > pkts bytes target prot opt in out source > destination > 1112 75928 LOG 0 -- * * 0.0.0.0/0 > 0.0.0.0/0 LOG flags 0 level 4 > 15 860 DNAT tcp -- xenbr0 * 0.0.0.0/0 > 0.0.0.0/0 tcp dpt:222 to:192.168.0.2:22 > 32 1536 DNAT tcp -- xenbr0 * 0.0.0.0/0 > 0.0.0.0/0 tcp dpt:443 to:192.168.0.2:443 > > Chain POSTROUTING (policy ACCEPT 6785 packets, 418K bytes) > pkts bytes target prot opt in out source > destination > 1058 75471 MASQUERADE 0 -- * eth0 0.0.0.0/0 > 0.0.0.0/0Well, actually I don''t see the reason but your NAT is quite strange!? Shouldn''t you only DNAT for eth0 port 222? like: iptables -A PREROUTING -i eth0 -p tcp --dport 222 -j DNAT --to-destination 192.168.0.2:22 Don''t do this at the bridge and I think you don''t need masquerade either. Regards, Holger> Chain OUTPUT (policy ACCEPT 5342 packets, 313K bytes) > pkts bytes target prot opt in out source > destination > > root@tachyon:~# iptables -L -v -n > Chain INPUT (policy ACCEPT 97072 packets, 7289K bytes) > pkts bytes target prot opt in out source > destination > > Chain FORWARD (policy ACCEPT 1649K packets, 100M bytes) > pkts bytes target prot opt in out source > destination > 0 0 ACCEPT 0 -- * * 0.0.0.0/0 > 0.0.0.0/0 PHYSDEV match --physdev-in vif2.0 > 7 1489 ACCEPT 0 -- * * 0.0.0.0/0 > 0.0.0.0/0 PHYSDEV match --physdev-in vif3.0 > 90 6972 ACCEPT 0 -- * * 192.168.0.2 > 0.0.0.0/0 PHYSDEV match --physdev-in vif4.0 > 0 0 ACCEPT udp -- * * 0.0.0.0/0 > 0.0.0.0/0 PHYSDEV match --physdev-in vif4.0 udp spt:68 > dpt:67 > > Chain OUTPUT (policy ACCEPT 81259 packets, 21M bytes) > pkts bytes target prot opt in out source > destination > > > > -- > GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! > Jetzt dabei sein: http://www.shortview.de/?mc=sv_ext_mf@gmx > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
DBadmin Rottmann
2008-Mar-05 15:24 UTC
Re: [Xen-users] combination of bridged and routed networking problem
Hi,
I have placed the following log-targets
to the tables:
iptables -I OUTPUT -j LOG --log-prefix "OUTPUT: "
iptables -I FORWARD -j LOG --log-prefix "FORWARD: "
iptables -t nat -I PREROUTING -j LOG --log-prefix "PREROUTING: "
iptables -t nat -I POSTROUTING -j LOG --log-prefix "POSTROUTING: "
iptables -I INPUT -j LOG --log-prefix "INPUT: "
The results below show, that the PREROUTING chain is visited
once by the packet with incoming interface xenbr0. Therefore,
I have placed the DNAT target into the chain as described in
my initial posting. Here, I have placed the DNAT as suggested by
Holger (b52@xxxx).
Chain PREROUTING (policy ACCEPT 2213 packets, 155K bytes)
pkts bytes target prot opt in out source destination
66 4447 LOG 0 -- * * 0.0.0.0/0 0.0.0.0/0
LOG flags 0 level 4 prefix `PREROUTING: ''
0 0 DNAT tcp -- eth0 * 0.0.0.0/0 10.32.18.38
tcp dpt:443 to:192.168.0.2:443
0 0 DNAT tcp -- eth0 * 0.0.0.0/0 10.32.18.38
tcp dpt:222 to:192.168.0.2:22
(The ssh client has the IP 10.43.2.15 and the server 10.32.18.38.)
Mar 5 15:52:27 tachyon kernel: [718938.533000] PREROUTING: IN=xenbr0 OUT=
PHYSIN=peth0 MAC=00:1e:0b:70:f6:96:00:0a:f3:e2:d4:0a:08:00 SRC=10.43.2.15
DST=10.32.18.38 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=43542 DF PROTO=TCP SPT=53416
DPT=222 WINDOW=5840 RES=0x00 SYN URGP=0
Mar 5 15:52:27 tachyon kernel: [718938.533016] FORWARD: IN=xenbr0 OUT=xenbr0
PHYSIN=peth0 PHYSOUT=vif0.0 SRC=10.43.2.15 DST=10.32.18.38 LEN=60 TOS=0x00
PREC=0x00 TTL=63 ID=43542 DF PROTO=TCP SPT=53416 DPT=222 WINDOW=5840 RES=0x00
SYN URGP=0
Mar 5 15:52:27 tachyon kernel: [718938.533025] POSTROUTING: IN= OUT=xenbr0
PHYSIN=peth0 PHYSOUT=vif0.0 SRC=10.43.2.15 DST=10.32.18.38 LEN=60 TOS=0x00
PREC=0x00 TTL=63 ID=43542 DF PROTO=TCP SPT=53416 DPT=222 WINDOW=5840 RES=0x00
SYN URGP=0
Mar 5 15:52:27 tachyon kernel: [718938.533048] INPUT: IN=eth0 OUT=
MAC=00:1e:0b:70:f6:96:00:0a:f3:e2:d4:0a:08:00 SRC=10.43.2.15 DST=10.32.18.38
LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=43542 DF PROTO=TCP SPT=53416 DPT=222
WINDOW=5840 RES=0x00 SYN URGP=0
Mar 5 15:52:27 tachyon kernel: [718938.533075] OUTPUT: IN= OUT=eth0
SRC=10.32.18.38 DST=10.43.2.15 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF
PROTO=TCP SPT=222 DPT=53416 WINDOW=0 RES=0x00 ACK RST URGP=0
Mar 5 15:52:27 tachyon kernel: [718938.533090] FORWARD: IN=xenbr0 OUT=xenbr0
PHYSIN=vif0.0 PHYSOUT=peth0 SRC=10.32.18.38 DST=10.43.2.15 LEN=40 TOS=0x00
PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=222 DPT=53416 WINDOW=0 RES=0x00 ACK RST
URGP=0
Best regards,
Valentin
--
Ist Ihr Browser Vista-kompatibel? Jetzt die neuesten
Browser-Versionen downloaden: http://www.gmx.net/de/go/browser
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users