Diego Alvarez
2006-Apr-28 12:54 UTC
[Xen-users] Solved: Re: Communication problem with virtual DMZ
Once again I was mistaken, the real problema was an incorrect TCP Checksum (discovered with tcpdump), so the solution was to use "ethtool -K eth0 tx off" on agustina. I learned this from the list archives. On 4/27/06, Diego Alvarez <arcane.lord@gmail.com> wrote:> Hi all, > I am running Xen 3.0.2-2 (taken from XenSource) with Linux kernel > 2.6.16 (taken from Debian Sid), I compiled Xen and 2 kernels > (dom0 and domU). > > Here is the ascii-art of my setup: > > ------------ ------------- > | LAN |------------------------| waste | 192.168.0.94/24 > ------------ ------------- > | > ····························· > · | · > · | Dom0 · > · | · > · --------- · ································ > · | peth0 | · · · > · --------- · · DomU hades · > · | · · (Firewall) · > · | · · · > · ----------- ---------- · · -------- · > · | br-inet |---| hades0 |============| eth0 | 192.168.0.34/24 · > · ----------- | (vif) | · · -------- · > · | ---------- · · · > · | · · -------- · > · ----------- · · | eth1 | 192.168.0.34/32 · > · | vif0.0 | · · -------- · > · ----------- · · || · > · || · ········||······················ > · || · || > · || ·············||··········· > · || || · > · -------- || · > · | eth0 | 192.168.0.22/24 ---------- · > · -------- | hades1 | · > · | (vif) | · > · ---------- · > · | · > · ----------- ---------- · > · | pdummy0 |---------| br-dmz | · > · ----------- ---------- · > · | · > · ------------- · > · | agustina0 | · > · | (vif) | · > · ------------- · > · || · > · || · > ·········································||··········· > || > ·-·-·-·-·-·-·-·-·-·||·-·-·-·-·-·-·-·-· > · || · > | ··········||········ | > · · || · · > | · -------- · | > · · | eth0 | · · > | · -------- · | > · · 192.168.0.39/32 · · > | · · | > · · DomU agustina · · > | · (DMZ Server) · | > · · · · > | ···················· | > · · > | Virtual DMZ | > · · > ·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·- > > -------- > > Network configuration for Dom0: > > auto eth0 > iface eth0 inet static > address 192.168.0.22 > netmask 255.255.255.0 > gateway 192.168.0.2 > > auto dummy0 > iface dummy0 inet static > address 10.1.1.1 > netmask 255.255.255.255 > up ifconfig dummy0 0.0.0.0 up > > -------- > > Network configuration for DomU agustina (DMZ Server): > > auto eth0 > iface eth0 inet static > address 192.168.0.39 > netmask 255.255.255.255 > up route add -host 192.168.0.34 dev eth0 > up route add default gw 192.168.0.34 dev eth0 > > ------- > > Network configuration for DomU hades (Firewall): > > auto eth0 > iface eth0 inet static > address 192.168.0.34 > netmask 255.255.255.0 > gateway 192.168.0.2 > up arp -Ds 192.168.0.39 eth0 pub > > auto eth1 > iface eth1 inet static > address 192.168.0.34 > netmask 255.255.255.255 > up route add -host 192.168.0.39 dev eth1 > > It also have ip_forward activated by sysctl > > ------ > > In dom0, I do the following things: > > In /etc/xen/xend-config.sxp I have: > > (network-script ''network-bridge bridge=br-inet'') > (vif-script ''vif-bridge bridge=br-inet'') > > > I also have a script which brings up br-dmz bridge on dummy0 > > # brctl show: > bridge name bridge id STP enabled interfaces > > br-dmz 8000.feffffffffff no agustina0 > hades1 > pdummy0 > > br-inet 8000.feffffffffff no hades0 > peth0 > vif0.0 > > Here is the configuration for hades and agustina: > > /etc/xen/auto/hades: > name="hades" > memory=128 > kernel="/boot/vmlinuz-2.6.16-xenU" > vif = [ ''mac=00:16:3e:00:01:01,bridge=br-inet,vifname=hades0'', > ''mac=00:16:3e:00:00:02,bridge=br-dmz,vifname=hades1'' ] > disk=[''phy:/dev/xen/hades-OS,hda1,w'',''phy:/dev/xen/hades-SWAP,hda2,w''] > root="/dev/hda1 ro" > on_crash="restart" > > /etc/xen/auto/agustina: > name="agustina" > memory=64 > kernel="/boot/vmlinuz-2.6.16-xenU" > vif = [ ''mac=00:16:3e:00:00:07,bridge=br-dmz,vifname=agustina0'' ] > disk=[''phy:/dev/xen/Agustina-OS,hda1,w'',''phy:/dev/xen/Agustina-SWAP,hda2,w''] > root="/dev/hda1 ro" > on_crash="restart" > > ------- > > So.... what is the problem? > well: > - routing is Ok > - ping works in all directions > - ssh from waste (lan machine) to Dom0 works > - ssh from Dom0 to waste works > - ssh from waste to hades works > - ssh from hades to waste works > - ssh from Dom0 to agustina works > - ssh from hades to agustina works > - ssh from agustina to Dom0 works > - ssh from agustina to hades works > > but: > - ssh from waste to agustina does not work > - ssh from agustina to waste does not work > > Here are is a tcpdump taken from agustina''s eth0: > > agustina:~# tcpdump -i eth0 -n host waste > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes > 19:45:15.242301 IP waste.4331 > agustina.22: S 30038281:30038281(0) > win 5840 <mss 1460,sackOK,timestamp 23876432 0,nop,wscale 2> > 19:45:15.251956 IP agustina.22 > waste.4331: S > 3550608405:3550608405(0) ack 30038282 win 5792 <mss > 1460,sackOK,timestamp 867394 23876432,nop, wscale 1> > 19:45:15.245850 IP waste.4331 > agustina.22: . ack 1 win 1460 > <nop,nop,timestamp 23876783 867394> > 19:45:15.255867 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 867394 23876783> > 19:45:15.468349 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 867417 23876783> > 19:45:15.888650 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 867459 23876783> > 19:45:16.728328 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 867543 23876783> > 19:45:18.408341 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 867711 23876783> > 19:45:21.768338 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 868047 23876783> > 19:45:28.491449 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 > <nop,nop,timestamp 868719 23876783> > > And that goes and goes until timeout. > > Those packets from ''agustina'' _are_ received by ''waste'' in the same way > (I verified that with tcpdump too), and then are dropped by ''waste'' > (netfilter conntrack say they are INVALID), so TCP socket is established, > but there is no communication. > > The problem I see there is the tcp window size of agustina''s reply, > which is bigger than waste first ACK packet, or I am wrong? > > The strange thing is that agustina does not have any strange > configuration, and if I connect it to ''br-inet'' bridge and change his > netmask and gateway, it work as expected. > > There is no firewall on Dom[0U]. > > Does any of you have any idea of what could be the problem? > > Regards, > Diego. > > PS1: sorry for the large mail. > > PS2: I have also tried packages from > http://packages.debianbase.de/sid/i386/xen3, > with xen-3.0.1 and kernel 2.6.12, and have the same results. >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nivedita Singhvi
2006-Apr-28 14:21 UTC
Re: [Xen-users] Solved: Re: Communication problem with virtual DMZ
Diego Alvarez wrote:> Once again I was mistaken, the real problema was an incorrect TCP > Checksum (discovered with tcpdump), so the solution was to use > "ethtool -K eth0 tx off" on agustina. > I learned this from the list archives.Folks, In recent days, quite a few fixes have gone in to xen-unstable and backported to 3.0-testing to fix the offload problems. In particular, if you were operating in a VLAN or IPSec environment, this is believed fixed. It would really help us if those of you who were having problems and needed to use ethtool to turn offload off, could try latest bits without doing that. If you are still experiencing problems and turning offload off still is a workaround, please let us know! thanks, Nivedita> On 4/27/06, Diego Alvarez <arcane.lord@gmail.com> wrote: > >>Hi all, >>I am running Xen 3.0.2-2 (taken from XenSource) with Linux kernel >>2.6.16 (taken from Debian Sid), I compiled Xen and 2 kernels >>(dom0 and domU). >> >>Here is the ascii-art of my setup: >> >> ------------ ------------- >> | LAN |------------------------| waste | 192.168.0.94/24 >> ------------ ------------- >> | >>····························· >>· | · >>· | Dom0 · >>· | · >>· --------- · ································ >>· | peth0 | · · · >>· --------- · · DomU hades · >>· | · · (Firewall) · >>· | · · · >>· ----------- ---------- · · -------- · >>· | br-inet |---| hades0 |============| eth0 | 192.168.0.34/24 · >>· ----------- | (vif) | · · -------- · >>· | ---------- · · · >>· | · · -------- · >>· ----------- · · | eth1 | 192.168.0.34/32 · >>· | vif0.0 | · · -------- · >>· ----------- · · || · >>· || · ········||······················ >>· || · || >>· || ·············||··········· >>· || || · >>· -------- || · >>· | eth0 | 192.168.0.22/24 ---------- · >>· -------- | hades1 | · >>· | (vif) | · >>· ---------- · >>· | · >>· ----------- ---------- · >>· | pdummy0 |---------| br-dmz | · >>· ----------- ---------- · >>· | · >>· ------------- · >>· | agustina0 | · >>· | (vif) | · >>· ------------- · >>· || · >>· || · >>·········································||··········· >> || >> ·-·-·-·-·-·-·-·-·-·||·-·-·-·-·-·-·-·-· >> · || · >> | ··········||········ | >> · · || · · >> | · -------- · | >> · · | eth0 | · · >> | · -------- · | >> · · 192.168.0.39/32 · · >> | · · | >> · · DomU agustina · · >> | · (DMZ Server) · | >> · · · · >> | ···················· | >> · · >> | Virtual DMZ | >> · · >> ·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·-·- >> >>-------- >> >>Network configuration for Dom0: >> >>auto eth0 >>iface eth0 inet static >> address 192.168.0.22 >> netmask 255.255.255.0 >> gateway 192.168.0.2 >> >>auto dummy0 >>iface dummy0 inet static >> address 10.1.1.1 >> netmask 255.255.255.255 >> up ifconfig dummy0 0.0.0.0 up >> >>-------- >> >>Network configuration for DomU agustina (DMZ Server): >> >>auto eth0 >>iface eth0 inet static >> address 192.168.0.39 >> netmask 255.255.255.255 >> up route add -host 192.168.0.34 dev eth0 >> up route add default gw 192.168.0.34 dev eth0 >> >>------- >> >>Network configuration for DomU hades (Firewall): >> >>auto eth0 >>iface eth0 inet static >> address 192.168.0.34 >> netmask 255.255.255.0 >> gateway 192.168.0.2 >> up arp -Ds 192.168.0.39 eth0 pub >> >>auto eth1 >>iface eth1 inet static >> address 192.168.0.34 >> netmask 255.255.255.255 >> up route add -host 192.168.0.39 dev eth1 >> >>It also have ip_forward activated by sysctl >> >>------ >> >>In dom0, I do the following things: >> >>In /etc/xen/xend-config.sxp I have: >> >>(network-script ''network-bridge bridge=br-inet'') >>(vif-script ''vif-bridge bridge=br-inet'') >> >> >>I also have a script which brings up br-dmz bridge on dummy0 >> >># brctl show: >>bridge name bridge id STP enabled interfaces >> >>br-dmz 8000.feffffffffff no agustina0 >> hades1 >> pdummy0 >> >>br-inet 8000.feffffffffff no hades0 >> peth0 >> vif0.0 >> >>Here is the configuration for hades and agustina: >> >>/etc/xen/auto/hades: >>name="hades" >>memory=128 >>kernel="/boot/vmlinuz-2.6.16-xenU" >>vif = [ ''mac=00:16:3e:00:01:01,bridge=br-inet,vifname=hades0'', >>''mac=00:16:3e:00:00:02,bridge=br-dmz,vifname=hades1'' ] >>disk=[''phy:/dev/xen/hades-OS,hda1,w'',''phy:/dev/xen/hades-SWAP,hda2,w''] >>root="/dev/hda1 ro" >>on_crash="restart" >> >>/etc/xen/auto/agustina: >>name="agustina" >>memory=64 >>kernel="/boot/vmlinuz-2.6.16-xenU" >>vif = [ ''mac=00:16:3e:00:00:07,bridge=br-dmz,vifname=agustina0'' ] >>disk=[''phy:/dev/xen/Agustina-OS,hda1,w'',''phy:/dev/xen/Agustina-SWAP,hda2,w''] >>root="/dev/hda1 ro" >>on_crash="restart" >> >>------- >> >>So.... what is the problem? >>well: >> - routing is Ok >> - ping works in all directions >> - ssh from waste (lan machine) to Dom0 works >> - ssh from Dom0 to waste works >> - ssh from waste to hades works >> - ssh from hades to waste works >> - ssh from Dom0 to agustina works >> - ssh from hades to agustina works >> - ssh from agustina to Dom0 works >> - ssh from agustina to hades works >> >>but: >> - ssh from waste to agustina does not work >> - ssh from agustina to waste does not work >> >>Here are is a tcpdump taken from agustina''s eth0: >> >>agustina:~# tcpdump -i eth0 -n host waste >>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >>listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes >>19:45:15.242301 IP waste.4331 > agustina.22: S 30038281:30038281(0) >>win 5840 <mss 1460,sackOK,timestamp 23876432 0,nop,wscale 2> >>19:45:15.251956 IP agustina.22 > waste.4331: S >>3550608405:3550608405(0) ack 30038282 win 5792 <mss >>1460,sackOK,timestamp 867394 23876432,nop, wscale 1> >>19:45:15.245850 IP waste.4331 > agustina.22: . ack 1 win 1460 >><nop,nop,timestamp 23876783 867394> >>19:45:15.255867 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 867394 23876783> >>19:45:15.468349 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 867417 23876783> >>19:45:15.888650 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 867459 23876783> >>19:45:16.728328 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 867543 23876783> >>19:45:18.408341 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 867711 23876783> >>19:45:21.768338 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 868047 23876783> >>19:45:28.491449 IP agustina.22 > waste.4331: P 1:42(41) ack 1 win 2896 >><nop,nop,timestamp 868719 23876783> >> >>And that goes and goes until timeout. >> >>Those packets from ''agustina'' _are_ received by ''waste'' in the same way >>(I verified that with tcpdump too), and then are dropped by ''waste'' >>(netfilter conntrack say they are INVALID), so TCP socket is established, >>but there is no communication. >> >>The problem I see there is the tcp window size of agustina''s reply, >>which is bigger than waste first ACK packet, or I am wrong? >> >>The strange thing is that agustina does not have any strange >>configuration, and if I connect it to ''br-inet'' bridge and change his >>netmask and gateway, it work as expected. >> >>There is no firewall on Dom[0U]. >> >>Does any of you have any idea of what could be the problem? >> >>Regards, >>Diego. >> >>PS1: sorry for the large mail. >> >>PS2: I have also tried packages from >>http://packages.debianbase.de/sid/i386/xen3, >> with xen-3.0.1 and kernel 2.6.12, and have the same results. >> > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga
2006-Apr-28 18:53 UTC
[Xen-users] still checksum errors with xen-3.0-testing.hg
Quoting Nivedita Singhvi <niv@us.ibm.com>:> It would really help us if those of you who were having > problems and needed to use ethtool to turn offload off, > could try latest bits without doing that. > > If you are still experiencing problems and turning offload > off still is a workaround, please let us know! >Without ethtool I still get checksum errors. I''m using changeset 9646:a8da66acde0c from xen-3.0-testing.hg While not the very latest changeset I do think (judging from the changelog) that this includes all checksum offload patches. My bridges are connected to dummy devices btw (real NICs are connected in domU''s), maybe this is unusual? Some systeminfo at the end of this mail Cheers, Peter root@bebop:~# xm info host : bebop release : 2.6.16.10-xen0 version : #1 Mon Apr 24 21:05:43 CEST 2006 machine : i686 nr_cpus : 1 nr_nodes : 1 sockets_per_node : 1 cores_per_socket : 1 threads_per_core : 1 cpu_mhz : 2261 hw_caps : afe9fbbf:00000000:00000000:00000040:00000180 total_memory : 1015 free_memory : 290 xen_major : 3 xen_minor : 0 xen_extra : .2-2 xen_caps : xen-3.0-x86_32 platform_params : virt_start=0xfc000000 xen_changeset : Sat Apr 22 11:42:34 2006 +0100 9646:a8da66acde0c cc_compiler : gcc version 4.0.3 (Ubuntu 4.0.3-1ubuntu4) cc_compile_by : peter cc_compile_domain : lan cc_compile_date : Mon Apr 24 20:54:03 CEST 2006 root@bebop:~# brctl show bridge name bridge id STP enabled interfaces xenbr1 8000.ca167f8e9b53 no dummy0 vif1.1 vif2.0 xenbr0 8000.feffffffffff no vif0.0 pdummy1 vif1.0 vif3.0 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nivedita Singhvi
2006-Apr-29 18:32 UTC
Re: [Xen-users] still checksum errors with xen-3.0-testing.hg
Peter Fokkinga wrote:> Quoting Nivedita Singhvi <niv@us.ibm.com>: > >> It would really help us if those of you who were having >> problems and needed to use ethtool to turn offload off, >> could try latest bits without doing that. >> >> If you are still experiencing problems and turning offload >> off still is a workaround, please let us know! >> > Without ethtool I still get checksum errors. I''m using changeset > 9646:a8da66acde0c from xen-3.0-testing.hg While not the very > latest changeset I do think (judging from the changelog) that > this includes all checksum offload patches.Thanks for the problem report, Peter. Yep, so you are using the recently added PCI passthrough stuff? We have not yet added any tests for that environment or looked into it yet. Would be good to add that case to an xm-test configuration. Will let you know once we get to that one. At the moment, our assumption has been that the physical NICs are in dom0. There is probably other stuff to shake out in driver domain support. thanks, Nivedita> My bridges are connected to dummy devices btw (real NICs are > connected in domU''s), maybe this is unusual? Some systeminfo > at the end of this mail > > Cheers, Peter > > > root@bebop:~# xm info > host : bebop > release : 2.6.16.10-xen0 > version : #1 Mon Apr 24 21:05:43 CEST 2006 > machine : i686 > nr_cpus : 1 > nr_nodes : 1 > sockets_per_node : 1 > cores_per_socket : 1 > threads_per_core : 1 > cpu_mhz : 2261 > hw_caps : afe9fbbf:00000000:00000000:00000040:00000180 > total_memory : 1015 > free_memory : 290 > xen_major : 3 > xen_minor : 0 > xen_extra : .2-2 > xen_caps : xen-3.0-x86_32 > platform_params : virt_start=0xfc000000 > xen_changeset : Sat Apr 22 11:42:34 2006 +0100 > 9646:a8da66acde0c > cc_compiler : gcc version 4.0.3 (Ubuntu 4.0.3-1ubuntu4) > cc_compile_by : peter > cc_compile_domain : lan > cc_compile_date : Mon Apr 24 20:54:03 CEST 2006 > > > root@bebop:~# brctl show > bridge name bridge id STP enabled interfaces > xenbr1 8000.ca167f8e9b53 no dummy0 > vif1.1 > vif2.0 > xenbr0 8000.feffffffffff no vif0.0 > pdummy1 > vif1.0 > vif3.0 > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Fokkinga
2006-Apr-30 19:01 UTC
Re: [Xen-users] still checksum errors with xen-3.0-testing.hg
Quoting Nivedita Singhvi <nsnix@comcast.net>:> Thanks for the problem report, Peter. Yep, so you > are using the recently added PCI passthrough stuff? >Yes, one domU acting as a firewall with a Intel eePro100 NIC, and one domU using the two Marvell Yukon2 NICs that are integrated on the mainboard. All using PCI passthrough. Works like a charm when I use ethtool.> We have not yet added any tests for that environment > or looked into it yet. Would be good to add that case > to an xm-test configuration. Will let you know once > we get to that one. >Thanks, looking forward to it. This system is still in its testing stages, so it''s no problem to install a new kernel (and rebooting the machine). Regards, Peter _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users