András Tarsoly
2008-Mar-18 17:24 UTC
Strange connection problem from a shorewalled / routed Xen DomU
Hi guys, I have an issue with my Shorewall setup, which I doesn''t seem to be able to resolve. I have a 3 box setup with the following parameters: Box 1: The router, running Shorewall 4.0.3. It has 3 802.1q interfaces: vlan100, vlan200, vlan300 on eth0 which is connected into a trunk port on my L2 switch. VLAN100 is the NET. VLAN200 is DMZ, VLAN300 and ETH3 is LOC. Box 1 also have 1 Xen DomU, routed to vlan300, interface name eth3. This is the VM which I have problems with. VLAN100 (EXT_IF) has the subnet 195.228.157.16/28 and ip 195.228.157.17. VLAN200 (DMZ_IF) has the subnet 10.20.0.0/24 and ip 10.20.0.1 VLAN300 (INT_IF) has the subnet 10.30.0.0/24 and ip 10.30.0.1 ETH0 is used only internal monitoring for the switch management port, has the ip 10.0.0.1. ETH3 (VM_2_IF) which is the Xen routed interface has the ip 10.30.0.1 The Xen DomU, call it DB-0, has an eth0 and ip 10.30.0.10. A split Bind9 running on it also, serving queries into INTERNAL and EXTERNAL views. Box 2: Connected into the 200 vlan, which is the DMZ. Has one interface, eth0. 2 Xen DomU-s running on this box, respectively 10.20.0.10 and 10.20.0.20, routed to 10.20.0.1 via 10.20.0.2 Box 3: Connected into the 300 clan, which is the LOC. This box is not used currently. I have general accept all to all from all policies and rules for testing purposes, logging everything. NAT: 195.228.157.18 vlan100 10.20.0.10 no no 195.228.157.19 eth3 10.30.0.10 no no MASQ: $EXT_IF $DMZ_IF 195.228.157.17 $EXT_IF $VM_2_IF 195.228.157.19 all My problem: DB-0 can ping everywhere properly, I get normal ICMP replies. It properly resolves everything due to the internal DNS server. However, I cannot establish connections from inside the VM: SSH, FTP, HTTP just hangs after Connection Established messages in verbose mode, then the connection just close after 1-2 minutes. The same stuff is working without a hitch on Box-2, I can connect everywhere properly from each of the 2 VMs running on it. Their configuration and the system is exactly the same, Gentoo Linuxes with 2.6.20-r6 kernels, one interface with an ip address routed through the vlan''s default gateway. My guess is somehow the Xen routing could be the problem when the DomU I''m trying to route is on the same host with the Shorewall setup, but I''m not smart enough to figure it on my own. I''ve attached the shorewall dump (after a reset and tried connections) to this message. Any ideas welcome. Thanks for your help in advance, András -- Some more information: TCPDUMP: After trying to ssh pixelszabaszat.hu from the DB-0 VM: PIX-APP-0 ~ # tcpdump -i eth3 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3, link-type EN10MB (Ethernet), capture size 96 bytes 19:07:33.172043 arp who-has 10.30.0.1 tell 10.30.0.10 19:07:33.172065 arp reply 10.30.0.1 is-at fe:ff:ff:ff:ff:ff (oui Unknown) 19:07:33.172102 IP 10.30.0.10.32784 > 10.30.0.1.domain: 46764+ AAAA? pixelszabaszat.hu. (35) 19:07:33.172440 IP 10.30.0.1.domain > 10.30.0.10.32784: 46764 0/1/0 (82) 19:07:33.172527 IP 10.30.0.10.32784 > 10.30.0.1.domain: 42920+ AAAA? pixelszabaszat.hu. (35) 19:07:33.172585 IP 10.30.0.1.domain > 10.30.0.10.32784: 42920 0/1/0 (82) 19:07:33.172640 IP 10.30.0.10.32784 > 10.30.0.1.domain: 7400+ A? pixelszabaszat.hu. (35) 19:07:33.172727 IP 10.30.0.1.domain > 10.30.0.10.32784: 7400 1/2/1 A pixelszabaszat.hu (110) 19:07:33.172833 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: S 157951987:157951987(0) win 5840 <mss 1460,sackOK,timestamp 182185 0,nop,wscale 6> 19:07:33.173112 IP pixelszabaszat.hu.ssh > 10.30.0.10.54999: S 1718828997:1718828997(0) ack 157951988 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 2068278201 182185,sackOK,eol> 19:07:33.173165 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: . ack 1 win 92 <nop,nop,timestamp 182185 2068278201> 19:07:33.193088 IP pixelszabaszat.hu.ssh > 10.30.0.10.54999: P 1:40(39) ack 1 win 33304 <nop,nop,timestamp 2068278221 182185> 19:07:33.193126 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: . ack 40 win 92 <nop,nop,timestamp 182187 2068278221> 19:07:33.193261 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182187 2068278221> 19:07:33.402034 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182208 2068278221> 19:07:33.822045 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182250 2068278221> 19:07:34.662033 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182334 2068278221> 19:07:36.342061 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182502 2068278221> 19:07:38.172104 arp who-has 10.30.0.10 tell 10.30.0.1 19:07:38.172142 arp reply 10.30.0.10 is-at 00:16:3e:32:37:29 (oui Unknown) 19:07:39.702114 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 182838 2068278221> 19:07:46.422246 IP 10.30.0.10.54999 > pixelszabaszat.hu.ssh: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 183510 2068278221> LOG: Mar 18 19:07:33 PIX-APP-0 Shorewall:loc2net:ACCEPT:IN=eth3 OUT=vlan100 SRC10.30.0.10 DST=80.249.168.21 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57434 DF PROTO=TCP SPT=54999 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Mar 18 19:07:34 PIX-APP-0 Shorewall:loc2net:ACCEPT:IN=eth3 OUT=vlan100 SRC10.30.0.10 DST=80.249.168.21 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57434 DF PROTO=TCP SPT=54999 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 -- András Tarsoly tarsolya@gmail.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Tom Eastep
2008-Mar-18 22:13 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
András Tarsoly wrote:> > TCPDUMP: > > After trying to ssh pixelszabaszat.hu <http://pixelszabaszat.hu> from > the DB-0 VM: > > PIX-APP-0 ~ # tcpdump -i eth3Please get another tcpdump, this time on vlan100; and please use the -n option. Thanks, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
András Tarsoly
2008-Mar-19 16:19 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
Hi Tom, On Tue, Mar 18, 2008 at 11:13 PM, Tom Eastep <teastep@shorewall.net> wrote:> Please get another tcpdump, this time on vlan100; and please use the -n > option.Here is the new tcpdump as you requested. I had to filter out the destination host, becasue there is a lot of traffic otherwise. Tell me if this is not enough. PIX-APP-0 ~ # tcpdump -i vlan100 -n port 22 | grep 80.249.168.21 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan100, link-type EN10MB (Ethernet), capture size 96 bytes 11:22:04.946716 IP 195.228.157.19.54144 > 80.249.168.21.22: S 1758632054:1758632054(0) win 5840 <mss 1460,sackOK,timestamp 6029274 0,nop,wscale 6> 11:22:04.946976 IP 80.249.168.21.22 > 195.228.157.19.54144: S 1498542778:1498542778(0) ack 1758632055 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 2126737310 6029274,sackOK,eol> 11:22:04.947031 IP 195.228.157.19.54144 > 80.249.168.21.22: . ack 1 win 92 <nop,nop,timestamp 6029274 2126737310> 11:22:04.961238 IP 80.249.168.21.22 > 195.228.157.19.54144: P 1:40(39) ack 1 win 33304 <nop,nop,timestamp 2126737324 6029274> 11:22:04.961288 IP 195.228.157.19.54144 > 80.249.168.21.22: . ack 40 win 92 <nop,nop,timestamp 6029275 2126737324> 11:22:04.961389 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029275 2126737324> 11:22:05.164057 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029296 2126737324> 11:22:05.584079 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029338 2126737324> 11:22:06.424099 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029422 2126737324> 11:22:08.104129 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029590 2126737324> 11:22:11.464242 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6029926 2126737324> 11:22:18.204283 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6030599 2126737324> 11:22:31.634460 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6031943 2126737324> 11:22:58.514850 IP 195.228.157.19.54144 > 80.249.168.21.22: P 1:21(20) ack 40 win 92 <nop,nop,timestamp 6034631 2126737324> 11:23:04.977583 IP 80.249.168.21.22 > 195.228.157.19.54144: F 40:40(0) ack 1 win 33304 <nop,nop,timestamp 2126797325 6029275> 11:23:04.978422 IP 195.228.157.19.54144 > 80.249.168.21.22: FP 21:813(792) ack 41 win 92 <nop,nop,timestamp 6035277 2126797325> Thank you, András -- András Tarsoly tarsolya@gmail.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Tom Eastep
2008-Mar-19 20:36 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
András Tarsoly wrote:> Hi Tom, > > On Tue, Mar 18, 2008 at 11:13 PM, Tom Eastep <teastep@shorewall.net > <mailto:teastep@shorewall.net>> wrote: > > > Please get another tcpdump, this time on vlan100; and please use the > -n option. > > > Here is the new tcpdump as you requested. I had to filter out the > destination host, becasue there is a lot of traffic otherwise. Tell me > if this is not enough.Please try this on the firewall: echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
András Tarsoly
2008-Mar-20 14:19 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
HI Tom, On Wed, Mar 19, 2008 at 9:36 PM, Tom Eastep <teastep@shorewall.net> wrote:> Please try this on the firewall: > > echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal >This been set in init already. I''ve double checked and this is active already. Regards, András -- András Tarsoly tarsolya@gmail.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Tom Eastep
2008-Mar-20 23:29 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
András Tarsoly wrote:> HI Tom, > > On Wed, Mar 19, 2008 at 9:36 PM, Tom Eastep <teastep@shorewall.net > <mailto:teastep@shorewall.net>> wrote: > > > Please try this on the firewall: > > echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal > > > This been set in init already. I''ve double checked and this is active > already.Have you looked at the outgoing packets on vlan100 using the -vv tcpdump option to be sure that the checksums are correct? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
András Tarsoly
2008-Mar-21 16:05 UTC
Re: Strange connection problem from a shorewalled / routed Xen DomU
Hi Tom, spot on. I''ve found this also now in the Strong Firewall in a Routed Dom0 and setting TX off on the DomU interface did the trick. Thanks for your help and patience. Have a nice day, my best regards: András On Fri, Mar 21, 2008 at 12:29 AM, Tom Eastep <teastep@shorewall.net> wrote:> András Tarsoly wrote: > > HI Tom, > > > > On Wed, Mar 19, 2008 at 9:36 PM, Tom Eastep <teastep@shorewall.net > > <mailto:teastep@shorewall.net>> wrote: > > > > > > Please try this on the firewall: > > > > echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal > > > > > > This been set in init already. I''ve double checked and this is active > > already. > > Have you looked at the outgoing packets on vlan100 using the -vv tcpdump > option to be sure that the checksums are correct? > > -Tom > -- > Tom Eastep \ Nothing is foolproof to a sufficiently talented fool > Shoreline, \ http://shorewall.net > Washington USA \ teastep@shorewall.net > PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >-- András Tarsoly tarsolya@gmail.com ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/