Hello all, I have a problem with netfilter and connection tracking on Xen. My config is: xen-3.0.3 linux-2.6.18 Debian Etch AMD64 2x Xeon with Hyper-Threading enabled Network configuration in dom0 is like this: eth0, eth0:1, eth0:2,... (public IPs) xenbr0 (private IPs)=vif1.x, vif2.x, vif3.x,... I am not using netloop (vif0.x and veth0). I DNAT selected IPs/ports from public interface to different domU hosts (one is webserver, other is mailserver, jabber server, FTP server, etc). Connections from domU to internet a SNATed to one of public IPs. One problem is that ip_nat_ftp does not work. When someone connects with passive FTP, and tries to open data connection, it connects to private address. It seems like ip_nat_ftp is not working at all. (Active ftp is OK). I have used Xen 2.0.4 with kernel 2.6.10 (i386) and ip_nat_ftp worked fine. Another problem I noticed is that connection tracking marks a lot of packets as INVALID. (iptables -A INPUT -m state --state INVALID -j DROP) These packets are often part of ESTABLISHED connections to servers in domU, and somehow they are not DNATed and intead of getting into FORWARD chain, they end up in INPUT. So instead of routing them to proper domU, they hit dom0. I looks like the same problem I had on xen 2.0.4 with kernel 2.6.10 which involved tcp window tracking and I got rid of it by setting sysctl variables: net/ipv4/netfilter/ip_conntrack_tcp_be_liberal=1 net/ipv4/netfilter/ip_conntrack_log_invalid=1 But in xen 3.0.3 with kernel 2.6.18 it does nothing. No logging, and still a lot of INVALID packets. I spent whole day googling, and found only some loosely related problems and no solution proposed for others worked for me. Does anyone know what can be wrong with netfilter / conntrack? Moreover I found some vague note about possible deadlock if I use bridging without netloop. Can someone shed more light on this? Thanks for all help Regards Vladislav Kurz P.S. Thanks to xen developers for the good work. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello all, I have a problem with netfilter and connection tracking on Xen. My config is: xen-3.0.3 linux-2.6.18 Debian Etch AMD64 2x Xeon with Hyper-Threading enabled Network configuration in dom0 is like this: eth0, eth0:1, eth0:2,... (public IPs) xenbr0 (private IPs)=vif1.x, vif2.x, vif3.x,... I am not using netloop (vif0.x and veth0). I DNAT selected IPs/ports from public interface to different domU hosts (one is webserver, other is mailserver, jabber server, FTP server, etc). Connections from domU to internet a SNATed to one of public IPs. One problem is that ip_nat_ftp does not work. When someone connects with passive FTP, and tries to open data connection, it connects to private address. It seems like ip_nat_ftp is not working at all. (Active ftp is OK). I have used Xen 2.0.4 with kernel 2.6.10 (i386) and ip_nat_ftp worked fine. Another problem I noticed is that connection tracking marks a lot of packets as INVALID. (iptables -A INPUT -m state --state INVALID -j DROP) These packets are often part of ESTABLISHED connections to servers in domU, and somehow they are not DNATed and intead of getting into FORWARD chain, they end up in INPUT. So instead of routing them to proper domU, they hit dom0. I looks like the same problem I had on xen 2.0.4 with kernel 2.6.10 which involved tcp window tracking and I got rid of it by setting sysctl variables: net/ipv4/netfilter/ip_conntrack_tcp_be_liberal=1 net/ipv4/netfilter/ip_conntrack_log_invalid=1 But in xen 3.0.3 with kernel 2.6.18 it does nothing. No logging, and still a lot of INVALID packets. I spent whole day googling, and found only some loosely related problems and no solution proposed for others worked for me. Does anyone know what can be wrong with netfilter / conntrack? Moreover I found some vague note about possible deadlock if I use bridging without netloop. Can someone shed more light on this? Thanks for all help Regards Vladislav Kurz P.S. Thanks to xen developers for the good work. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexander Wilms
2007-May-30 06:34 UTC
Re: [Xen-users] netfilter, conntrack, ip_nat_ftp problem
Hi Vladislav, this all sounds familiar to me. Both problems seem to be related to the TCP/UDP Checksum problem. If you would look with wireshark into your packets you would see a lot of wrong checksums. And this explains both: Because of this the FTP nat helper doesn''t rewrite the re-transmitted packets anymore and also confuses the rest of the connection tracking. Solution is quite simple. Switch of tx checksumming of your nic(s). E.g. "ethtool -K eth0 tx off" You have to find out which of your nics need it. In my setup I had to switch it off in dom0 and domU on all physical nics. HTH, Alex On Montag 28 Mai 2007, Vladislav Kurz wrote:> Hello all, > > I have a problem with netfilter and connection tracking on Xen. > > My config is: > xen-3.0.3 > linux-2.6.18 > Debian Etch AMD64 > 2x Xeon with Hyper-Threading enabled > > Network configuration in dom0 is like this: > > eth0, eth0:1, eth0:2,... (public IPs) > xenbr0 (private IPs)=vif1.x, vif2.x, vif3.x,... > I am not using netloop (vif0.x and veth0). > > I DNAT selected IPs/ports from public interface to different domU hosts > (one is webserver, other is mailserver, jabber server, FTP server, etc). > Connections from domU to internet a SNATed to one of public IPs. > > One problem is that ip_nat_ftp does not work. When someone connects with > passive FTP, and tries to open data connection, it connects to private > address. It seems like ip_nat_ftp is not working at all. (Active ftp is > OK). > > I have used Xen 2.0.4 with kernel 2.6.10 (i386) and ip_nat_ftp worked fine. > > > Another problem I noticed is that connection tracking marks a lot of > packets as INVALID. (iptables -A INPUT -m state --state INVALID -j DROP) > These packets are often part of ESTABLISHED connections to servers in domU, > and somehow they are not DNATed and intead of getting into FORWARD chain, > they end up in INPUT. So instead of routing them to proper domU, they hit > dom0. > > I looks like the same problem I had on xen 2.0.4 with kernel 2.6.10 which > involved tcp window tracking and I got rid of it by setting sysctl > variables: net/ipv4/netfilter/ip_conntrack_tcp_be_liberal=1 > net/ipv4/netfilter/ip_conntrack_log_invalid=1 > > But in xen 3.0.3 with kernel 2.6.18 it does nothing. No logging, and still > a lot of INVALID packets. > > I spent whole day googling, and found only some loosely related problems > and no solution proposed for others worked for me. Does anyone know what > can be wrong with netfilter / conntrack? > > Moreover I found some vague note about possible deadlock if I use bridging > without netloop. Can someone shed more light on this? > > Thanks for all help > Regards > Vladislav Kurz > > P.S. Thanks to xen developers for the good work. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- =======================================::: NEUE ANSCHRIFT AB 01. JUNI 2007 ::: ::: Güterstr. 20 | D-42117 Wuppertal ::: ======================================= -- Mit freundlichen Grüssen Alexander Wilms \\\______________________________________________________ ...prosem ...Dipl.-Ing. Christian Boss ...Vohwinkeler Str.101 ...D-42329 Wuppertal ...fon: +49.202.737939_77 ...fax: +49.202.737939_80 ...mailto:a.wilms@prosem.net ...http://www.prosem.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Vladislav Kurz
2007-May-31 16:30 UTC
Re: [Xen-users] netfilter, conntrack, ip_nat_ftp problem
On Wednesday 30 May 2007 08:34, Alexander Wilms wrote:> Hi Vladislav, > > this all sounds familiar to me. Both problems seem to be related to the > TCP/UDP Checksum problem. If you would look with wireshark into your > packets you would see a lot of wrong checksums. And this explains both: > Because of this the FTP nat helper doesn''t rewrite the re-transmitted > packets anymore and also confuses the rest of the connection tracking. > > > Solution is quite simple. Switch of tx checksumming of your nic(s). E.g. > "ethtool -K eth0 tx off" > You have to find out which of your nics need it. In my setup I had to > switch it off in dom0 and domU on all physical nics. > > HTH, > AlexThanks a lot Alex, I switched off checksum offloading on domU and FTP NAT helper started to work. I still get some INVALID packets with FIN & RST flag set, and some bad tcp checksum in dom0 - domU traffic, so I will monitor it and perhaps switch off checksum on the real eth0 and xen-br0 (or the vifX) in dom0. Anyway I think this must have affected quite a lot of xen users. TCP checksum offloading must break any statefull firewall in dom0, or do I miss something? Why there is no note about this in docs? Or is our configuration so unusual? (dom0 as a firewall in front of domU guests) Thanks Vladislav Kurz> On Montag 28 Mai 2007, Vladislav Kurz wrote: > > Hello all, > > > > I have a problem with netfilter and connection tracking on Xen. > > > > My config is: > > xen-3.0.3 > > linux-2.6.18 > > Debian Etch AMD64 > > 2x Xeon with Hyper-Threading enabled > > > > Network configuration in dom0 is like this: > > > > eth0, eth0:1, eth0:2,... (public IPs) > > xenbr0 (private IPs)=vif1.x, vif2.x, vif3.x,... > > I am not using netloop (vif0.x and veth0). > > > > I DNAT selected IPs/ports from public interface to different domU hosts > > (one is webserver, other is mailserver, jabber server, FTP server, etc). > > Connections from domU to internet a SNATed to one of public IPs. > > > > One problem is that ip_nat_ftp does not work. When someone connects with > > passive FTP, and tries to open data connection, it connects to private > > address. It seems like ip_nat_ftp is not working at all. (Active ftp is > > OK). > > > > I have used Xen 2.0.4 with kernel 2.6.10 (i386) and ip_nat_ftp worked > > fine. > > > > > > Another problem I noticed is that connection tracking marks a lot of > > packets as INVALID. (iptables -A INPUT -m state --state INVALID -j DROP) > > These packets are often part of ESTABLISHED connections to servers in > > domU, and somehow they are not DNATed and intead of getting into FORWARD > > chain, they end up in INPUT. So instead of routing them to proper domU, > > they hit dom0. > > > > I looks like the same problem I had on xen 2.0.4 with kernel 2.6.10 which > > involved tcp window tracking and I got rid of it by setting sysctl > > variables: net/ipv4/netfilter/ip_conntrack_tcp_be_liberal=1 > > net/ipv4/netfilter/ip_conntrack_log_invalid=1 > > > > But in xen 3.0.3 with kernel 2.6.18 it does nothing. No logging, and > > still a lot of INVALID packets. > > > > I spent whole day googling, and found only some loosely related problems > > and no solution proposed for others worked for me. Does anyone know what > > can be wrong with netfilter / conntrack? > > > > Moreover I found some vague note about possible deadlock if I use > > bridging without netloop. Can someone shed more light on this? > > > > Thanks for all help > > Regards > > Vladislav Kurz > > > > P.S. Thanks to xen developers for the good work. > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > -- > =======================================> > ::: NEUE ANSCHRIFT AB 01. JUNI 2007 ::: > ::: Güterstr. 20 | D-42117 Wuppertal ::: > > =======================================_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alexander Wilms
2007-May-31 16:56 UTC
Re: [Xen-users] netfilter, conntrack, ip_nat_ftp problem
On Donnerstag 31 Mai 2007, Vladislav Kurz wrote:> On Wednesday 30 May 2007 08:34, Alexander Wilms wrote: > > Hi Vladislav, > > > > this all sounds familiar to me. Both problems seem to be related to the > > TCP/UDP Checksum problem. If you would look with wireshark into your > > packets you would see a lot of wrong checksums. And this explains both: > > Because of this the FTP nat helper doesn''t rewrite the re-transmitted > > packets anymore and also confuses the rest of the connection tracking. > > > > > > Solution is quite simple. Switch of tx checksumming of your nic(s). E.g. > > "ethtool -K eth0 tx off" > > You have to find out which of your nics need it. In my setup I had to > > switch it off in dom0 and domU on all physical nics. > > > > HTH, > > Alex > > Thanks a lot Alex, > > I switched off checksum offloading on domU and FTP NAT helper started to > work. I still get some INVALID packets with FIN & RST flag set, and some > bad tcp checksum in dom0 - domU traffic, so I will monitor it and perhaps > switch off checksum on the real eth0 and xen-br0 (or the vifX) in dom0. > > Anyway I think this must have affected quite a lot of xen users. TCP > checksum offloading must break any statefull firewall in dom0, or do I miss > something? Why there is no note about this in docs? Or is our configuration > so unusual? (dom0 as a firewall in front of domU guests) > > Thanks > Vladislav KurzHi Vladislav, no, not so unusual. So also I don''t understand why not more people reporting this issue on the xen lists. (There was only one thread that I remember that was related to that issue. It was like: Everything works, but DNS resolution fails. This was also related to offloading features of the nic.) But in shorewall list we discussed it and also in a xen book I read it was topic. So the best hint so far you can find is maybe in the shorewall documentation written by Tom Eastep. http://www.shorewall.net/XenMyWay.html Btw. my way of xen is a bit different. I''m running my firewall in a domU (with PCI passthrough''ed nics) and (of course) have still same effects. HTH, Alex _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Vladislav Kurz
2007-Jun-04 12:13 UTC
Re: [Xen-users] netfilter, conntrack, ip_nat_ftp problem
On Thursday 31 May 2007 18:56, Alexander Wilms wrote:> On Donnerstag 31 Mai 2007, Vladislav Kurz wrote: > > On Wednesday 30 May 2007 08:34, Alexander Wilms wrote: > > > Hi Vladislav, > > > > > > this all sounds familiar to me. Both problems seem to be related to the > > > TCP/UDP Checksum problem. If you would look with wireshark into your > > > packets you would see a lot of wrong checksums. And this explains both: > > > Because of this the FTP nat helper doesn''t rewrite the re-transmitted > > > packets anymore and also confuses the rest of the connection tracking. > > > > > > > > > Solution is quite simple. Switch of tx checksumming of your nic(s). > > > E.g. "ethtool -K eth0 tx off" > > > You have to find out which of your nics need it. In my setup I had to > > > switch it off in dom0 and domU on all physical nics. > > > > > > HTH, > > > Alex > > > > Thanks a lot Alex, > > > > I switched off checksum offloading on domU and FTP NAT helper started to > > work. I still get some INVALID packets with FIN & RST flag set, and some > > bad tcp checksum in dom0 - domU traffic, so I will monitor it and perhaps > > switch off checksum on the real eth0 and xen-br0 (or the vifX) in dom0.I have switched off all checksum offloading (tx and rx), on all interfaces (physical, virtual, even bridge) and still I have a lot of invalid packets. NAT FPT helper works, but some packets miss DNAT rules and get logged as INVALID in INPUT (instead of being forwarded to domU host). Most of them are packets with flag FIN set. They correlate with records in apache access log, and often appear multiple times as the other party retransmits them. Some packets in the oposite direction (domU -> internet) are marked INVALID as well, mostly ACK+FIN and ACK+SYN, I''m not sure if they pass trhough SNAT, or just get into the wild with private source IP. Sniffing for bad checksum (tshark -i eth0 -R "tcp.checksum_bad == 1") does not yield any results, so this might be completely different issue. Has anyone seen this behavior and found a solution for it.> But in shorewall list we discussed it and also in a xen book I read it was > topic.Could you specify which book? Is it available on-line?> So the best hint so far you can find is maybe in the shorewall > documentation written by Tom Eastep. > http://www.shorewall.net/XenMyWay.htmlThanks, but this does not help any more than you already did :-)> Btw. my way of xen is a bit different. I''m running my firewall in a domU > (with PCI passthrough''ed nics) and (of course) have still same effects. > > HTH, > Alex > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- S pozdravem Vladislav Kurz === WebStep, s.r.o. (Ltd.) ========= a step to the Web ==address: Mezirka 1, 602 00 Brno, CZ, tel: +420 548 214 711 === www.webstep.net ======= vladislav.kurz@webstep.net == _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users