Hi! I''ve built a xen domain without bridging, and it seems that tcp is broken. ICMP works fine (I can ping either end), but TCP doesn''t ACK in the xen client. I found one mention this on the mailing list back in March, but it didn''t have an answer. I watched the communication in ethereal, and the client simply ignored the SYN,ACK packets from the host. I''m running: Linux host 2.6.11.10-xen0 #1 Sun May 22 11:38:50 BST 2005 i686 GNU/Linux I''m intentionally isolating my xen clients, but I need them to talk to the host. What sort of diagnostics can I provide to help track this down? Thanks! -- Kees Cook @outflux.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kees Cook wrote:> Hi! I''ve built a xen domain without bridging, and it seems that tcp is > broken. ICMP works fine (I can ping either end), but TCP doesn''t ACK in > the xen client. I found one mention this on the mailing list back in > March, but it didn''t have an answer. I watched the communication in > ethereal, and the client simply ignored the SYN,ACK packets from the > host.Can you tell where they were being dropped? Was netstat -s showing anything? Have you tried different TCP apps? ftp, hping, rsync, ssh? Are you running IPSec or some other filtering?> I''m running: > Linux host 2.6.11.10-xen0 #1 Sun May 22 11:38:50 BST 2005 i686 GNU/Linux > > I''m intentionally isolating my xen clients, but I need them to talk to > the host. What sort of diagnostics can I provide to help track this > down? > > Thanks! >This is usually caused by a misconfiguration of the firewall or having some needed port not open. Could you collect the ethereal or tcpdump trace of that happening, and also ifconfig, netstat -tan, netstat -s, sysctl -a, ip show rule, output? thanks, Nivedita _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jun 21, 2005 at 12:35:43PM -0700, Nivedita Singhvi wrote:> Can you tell where they were being dropped? Was netstat -s showing > anything? Have you tried different TCP apps? ftp, hping, rsync, ssh? > Are you running IPSec or some other filtering?I''m not running any ipsec, or iptables. I''m not sure what to look for out of netstat -s. (See attached.) I didn''t try other TCP apps, since it seemed to be down in the kernel (no ACK coming back).> This is usually caused by a misconfiguration of the firewall > or having some needed port not open.That''s what I had hoped, and I''ve trimmed it down to bare metal now. Originally I thought it was just related to bridging, and I totally removed that from the equation.> Could you collect the ethereal or tcpdump trace of that happening, and also > ifconfig, netstat -tan, netstat -s, sysctl -a, ip show rule, output?Sure! See attached. It''s got two directories, with the output from all mentioned commands (and route -n) along with xenU.pcap and xen0.pcap. To me, it just looks like xenU is just plain ignoring the SYN,ACK that comes in. :( -- Kees Cook @outflux.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kees Cook wrote:> On Tue, Jun 21, 2005 at 12:35:43PM -0700, Nivedita Singhvi wrote: > >>Can you tell where they were being dropped? Was netstat -s showing >>anything? Have you tried different TCP apps? ftp, hping, rsync, ssh? >>Are you running IPSec or some other filtering? > > > I''m not running any ipsec, or iptables. I''m not sure what to look for > out of netstat -s. (See attached.) I didn''t try other TCP apps, since > it seemed to be down in the kernel (no ACK coming back).Looks like you need to add a default route on domU. Could you fix that and try again? thanks, Nivedita>>This is usually caused by a misconfiguration of the firewall >>or having some needed port not open. > > > That''s what I had hoped, and I''ve trimmed it down to bare metal now. > Originally I thought it was just related to bridging, and I totally > removed that from the equation. > > >>Could you collect the ethereal or tcpdump trace of that happening, and also >>ifconfig, netstat -tan, netstat -s, sysctl -a, ip show rule, output? > > > Sure! See attached. It''s got two directories, with the output from all > mentioned commands (and route -n) along with xenU.pcap and xen0.pcap. > To me, it just looks like xenU is just plain ignoring the SYN,ACK that > comes in. :( > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jun 21, 2005 at 03:28:16PM -0700, Nivedita Singhvi wrote:> Looks like you need to add a default route on domU. > > Could you fix that and try again?Sure, added, but no change in behavior. (And shouldn''t: the packets get to and are responded to by the host, but when the client gets the SYN,ACK it just ignores it.) This is running the stable binrary release of the Xen website, if that helps. I noticed the xenU doesn''t have any kernel network modules built. Is it possible something is misconfigured in the build? Are other people able to do TCP between host/client in a routed config running the 2.0.6 stable release? -- Kees Cook @outflux.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kees Cook wrote:> On Tue, Jun 21, 2005 at 03:28:16PM -0700, Nivedita Singhvi wrote: > >>Looks like you need to add a default route on domU. >> >>Could you fix that and try again? > > > Sure, added, but no change in behavior. (And shouldn''t: the packets get > to and are responded to by the host, but when the client gets the > SYN,ACK it just ignores it.)The next issue is that your tcp_mem seems to be on the disfunctionally low. Your netstat -s output seems truncated on domU (only had the first 2 Linux MIB counters). You might have otherwise been able to confirm whether you could even open a socket or were hitting memory pressure. Mainline used to disallow tcp allocations if total usage in the system was at sysctl.tcp_mem[2], and entered memory pressure at sysctl.tcp_mem[1]. net.ipv4.tcp_mem = 3072 4096 6144 How did this get so low? Bump that up by a factor of 100 each (assuming your domain has enough memory allocated, forgot to ask). Admittedly, that was older mainline behaviour, might have changed recently (post 2.6.11) and not sure exactly what 2.0.6 is running. this wee> This is running the stable binrary release of the Xen website, if that > helps. I noticed the xenU doesn''t have any kernel network modules > built. Is it possible something is misconfigured in the build? Are > other people able to do TCP between host/client in a routed config > running the 2.0.6 stable release? >It used to work fine on unstable about 2 months back. Haven''t tested the testing tree myself, unfortunately. thanks, Nivedita _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jun 21, 2005 at 04:37:04PM -0700, Nivedita Singhvi wrote:> The next issue is that your tcp_mem seems to be > on the disfunctionally low.Hm. It''s the same on xenU and xen0 (but way lower than another machine I looked at). Both xenU and xen0 have 96M allocated for main memory. (And, xen0 has no problems with TCP connections to other hosts.)> Your netstat -s output seems truncated on domU (only had the first 2 > Linux MIB counters). You might have otherwise been able to confirm > whether you could even open a socket or were hitting memory pressure.Hm, that is interesting. It''s actually not truncated; that''s how it looks. If I do a "telnet localhost 22" (which connects fine) the "TcpExt" section changes to: TcpExt: 1 packets directly queued to recvmsg prequeue. 1 of bytes directly received from prequeue 1 packet headers predicted 2 acknowledgments not containing data received 3 predicted acknowledgments 0 TCP data loss events> How did this get so low? Bump that up by a factor of 100 > each (assuming your domain has enough memory allocated, forgot > to ask).I bumped it; no luck.> It used to work fine on unstable about 2 months back. Haven''t tested > the testing tree myself, unfortunately.How strange. I guess I need to build my very own to see if recent stuff fixes it. Pretty strange. Thanks for looking at this. I''m quite baffled. -- Kees Cook @outflux.net _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Kees Cook wrote:> On Tue, Jun 21, 2005 at 04:37:04PM -0700, Nivedita Singhvi wrote: >> The next issue is that your tcp_mem seems to be >> on the disfunctionally low. > > Hm. It''s the same on xenU and xen0 (but way lower than another machine > I looked at). Both xenU and xen0 have 96M allocated for main memory. > (And, xen0 has no problems with TCP connections to other hosts.) > >> Your netstat -s output seems truncated on domU (only had the first 2 >> Linux MIB counters). You might have otherwise been able to confirm >> whether you could even open a socket or were hitting memory pressure. > > Hm, that is interesting. It''s actually not truncated; that''s how it > looks. If I do a "telnet localhost 22" (which connects fine) the > "TcpExt" section changes to: > > TcpExt: > 1 packets directly queued to recvmsg prequeue. > 1 of bytes directly received from prequeue > 1 packet headers predicted > 2 acknowledgments not containing data received > 3 predicted acknowledgments > 0 TCP data loss events > >> How did this get so low? Bump that up by a factor of 100 >> each (assuming your domain has enough memory allocated, forgot >> to ask). > > I bumped it; no luck. > >> It used to work fine on unstable about 2 months back. Haven''t tested >> the testing tree myself, unfortunately. > > How strange. I guess I need to build my very own to see if recent stuff > fixes it. Pretty strange. Thanks for looking at this. I''m quite > baffled.Sorry to resurrect an old thread (http://thread.gmane.org/gmane.comp.emulators.xen.user/1548 for the full discussion), but did you ever get any result with this, Kees? I believe i''m having exactly the same problem as you were on my current setup. (http://permalink.gmane.org/gmane.comp.emulators.xen.user/18651 for full details) Any pointers would be greatly appreciated. -- Paul <http://paulgear.webhop.net> -- OpenOffice.org is the no-strings-attached office package. If you enjoy the test drive, keep the car! http://why.openoffice.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users