Vikas
2009-Jan-18 18:49 UTC
[Xen-users] Dom0 to DomU and reverse works but Outside world to DomU only ping works
Summary problem description: What works ? A. ping, ssh, http access - works from Dom0 to DomU and also from DomU to Dom0 B. ping, ssh, http access - works from outside world to Dom0 and from Dom0 to outside world C. ping from outside world to DomU What does not work ? A. ssh and http access from outside world to DomU and also from DomU to outside world Diagnostics:>From DomU I gave the command: > lynx 74.125.19.147and I got the error message: Making HTTP connection to 74.125.19.147 Sending HTTP request. HTTP request sent; waiting for response. Alert!: Unexpected network read error; connection aborted. Can''t Access `http://74.125.19.147/'' Alert!: Unable to access document. lynx: Can''t access startfile In Dom0 I had tcpdump running and it gave me the following output: [root@ps1 ~]# tcpdump -vvvv host 74.125.19.147 tcpdump: WARNING: peth0: no IPv4 address assigned tcpdump: listening on peth0, link-type EN10MB (Ethernet), capture size 96 bytes 06:43:55.941963 IP (tos 0x0, ttl 64, id 20913, offset 0, flags [DF], proto: TCP (6), length: 60) 72.52.93.9.51415 > cf-in-f147.google.com.http: S, cksum 0x249c (correct), 2918592937:2918592937(0) win 5840 <mss 1460,sackOK,timestamp 6774808 0,nop,wscale 7> 06:43:55.944002 IP (tos 0x0, ttl 59, id 61335, offset 0, flags [none], proto: TCP (6), length: 60) cf-in-f147.google.com.http > 72.52.93.9.51415: S, cksum 0x81ea (correct), 2521226213:2521226213(0) ack 2918592938 win 5672 <mss 1430,sackOK,timestamp 2113385283 6774808,nop,wscale 6> 06:43:55.944032 IP (tos 0x0, ttl 64, id 20914, offset 0, flags [DF], proto: TCP (6), length: 52) 72.52.93.9.51415 > cf-in-f147.google.com.http: ., cksum 0xc691 (correct), 1:1(0) ack 1 win 46 <nop,nop,timestamp 6774808 2113385283> 06:43:55.945897 IP (tos 0x0, ttl 64, id 20915, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6774809 2113385283> 06:43:56.149909 IP (tos 0x0, ttl 64, id 20916, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6774860 2113385283> 06:43:56.557931 IP (tos 0x0, ttl 64, id 20917, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6774962 2113385283> 06:43:57.373983 IP (tos 0x0, ttl 64, id 20918, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6775166 2113385283> 06:43:59.006087 IP (tos 0x0, ttl 64, id 20919, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6775574 2113385283> 06:44:02.270294 IP (tos 0x0, ttl 64, id 20920, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 <nop,nop,timestamp 6776390 2113385283> 06:44:05.951449 IP (tos 0x0, ttl 59, id 61336, offset 0, flags [none], proto: TCP (6), length: 52) cf-in-f147.google.com.http > 72.52.93.9.51415: F, cksum 0x9f4e (correct), 1:1(0) ack 1 win 89 <nop,nop,timestamp 2113395290 6774808> 06:44:05.954517 IP (tos 0x0, ttl 64, id 20921, offset 0, flags [DF], proto: TCP (6), length: 52) 72.52.93.9.51415 > cf-in-f147.google.com.http: ., cksum 0x9486 (correct), 301:301(0) ack 2 win 46 <nop,nop,timestamp 6777311 2113395290> 06:44:08.798696 IP (tos 0x0, ttl 64, id 20922, offset 0, flags [DF], proto: TCP (6), length: 352) 72.52.93.9.51415 > cf-in-f147.google.com.http: P 1:301(300) ack 2 win 46 <nop,nop,timestamp 6778022 2113395290> 06:44:08.954717 IP (tos 0x0, ttl 64, id 20923, offset 0, flags [DF], proto: TCP (6), length: 52) 72.52.93.9.51415 > cf-in-f147.google.com.http: F, cksum 0x9197 (correct), 301:301(0) ack 2 win 46 <nop,nop,timestamp 6778061 2113395290> 06:44:08.956809 IP (tos 0x0, ttl 59, id 61337, offset 0, flags [none], proto: TCP (6), length: 40) cf-in-f147.google.com.http > 72.52.93.9.51415: R, cksum 0x713d (correct), 2521226215:2521226215(0) win 0 14 packets captured 30 packets received by filter 0 packets dropped by kernel Software environment: Name : xen x86_64 Version: 3.0.3 Release: 64.el5_2.9 Hardware environment: Supermicro motherboard-X7DWN+ Things I have tried: 1. Shutting down the IPtables in Dom0 2. Reinstalling CentOS. How should I debug this problem further ? Thanks for your time, -- sysadmin http://www.debtconsolidationcare.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Vikas
2009-Jan-19 23:55 UTC
[Xen-users] Re: Dom0 to DomU and reverse works but Outside world to DomU only ping works
For others who might get stuck in the similar situation, the solution was to stop tx checsum in the DomU $ethtool -K eth0 tx off I am wondering why do I need to do this ? sysadmin http://www.debtconsolidationcare.com On Sun, Jan 18, 2009 at 12:49 PM, Vikas <kedia.vikas@gmail.com> wrote:> Summary problem description: > What works ? > A. ping, ssh, http access - works from Dom0 to DomU and also from DomU to Dom0 > B. ping, ssh, http access - works from outside world to Dom0 and from > Dom0 to outside world > C. ping from outside world to DomU > > What does not work ? > A. ssh and http access from outside world to DomU and also from DomU > to outside world > > Diagnostics: > From DomU I gave the command: >> lynx 74.125.19.147 > and I got the error message: > Making HTTP connection to 74.125.19.147 > Sending HTTP request. > HTTP request sent; waiting for response. > Alert!: Unexpected network read error; connection aborted. > Can''t Access `http://74.125.19.147/'' > Alert!: Unable to access document. > lynx: Can''t access startfile > > > In Dom0 I had tcpdump running and it gave me the following output: > [root@ps1 ~]# tcpdump -vvvv host 74.125.19.147 > tcpdump: WARNING: peth0: no IPv4 address assigned > tcpdump: listening on peth0, link-type EN10MB (Ethernet), capture size 96 bytes > 06:43:55.941963 IP (tos 0x0, ttl 64, id 20913, offset 0, flags [DF], > proto: TCP (6), length: 60) 72.52.93.9.51415 > > cf-in-f147.google.com.http: S, cksum 0x249c (correct), > 2918592937:2918592937(0) win 5840 <mss 1460,sackOK,timestamp 6774808 > 0,nop,wscale 7> > 06:43:55.944002 IP (tos 0x0, ttl 59, id 61335, offset 0, flags > [none], proto: TCP (6), length: 60) cf-in-f147.google.com.http > > 72.52.93.9.51415: S, cksum 0x81ea (correct), 2521226213:2521226213(0) > ack 2918592938 win 5672 <mss 1430,sackOK,timestamp 2113385283 > 6774808,nop,wscale 6> > 06:43:55.944032 IP (tos 0x0, ttl 64, id 20914, offset 0, flags [DF], > proto: TCP (6), length: 52) 72.52.93.9.51415 > > cf-in-f147.google.com.http: ., cksum 0xc691 (correct), 1:1(0) ack 1 > win 46 <nop,nop,timestamp 6774808 2113385283> > 06:43:55.945897 IP (tos 0x0, ttl 64, id 20915, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6774809 2113385283> > 06:43:56.149909 IP (tos 0x0, ttl 64, id 20916, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6774860 2113385283> > 06:43:56.557931 IP (tos 0x0, ttl 64, id 20917, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6774962 2113385283> > 06:43:57.373983 IP (tos 0x0, ttl 64, id 20918, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6775166 2113385283> > 06:43:59.006087 IP (tos 0x0, ttl 64, id 20919, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6775574 2113385283> > 06:44:02.270294 IP (tos 0x0, ttl 64, id 20920, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 1 win 46 > <nop,nop,timestamp 6776390 2113385283> > 06:44:05.951449 IP (tos 0x0, ttl 59, id 61336, offset 0, flags > [none], proto: TCP (6), length: 52) cf-in-f147.google.com.http > > 72.52.93.9.51415: F, cksum 0x9f4e (correct), 1:1(0) ack 1 win 89 > <nop,nop,timestamp 2113395290 6774808> > 06:44:05.954517 IP (tos 0x0, ttl 64, id 20921, offset 0, flags [DF], > proto: TCP (6), length: 52) 72.52.93.9.51415 > > cf-in-f147.google.com.http: ., cksum 0x9486 (correct), 301:301(0) ack > 2 win 46 <nop,nop,timestamp 6777311 2113395290> > 06:44:08.798696 IP (tos 0x0, ttl 64, id 20922, offset 0, flags [DF], > proto: TCP (6), length: 352) 72.52.93.9.51415 > > cf-in-f147.google.com.http: P 1:301(300) ack 2 win 46 > <nop,nop,timestamp 6778022 2113395290> > 06:44:08.954717 IP (tos 0x0, ttl 64, id 20923, offset 0, flags [DF], > proto: TCP (6), length: 52) 72.52.93.9.51415 > > cf-in-f147.google.com.http: F, cksum 0x9197 (correct), 301:301(0) ack > 2 win 46 <nop,nop,timestamp 6778061 2113395290> > 06:44:08.956809 IP (tos 0x0, ttl 59, id 61337, offset 0, flags > [none], proto: TCP (6), length: 40) cf-in-f147.google.com.http > > 72.52.93.9.51415: R, cksum 0x713d (correct), 2521226215:2521226215(0) > win 0 > > 14 packets captured > 30 packets received by filter > 0 packets dropped by kernel > > Software environment: > Name : xen x86_64 > Version: 3.0.3 > Release: 64.el5_2.9 > > Hardware environment: > Supermicro motherboard-X7DWN+ > > Things I have tried: > 1. Shutting down the IPtables in Dom0 > 2. Reinstalling CentOS. > > How should I debug this problem further ? > > > Thanks for your time, > -- > sysadmin > http://www.debtconsolidationcare.com >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
James Harper
2009-Jan-20 00:24 UTC
RE: [Xen-users] Re: Dom0 to DomU and reverse works but Outside world toDomU only ping works
> > For others who might get stuck in the similar situation, the solution > was to stop tx checsum in the DomU > > $ethtool -K eth0 tx off > > I am wondering why do I need to do this ? >This is a common problem, and appears to be related to bugs in one or more of: . Xen frontend network driver (unlikely) . Xen backend network driver . Linux bridge code . Linux routing code . Linux physical NIC adapter I suspect it is the last one, but it''s complicated enough that the bug could lie elsewhere but only show up on certain NIC''s. You can be thankful that your problem was consistent. I have a setup where the packets from DomU are routed in Dom0 to one of 6 GRE tunnels, and occasionally for 30 minutes or so TCP packets stopped working. Took me ages to figure out that turning off tx checksum offload in the DomU would resolve the problem. It was made harder because by the time I could connect to the client to investigate, it had started working again so all I could do was to set up tcpdumps of some fairly high volume links and wade through those based on the times that the problem was occurring... James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Vikas
2009-Jan-20 00:26 UTC
Re: [Xen-users] Re: Dom0 to DomU and reverse works but Outside world toDomU only ping works
Is it ok to run a production machine with tx checsum stopped in the DomU ? Thanks, -- sysadmin http://www.debtconsolidationcare.com On Mon, Jan 19, 2009 at 6:24 PM, James Harper <james.harper@bendigoit.com.au> wrote:>> >> For others who might get stuck in the similar situation, the solution >> was to stop tx checsum in the DomU >> >> $ethtool -K eth0 tx off >> >> I am wondering why do I need to do this ? >> > > This is a common problem, and appears to be related to bugs in one or > more of: > . Xen frontend network driver (unlikely) > . Xen backend network driver > . Linux bridge code > . Linux routing code > . Linux physical NIC adapter > > I suspect it is the last one, but it''s complicated enough that the bug > could lie elsewhere but only show up on certain NIC''s. > > You can be thankful that your problem was consistent. I have a setup > where the packets from DomU are routed in Dom0 to one of 6 GRE tunnels, > and occasionally for 30 minutes or so TCP packets stopped working. Took > me ages to figure out that turning off tx checksum offload in the DomU > would resolve the problem. It was made harder because by the time I > could connect to the client to investigate, it had started working again > so all I could do was to set up tcpdumps of some fairly high volume > links and wade through those based on the times that the problem was > occurring... > > James >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Alain RICHARD
2009-Jan-21 08:46 UTC
Re: [Xen-users] Re: Dom0 to DomU and reverse works but Outside world toDomU only ping works
Le 20 janv. 09 à 01:24, James Harper a écrit :> This is a common problem, and appears to be related to bugs in one or > more of: > . Xen frontend network driver (unlikely) > . Xen backend network driver > . Linux bridge code > . Linux routing code > . Linux physical NIC adapter > > I suspect it is the last one, but it''s complicated enough that the bug > could lie elsewhere but only show up on certain NIC''s. > > You can be thankful that your problem was consistent. I have a setup > where the packets from DomU are routed in Dom0 to one of 6 GRE > tunnels, > and occasionally for 30 minutes or so TCP packets stopped working. > Took > me ages to figure out that turning off tx checksum offload in the DomU > would resolve the problem. It was made harder because by the time I > could connect to the client to investigate, it had started working > again > so all I could do was to set up tcpdumps of some fairly high volume > links and wade through those based on the times that the problem was > occurring... > > JamesI had the same kind of issue under redhat 5.2 and succeed in reproducing it 100% of the time. It seams to be an interraction between tx offload and netfilter nat. I have opened a case : http://bugzilla.redhat.com/show_bug.cgi?id=474191 I have got a reply from Herbet Xu indicating that effectively, currently there is an issue in the netfilter infrastructure. I don''t know if this issue is well identified in the xen or netfilter projects. Regards, -- Alain RICHARD <mailto:alain.richard@equation.fr> EQUATION SA <http://www.equation.fr/> Tel : +33 477 79 48 00 Fax : +33 477 79 48 01 E-Liance, Opérateur des entreprises et collectivités, Liaisons Fibre optique, SDSL et ADSL <http://www.e-liance.fr> _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users