I''m running Xen 4.1.4 on Fedora 17. I have some CentOS 6 DomUs - an haproxy machine and some tomcat VMs. When clients send requests with: POST /ProposalInterface HTTP/1.1 Content-Type: text/xml Host: www.myhost.co.uk Content-Length: 2099 Expect: 100-continue Connection: Keep-Alive The "continuation" doesn''t happen. The POST is truncated at 1449 bytes, and the request fails. Running identical VMs, with identical versions of tomcat, haproxy, java, and Centos on VMware works fine. The only difference is that these machines are on Xen rather than VMware. I have absolutely no idea where to start looking, but I sense there must be some setting somewhere which is impacting this. Any hints would be most gratefully received, as is is a show-stopper for a go-live today. I really don''t fancy replacing the whole infrastructure with VMware... and I''d really like to understand what''s going on. Thanks! S. -- Stephen Nelson-Smith Technical Director Atalanta Systems Ltd www.atalanta-systems.com _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 2013-04-03 13:22:12 Stephen Nelson-Smith wrote:> I''m running Xen 4.1.4 on Fedora 17. > > I have some CentOS 6 DomUs - an haproxy machine and some tomcat VMs. > > When clients send requests with: > > POST /ProposalInterface HTTP/1.1 > Content-Type: text/xml > Host: www.myhost.co.uk > Content-Length: 2099 > Expect: 100-continue > Connection: Keep-Alive > > The "continuation" doesn''t happen. The POST is truncated at 1449 bytes, > and the request fails.I would have a look at MTU settings of your network interfaces, as 1449 is just below the usual 1500 bytes MTU size and the symptoms look just like it. We''ve had problems concerning MTU when sizes did not match and path discovery didn''t work for whatever reason. But my knowlegde ends there - so I can just give you the hint to take a look ... - peter.
Hi, On Wed, Apr 3, 2013 at 2:06 PM, Peter Gansterer < peter.gansterer@paradigma.net> wrote:> I would have a look at MTU settings of your network interfaces, as 1449 is > just below the usual 1500 bytes MTU size and the symptoms look just like it. >Yeah - MTU seems to be 1500 everywhere - on the DomU and the Dom0. S. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On 2013-04-03 14:20:55 Stephen Nelson-Smith wrote:> Hi, > > On Wed, Apr 3, 2013 at 2:06 PM, Peter Gansterer < > peter.gansterer@paradigma.net> wrote: > > > I would have a look at MTU settings of your network interfaces, as 1449 is > > just below the usual 1500 bytes MTU size and the symptoms look just like it. > > > > Yeah - MTU seems to be 1500 everywhere - on the DomU and the Dom0.Also on your clients? Your working VMWare setup is Linux? On the same machine? Same switch? Do you have similar symptoms when you transfer with TCP using other means (eg. scp some file from client to server)? Or is it just your HTTP connection? ... just a few things to check as long as you don''t get better input :-) - peter.
> Yeah - MTU seems to be 1500 everywhere - on the DomU and the Dom0.Do you use VLANs?
Hi, On Wed, Apr 3, 2013 at 2:51 PM, Peter Gansterer < peter.gansterer@paradigma.net> wrote:> Also on your clients? >On the client I control, and all upstream firewalls / routers / switches.> Your working VMWare setup is Linux? On the same machine? Same switch? >Working setup is: - ESX6.1 - CentOS 6 on haproxy and tomcat - All on same box - On same switch as the Xen machines> Do you have similar symptoms when you transfer with TCP using other means > (eg. scp some file from client to server)? Or is it just your HTTP > connection? >Only see it with HTTP POSTs. Also - further testing - the size of the post needs to be around 2000 bytes. When it fails, the POST is truncated at 1250-1450 bytes. S. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Wed, Apr 3, 2013 at 3:03 PM, Mike <debian@good-with-numbers.com> wrote:> > Yeah - MTU seems to be 1500 everywhere - on the DomU and the Dom0. > > Do you use VLANs? >We don''t. S. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Maybe someone comes up with an idea if you post more info about your Xen network setup. Like: brctl show route -n ifconfig -a ... - peter. On 2013-04-03 15:15:55 Stephen Nelson-Smith wrote:> Hi, > > > On Wed, Apr 3, 2013 at 2:51 PM, Peter Gansterer < > peter.gansterer@paradigma.net> wrote: > > > Also on your clients? > > > > On the client I control, and all upstream firewalls / routers / switches. > > > > Your working VMWare setup is Linux? On the same machine? Same switch? > > > > Working setup is: > > - ESX6.1 > - CentOS 6 on haproxy and tomcat > - All on same box > - On same switch as the Xen machines > > > > Do you have similar symptoms when you transfer with TCP using other means > > (eg. scp some file from client to server)? Or is it just your HTTP > > connection? > > > > Only see it with HTTP POSTs. > > Also - further testing - the size of the post needs to be around 2000 > bytes. When it fails, the POST is truncated at 1250-1450 bytes. > > S.
Hi, On Wed, Apr 3, 2013 at 3:40 PM, Peter Gansterer < peter.gansterer@paradigma.net> wrote:> Maybe someone comes up with an idea if you post more info about your Xen > network setup. > Like: > > brctl show >[root@dom0-a ~]# brctl show bridge name bridge id STP enabled interfaces br0 8000.90b11c26fac3 no em1 p1p1 vif1.0 vif2.0 vif3.0 virbr0 8000.000000000000 yes> route -n >[root@dom0-a ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.1.1.1 0.0.0.0 UG 0 0 0 br0 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br0 169.254.0.0 0.0.0.0 255.255.0.0 U 1006 0 0 br0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0> ifconfig -a ># ifconfig -a br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.1.1.102 netmask 255.255.255.0 broadcast 10.1.1.255 inet6 fe80::92b1:1cff:fe26:fac3 prefixlen 64 scopeid 0x20<link> ether 90:b1:1c:26:fa:c3 txqueuelen 0 (Ethernet) RX packets 5609 bytes 888376 (867.5 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 739 bytes 103749 (101.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 em1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether 90:b1:1c:26:fa:c3 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 16 em2: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether 90:b1:1c:26:fa:c4 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 456 bytes 62928 (61.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 456 bytes 62928 (61.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p1p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether a0:36:9f:0b:5b:5c txqueuelen 1000 (Ethernet) RX packets 166067 bytes 35247039 (33.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 155732 bytes 47672875 (45.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xd4d00000-d4e00000 p1p2: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether a0:36:9f:0b:5b:5d txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xd4e00000-d4f00000 vif1.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::fcff:ffff:feff:ffff prefixlen 64 scopeid 0x20<link> ether fe:ff:ff:ff:ff:ff txqueuelen 32 (Ethernet) RX packets 11 bytes 2054 (2.0 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4868 bytes 880588 (859.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vif2.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::fcff:ffff:feff:ffff prefixlen 64 scopeid 0x20<link> ether fe:ff:ff:ff:ff:ff txqueuelen 32 (Ethernet) RX packets 158626 bytes 61007135 (58.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 166815 bytes 34685114 (33.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 vif3.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::fcff:ffff:feff:ffff prefixlen 64 scopeid 0x20<link> ether fe:ff:ff:ff:ff:ff txqueuelen 32 (Ethernet) RX packets 19319 bytes 2725065 (2.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 15096 bytes 18264428 (17.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ether 22:34:6a:c1:1b:23 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # uname -a Linux dom0-a.aha.biz 3.7.9-104.fc17.x86_64 #1 SMP Sun Feb 24 19:19:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux [root@dom0-a ~]# cat /etc/fedora-release Fedora release 17 (Beefy Miracle) # xm dmesg | grep -i ''xen version'' (XEN) Xen version 4.1.4 (mockbuild@[unknown]) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) Fri Feb 22 16:32:39 UTC 2013 S. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
On Wed, Apr 3, 2013 at 4:57 PM, Mike <debian@good-with-numbers.com> wrote:> > Only see it with HTTP POSTs. > > > > Also - further testing - the size of the post needs to be around 2000 > > bytes. When it fails, the POST is truncated at 1250-1450 bytes. > > tcpdump the packets at the various points in your network to see where > the data drops. >Yep - done that, viz: Test ---> HAProxy (vmware) ---> Tomcat (vmware) # Works Test ---> HAProxy (vmware) ---> Tomcat (xen) # Data dropped between HAProxy and Tomcat Test ---> HAProxy (xen) ---> Tomcat (vmware) # Data droped at Haproxy Test ---> HAProxy(xen) ---> HAProxy (xen) # Data dropped at Haproxy I''ll verify this again. It''s hard to see the whole XML Post. The XML being posted is about 2000 bytes in size - up to 4000. Up to a certain size (circza 1800 or so) it works reliably. Above that size, we see the XML hit the LB, but the full XML isn''t reached by the Tomcats. S. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users