Syslinux - May 2012 - Syslinux 4.04 gpxelinux.0 http performance problem with VMware VMs

Hello,

In my testing environment I have two VMs on ESXi 5.0.

VM A = dhcp/tftp/PXE/http server, running CentOS 6.2. Syslinux 4.04 with the
included gpxelinux.0.
VM B = PXE boot client.

If I run CentOS 6.2 also on the VM B, I can easily transfer 50+ MB/sec over http
between the VMs (wget, links).

Now, if I PXE boot gpxelinux.0 on the VM B, and start to download bigger initrd
image
over http the download speed is very slow: only 200 - 500 kB/sec.

I tried using both Intel e1000 and vmxnet3 NICs on the VM B. 
No big difference, both were slow.

I tried moving the VMs on the same host, and also to separate hosts - no
difference.

Then I noticed there's a lot of TCP retransmit happening when gpxelinux.0
client
downloads files from the http server.. dozens or even hundreds of retransmit per
second.
Any ideas what could cause that? 

Then I noticed if I use ethtool to disable all offloads (lro/gro/rx/tx) on the
VM A NIC
the download speed jumps to 1700 - 2800 kB/sec. So that's WAY better speed. 
But still there's a lot tcp retransmits happening all the time..

Any ideas what's causing the bad performance and those continuous
retransmits?
Anything I should try to debug this? 

On baremetal with Intel NICs I've seen much better gpxelinux.0 download
speeds over http.

Thanks,

-- Pasi

On Mon, May 7, 2012 at 4:40 PM, Pasi K?rkk?inen <pasik at iki.fi>
wrote:
> Hello,
>
> In my testing environment I have two VMs on ESXi 5.0.
>
> VM A = dhcp/tftp/PXE/http server, running CentOS 6.2. Syslinux 4.04 with
the included gpxelinux.0.
> VM B = PXE boot client.
> Any ideas what's causing the bad performance and those continuous
retransmits?
> Anything I should try to debug this?
>
> On baremetal with Intel NICs I've seen much better gpxelinux.0 download
speeds over http.
Syslinux 4.04 is obsolete and probably not worth debugging.  4.05 is
current and 4.10-pre20 is a beta with lwIP, a lightweight IP stack,
eliminating the need for gPXE if you only want HTTP downloads.  I've
already been testing 4.10-pre* in VMware VMs with success after my
patch.  I've seen a solid 45-70 MB/s after that patch on VMware VMs.

If I recall correctly, the issue you notice is actually how that
version of gPXE talks with the VMware VM PXE stack.

-- 
-Gene

On Tue, Sep 18, 2012 at 5:38 AM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> On Sep 17, 2012 6:18 PM, "H. Peter Anvin" <hpa at
zytor.com> wrote:
>>
>> On 09/17/2012 10:22 AM, Pasi K?rkk?inen wrote:
>> >
>> > I upgraded syslinux to benchmark the latest 4.10-pre22, but when
using
>> > pxelinux.0
>> > from 4.10-pre22 I get this error:
>> >
>> > ..
>> > Trying to load: pxelinux.cfg/01-mac-address
>> >
>> > netconn_write error -5
>> > LWIP(lwip/src/api/api_msg.c,831,0x00107b79): blocking connect in
>> > progress
>> > Boot failed: prss a key to retry, or wait for reset...
>> >
>> >
>> > Using pxelinux.0 from syslinux 4.10-pre20 works OK with the same
>> > config/setup.
>> >
>> > Any ideas?
>> >
>> > Thanks,
>> >
>>
>> Hi Gene,
>>
>> I suspect this is related to the lwip update... any chance you might be
>> able to look at it?
>
> Yes.
>
> --Gene
First glance shows the precompiled 4.10-pre22 pxelinux.0 binary as
functional on one VMware platform loading when config from TFTP within
the local NAT vmnet.

Could I get more details?  Still 1 subnet as before?  All in 1 vSwitch
or does it go out to a physical network?  Is your config on TFTP or
HTTP (with one of the DHCP options)?  Which vNIC?  VMXNET3 still?
Tried e1000/e1000e?  What VMware product is your host system running?
Build number would be nice.

-- 
-Gene