Syslinux - Jun 2015 - Fwd: EFI & PXE-booting: very slow TFTP performance on a VMWare test setup

>> Are there any known problems with the performance of TFTP in (U)EFI
>> environments in general or maybe just on VMWare?
> I don't believe so.  I use Workstation 10.0.2 on Ubuntu 12.04 for the
> moment.
<stupid question> and have EFI enabled? </stupid>
> I hope this is a hex-core host with 6+GiB of RAM.
Fedora runs on an AMD A8-6600K at 3Ghz and has 8G of RAM. Ubuntu runs
on an Intel i7-2600K at 3.4Ghz with 16G RAM. I think they perform at
least "good enough" to quikly boot and install in BIOS mode. The
trouble only starts when I toggle the EFI switch.

> Have you considered single vCPU client VMs?
Yes. That's what I started with. But then I saw this piece of code in
efi/udp.c and thought it would be wise to enable an extra CPU to
receive the UDP packet. A single CPU has a 100% load here:

    start = jiffies();
    while (cb_status == -1) {
        /* 15ms receive timeout... */
        if (jiffies() - start >= 30) {
            if (jiffies() - start >= 60)
                printf("Failed to cancel UDP\n");

            uefi_call_wrapper(udp->Cancel, 2, udp, &token);
            printf("core_udp_recv: timed out\n");
        }
        uefi_call_wrapper(udp->Poll, 1, udp);
    }

Just to rule out some stupid missed interrupt or so, I gave it an extra CPU.

> I presume you disabled VMware's isc-dhcp on vmnet1?
Yes. I should have mentioned that. Found out the hard way.. . ;-)

> Good idea to serve from a subtree.
Thanks. We need to support both modes, so we now have /tftpboot/bios
and ../efi64.

>> # make spotless
>> # make bios
>> # make efi64
>
> 'make bios' should be unnecessary.
I have to test both modes, but while debugging I mostly did just "make
efi64 && install-stuff-to-/tftpboot/efi64".
>> The server does just fine. It typically reacts within 0.000100 seconds.
>> The client also starts out fine, reacting within about 0.000250
seconds,
>> but this delay slowly increases.
>
> This makes me wonder the source and there's at least 3 factors: VMware
UEFI
> on multiple vCPU, memory or other leak, a strange bug in Syslinux.
This also happens on a single vCPU.

I did check efi/udp.c over and over again (especially the event
handling) but I don't see a place that could cause this behaviour.

One thing I did just now is a bit - ehm - hackish. I symlinked
/tftpboot/efi64/syslinux.efi to /dev/zero and tried booting from that.
I wanted to see if the "built-in" tftp-client of VMWare's UEFI
firmware also showed this behaviour, but it didn't. Well, it got a bit
slower, but not as much as the client from syslinux.

> I recall finding the UEFI 2.4 spec really for free.
I recall running into some NDA that had to be manually approved before
I could download anything. I also remember preferring to go on hacking
in more debug statements to find out where the delay comes from.
Unfortunately I still have no clue and no specs as a result. Even with
the specs, my guess is it will take some time understanding them. I'm
just becoming a bit more comfortable reading syslinux code.
> By the way, what version/commit?
A fresh syslinux-6.03.tar.gz from kernel.org.
I also tried 81ad566f155fac31089fde69c87059b217e7e9b6 with the same results.

-- 

Oscar Roozen
Linux Developer
Bright Computing BV

On Wed, Jun 17, 2015 at 2:20 PM, Oscar Roozen via Syslinux
<syslinux at zytor.com> wrote:
>>> Are there any known problems with the performance of TFTP in (U)EFI
>>> environments in general or maybe just on VMWare?
>> I don't believe so.  I use Workstation 10.0.2 on Ubuntu 12.04 for
the
>> moment.
>
> <stupid question> and have EFI enabled? </stupid>
>
>> I hope this is a hex-core host with 6+GiB of RAM.
>
> Fedora runs on an AMD A8-6600K at 3Ghz and has 8G of RAM. Ubuntu runs
> on an Intel i7-2600K at 3.4Ghz with 16G RAM. I think they perform at
> least "good enough" to quikly boot and install in BIOS mode. The
> trouble only starts when I toggle the EFI switch.
>
>
>> Have you considered single vCPU client VMs?
>
> Yes. That's what I started with. But then I saw this piece of code in
> efi/udp.c and thought it would be wise to enable an extra CPU to
> receive the UDP packet. A single CPU has a 100% load here:
>
>     start = jiffies();
>     while (cb_status == -1) {
>         /* 15ms receive timeout... */
>         if (jiffies() - start >= 30) {
>             if (jiffies() - start >= 60)
>                 printf("Failed to cancel UDP\n");
>
>             uefi_call_wrapper(udp->Cancel, 2, udp, &token);
>             printf("core_udp_recv: timed out\n");
>         }
>         uefi_call_wrapper(udp->Poll, 1, udp);
>     }
>
> Just to rule out some stupid missed interrupt or so, I gave it an extra
CPU.
>
>
>> I presume you disabled VMware's isc-dhcp on vmnet1?
>
> Yes. I should have mentioned that. Found out the hard way.. . ;-)
>
>
>> Good idea to serve from a subtree.
>
> Thanks. We need to support both modes, so we now have /tftpboot/bios
> and ../efi64.
>
>
>>> # make spotless
>>> # make bios
>>> # make efi64
>>
>> 'make bios' should be unnecessary.
>
> I have to test both modes, but while debugging I mostly did just "make
> efi64 && install-stuff-to-/tftpboot/efi64".
>
>>> The server does just fine. It typically reacts within 0.000100
seconds.
>>> The client also starts out fine, reacting within about 0.000250
seconds,
>>> but this delay slowly increases.
>>
>> This makes me wonder the source and there's at least 3 factors:
VMware UEFI
>> on multiple vCPU, memory or other leak, a strange bug in Syslinux.
>
> This also happens on a single vCPU.
>
> I did check efi/udp.c over and over again (especially the event
> handling) but I don't see a place that could cause this behaviour.
>
> One thing I did just now is a bit - ehm - hackish. I symlinked
> /tftpboot/efi64/syslinux.efi to /dev/zero and tried booting from that.
> I wanted to see if the "built-in" tftp-client of VMWare's
UEFI
> firmware also showed this behaviour, but it didn't. Well, it got a bit
> slower, but not as much as the client from syslinux.
>
>
>> I recall finding the UEFI 2.4 spec really for free.
>
> I recall running into some NDA that had to be manually approved before
> I could download anything. I also remember preferring to go on hacking
> in more debug statements to find out where the delay comes from.
> Unfortunately I still have no clue and no specs as a result. Even with
> the specs, my guess is it will take some time understanding them. I'm
> just becoming a bit more comfortable reading syslinux code.
>
>> By the way, what version/commit?
>
> A fresh syslinux-6.03.tar.gz from kernel.org.
> I also tried 81ad566f155fac31089fde69c87059b217e7e9b6 with the same
results.
I see the nice exponential-like decay of IO rate :(

-- 
-Gene

On Thu, Jun 18, 2015 at 1:18 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> On Wed, Jun 17, 2015 at 2:20 PM, Oscar Roozen via Syslinux
> <syslinux at zytor.com> wrote:
>>>> Are there any known problems with the performance of TFTP in
(U)EFI
>>>> environments in general or maybe just on VMWare?
>>> I don't believe so.  I use Workstation 10.0.2 on Ubuntu 12.04
for the
>>> moment.
>>
>> <stupid question> and have EFI enabled? </stupid>
>>
>>> I hope this is a hex-core host with 6+GiB of RAM.
>>
>> Fedora runs on an AMD A8-6600K at 3Ghz and has 8G of RAM. Ubuntu runs
>> on an Intel i7-2600K at 3.4Ghz with 16G RAM. I think they perform at
>> least "good enough" to quikly boot and install in BIOS mode.
The
>> trouble only starts when I toggle the EFI switch.
>>
>>
>>> Have you considered single vCPU client VMs?
>>
>> Yes. That's what I started with. But then I saw this piece of code
in
>> efi/udp.c and thought it would be wise to enable an extra CPU to
>> receive the UDP packet. A single CPU has a 100% load here:
>>
>>     start = jiffies();
>>     while (cb_status == -1) {
>>         /* 15ms receive timeout... */
>>         if (jiffies() - start >= 30) {
>>             if (jiffies() - start >= 60)
>>                 printf("Failed to cancel UDP\n");
>>
>>             uefi_call_wrapper(udp->Cancel, 2, udp, &token);
>>             printf("core_udp_recv: timed out\n");
>>         }
>>         uefi_call_wrapper(udp->Poll, 1, udp);
>>     }
>>
>> Just to rule out some stupid missed interrupt or so, I gave it an extra
CPU.
Boot loader environments rarely benefit from a second CPU.  The best
way to not miss an interrupt in a VM boot environment is to not
overcommit.
>>> I presume you disabled VMware's isc-dhcp on vmnet1?
>>
>> Yes. I should have mentioned that. Found out the hard way.. . ;-)
>>
>>
>>> Good idea to serve from a subtree.
>>
>> Thanks. We need to support both modes, so we now have /tftpboot/bios
>> and ../efi64.
>>
>>
>>>> # make spotless
>>>> # make bios
>>>> # make efi64
>>>
>>> 'make bios' should be unnecessary.
>>
>> I have to test both modes, but while debugging I mostly did just
"make
>> efi64 && install-stuff-to-/tftpboot/efi64".
>>
>>>> The server does just fine. It typically reacts within 0.000100
seconds.
>>>> The client also starts out fine, reacting within about 0.000250
seconds,
>>>> but this delay slowly increases.
>>>
>>> This makes me wonder the source and there's at least 3 factors:
VMware UEFI
>>> on multiple vCPU, memory or other leak, a strange bug in Syslinux.
>>
>> This also happens on a single vCPU.
>>
>> I did check efi/udp.c over and over again (especially the event
>> handling) but I don't see a place that could cause this behaviour.
>>
>> One thing I did just now is a bit - ehm - hackish. I symlinked
>> /tftpboot/efi64/syslinux.efi to /dev/zero and tried booting from that.
>> I wanted to see if the "built-in" tftp-client of VMWare's
UEFI
>> firmware also showed this behaviour, but it didn't. Well, it got a
bit
>> slower, but not as much as the client from syslinux.
>>
>>
>>> I recall finding the UEFI 2.4 spec really for free.
>>
>> I recall running into some NDA that had to be manually approved before
>> I could download anything. I also remember preferring to go on hacking
>> in more debug statements to find out where the delay comes from.
>> Unfortunately I still have no clue and no specs as a result. Even with
>> the specs, my guess is it will take some time understanding them.
I'm
>> just becoming a bit more comfortable reading syslinux code.
>>
>>> By the way, what version/commit?
>>
>> A fresh syslinux-6.03.tar.gz from kernel.org.
>> I also tried 81ad566f155fac31089fde69c87059b217e7e9b6 with the same
results.
>
> I see the nice exponential-like decay of IO rate :(
Consider one of the following workarounds:

1) HTTP/FTP.  Loading individual files over HTTP/FTP would be
suggested for a large file anyways.  For your situation, I'd suggest
trying to use DHCP option 210, either embedded in option 43 or
hard-coded into syslinux.efi as an after-DHCP option with
pxelinux-options.  Setting the plain option would require forcing your
dhcpd to always insert it (isc-dhcpd after v3 I think can do this) as
no PXE client will ever request it.  I have http://myip/tftp/ pointing
at my tftpd root just for this purpose.

2) Load multiple files.  If the initrd is a CPIO (compressed or not)
for Linux initramfs, split it to multiple files and only put a portion
of the file system into each.  I often load 3-10 GZipped CPIO archves,
each one an independent CPIO file that's extracted over the last.

-- 
-Gene

[re-sent to list]

On Thu, Jun 18, 2015 at 7:18 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> I see the nice exponential-like decay of IO rate :(
Did you see this only on vmware or also in other environments?


On Sat, Jun 20, 2015 at 1:44 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> Boot loader environments rarely benefit from a second CPU.  The best
> way to not miss an interrupt in a VM boot environment is to not
> overcommit.
I'm still a bit worried about the 100% CPU load while waiting for UDP
packets to arrive..
Wouldn't it be better to just 'hlt' somehow and have the interrupts
wake the CPU up?
> Consider one of the following workarounds:
>
> 1) HTTP/FTP.  Loading individual files over HTTP/FTP would be
> suggested for a large file anyways.  For your situation, I'd suggest
> trying to use DHCP option 210, either embedded in option 43 or
> hard-coded into syslinux.efi as an after-DHCP option with
> pxelinux-options.  Setting the plain option would require forcing your
> dhcpd to always insert it (isc-dhcpd after v3 I think can do this) as
> no PXE client will ever request it.  I have http://myip/tftp/ pointing
> at my tftpd root just for this purpose.
I went the HTTP-way for now:

if option architecture-type = 00:09 {
    filename "efi64/syslinux.efi";
    option pxelinux.pathprefix  "http://myserverip/tftp/efi64/";
} elsif option architecture-type = 00:07 {
    # efi32 not supported.
} else {
    filename "bios/pxelinux.0";
}

Thanks!

--

Oscar Roozen
Linux Developer




oscar.roozen at brightcomputing.com

Bright Computing BV
Kingsfordweg 151
1043 GR Amsterdam
The Netherlands
Tel: +31 20 491 9324

www.BrightComputing.com

On Jun 23, 2015 5:47 AM, "Oscar Roozen"
<oscar.roozen at brightcomputing.com> wrote:
> On Thu, Jun 18, 2015 at 7:18 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> > I see the nice exponential-like decay of IO rate :(
>
> Did you see this only on vmware or also in other environments?
I haven't tested anything else yet.  My only physical EFI machine is
my main machine making it a little challenging.

I did however notice some strange patterns on HTTP transfer at a
millisecond scale in my capture.
> On Sat, Jun 20, 2015 at 1:44 PM, Gene Cumm <gene.cumm at gmail.com>
wrote:
> > Boot loader environments rarely benefit from a second CPU.  The best
> > way to not miss an interrupt in a VM boot environment is to not
> > overcommit.
>
> I'm still a bit worried about the 100% CPU load while waiting for UDP
> packets to arrive..
> Wouldn't it be better to just 'hlt' somehow and have the
interrupts
> wake the CPU up?
I haven't looked at the scheduler with respect to EFI.
> > Consider one of the following workarounds:
> >
> > 1) HTTP/FTP.  Loading individual files over HTTP/FTP would be
> > suggested for a large file anyways.  For your situation, I'd
suggest
> > trying to use DHCP option 210, either embedded in option 43 or
> > hard-coded into syslinux.efi as an after-DHCP option with
> > pxelinux-options.  Setting the plain option would require forcing your
> > dhcpd to always insert it (isc-dhcpd after v3 I think can do this) as
> > no PXE client will ever request it.  I have http://myip/tftp/ pointing
> > at my tftpd root just for this purpose.
>
> I went the HTTP-way for now:
>
> if option architecture-type = 00:09 {
>     filename "efi64/syslinux.efi";
>     option pxelinux.pathprefix  "http://myserverip/tftp/efi64/";
> } elsif option architecture-type = 00:07 {
>     # efi32 not supported.
> } else {
>     filename "bios/pxelinux.0";
> }
>
> Thanks!
You're welcome.

--Gene