Syslinux - Aug 2015 - EFI: ipxe + syslinux = Failed to read blocks: 0xC

Hello dear list,

Using VMware I built a test setup of a server with dhcp, tftp and a 
http-server and several pxe-clients with EFI mode turned on. This setup 
worked, albeit with the exponential-like decay of IO rate I described 
earlier. The work-around of using HTTP works beautifully though.

So it was time for the next step. Migrate this setup to our automated 
testing environment that runs under qemu-kvm. For this I had to use OVMF 
so the nodes would boot in EFI mode (or should I say "UEFI mode" ?). 
This setup uses ipxe roms  to boot over the network.

The short story is that I never got further than syslinux proclaiming 
"Failed to read blocks: 0xC" before even getting to the stage where it
tries to load ldlinux.e64. Searching for answers I found a thread on the 
qemu-devel list discussing this problem. The user mjt (Michael Tokarev, 
also on this list) found a branch of ipxe that contains two patches that 
should solve this problem. See 
http://www.syslinux.org/archives/2015-June/023582.html for more on this. 
Those patches haven't been merged to the master branch on ipxe.org yet.

Unfortunately whichever version of IPXE or syslinux I try, the result is 
always:

    iPXE 1.0.0+ (87981) -- Open Source Network Boot Firmware --
    http://ipxe.org
    Features: DNS HTTP TFTP EFI Menu

    net0: ec:b1:d7:75:e5:24 using NII on NII-PCI02:00.0 (open)
       [Link:down, TX:0 TXE:0 RX:0 RXE:0]
       [Link status: Unknown (http://ipxe.org/1a086194)]
    Configuring (net0 ec:b1:d7:75:e5:24)...... ok
    net0: 10.X.167.253/255.255.0.0 gw 10.X.255.254
    Next server: 10.X.255.254
    Filename: efi64/syslinux-87020-debug.efi
    tftp://10.X.255.254/efi64/syslinux-87020-debug.efi... ok
    efi_rdwr_sectors: read 1 bytes
    Failed to read blocks: 0xC

In this screenshot the iPXE version is 87981 for commit 87981bb6. The 
syslinux version here is 87020-or for commit 8702009f plus one extra 
printf I added, that shows me the error happens on the very first call 
to efi_rdwr_sectors while reading one byte.

This iPXE comes from git://git.kraxel.org/ipxe, but I also tried stock 
version 1.0.0+ and 2b15ae55 and the newest ae7f22eb2  from 
git://git.ipxe.org/ipxe.git. For syslinux I used the precompiled binary 
from syslinux-6.03.tar.gz, the result of 'make spotless; make efi64' on 
that tar. Furthermore I checked out the master branch from 
git://repo.or.cz/syslinux.git which gave me the 8702009f commit. I also 
tried 23b2707b for reasons I forgot.

For OVMF I tried both edk2.git-ovmf-x64-0-20150716.b1120.g387e7c1.noarch 
from https://www.kraxel.org/repos/jenkins/ and a freshly built version 
of commit 2ad9cf37 from https://github.com/tianocore/edk2.git.

But I think I can rule out any influence from either qemu or 
ovmf/tianocore/edk2 or some misconfiguration of libvirt on my behalf, 
because while I was busy troubleshooting this, a shiny new piece of real 
hardware arrived: a HP Proliant DL360 gen9. So I got to play with real 
hardware!

Unfortunately, the error above was actually cut and pasted from the 
ipmi-console session to this HP...

Needless to say I tried all combinations of ipxe + syslinux on this HP 
as well, but no luck whatsoever. Always:

     >> Booting Embedded LOM 1 Port 1 : HP Ethernet 1Gb 4-port 331i
    Adapter - NIC (PXE IPv4)

     >> Booting PXE over IPv4.
       Station IP address is 10.X.167.251

       Server IP address is 10.X.255.254
       NBP filename is efi64/snponly.efi
       NBP filesize is 119488 Bytes
      Downloading NBP file...

       NBP file downloaded successfully.
    iPXE initialising devices...ok



    iPXE 1.0.0+ (87981) -- Open Source Network Boot Firmware --
    http://ipxe.org
    Features: DNS HTTP TFTP EFI Menu

    net0: ec:b1:d7:75:e5:24 using NII on NII-PCI02:00.0 (open)
       [Link:down, TX:0 TXE:0 RX:0 RXE:0]
       [Link status: Unknown (http://ipxe.org/1a086194)]
    Configuring (net0 ec:b1:d7:75:e5:24)...... ok
    net0: 10.X.167.253/255.255.0.0 gw 10.X.255.254
    Next server: 10.X.255.254
    Filename: efi64/syslinux-87020-or.efi
    tftp://10.X.255.254/efi64/syslinux-87020-or.efi... ok
    efi_rdwr_sectors: read 1 bytes
    Failed to read blocks: 0xC

While playing with this HP I found there is a misunderstanding worldwide 
about the contents of dhcp option 93, but this was also discussed 
before: http://www.syslinux.org/archives/2014-October/022684.html. It 
looks like all open source authors correctly use the definition from RFC 
4578 whereas most hardware vendors stick with the definition from the 
DHCPv6 specs which has x64 and EBC the wrong way around. The HP is no 
exception here, but this made it very easy to chainload syslinux after 
ipxe. If dhcpd receives a 0x07, it will respond with ipxe.efi. This will 
then do a request again with the correct (0x09) architecture upon which 
dhcpd responds with syslinux.efi.

My questions:

What can be the cause of error 0x0C which translates to EFI_NO_MEDIA ?

How can I enable debugging on the various components involved?

If some you have iPXE + syslinux working with UEFI somehow, either 
through roms in qemu or by chainloading from snponly.efi, ipxe.efi or 
any other way, please tell me more about your setup?

Can there be some mis-configuration somewhere that can lead to this error?

Or the TL;DR version: HELP! ;-)

In any case, thanks for reading this far into this mail!

Regards,

Oscar Roozen
Linux Developer

>>>
 Using VMware I built a test setup of a server with dhcp, tftp and a 
 http-server and several pxe-clients with EFI mode turned on. This setup 
 worked, albeit with the exponential-like decay of IO rate I described 
 earlier. The work-around of using HTTP works beautifully though.
<<<

""exponential-like decay of IO rate"" could you please post
a link on
your comments on this?
 
>>>
 So it was time for the next step. Migrate this setup to our automated 
 testing environment that runs under qemu-kvm. For this I had
 to use OVMF  so the nodes would boot in EFI mode (or should I say "UEFI
 mode" ?).  This setup uses ipxe roms? to boot over the network.
 
 The short story is that I never got further than syslinux proclaiming 
 "Failed to read blocks: 0xC" before even getting to the stage where
it
 tries to load ldlinux.e64. Searching for answers I found a thread on the 
 qemu-devel list discussing this problem. The user mjt (Michael Tokarev, 
 also on this list) found a branch of ipxe that contains two patches that 
 should solve this problem. See 
 http://www.syslinux.org/archives/2015-June/023582.html for more on this. 
 Those patches haven't been merged to the master branch on ipxe.org yet.

 Unfortunately whichever version of IPXE or syslinux I try, the result is 
 always:
 
 ? ? iPXE 1.0.0+ (87981) -- Open Source Network
 Boot Firmware --
 ? ? http://ipxe.org
 ? ? Features: DNS HTTP TFTP EFI Menu
 
 ? ? net0: ec:b1:d7:75:e5:24 using NII on NII-PCI02:00.0 (open)
 ? ? ???[Link:down, TX:0 TXE:0 RX:0 RXE:0]
 ? ? ???[Link status: Unknown (http://ipxe.org/1a086194)]
 ? ? Configuring (net0 ec:b1:d7:75:e5:24)...... ok
 ? ? net0: 10.X.167.253/255.255.0.0 gw 10.X.255.254
 ? ? Next server: 10.X.255.254
 ? ? Filename: efi64/syslinux-87020-debug.efi
 ? ?
 tftp://10.X.255.254/efi64/syslinux-87020-debug.efi... ok
 ? ? efi_rdwr_sectors: read 1 bytes
 ? ? Failed to read blocks: 0xC
 
 In this screenshot the iPXE version is 87981 for commit 87981bb6. The 
 syslinux version here is 87020-or for commit 8702009f plus one extra 
 printf I added, that shows me the error happens on the very first call 
 to efi_rdwr_sectors while reading one byte.
 
 This iPXE comes from git://git.kraxel.org/ipxe, but I also tried stock 
 version 1.0.0+ and 2b15ae55 and the newest ae7f22eb2? from 
 git://git.ipxe.org/ipxe.git. For syslinux I used the precompiled binary 
 from syslinux-6.03.tar.gz, the result of 'make spotless; make efi64' on
 that tar. Furthermore I checked out the master branch from 
 git://repo.or.cz/syslinux.git which gave me the 8702009f
 commit. I also  tried 23b2707b for reasons I forgot.
 
 For OVMF I tried both
 edk2.git-ovmf-x64-0-20150716.b1120.g387e7c1.noarch 
 from https://www.kraxel.org/repos/jenkins/ and a freshly built version 
 of commit 2ad9cf37 from https://github.com/tianocore/edk2.git.
 
 But I think I can rule out any influence from either qemu or
 ovmf/tianocore/edk2 or some misconfiguration of libvirt on my behalf, 
 because while I was busy troubleshooting this, a shiny new piece of real 
 hardware arrived: a HP Proliant DL360 gen9. So I got to play with real 
 hardware!
 
 Unfortunately, the error above was actually cut and pasted from the 
 ipmi-console session to this HP...
 
 Needless to say I tried all combinations of ipxe + syslinux on this HP 
 as well, but no luck whatsoever. Always:
 
 ? ???>> Booting Embedded LOM 1
 Port 1 : HP Ethernet 1Gb 4-port 331i ? ? Adapter - NIC (PXE IPv4)
 
 ? ???>> Booting PXE over IPv4.
 ? ? ???Station IP address is 10.X.167.251
 
 ? ? ???Server IP address is 10.X.255.254
 ? ? ???NBP filename is efi64/snponly.efi
 ? ? ???NBP filesize is 119488 Bytes
 ? ? ? Downloading NBP file...
 
 ? ? ???NBP file downloaded successfully.
 ? ? iPXE initialising devices...ok
 
 
 
 ? ? iPXE 1.0.0+ (87981) -- Open Source Network
 Boot Firmware -- ? ? http://ipxe.org
 ? ? Features: DNS HTTP TFTP EFI Menu
 
 ? ? net0: ec:b1:d7:75:e5:24 using NII on NII-PCI02:00.0 (open)
 ? ? ???[Link:down, TX:0 TXE:0 RX:0 RXE:0]
 ? ? ???[Link status: Unknown (http://ipxe.org/1a086194)]
 ? ? Configuring (net0 ec:b1:d7:75:e5:24)...... ok
 ? ? net0: 10.X.167.253/255.255.0.0 gw 10.X.255.254
 ? ? Next server: 10.X.255.254
 ? ? Filename: efi64/syslinux-87020-or.efi
 ? ?
 tftp://10.X.255.254/efi64/syslinux-87020-or.efi... ok
 ? ? efi_rdwr_sectors: read 1 bytes
 ? ? Failed to read blocks: 0xC
 
 While playing with this HP I found there is a misunderstanding worldwide 
 about the contents of dhcp option 93, but this was also discussed 
 before: http://www.syslinux.org/archives/2014-October/022684.html.
 It  looks like all open source authors correctly use the definition from 
RFC  4578 whereas most hardware vendors stick with the definition
 from the  DHCPv6 specs which has x64 and EBC the wrong way around. 
The HP is no  exception here, but this made it very easy to chainload
 syslinux after ipxe. If dhcpd receives a 0x07, it will respond with
 ipxe.efi. This will  then do a request again with the correct (0x09)
architecture
 upon which dhcpd responds with syslinux.efi. 
>>>
OK I got lost.
are you saying you finally were able to chainload syslinux.efi on the HP??
by doing what?

Best,
Patrick

On 31-07-15 14:25, Patrick Masotta wrote:
> ""exponential-like decay of IO rate"" could you please
post a link on
> your comments on this?
Sure, see the thread starting at:
	http://www.syslinux.org/archives/2015-June/023621.html
And Gene's confirmation at:
	http://www.syslinux.org/archives/2015-June/023629.html
> OK I got lost.
> are you saying you finally were able to chainload syslinux.efi on the HP??
> by doing what?
Some magic in dhcpd.conf:

       if option architecture-type = 00:09 {
         filename "efi64/syslinux-87020-or.efi";
         option pxelinux.pathprefix "http://bla/efi64/";
       } elsif option architecture-type = 00:07 {
         filename "efi64/ipxe-git87981.efi";
       } else {
         filename "pxelinux.0";
       }

Hello,

I can't help with the problems on real hardware but have come across the 
qemu problems before and maybe I can give some hints on that.

On Fri, 31 Jul 2015, Oscar Roozen via Syslinux wrote:
> So it was time for the next step. Migrate this setup to our automated
testing
> environment that runs under qemu-kvm. For this I had to use OVMF so the
nodes
> would boot in EFI mode (or should I say "UEFI mode" ?). This
setup uses ipxe
> roms  to boot over the network.
>
> The short story is that I never got further than syslinux proclaiming
"Failed
> to read blocks: 0xC" before even getting to the stage where it tries
to load
> ldlinux.e64. Searching for answers I found a thread on the qemu-devel list
This is happening if you are trying to boot syslinux.efi from iPXE that 
was not patched. In my understanding iPXE patches some UEFI network 
services in order to provide a better user experience but this 
unfortunately breaks syslinux's usage of UEFI services where it thinks 
it's not booted from PXE (cannot find PxeBaseCodeProtocol) and tries to 
read from disk which fails with the above error.
> discussing this problem. The user mjt (Michael Tokarev, also on this list) 
> found a branch of ipxe that contains two patches that should solve this 
> problem. See http://www.syslinux.org/archives/2015-June/023582.html for
more
> on this. Those patches haven't been merged to the master branch on
ipxe.org
> yet.
If you go back one message in that thread you'll find a link to another 
thread explaining what's happening and what workarounds are available 
here: http://sourceforge.net/p/edk2/mailman/message/33735116/

The easiest probably is to use virtio-net with the romfile= option to 
completely disable iPXE roms and use the OVMF built in driver. You can 
tell it's using that if you don't see the iPXE header just syslinux.efi 
booting. This unfortunately only works with virtio-net as that's the only 
network inferface that OVMF has a driver. Or you could try a patched iPXE 
rom or a recent version of qemu (maybe git or latest rc releases) that 
possibly have patched iPXE roms which may work better. Using older qemu or 
original upstream iPXE probably will not work and lead to this error.
> But I think I can rule out any influence from either qemu or 
> ovmf/tianocore/edk2 or some misconfiguration of libvirt on my behalf,
because
> while I was busy troubleshooting this, a shiny new piece of real hardware 
> arrived: a HP Proliant DL360 gen9. So I got to play with real hardware!
>
> Unfortunately, the error above was actually cut and pasted from the 
> ipmi-console session to this HP...
>
> Needless to say I tried all combinations of ipxe + syslinux on this HP as 
> well, but no luck whatsoever. Always:
>
>    >> Booting Embedded LOM 1 Port 1 : HP Ethernet 1Gb 4-port 331i
>   Adapter - NIC (PXE IPv4)
>
>    >> Booting PXE over IPv4.
>      Station IP address is 10.X.167.251
>
>      Server IP address is 10.X.255.254
>      NBP filename is efi64/snponly.efi
>      NBP filesize is 119488 Bytes
>     Downloading NBP file...
>
>      NBP file downloaded successfully.
>   iPXE initialising devices...ok
>
>
>
>   iPXE 1.0.0+ (87981) -- Open Source Network Boot Firmware --
>   http://ipxe.org
>   Features: DNS HTTP TFTP EFI Menu
Since iPXE is involved it can be the same problem as with qemu but it's 
possible that without iPXE you find other problems caused by multiple 
network interfaces. I'm not sure about that. After you've ruled out iPXE
maybe you could try to boot from different interfaces and see if they 
behave differently.

Regards,
BALATON Zoltan

> I can't help with the problems on real hardware but have come across
the
> qemu problems before and maybe I can give some hints on that.
Thank you, your help is much appreciated!

>> The short story is that I never got further than syslinux proclaiming
>> "Failed to read blocks: 0xC" before even getting to the stage
where it
>> tries to load ldlinux.e64. Searching for answers I found a thread on
>> the qemu-devel list
>
> This is happening if you are trying to boot syslinux.efi from iPXE that
> was not patched.
I was using commit 87981bb from the branch 'qemu' at
git.kraxel.org/ipxe.

The last two log messages for this branch are:
87981bb [efi] make load file protocol optional
ca65a1f efi_snp: improve compliance with the EFI_SIMPLE_NETWORK_PROTOCOL 
spec

These are the patches Michael mentioned in his post. Did I miss other 
patches?

> If you go back one message in that thread you'll find a link to another
> thread explaining what's happening and what workarounds are available
> here: http://sourceforge.net/p/edk2/mailman/message/33735116/
Thanks for linking this thread. I'll read up on it. I think it is one of 
the links that got hit by sourceforge's horrible downtime last month. I 
remember hitting their "we're rebuilding our site" page quite
often
while searching around...

> The easiest probably is to use virtio-net with the romfile= option to
> completely disable iPXE roms and use the OVMF built in driver. You can
> tell it's using that if you don't see the iPXE header just
syslinux.efi
> booting.
I did not succeed in doing this, because I'm running qemu from libvirt. 
There is a new '<rom file="some.rom"/> option, but it
won't take an
empty filename. Can this be done from virsh somehow?
> Or you could try a
> patched iPXE rom or a recent version of qemu (maybe git or latest rc
> releases) that possibly have patched iPXE roms which may work better.
I did try a patched iPXE ROM. At least, I think I did, see above. The 
iPXE ROM identified itself as 1.0.0+ (87981). I created it by running 
'make bin-x86_64-efi/1af41000.efirom' and referenced the resulting file 
with the '<rom file='path'/>' in the device section of the
vm.xml.

On 31/07/15 21:31, BALATON Zoltan via Syslinux wrote:
> On Fri, 31 Jul 2015, Oscar Roozen via Syslinux wrote:
>> The short story is that I never got further than syslinux proclaiming
>> "Failed to read blocks: 0xC" before even getting to the stage
where it
>> tries to load ldlinux.e64. Searching for answers I found a thread on
>> the qemu-devel list
>
> This is happening if you are trying to boot syslinux.efi from iPXE that
> was not patched. In my understanding iPXE patches some UEFI network
> services in order to provide a better user experience but this
> unfortunately breaks syslinux's usage of UEFI services where it thinks
> it's not booted from PXE (cannot find PxeBaseCodeProtocol) and tries to
> read from disk which fails with the above error.
As a heads-up: iPXE will be gaining support for PxeBaseCodeProtocol 
within the next couple of weeks.

Michael