Syslinux - May 2019 - lpxelinux.0 issues with larger initrd.img files from RHEL >= 7.5 on UCS servers?

Hello,

I am using lpxelinux.0 (latest stable version 6.03, using the official binaries
from kernel.org) to kickstart servers as http transfers really helps over links
with poor latencies... These servers are being booted in legacy mode,
not in UEFI.

This has worked very well until recently. Starting with RHEL 7.5, on
some servers, we would see the machine rebooting while pxelinux is in the
middle of downloading the initrd.img file.

I quick workaround was to tell people to boot/kickstart using a previous
minor (last working was 7.4): the install process taking care of updating to
the latest version.

I experienced the same issue again yesterday and had time to think about
it again.

On rhel 7.4, the last working version in my case, the file is that big:
   -rw-r--r--   1 root     root     49763300 Dec  1  2017 initrd.img

I saw on 7.5 and 7.6 they're slightly larger, respectively:
   -r--r--r--   1 root     root     54525200 Mar 22  2018 initrd.img
and
   -r--r--r--   1 root     root     54799220 Oct 10  2018 initrd.img

Because I have no trace on the screen to explain the reboot (same thing
with a recorded session over a serial console), I was like: what if the
size is a factor? Given I had nothing else to try, I looked at the files
and saw they were compressed with xz.

So I wondered, what if I compressed them more (I guessed they were compressed
with the default compression preset)? After uncompressing and compressing with
xz -9 -C crc32, here's what I get for 7.5 and 7.6 respectively:
   -r--r--r--   1 root     root     48823576 May 21 11:45 initrd.img
and
   -r--r--r--   1 root     root     49812292 May 21 11:27 initrd.img

Pretty close to what I had in 7.4. And to my suprise, with these smaller
files, lpxelinux doesn't reboot while downloading the file over http. The
kernel boots up and the OS is installed properly....

I haven't had the time to reproduce this issue over regular
pxelinux/tftp so I don't know if it's just tied to lpxelinux/http or
not.

Also, so far this bug only seems to be triggered on some  Cisco UCS
servers such as UCSB-B200-M3 like the one described below. So maybe it
could be related to BIOS or memory maps, I am not sure!?

I'd hate to go back to tftp because the switch to http was such a huge
step forward.

Is there something I could do or provide to help debug this issue? I
read the "Hardware Compatibility" and "Common Problems"
pages on the
wiki and found nothing close to what I'm seeing. I started reading the
"Development/Debugging" but while I could use "COM32
debug.c32" to get
more details, but I don't know which functions I should be tracing?

Please let me know.

Cheers,
Mathieu

BIOS Information
           Vendor: Cisco Systems, Inc.
           Version: B200M3.2.2.6f.0.052120182033
           Release Date: 05/21/2018
           Address: 0xF0000
           Runtime Size: 64 kB
           ROM Size: 4096 kB
           Characteristics:
                   PCI is supported
                   BIOS is upgradeable
                   BIOS shadowing is allowed
                   Boot from CD is supported
                   Selectable boot is supported
                   BIOS ROM is socketed
                   EDD is supported
                   5.25"/1.2 MB floppy services are supported (int 13h)
                   3.5"/720 kB floppy services are supported (int 13h)
                   3.5"/2.88 MB floppy services are supported (int 13h)
                   Print screen service is supported (int 5h)
                   8042 keyboard services are supported (int 9h)
                   Serial services are supported (int 14h)
                   Printer services are supported (int 17h)
                   ACPI is supported
                   USB legacy is supported
                   BIOS boot specification is supported
                   Targeted content distribution is supported
                   UEFI is supported
           BIOS Revision: 4.6

System Information
           Manufacturer: Cisco Systems Inc
           Product Name: UCSB-B200-M3
           Version: 1
           Serial Number: MYSERIALNUMBER
           UUID: SOMEUUID
           Wake-up Type: Other
           SKU Number:
           Family:

-- 
Mathieu Chouquet-Stringer                               m+syslinux at thi.eu.com
            The sun itself sees not till heaven clears.
	             -- William Shakespeare --

> Is there something I could do
 
 
A_ Instead of 6.03, try 6.04-pre1 (not pre2 and not pre3, but pre1). 
Please remember to change _all_ related files to the same version, not 
just lpxelinux.0.

Alternatively (and in many cases, even better) you could try current 
Debian's "Unstable" packages, which are "more
up-to-date" than
6.04-pre1.

B_ In the configuration file, instead of using:

###
LABEL mylabel
KERNEL mykernel
INITRD myinitrd
APPEND myoptions
###


try:

###
LABEL mylabel
LINUX mykernel
INITRD myinitrd
APPEND myoptions
###


and/or also try:

###
LABEL mylabel2
COM32 linux.c32
APPEND initrd=myinitrd myoptions
###

(Reminder: use linux.c32 and the lib*.c32 modules from the same version 
as lpxelinux.0 and ldlinux.c32.)

Using the LINUX directive and/or the linux.c32 module (instead of the 
KERNEL directive) _might_ help in some "(files') size too big" (or
similar) cases.

Regards,
Ady.

Hello Ady,

thanks for getting back to me.

On Wed, May 22, 2019 at 01:33:50AM +0000, Ady Ady via Syslinux
wrote:
> > Is there something I could do
>  
> A_ Instead of 6.03, try 6.04-pre1 (not pre2 and not pre3, but pre1). 
> Please remember to change _all_ related files to the same version, not 
> just lpxelinux.0.
>
> Alternatively (and in many cases, even better) you could try current 
> Debian's "Unstable" packages, which are "more
up-to-date" than
> 6.04-pre1.
I downloaded pxelinux and syslinux 6.04 from sid, this version has the
same issue. I had also tried official 6.04-pre but not all of them as
some are known to be broken: they didn't work either...
> B_ In the configuration file, instead of using:
> [...]
> try:
> 
> ###
> LABEL mylabel
> LINUX mykernel
> INITRD myinitrd
> APPEND myoptions
> ###
Because our installer has a long history, we were still using:
LABEL
KERNEL
APPEND

I tried using LINUX with INITRD and APPEND but the behavior is the same:
the host reboots.
> and/or also try:
> 
> ###
> LABEL mylabel2
> COM32 linux.c32
> APPEND initrd=myinitrd myoptions
> ###
Tried that as well, both using 6.03 and 6.04 from Debian, doesn't work
either.

On a graphical console, I do see a "Probing EDD (edd=off to disable)"
right before the server reboots...

I did try a edd=off but it doesn't make a difference either...

Any other ideas? ;-)

Cheers,
-- 
Mathieu Chouquet-Stringer                               m+syslinux at thi.eu.com
            The sun itself sees not till heaven clears.
	             -- William Shakespeare --