Syslinux - Apr 2018 - EFI Clients Unable to Load kernel/initrd Not Stored on PXE Server

Issue Statement:
PXE booting from BIOS systems works fine on both 6.03 and 6.04-pre1. New clients
only support (u)EFI which results in the inability to load remote kernel or
initrd. Tested with both Syslinux 6.03 and 6.04-pre1. So far unable to boot any
(u)EFI clients unless both initrd and kernel are on PXE server.

Using packet captures I see that the URI request for the remote file on the repo
mirror looks correct ( http://192.168.1.10/centos74/isolinux/vmlinuz ), but the
client is asking the PXE server for the files instead of the repo mirror. The
PXE server responds with a http 404 packet. User sees "File not found"
message on client. Duplicating the directory structure of the Linux Repo Mirror
in the HTTP directory of the PXE server results in successful file transfers.

--Expected Behavior--
*Menu item set to retrieve files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"
*Client looks for files at
"http://192.168.1.10/centos74/isolinux/vmlinuz"
*Client gets files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"

--Actual Behavior--
*Menu item set to retrieve files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"
*Client looks for files at
"http://192.168.1.8/centos74/isolinux/vmlinuz"
*Client gets files from
"http://192.168.1.8/centos74/isolinux/vmlinuz", if the directory
structure has been copied in /var/www/html/


System used for initial testing
*ESXi 5.5U3 VM Version 8 (EFI mode)
	NIC - vmxnet3
*Dell R7425 (AMD Epyc based), does not offer a "BIOS" mode
	NIC - Broadcom 57412 (10G SFP+), FW 20.06.05.11
	BIOS - 1.1.3

Example Systems:
*dhcpd Server - 192.168.1.11
*Internal Linux Repo Mirror - 192.168.1.10
*Prod BIOS PXE (syslinux-6.03) - 192.168.1.13
*Test UEFI PXE (syslinux-6.04-pre1) - 192.168.1.8
*ESXi 5.5U3 VM Version 8 (EFI mode) - 192.168.1.120


--Steps to Reproduce--

##Set up dhcpd server
*Install CentOS 7.4
[dhcpd-server]$ yum install dhcpd -y
*Edit /etc/dhcp/dhcpd.conf to include a DHCP range and the below PXE section
(adjust IPs for PXE server accordingly)
[dhcpd-server]$ systemctl start dhcpd


##Set up PXE server
*Install CentOS 7.4
*Set static IP (192.168.1.8)
[pxe-server]$ yum install tftp-server httpd -y
[pxe-server]$ systemctl start tftp
[pxe-server]$ systemctl start httpd
[pxe-server]$ mkdir ~/syslinux604
[pxe-server]$ cd ~/syslinux604/;wget
https://cdn.kernel.org/pub/linux/utils/boot/syslinux/Testing/6.04/syslinux-6.04-pre1.zip
[pxe-server]$ unzip syslinux-6.04-pre1.zip
[pxe-server]$ cd efi64/
[pxe-server]$ cp com32/elflink/ldlinux/ldlinux.e* com32/lib/libcom32.*
com32/libutil/libutil.c32 com32/menu/vesamenu.c32 /var/lib/tftpboot/
[pxe-server]$ cp efi/syslinux.efi /var/lib/tftpboot/syslinux64.efi
[pxe-server]$ mkdir /var/lib/tftpboot/pxelinux.cfg
*Create a file named '/var/lib/tftpboot/pxelinux.cfg/default', copy
contents from below - adjust IPs accordingly.


##Repo server setup
*Install CentOS 7.4
*Set static IP (192.168.1.10)
[repo-server]$ yum install httpd -y
[repo-server]$ systemctl start httpd
*Copy CentOS ISO file to ~
[repo-server]$ mkdir /mnt/iso
[repo-server]$ mount -t iso9660 -o loop
/home/username/CentOS-7-x86_64-Minimal-1708.iso /mnt/iso/
[repo-server]$ mkdir /var/www/html/centos74/
[repo-server]$ cp -R /mnt/iso/* /var/www/html/centos74/


##Try to PXE boot client
*Boot EFI client and Network boot
*Select menu option for CentOS-7.4
*Notice menu is reloaded and no files are being transferred


##Copy vmlinuz and initrd.img to PXE server's httpd folder
[pxe-server]# mkdir /var/www/html/centos74
[pxe-server]# cd /var/www/html/centos74
[pxe-server]# wget http://192.168.1.10/centos74/isolinux/vmlinuz
[pxe-server]# wget http://192.168.1.10/centos74/isolinux/initrd.img


##Retry PXE boot on client
*Boot EFI client and Network boot
*Select menu option for CentOS-7.4
*Note that files are transferring now




dhcpd.conf PXE section:
	...
	option arch code 93 = unsigned integer 16;
	if option arch = 00:09 {
		next-server 192.168.1.8;
		filename "syslinux64.efi";
	} elsif option arch = 00:07 {
		next-server 192.168.1.8;
		filename "syslinux64.efi";
	} else {
	next-server 192.168.1.13;
	filename "bios/lpxelinux.0";
	}
	...

	
UEFI PXE structure
  /var/lib/tftpboot/
	??? boot
	??? ??? initrd.img
	??? ??? vmlinuz
	??? bg-640x480.jpg
	??? ldlinux.e64
	??? ldlinux.elf
	??? libcom32.c32
	??? libcom32.elf
	??? libutil.c32
	??? menu.c32
	??? pxelinux.cfg
	??? ??? default
	??? sysdump.c32
	??? syslinux64.efi
	??? vesamenu.c32
	

pxelinux.cfg/default Contents
	UI vesamenu.c32
	PROMPT 0

	menu title Linux EFI PXE Boot Menu
	MENU RESOLUTION 640 480
	MENU BACKGROUND bg-640x480.jpg

	TIMEOUT 300
	TOTALTIMEOUT 3000

	label C74
	  MENU LABEL CentOS-7.4
	  MENU DEFAULT
	  KERNEL http://192.168.1.10/centos74/isolinux/vmlinuz
	  initrd http://192.168.1.10/centos74/isolinux/initrd.img



Nathan Wittie

On Fri, Apr 6, 2018 at 3:28 PM, Nathan.Wittie--- via Syslinux
<syslinux at zytor.com> wrote:
> Issue Statement:
> PXE booting from BIOS systems works fine on both 6.03 and 6.04-pre1. New
clients only support (u)EFI which results in the inability to load remote kernel
or initrd. Tested with both Syslinux 6.03 and 6.04-pre1. So far unable to boot
any (u)EFI clients unless both initrd and kernel are on PXE server.
>
> Using packet captures I see that the URI request for the remote file on the
repo mirror looks correct ( http://192.168.1.10/centos74/isolinux/vmlinuz ), but
the client is asking the PXE server for the files instead of the repo mirror.
The PXE server responds with a http 404 packet. User sees "File not
found" message on client. Duplicating the directory structure of the Linux
Repo Mirror in the HTTP directory of the PXE server results in successful file
transfers.
>
> --Expected Behavior--
> *Menu item set to retrieve files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"
> *Client looks for files at
"http://192.168.1.10/centos74/isolinux/vmlinuz"
> *Client gets files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"
>
> --Actual Behavior--
> *Menu item set to retrieve files from
"http://192.168.1.10/centos74/isolinux/vmlinuz"
> *Client looks for files at
"http://192.168.1.8/centos74/isolinux/vmlinuz"
> *Client gets files from
"http://192.168.1.8/centos74/isolinux/vmlinuz", if the directory
structure has been copied in /var/www/html/
This is certainly an issue however we need to try to separate this
into either a Syslinux issue or a firmware issue.

Did you happen to test HTTP versus TFTP?
> System used for initial testing
> *ESXi 5.5U3 VM Version 8 (EFI mode)
>         NIC - vmxnet3
At one point, I remember having a lot of issues with a VMHW v9 VM when
I was working on a multi-NIC issue and changed to doing most of
testing on VMHW v10 with VMware Workstation 12 or ESXi 6.5 however I
still tend to test with VMHW v8 on BIOS.
> *Dell R7425 (AMD Epyc based), does not offer a "BIOS" mode
>         NIC - Broadcom 57412 (10G SFP+), FW 20.06.05.11
>         BIOS - 1.1.3
>
> Example Systems:
> *dhcpd Server - 192.168.1.11
> *Internal Linux Repo Mirror - 192.168.1.10
> *Prod BIOS PXE (syslinux-6.03) - 192.168.1.13
> *Test UEFI PXE (syslinux-6.04-pre1) - 192.168.1.8
> *ESXi 5.5U3 VM Version 8 (EFI mode) - 192.168.1.120
Snipping off the instructions for now.

And thank you for so carefully documenting your test procedure.

-- 
-Gene

Dell - Internal Use - Confidential  
>This is certainly an issue however we need to try to separate this into
either a Syslinux issue or a firmware issue.
>Did you happen to test HTTP versus TFTP?

I set up and tried with tftp on a separate server. Still seeing the file request
packets go to the main PXE server instead of the 'remote repository'.
>At one point, I remember having a lot of issues with a VMHW v9 VM when I was
working on a multi-NIC issue and changed to doing most of testing on VMHW v10
with VMware Workstation 12 or ESXi 6.5 however I still tend to test with VMHW v8
on BIOS.

I upgraded the VM to v10, still no luck with HTTP or TFTP.

Thanks,
-Nathan