Syslinux - Apr 2017 - [PATCH 1/2] memdisk: Force ld output format to 32-bits

Hi there,

I've had issue with the integration of syslinux into the buildroot project
[0]. After some research, it turned out that we were wrongly using the host
toolchain instead of the target toolchain to build syslinux. This gave us
bigger issues (for instance) when trying to link syslinux with gnu-efi, that
was built using the target toolchain.

The first one forces ld to output i386 binary when processing the binary
files generated by nasm (for consistency with the other .o files), and the
second one explicitely uses the host toolchain to build the utilities under
utils/.

Those two patches turned out to be necessary in order to have a proper build.
please consider them for upstream inclusion.

There was an aditionnal third patch [1] I required, but that one isn't
authored by me. I would be grateful if you integrate it in your next release
as well.

Best Regards,
Ben.

[0] https://buildroot.org
[1]
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=846679;filename=syslinux_6.03%2Bdfsg-14.1.debdiff;msg=10

Beno?t Allard (2):
  memdisk: Force ld output format to 32-bits
  utils: Use the host toolchain to build.

 memdisk/Makefile | 2 +-
 utils/Makefile   | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)

-- 
2.1.4

On toolchains where the default output is x86_64, we need to be
consistent with the other .o files

Signed-off-by: Beno?t Allard <benoit.allard at greenbone.net>
---
 memdisk/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/memdisk/Makefile b/memdisk/Makefile
index 42e56e0..ccd5738 100644
--- a/memdisk/Makefile
+++ b/memdisk/Makefile
@@ -78,7 +78,7 @@ memdisk16.o: memdisk16.asm
 	$(NASM) -f bin $(NASMOPT) $(NFLAGS) $(NINCLUDE) -o $@ -l $*.lst $<
 
 memdisk_%.o: memdisk_%.bin
-	$(LD) -r -b binary -o $@ $<
+	$(LD) --oformat elf32-i386 -r -b binary -o $@ $<
 
 memdisk16.elf: $(OBJS16)
 	$(LD) -Ttext 0 -o $@ $^
-- 
2.1.4

The utilities are meant to run on the host machine, hence must be built using
the host toolchain.

Signed-off-by: Beno?t Allard <benoit.allard at greenbone.net>
---
 utils/Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/utils/Makefile b/utils/Makefile
index dfe6259..f31f117 100644
--- a/utils/Makefile
+++ b/utils/Makefile
@@ -17,8 +17,9 @@
 VPATH = $(SRC)
 include $(MAKEDIR)/syslinux.mk
 
-CFLAGS   = $(GCCWARN) -Os -fomit-frame-pointer -D_FILE_OFFSET_BITS=64 -I$(SRC)
-LDFLAGS  = -O2
+CC       = $(CC_FOR_BUILD)
+CFLAGS   = $(CFLAGS_FOR_BUILD) $(GCCWARN) -Os -fomit-frame-pointer
-D_FILE_OFFSET_BITS=64 -I$(SRC)
+LDFLAGS  = $(LDFLAGS_FOR_BUILD) -O2
 
 C_TARGETS	 = isohybrid gethostip memdiskfind
 SCRIPT_TARGETS	 = mkdiskimage
-- 
2.1.4

> Hi there,
> 
> I've had issue with the integration of syslinux into the buildroot
project
> [0]. After some research, it turned out that we were wrongly using the host
> toolchain instead of the target toolchain to build syslinux. This gave us
> bigger issues (for instance) when trying to link syslinux with gnu-efi,
that
> was built using the target toolchain.
> 
> The first one forces ld to output i386 binary when processing the binary
> files generated by nasm (for consistency with the other .o files), and the
> second one explicitely uses the host toolchain to build the utilities under
> utils/.
> 
> Those two patches turned out to be necessary in order to have a proper
build.
> please consider them for upstream inclusion.
> 
> There was an aditionnal third patch [1] I required, but that one isn't
> authored by me. I would be grateful if you integrate it in your next
release
> as well.
> 
> Best Regards,
> Ben.
> 
 
I'm not not sure the assumptions are always true (for every downstream 
use case / project / distro).

Expressions such as "The utilities are meant to run on the host 
machine, hence must be built using the host toolchain." might be 
relevant for certain use cases whereas not necessarily true for others.

What would happen if a user, while troubleshooting, wants to use a 
binary - just for the sake of simplicity, say we are talking about 
memdisk and/or the isohybrid tool - that was created on a different 
host than this user has/uses? We should want to be able to tell such 
user things like "try the pre-built official binaries included in the 
official distribution archives located at kernel.org".

In a similar (but not the same) line of thinking , there are Linux 
distributions that are no longer building packages for xi86, and so 
they don't build EFI binaries for UEFI IA32. I am sure that a PC with a 
UEFI IA32 firmware is not from the same era than a "686" PC, and they 
do not use such a similar microprocessor. UEFI IA32 binaries should 
still be built, but such distros don't do it anymore; IMHO, they still 
should.

We should be aware about the broad consequences (for others) when we 
assume that "my use-case is _the_ (only/best) case.

Regards,
Ady.

Replying to my own mail as I subscribe to the ML after the answer came,
and thus never received it in my mailbox.

On Wed,  5 Apr 2017 16:32:04 +0200
Beno?t Allard <benoit.allard at greenbone.net> wrote:
> Hi there,
> 
> I've had issue with the integration of syslinux into the buildroot
> project [0]. After some research, it turned out that we were wrongly
> using the host toolchain instead of the target toolchain to build
> syslinux. This gave us bigger issues (for instance) when trying to
> link syslinux with gnu-efi, that was built using the target toolchain.
> 
> The first one forces ld to output i386 binary when processing the
> binary files generated by nasm (for consistency with the other .o
> files), and the second one explicitely uses the host toolchain to
> build the utilities under utils/.
> 
> Those two patches turned out to be necessary in order to have a
> proper build. please consider them for upstream inclusion.
> 
> There was an aditionnal third patch [1] I required, but that one isn't
> authored by me. I would be grateful if you integrate it in your next
> release as well.
> 
> Best Regards,
> Ben.
> 
> [0] https://buildroot.org
> [1]
>
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=846679;filename=syslinux_6.03%2Bdfsg-14.1.debdiff;msg=10
> 
I identified three categories of binaries built when building
syslinux:

1. utilities required during the build (prepcore)
2. utilities meant to be run in the target operating-system (isohybrid,
  memdisk, the extlinux installer, ...)
3. utilities used to boot the operating system

1. and 2. can possibly run on very different systems (different libc for
instance), so at least there, a distinction has to be possible.

The toolchain used to build 3. has to be the same as the one you used
when buidling the dependencies (gnu-efi), which is most likely the one
of the target-system. While we have no issues with different libc there
(no dependency on them), the version of the tools used has to be the
same: you can't mix different ld version very well.

My first patch is actually unrelated to that problematic and fixes an
issue when building with a toolchain that defaults to 64-bits. In that
case, we have to force ld to convert the nasm binary format into
32-bits elf binaries (else it generates 64-bits elf binaries).

My second patch uses variables familiar to people using autoconf. I
agree that a fallback should be made if those variables are not
present. As it turned out, overriding the CC make variable is
error-prone, so in the meantime, I have a new version of that patch
that does it differently. I will send it once we agree on the need for
a fix in that direction.

I have a third patch that builds prepcore using the CC_FOR_BUILD
compiler as well. That one was forgotten in my first serie.

If you look inside the file ``mk/build.mk``, there is already a
comment that mention that the distinction between the build toolchain
and the host toolchain should be made. Can we at least agree that some
step forward have to be made there ?

Regards,
Ben.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL:
<http://www.zytor.com/pipermail/syslinux/attachments/20170410/9d31427a/attachment.sig>

> Replying to my own mail as I subscribe to the ML after the answer came,
> and thus never received it in my mailbox.
> 
 
(snip) 

 
> 
> Regards,
> Ben.
> 
 
Since you mentioned a prior reply, here is the link, just for 
reference:
 http://www.syslinux.org/archives/2017-April/025749.html 

Regarding the gnu-efi submodule, I recall past questions/suggestions, 
requesting to use the host version of gnu-efi when building Syslinux. 
The answer from Syslinux developers back then was that it could 
generate unpredictable behavior, difficult to replicate / track down.

On the other hand, there have been requests to update the gnu-efi 
submodule in Syslinux, so as to use an updated version / commit of it. 
I don't want to waste time reciting the reasons / circumstances, but 
this has not happened either (yet?); let's see some relevant reply to 
this email thread from a Syslinux developer.

FWIW, I referenced the same FTBFS issue in Debian a couple of months 
ago:
 "ldlinux.elf: Not enough room for program headers" 
 http://www.syslinux.org/archives/2017-February/025531.html 

with no reply, to this day.

Another FWIW, there is:
 http://www.syslinux.org/wiki/index.php/Building 

@Beno?t Allard,

IMHO, publicly posting up-to-date patches, explicitly and clearly 
expressing the commit / pre-release version being used as base, can be 
helpful at some point, even if Syslinux's developers don't reply 
immediately / on time. This might be even more important when we are 
talking about building-related patches.

TIA,
Ady.