Syslinux - Aug 2017 - bootloader installation improvements

I have some concerns about the Syslinux boot process, which I'm now
investigating in connection with my work on booting Live-CD images from
USB flashdrives. Some of these are related to what seems to be
inadequate documentation, but I think that there are also aspects of
Syslinux operation which could be fairly easily improved.

quoting from the official documentation:

    Since version 5.00, when a SYSLINUX or EXTLINUX installer is used,
    the relevant boot sector will be modified and two files will be
    added to the "installation directory": the ldlinux.sys boot loader
    file, and an auxiliary ldlinux.c32 file. Note that these two files
    are not necessary so as to execute the installers; they are already
    embedded in the installers themselves.

http://www.syslinux.org/wiki/index.php?title=Library_modules#All_Syslinux_variants_need_an_additional_ldlinux_module

With regard to the "ldlinux.c32" file, this information seems to be
out-of-date. At least, the Debian <syslinux-common> package, which I'm
using, contains a file with that name, whose contents are identical to
the file which the EXTLINUX installer copies into the boot directory. As
far as I'm concerned, this is an improvement.

The semi-magical (as far as you would learn from the documentation)
"ldlinux.sys" file, however, is nowhere to be found, so presumably it
remains embedded inside {SYS,EXT}LINUX. Experiment shows that the two
installers create "ldlinux.sys" files which have the same length, but
differ in the values of two bytes. The difference increases, but remains
small, with the same file length, when one of the installers is applied
to a FAT32 filesystem with a different format. This suggest that the
content of "ldlinux.sys" is mostly static, with a few adjustable
parameters.

(If the only remaining difference between these utilities is that
SYSLINUX wants to work on an unmounted filesystem, while EXTLINUX wants
to work on a mounted filesystem, then why isn't one of them removed?)

It seems to me that having executable files that magically appear out of
nowhere, because they're embedded inside another executable file, is, in
general, a BAD IDEA, for reasons of flexibility and transparency, if
nothing else. It would be better to have this data stored in a separate
file, as "ldlinux.c32" now is. The small number of adjustable
parameters
could be handled by having the first sector of the file dedicated to
environment strings of the form "VARIABLE=VALUE", or something like
that, which would be much more transparent than the current method.

Oddly, the best explanation of the Syslinux boot process that I've been
able to find is from the Arch Linux documentation:

    * Boot process overview *

    Stage 1 : Part 1 - Load MBR - At boot, the BIOS loads the 440 byte
    MBR boot code at the start of the disk
    (/usr/lib/syslinux/bios/mbr.bin or
    /usr/lib/syslinux/bios/gptmbr.bin).

    Stage 1 : Part 2 - Search active partition. The Stage 1 MBR boot
    code looks for the partition that is marked as active (boot flag in
    MBR disks). Let us assume this is the /boot partition, for example.

    Stage 2 : Part 1 - Execute volume boot record - The Stage 1 MBR boot
    code executes the Volume Boot Record (VBR) of the /boot partition.
    In the case of Syslinux, the VBR boot code is the starting sector of
    /boot/syslinux/ldlinux.sys which is created by the extlinux
    --install command. Note that ldlinux.sys is not the same as
    ldlinux.c32.

    Stage 2 : Part 2 - Execute /boot/syslinux/ldlinux.sys - The VBR will
    load the rest of /boot/syslinux/ldlinux.sys. The sector location of
    /boot/syslinux/ldlinux.sys should not change, otherwise syslinux
    will not boot.

    Note: In the case of Btrfs, the above method will not work since
    files move around resulting in changing of the sector location of
    ldlinux.sys. Therefore, in Btrfs the entire ldlinux.sys code is
    embedded in the space following the VBR and is not installed at
    /boot/syslinux/ldlinux.sys unlike the case of other filesystems.

    Stage 3 - Load /boot/syslinux/ldlinux.c32 - The
    /boot/syslinux/ldlinux.sys will load the /boot/syslinux/ldlinux.c32
    (core module) that contains the rest of the core part of syslinux
    that could not be fit into ldlinux.sys (due to file-size
    constraints). The ldlinux.c32 file should be present in every
    Syslinux installation and should match the version of ldlinux.sys
    installed in the partition. Otherwise Syslinux will fail to boot.

[what do they mean by "file-size constraints"? what is the hard limit
on
the size of "ldlinux.sys"? is it an 8086 16-bit segment? oh, the
horror.]

    Stage 4 - Search and Load configuration file - Once Syslinux is
    fully loaded, it looks for /boot/syslinux/syslinux.cfg (or
    /boot/syslinux/extlinux.conf in some cases) and loads it if it is
    found. If no configuration file is found, you will be dropped to a
    Syslinux boot: prompt. This step and the rest of non-core parts of
    Syslinux (/boot/syslinux/*.c32 modules, excluding lib*.c32 and
    ldlinux.c32) require /boot/syslinux/lib*.c32 (library) modules to be
    present. The lib*.c32 library modules and non-core *.c32 modules
    should match the version of ldlinux.sys installed in the partition.

https://wiki.archlinux.org/index.php/syslinux#Boot_process_overview

For me, the most interesting part of that is this: "in Btrfs the entire
ldlinux.sys code is embedded in the space following the VBR and is not
installed at /boot/syslinux/ldlinux.sys unlike the case of other
filesystems." (Is this true? There doesn't seem to be any hint of it in
the official Syslinux documentation.)

That sounds like a much better arrangement. Data that will be accessed
by non-filesystem means, is better located outside of the filesystem
data area, exactly to avoid problems like files that cannot be relocated
(and must be contiguous?), or else everything will break.

Why is this not an option for FAT32, which must now be by far the most
common filesystem used with Syslinux? (Does anybody actually use floppy
disks anymore?) The standard "mkfs.fat" filesystem creation utility,
from the <dosfstools> project, offers the "-R" option, which
will
provide as many "reserved sectors" as are required, before the
filesystem data area. "ldlinux.sys" could easily be placed there,
contiguously, safe from the OS filesystem driver, which does not
understand that it cannot be touched.

http://manpages.org/mkfsfat/8

https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Layout

Alternatively, why is there not an option to install "ldlinux.sys" in
a
separate partition, which is what GRUB does with its bootloader? A USB
flashdrive could easily be formatted with two partitions, a small one
for "ldlinux.sys", and a much bigger one for the FAT32 or other
filesystem.

To summarize, it seems to me that these changes would make the Syslinux
installation process more reliable, transparent, and understandable (I
didn't, until I read the Arch Linux wiki!):

- transfer of "ldlinux.sys" into a separate file, with a text-based
  parameter block at some fixed location. This could then be edited, not
  only by {SYS,EXT}LINUX, but if necessary, with a binary sector editor,
  or even "sed".

- reconciliation of SYSLINUX and EXTLINUX into a single utility, which
  operates only on the non-data parts of the filesystem. "ldlinux.c32"
  can be copied into the filesystem with the other Syslinux modules, by
  the normal operating system methods. So can "ldlinux.sys", with the
  requirement that {SYS,EXT}LINUX be run AFTER that, to inform the
  Volume Boot Record where to find it. (This is how LILO works.)

- an option to install "ldlinux.sys" in the reserved sectors of a
FAT32
  filesystem, or the equivalent for other filesystems, as is apparently
  already done for BTRFS.

- an option to install "ldlinux.sys" into a separate partition, as
GRUB
  does with its first-stage bootloader.

Thanks for your consideration of these suggestions for the improvement
of Syslinux. I volunteer to write the parameter block parser, if that
would be helpful.


-- Ian Bruce

On Fri, Aug 4, 2017 at 12:06 PM, ian_bruce--- via Syslinux
<syslinux at zytor.com> wrote:
> I have some concerns about the Syslinux boot process, which I'm now
> investigating in connection with my work on booting Live-CD images from
> USB flashdrives. Some of these are related to what seems to be
> inadequate documentation, but I think that there are also aspects of
> Syslinux operation which could be fairly easily improved.
>
> quoting from the official documentation:
>
>     Since version 5.00, when a SYSLINUX or EXTLINUX installer is used,
>     the relevant boot sector will be modified and two files will be
>     added to the "installation directory": the ldlinux.sys boot
loader
>     file, and an auxiliary ldlinux.c32 file. Note that these two files
>     are not necessary so as to execute the installers; they are already
>     embedded in the installers themselves.
>
>
http://www.syslinux.org/wiki/index.php?title=Library_modules#All_Syslinux_variants_need_an_additional_ldlinux_module
I thought the syslinux/extlinux nomenclature was adopted for the
installer but SYSLINUX and EXTLINUX for the bootloader itself.
</pedant>
> With regard to the "ldlinux.c32" file, this information seems to
be
> out-of-date. At least, the Debian <syslinux-common> package, which
I'm
> using, contains a file with that name, whose contents are identical to
> the file which the EXTLINUX installer copies into the boot directory. As
> far as I'm concerned, this is an improvement.
It is expected that a package like this will include this file since
you won't be able to build an ISO with ISOLINUX without it.  It is
both embedded in the installers and available as a plain file.  The
syslinux and extlinux installers contain ldlinux.c32 to ensure that a
system is fully bootable with a matching ldlinux.c32 upon successful
completion of the installer execution.
> The semi-magical (as far as you would learn from the documentation)
> "ldlinux.sys" file, however, is nowhere to be found, so
presumably it
> remains embedded inside {SYS,EXT}LINUX. Experiment shows that the two
> installers create "ldlinux.sys" files which have the same length,
but
> differ in the values of two bytes. The difference increases, but remains
> small, with the same file length, when one of the installers is applied
> to a FAT32 filesystem with a different format. This suggest that the
> content of "ldlinux.sys" is mostly static, with a few adjustable
> parameters.
I fully expect ldlinux.sys to not be in a package since it serves no
value.  You'll find it in the prebuilt binary/source archives mostly
to allow an installer-only rebuild.

I fully expect at least two bytes and up to something in the direction
of a full kilobyte to change.  It's a block map and rudimentary
checksum.  It's in the source.
> (If the only remaining difference between these utilities is that
> SYSLINUX wants to  work on an unmounted filesystem, while EXTLINUX wants
> to work on a mounted filesystem, then why isn't one of them removed?)
Yes, that is the precise purpose.  syslinux ONLY works on unmounted
FAT* file systems while extlinux works on any supported mounted file
system.
> It seems to me that having executable files that magically appear out of
> nowhere, because they're embedded inside another executable file, is,
in
> general, a BAD IDEA, for reasons of flexibility and transparency, if
> nothing else. It would be better to have this data stored in a separate
> file, as "ldlinux.c32" now is. The small number of adjustable
parameters
> could be handled by having the first sector of the file dedicated to
> environment strings of the form "VARIABLE=VALUE", or something
like
> that, which would be much more transparent than the current method.
Think portability.  With just a single binary, it applies everything
that's necessary without the need for external include files akin to a
library.  At this time, syslinux and exlinux include ldlinux.sys and
ldlinux.c32.  Without having both, SYSLINUX would never do anything
useful aside from complain at a user.

See also above.  Perhaps reading the source is in order
> Oddly, the best explanation of the Syslinux boot process that I've been
> able to find is from the Arch Linux documentation:
>
>     * Boot process overview *
>
>     Stage 1 : Part 1 - Load MBR - At boot, the BIOS loads the 440 byte
>     MBR boot code at the start of the disk
>     (/usr/lib/syslinux/bios/mbr.bin or
>     /usr/lib/syslinux/bios/gptmbr.bin).
>
>     Stage 1 : Part 2 - Search active partition. The Stage 1 MBR boot
>     code looks for the partition that is marked as active (boot flag in
>     MBR disks). Let us assume this is the /boot partition, for example.
>
>     Stage 2 : Part 1 - Execute volume boot record - The Stage 1 MBR boot
>     code executes the Volume Boot Record (VBR) of the /boot partition.
>     In the case of Syslinux, the VBR boot code is the starting sector of
>     /boot/syslinux/ldlinux.sys which is created by the extlinux
>     --install command. Note that ldlinux.sys is not the same as
>     ldlinux.c32.
>
>     Stage 2 : Part 2 - Execute /boot/syslinux/ldlinux.sys - The VBR will
>     load the rest of /boot/syslinux/ldlinux.sys. The sector location of
>     /boot/syslinux/ldlinux.sys should not change, otherwise syslinux
>     will not boot.
>
>     Note: In the case of Btrfs, the above method will not work since
>     files move around resulting in changing of the sector location of
>     ldlinux.sys. Therefore, in Btrfs the entire ldlinux.sys code is
>     embedded in the space following the VBR and is not installed at
>     /boot/syslinux/ldlinux.sys unlike the case of other filesystems.
>
>     Stage 3 - Load /boot/syslinux/ldlinux.c32 - The
>     /boot/syslinux/ldlinux.sys will load the /boot/syslinux/ldlinux.c32
>     (core module) that contains the rest of the core part of syslinux
>     that could not be fit into ldlinux.sys (due to file-size
>     constraints). The ldlinux.c32 file should be present in every
>     Syslinux installation and should match the version of ldlinux.sys
>     installed in the partition. Otherwise Syslinux will fail to boot.
>
> [what do they mean by "file-size constraints"? what is the hard
limit on
> the size of "ldlinux.sys"? is it an 8086 16-bit segment? oh, the
> horror.]
Several years back when btrfs support was first implemented, there was
only 64k available to install ldlinux.sys, since it needs to be
ensured that the blocks don't move even through the late-stage changes
to apply the block map.
>     Stage 4 - Search and Load configuration file - Once Syslinux is
>     fully loaded, it looks for /boot/syslinux/syslinux.cfg (or
>     /boot/syslinux/extlinux.conf in some cases) and loads it if it is
>     found. If no configuration file is found, you will be dropped to a
>     Syslinux boot: prompt. This step and the rest of non-core parts of
>     Syslinux (/boot/syslinux/*.c32 modules, excluding lib*.c32 and
>     ldlinux.c32) require /boot/syslinux/lib*.c32 (library) modules to be
>     present. The lib*.c32 library modules and non-core *.c32 modules
>     should match the version of ldlinux.sys installed in the partition.
>
> https://wiki.archlinux.org/index.php/syslinux#Boot_process_overview
>
> For me, the most interesting part of that is this: "in Btrfs the
entire
> ldlinux.sys code is embedded in the space following the VBR and is not
> installed at /boot/syslinux/ldlinux.sys unlike the case of other
> filesystems." (Is this true? There doesn't seem to be any hint of
it in
> the official Syslinux documentation.)
100% true.  It's in the source.
> That sounds like a much better arrangement. Data that will be accessed
> by non-filesystem means, is better located outside of the filesystem
> data area, exactly to avoid problems like files that cannot be relocated
> (and must be contiguous?), or else everything will break.
Except btrfs was designed for this.  Other file systems don't have
better provisions for this.

ldlinux.sys can be discontiguous but in most installs, one contiguous
block.  See also above.
> Why is this not an option for FAT32, which must now be by far the most
> common filesystem used with Syslinux? (Does anybody actually use floppy
> disks anymore?) The standard "mkfs.fat" filesystem creation
utility,
> from the <dosfstools> project, offers the "-R" option,
which will
> provide as many "reserved sectors" as are required, before the
> filesystem data area. "ldlinux.sys" could easily be placed there,
> contiguously, safe from the OS filesystem driver, which does not
> understand that it cannot be touched.
See above.  Also, it can not be expected that we install to a pristine
file system prepared in the perfect manner with a optional parameter.
Eating 64k on a 160k floppy is unfeasible.
> http://manpages.org/mkfsfat/8
>
> https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#Layout
>
> Alternatively, why is there not an option to install
"ldlinux.sys" in a
> separate partition, which is what GRUB does with its bootloader? A USB
> flashdrive could easily be formatted with two partitions, a small one
> for "ldlinux.sys", and a much bigger one for the FAT32 or other
> filesystem.
multifs patch is pending and scheduled for 6.10.
> To summarize, it seems to me that these changes would make the Syslinux
> installation process more reliable, transparent, and understandable (I
> didn't, until I read the Arch Linux wiki!):
>
> - transfer of "ldlinux.sys" into a separate file, with a
text-based
>   parameter block at some fixed location. This could then be edited, not
>   only by {SYS,EXT}LINUX, but if necessary, with a binary sector editor,
>   or even "sed".
I'm not keen on this idea.
> - reconciliation of SYSLINUX and EXTLINUX into a single utility, which
>   operates only on the non-data parts of the filesystem.
"ldlinux.c32"
>   can be copied into the filesystem with the other Syslinux modules, by
>   the normal operating system methods. So can "ldlinux.sys", with
the
>   requirement that {SYS,EXT}LINUX be run AFTER that, to inform the
>   Volume Boot Record where to find it. (This is how LILO works.)
Two utilities for two purposes.  EXTLINUX (loader) was already killed
and its functionality merged to SYSLINUX.  Further changes would
potentially break existing scripting.
> - an option to install "ldlinux.sys" in the reserved sectors of a
FAT32
>   filesystem, or the equivalent for other filesystems, as is apparently
>   already done for BTRFS.
This MIGHT be worthwhile, if the reserved block is large enough.  Then
again, Windows might use it for things and using it might break
chainloading to an alternate boot from a stored VBR.
> - an option to install "ldlinux.sys" into a separate partition,
as GRUB
>   does with its first-stage bootloader.
>
> Thanks for your consideration of these suggestions for the improvement
> of Syslinux. I volunteer to write the parameter block parser, if that
> would be helpful.
>
>
> -- Ian Bruce
Hope this helps clarify things.

-- 
-Gene

On 08/06/17 04:41, Gene Cumm via Syslinux wrote:
>>
>> [what do they mean by "file-size constraints"? what is the
hard limit on
>> the size of "ldlinux.sys"? is it an 8086 16-bit segment? oh,
the
>> horror.]
> 
> Several years back when btrfs support was first implemented, there was
> only 64k available to install ldlinux.sys, since it needs to be
> ensured that the blocks don't move even through the late-stage changes
> to apply the block map.
> 
And even today there is a hard limit on btrfs, as well as on PXE (in
both cases a few hundred kilobytes.)

	-hpa