Syslinux - Feb 2015 - isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox

Hi,

Ady wrote:
> Let's not forget that the goal usually is to select CHS values so that 
> the resulting (optical and/or other) media would boot correctly in as 
> many systems as possible 
Agreed. But we now have a report about failure of 255x63
on a popular pseudo-hardware, when the ISO is presented
as DVD-ROM. (Probably there is no complaint when it gets
presented as hard disk.)

We do have statements by hpa that cylinder geometry can
matter. But actually i am not aware of a particular
example, from which we could judge whether 255x63 is
essential for any popular hardware/firmware.

> Now, if the proposed (temporal?) workaround is to use a different pair 
> of '-h' and '-s' values, I would suggest choosing them
according to
> commonly-used conventions or standards for CHS translations (e.g. 
> h={15,16,32,64,128,240,255}; s={16,32,63})
Are there any standards ? We mainly have traditions
in this aspect.

C/H/S addressing is relying on an unstable concept which
actually assumes that the software can detect the cylinder
size from the storage device. Since decades those drum
storages and early hard disks are gone. There is no cylinder
any more. The number of physical read heads has decreased
to at most a handful, if not a single one. 

Partition editors make daring assumptions from the
comparison of LBA and CHS of the end addresses of
partitions. If these assumptions turn out to be inconsistent,
then e.g. fdisk issues complaints like:

  Partition 1 has different physical/logical beginnings (non-Linux?):
  Partition 1 has different physical/logical endings:

This is the main reason, which i personaly know, why it is
helpful to assume a particular CHS geometry and to aling
the ISO image to a full cylinder size.

I proposed -h 252 -s 63 because it yields the largest
possible cylinder size which is divisible by 4 (i.e.
the byte count is divisible by 512*4 = 2048).
-h 240 -s 63 would fulfill this constraint, too.
Size limits are 252*63*1024*512 = 8,323,596,288
resp.           240*63*1024*512 = 7,927,234,560 bytes.


The workaround cannot be temporal, because the problem
is fundamental.
We have two candidates for a remedy:
- Pad up to multiple of 2048 bytes (e.g. by truncate)
- Use a cylinder size in bytes which is divisible by 2048.

The pad-up method is equivalent to what a burn program
would have to do when putting the ISO onto a CD or DVD.


Have a nice day :)

Thomas

[I was dropped from cc in the initial response but I'd appreciate if you
could keep it this time around since I'm not subscribed to this list.
Thanks!]

Ady wrote:
> I do not know the exact complete procedure that the TAILS team is using 
> to build the original ISO image, and how it is transformed into an 
> isohybrid image (there are multiple tools for each, sometimes 
> combined).
We end up generating the (non-hybrid) ISO image with something like this:

    genisoimage -J -l -cache-inodes -allow-multidot -A "The Amnesic
Incognito Live System" -p "live-build 2.0.12-2;
http://packages.qa.debian.org/live-build" -publisher
"https://tails.boum.org/" -V "TAILS 1.2.2 - 20141212"
-no-emul-boot
-boot-load-size 4 -boot-info-table -r -b isolinux/isolinux.bin -c
isolinux/boot.cat -o binary.iso binary

Then we post-process it into a hybrid ISO image by running `isohybrid -h
255 -s 63 $IMAGE`.
> If the isohybrid method (whichever tool you are using) is not applied 
> to your original ISO image, is such ISO image' size a multiple of 2048 
> bytes?
I would say yes. We haven't had this issue before re-introducing
isohybrid in the mix. I say "re-introduced" because we used isohybrid
years ago but then it was not problematic either. Back then we ran it
with the `-entry 4 -type 1c` options only, so no -h or -s. Also, back
then all images (no matter which compression we used) were definitely
far below 1GiB in size, which may be relevant information for your later
question.
> What's the difference, in bytes, between the non-hybrid ISO image and 
> the isohybrid one? Could you please provide both exact sizes?
Our current stable release [1], which wasn't a hybrid ISO, is 951764992
bytes. If I run `isohybrid -h 255 -s 63` it becomes 954132480 bytes
which is a multiple of 1MiB (and hence 2048 bytes too). This seems to
happen for all of our stable releases I tried, and they're all below
1GiB in size...

[1]
http://dl.amnesia.boum.org/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso
> It is 
> possible that some "limit" (e.g. around the 1GiB) might be
triggering
> some condition (I am not saying that this is a bug in the isohybrid 
> tools, or that such case might trigger a problem; just assuming that 
> there might be some potential chance that some case might have not been 
> considered in the source code).
... so this would explain a lot. I've actually only seen the problem
with my development builds, for which I use a faster but less efficient
compression for the included squashfs, so the images end up being around
1.2GiB in size. I haven't got a good sample size, but I'd say there
seemed to be a 50% chance of isohybrid making the image size not
divisible by 2048.

If it's of any help, here are some examples of development builds's
sizes before and after isohybrid post-processing:

# Example 1
Before: 1263595520
After:  1266693120 which is divisible neither by 1024^2 nor 2048

# Example 2
Before: 1309853696
After:  1316044800 which is divisible by 2048 but not 1024^2

For the record, I had lot's of images that were similar to example 1's
1263595520 bytes in size, and they all ended up as 1266693120 bytes
after isohybrid:ing them. Also note that the padding is 3097600 bytes
and 6191104 bytes, respectively.

Thomas Schmitt wrote:
> I proposed -h 252 -s 63 because it yields the largest
> possible cylinder size which is divisible by 4 (i.e.
> the byte count is divisible by 512*4 = 2048).
> -h 240 -s 63 would fulfill this constraint, too.
> Size limits are 252*63*1024*512 = 8,323,596,288
> resp.           240*63*1024*512 = 7,927,234,560 bytes.
Excuse my ignorance, but you say "largest possible cylinder size". We
are not going to fill up to this theoretical maximum (our images are
currently ~910 MiB for releases, and ~1.2 GiB for development builds) so
I just want to double-check that what you propose also guarantees
divisibility by 2048 for images that doesn't fill up to this maximum.

Cheers!

> I proposed -h 252 -s 63 because it yields the largest
> possible cylinder size which is divisible by 4 (i.e.
> the byte count is divisible by 512*4 = 2048).
> -h 240 -s 63 would fulfill this constraint, too.
> Size limits are 252*63*1024*512 = 8,323,596,288
> resp.           240*63*1024*512 = 7,927,234,560 bytes.
> 
> 
> The workaround cannot be temporal, because the problem
> is fundamental.
> We have two candidates for a remedy:
> - Pad up to multiple of 2048 bytes (e.g. by truncate)
> - Use a cylinder size in bytes which is divisible by 2048.
> 
> The pad-up method is equivalent to what a burn program
> would have to do when putting the ISO onto a CD or DVD.
> 
> 
> Have a nice day :)
> 
> Thomas
> 
 
The largest "'head' multiple of 4" would be as valid as using
a
different pair of values that would cover the total size of the ISO 
image. And, with "temporal" I meant that users can supply whatever
they
want (to the isohybrid tools included in Syslinux) as 'heads' and 
'sectors_per_track' values without changing the source code (or without 
waiting for a new release).

So, while choosing 255/63 has some intentional purpose, choosing 252/63 
is as arbitrary as choosing, for instance, 240/63, or 128/32, with the 
difference that "252" does not coincide with any particular size 
limitation while "240" and "128" partially do (note that all
these
pairs would still cover the relevant size we are talking about in this 
example of TAILS, slightly over 1GiB). There are other potential 
advantages for smaller values, but successfully booting as many systems 
as possible should be the primary target.

And yet, all this doesn't really matter, as different BIOS could assume 
different values, whether because they use different algorithms to 
select CHS values / translation methods, or because some bug (in the 
BIOS; I would be shocked :o :D). This is the reason for the recommended 
values (255, 63) for a size from 1GiB and up.

The 'h' and 's' values are related to BIOS issues (CHS
translations and
size limitations). OTOH, the algorithm in the isohybrid tools being 
used to calculate the size of the resulting ISO image and the values 
for the boot code / partition table (to be inserted into the original 
ISO image by the isohybrid tool) are indeed related to partition 
editors (cylinder boundary), and to potential performance (and 
potential wear out) of storage media (e.g. so called "flash / pen 
drives" and SSDs).

So let's attempt to list the minimal conditions, and the additional 
desired (preferred) conditions, for the algorithm:

1_ The resulting ISO9660 file system size shall be a multiple of 2048 
(bytes_per_sector).
2_ The resulting size of the isohybrid image shall be a multiple of 
2048 bytes.
3_ The resulting partition size should better match the end of a 
complete cylinder (the latter is the value to be calculated and 
optimized).
4_ The additional zero-padding tail (so to match the calculated 
complete cylinder and partition size) should be as minimal as possible.
5_ Let's not forget that the primary goal is still maximum 
compatibility with as many firmwares as possible (so users should 
choose the values of 'h' and 's' accordingly and wisely).
6_ The resulting size of the isohybrid image should better be a 
multiple of 1MiB.
7_ The starting absolute offset of the resulting partition should 
better be a multiple of 4KiB.

If the amount of cylinders would be a multiple of 2048, we would 
achieve #1 and #2 (and then #3), but not always #4. One possibility to 
help with #4 would be to loop the expected calculations of cylinders 
and resulting image size (please avoid reading the following as real 
code or calculations; it is not):

 "cylsize = head * sector * 512*1" ... ;
 evaluate the expected resulting size (multiple of 2048 bytes, and 
complete cylinders for the partition) ;
 "cylsize = head * sector * 512*2" ... ;
 evaluate the expected resulting size (multiple of 2048 bytes, and 
complete cylinders for the partition) ;
"cylsize = head * sector * 512*3" ... ;
 evaluate the expected resulting size (multiple of 2048 bytes, and 
complete cylinders for the partition) ;
"cylsize = head * sector * 512*4" ... ;

The above is not a precise and adequate pseudo-code (not even close), 
but my intention is just to present the general idea. My point is, if 
the first loop achieves the minimal (necessary) conditions, then the 
ISO image can be modified using such calculated values; but if not, 
then re-calculate with the next loop... until the calculations fulfill 
the expected minimal conditions and use the last calculated values to 
actually modify the original ISO image.

BTW, let's not forget that applying the isohybrid tool to an 
already-isohybrid image will make a mess of the resulting image. 
Isohybrid tools should better inform the user and ask whether to 
continue anyway when the input image is not a "normal" (not isohybrid)
ISO image.

Regards,
Ady. 
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux
>

Hi,

it would be interesting to know for what solution
TAILS decides and whether any problems arise with it.

-------------------------------------------------------
On with the fundamental discussion of MBR cylinders:

Ady wrote:
> 1_ The resulting ISO9660 file system size shall be a multiple of 2048 
> (bytes_per_sector).
The known ISO production programs comply to this condition.
(Theoretically there could be other block sizes between
 512 byte and 64 KiB. But 2048 is the address granularity
 of optical media and so no other block sizes have been seen
 yet.) 

> 2_ The resulting size of the isohybrid image shall be a multiple of 
> 2048 bytes.
> 3_ The resulting partition size should better match the end of a 
> complete cylinder (the latter is the value to be calculated and 
> optimized).
Would #3 allow that the ISO image file size is larger than
the partition size ? Like the result of the truncate
workaround ?

> If the amount of cylinders would be a multiple of 2048,
You mean the size of a single cylinder, i assume.
> we would achieve #1 and #2 (and then #3),
If the ISO image file shall end exactly at the partition
end, then #2 and #3 go together if and only if the cylinder
size is a multiple of 2048, which is equivalent to the
product of -h and -s being divisible by 4.

> but not always #4.
> [...hopping back ...]
> 4_ The additional zero-padding tail (so to match the calculated 
> complete cylinder and partition size) should be as minimal as possible.
It is determined by the remaining space up to the next
cylinder end. 

There are only 2 exp 14 possible cylinder sizes of wich
only 2 exp 12 = 4096 are divisible by 2048. So this invites
brute force. :)) 
I.e. try all cylinder sizes which are larger than or equal
to (ISObytesize / 1024) and memorize the one with the least
waste. (The loop(s) can end prematurely if waste 0 is
found.)

> 6_ The resulting size of the isohybrid image should better be a 
> multiple of 1MiB.
I am not sure whether this can be deduced from the
traditions. We only know that a cylinder size of exactly
1 MiB gives much hope for success.
But i understand that TAILS ISO production uses
-h 255 -s 63 because the ISO grew above 1024 MiB.
So cylinder size 1 MiB is not suitable.

> 7_ The starting absolute offset of the resulting partition should 
> better be a multiple of 4KiB.
This one is new to me.
What can possibly happen if it is not fulfilled ?

Well, isohybrid MBR partition 1 has to begin at LBA 0
in order to make it mountable, unless the ISO was
produced by xorriso with -partition_offset to create
a suitable second superblock and file tree.
The advisable -partition_offset is 16 * 2 KiB.

So the 4 KiB condition is more or less always fulfilled
for the first partition of an ISO, after isohybrid is
done with it.
But it is not fulfilled with partitions which emerge
because options --gpt and/or --apm are used.

> BTW, let's not forget that applying the isohybrid tool to an 
> already-isohybrid image will make a mess of the resulting image. 
Does it ? It should not.
What are the symptoms of mess ?

> Isohybrid tools should better inform the user and ask whether to 
> continue anyway when the input image is not a "normal" (not
isohybrid)
> ISO image.
The only condition an ISO has to fulfill for isohybrid
is that it bears as first El Torito boot image the
isolinux.bin BIOS boot image from a SYSLINUX release which
is modern enough to also offer the isohybrid tool.

The tools check this and eventually complain:

http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.in
  if ($ibsig ne "\xfb\xc0\x78\x70") {
      die "$0: $file: bootloader does not have a isolinux.bin hybrid
signature.".
          "Note that isolinux-debug.bin does not support hybrid
booting.\n";
  }

http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c
    if (memcmp(buf, "\xFB\xC0\x78\x70", 4))
        errx(1, "%s: boot loader does not have an isolinux.bin hybrid
" \
                 "signature. Note that isolinux-debug.bin does not support
" \
                 "hybrid booting", argv[0]);

isohybrid.c with combined options --gpt and --apm fails
on xorriso produced ISOs, because of inappropriate
assumptions about the structure of an El Torito boot
catalog. See this thread:
  http://www.syslinux.org/archives/2014-February/021676.html


Have a nice day :)

Thomas

Hi,

anonym <anonym at riseup.net> wrote:
> I've actually only seen the problem with my development builds, 
> or which I use a faster but less efficient
> compression for the included squashfs, so the images end up being around
> 1.2GiB in size. I haven't got a good sample size, but I'd say there
> seemed to be a 50% chance of isohybrid making the image size not
> divisible by 2048.
It should rather be 75 % failure. Theoretically ...

Since the size of the 255x63 cylinders is not divisible
by 2048 but only by 512, the resulting image size is
divisible by 2048 only if the number of cylinders is
divisible by 4.

So the failure comes and goes in cylinder size steps.
3 cylinders bad, fourth cylinder good again.
In case of 255x63, the steps are 8,225,280 bytes each.

I am reading in the original bug report
  https://labs.riseup.net/code/issues/8891
> One can fix the image by running either of the following:
>     * isohybrid $ISO
>     * isohybrid $ISO -h 253 -s 63
>     * isohybrid $ISO -h 255 -s 62
The first one produces a cylinder size divisible by 2048.
The second one obviously was lucky to cause a number of
cylinders that was divisible by 4.
The third one decreased the probability of failure from
75 percent to 50 percent, and was lucky to produce a number
of cylinders which was divisible by 2.

> Thomas Schmitt wrote:
> > I proposed -h 252 -s 63 because it yields the largest
> > possible cylinder size which is divisible by 4 [blocks of 512]
> Excuse my ignorance, but you say "largest possible cylinder
size". We
> are not going to fill up to this theoretical maximum (our images are
> currently ~910 MiB for releases, and ~1.2 GiB for development builds) so
> I just want to double-check that what you propose also guarantees
> divisibility by 2048 for images that doesn't fill up to this maximum.
Yes, it guarantees byte size divisibility by 2048.
That's because the ISO gets padded up to a full cylinder
and each cylinder is divisible by 4 blocks = 2048 bytes.

You can take into respect Ady's concerns and use

  -h 128 -s 32

which produces cylinders of 2 MiB size (and a maximum
padding waste of 2 MiB - 2 KiB).
That's enough for ISO images up to 2048 MiB.

Funnily, 4 MiB is not a possible cylinder size.
(One would have to add a factor of 2 to either -h or -s,
 but each of them would overflow. Since 2 is a prime,
 we cannot smear it over both options.)


Have a nice day :)

Thomas

> [I was dropped from cc in the initial response but I'd appreciate if
you
> could keep it this time around since I'm not subscribed to this list.
> Thanks!]
 
My guess is that the moderator allowed this email, but I do not know 
whether in addition your email address has been added to the mailing 
list. Is it?

I can cc' your email address in this occasion (which I usually would 
avoid). I do not know whether other subscribers / senders will also do 
it. In the meantime, additional replies (that were already sent, and 
future ones) can be accessed by means of the public archive (see the 
links at the end of this email).
 
> 
> Ady wrote:
> > I do not know the exact complete procedure that the TAILS team is
using
> > to build the original ISO image, and how it is transformed into an 
> > isohybrid image (there are multiple tools for each, sometimes 
> > combined).
> 
> We end up generating the (non-hybrid) ISO image with something like this:
> 
>     genisoimage -J -l -cache-inodes -allow-multidot -A "The Amnesic
> Incognito Live System" -p "live-build 2.0.12-2;
> http://packages.qa.debian.org/live-build" -publisher
> "https://tails.boum.org/" -V "TAILS 1.2.2 - 20141212"
-no-emul-boot
> -boot-load-size 4 -boot-info-table -r -b isolinux/isolinux.bin -c
> isolinux/boot.cat -o binary.iso binary
> 
> Then we post-process it into a hybrid ISO image by running `isohybrid -h
> 255 -s 63 $IMAGE`.
> 
 
Please be aware that genisoimage is not really maintained, and that 
there are 2 "isohybrid" tools included in Syslinux: one is a Perl 
script (older, less recommended) and one program (built from 
isohybrid.c). Additionally, xorriso builds ISO images and has optional 
isohybrid options too (there is no "post-process"). There are other 
methods, but these are the relevant ones in this scenario.

Considering the existence of alternative methods, it was important to 
clarify which of them we are talking about. Now we know :).
 
> > If the isohybrid method (whichever tool you are using) is not applied 
> > to your original ISO image, is such ISO image' size a multiple of
2048
> > bytes?
> 
> I would say yes. We haven't had this issue before re-introducing
> isohybrid in the mix. I say "re-introduced" because we used
isohybrid
> years ago but then it was not problematic either. Back then we ran it
> with the `-entry 4 -type 1c` options only, so no -h or -s. Also, back
> then all images (no matter which compression we used) were definitely
> far below 1GiB in size, which may be relevant information for your later
> question.
> 
> > What's the difference, in bytes, between the non-hybrid ISO image
and
> > the isohybrid one? Could you please provide both exact sizes?
> 
> Our current stable release [1], which wasn't a hybrid ISO, is 951764992
> bytes. If I run `isohybrid -h 255 -s 63` it becomes 954132480 bytes
> which is a multiple of 1MiB (and hence 2048 bytes too). This seems to
> happen for all of our stable releases I tried, and they're all below
> 1GiB in size...
 
Hmm. FWIW...
951764992 / 2048 = 464729 -> correct
954132480 / (1024)^2 = 909.931... -> not a multiple of 1MiB.
Perhaps you were using a 64/32 geometry when the resulting isohybrid 
image size was a multiple of 1MiB (which would make more sense).
 
> 
> [1]
>
http://dl.amnesia.boum.org/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso
> 
> > It is 
> > possible that some "limit" (e.g. around the 1GiB) might be
triggering
> > some condition (I am not saying that this is a bug in the isohybrid 
> > tools, or that such case might trigger a problem; just assuming that 
> > there might be some potential chance that some case might have not
been
> > considered in the source code).
> 
> ... so this would explain a lot. I've actually only seen the problem
> with my development builds, for which I use a faster but less efficient
> compression for the included squashfs, so the images end up being around
> 1.2GiB in size. I haven't got a good sample size, but I'd say there
> seemed to be a 50% chance of isohybrid making the image size not
> divisible by 2048.
 
Generally speaking, I would say that the chances (as of version 6.03) 
of getting an isohybrid image size multiple of 2048 bytes are about %25 
for the recommended values of '-h 255' and '-s 63' and size
bigger than
1GiB. We should attempt to make it %100, even if this issue would be 
mostly relevant for virtual optical media (such as in VMs). There might 
be other important matters of influence for real HW (and we should try 
to help resolve those too), but they are not critical for _booting_.
 
> 
> If it's of any help, here are some examples of development builds's
> sizes before and after isohybrid post-processing:
> 
> # Example 1
> Before: 1263595520
> After:  1266693120 which is divisible neither by 1024^2 nor 2048
> 
> # Example 2
> Before: 1309853696
> After:  1316044800 which is divisible by 2048 but not 1024^2
> 
> For the record, I had lot's of images that were similar to example
1's
> 1263595520 bytes in size, and they all ended up as 1266693120 bytes
> after isohybrid:ing them. Also note that the padding is 3097600 bytes
> and 6191104 bytes, respectively.
> 
> Thomas Schmitt wrote:
> > I proposed -h 252 -s 63 because it yields the largest
> > possible cylinder size which is divisible by 4 (i.e.
> > the byte count is divisible by 512*4 = 2048).
> > -h 240 -s 63 would fulfill this constraint, too.
> > Size limits are 252*63*1024*512 = 8,323,596,288
> > resp.           240*63*1024*512 = 7,927,234,560 bytes.
> 
> Excuse my ignorance, but you say "largest possible cylinder
size". We
> are not going to fill up to this theoretical maximum (our images are
> currently ~910 MiB for releases, and ~1.2 GiB for development builds) so
> I just want to double-check that what you propose also guarantees
> divisibility by 2048 for images that doesn't fill up to this maximum.
 
I think that reading (some of) the prior emails in this email thread 
(see the last link in this email for the mailing list archives for 
those that you might had missed) might help. In the meantime, for a 
_max_ size of:
 1024 X 64 X 32 X 512 = 1GiB  ( > 910MiB ) 
you can use '-h 64 -s 32', and most BIOS would correctly boot (whether 
using optical media or a flash drive). Of course, you cannot truthfully 
test your development images (>1GiB) with these same parameters, so 
this is not a solution for your case.

To be clear, IM_H_O, the _user_ (in this case, TAILS' developers) 
padding with additional zeroes the resulting isohybrid image was a 
better _workaround_ than using arbitrary CHS geometries; emphasizing 
_user_ and _workaround_, not as a permanent solution in isohybrid.c.

As I said, I think the isohybrid tools should (eventually) solve the 
situation anyway.

Regards,
Ady.

PS: During the time I was writing this email, I see new emails related 
to this thread. I have not read them yet.
 
> 
> Cheers!
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux
>