Syslinux - Feb 2015 - isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox

Hi,

it would be interesting to know for what solution
TAILS decides and whether any problems arise with it.

-------------------------------------------------------
On with the fundamental discussion of MBR cylinders:

Ady wrote:
> 1_ The resulting ISO9660 file system size shall be a multiple of 2048 
> (bytes_per_sector).
The known ISO production programs comply to this condition.
(Theoretically there could be other block sizes between
 512 byte and 64 KiB. But 2048 is the address granularity
 of optical media and so no other block sizes have been seen
 yet.) 

> 2_ The resulting size of the isohybrid image shall be a multiple of 
> 2048 bytes.
> 3_ The resulting partition size should better match the end of a 
> complete cylinder (the latter is the value to be calculated and 
> optimized).
Would #3 allow that the ISO image file size is larger than
the partition size ? Like the result of the truncate
workaround ?

> If the amount of cylinders would be a multiple of 2048,
You mean the size of a single cylinder, i assume.
> we would achieve #1 and #2 (and then #3),
If the ISO image file shall end exactly at the partition
end, then #2 and #3 go together if and only if the cylinder
size is a multiple of 2048, which is equivalent to the
product of -h and -s being divisible by 4.

> but not always #4.
> [...hopping back ...]
> 4_ The additional zero-padding tail (so to match the calculated 
> complete cylinder and partition size) should be as minimal as possible.
It is determined by the remaining space up to the next
cylinder end. 

There are only 2 exp 14 possible cylinder sizes of wich
only 2 exp 12 = 4096 are divisible by 2048. So this invites
brute force. :)) 
I.e. try all cylinder sizes which are larger than or equal
to (ISObytesize / 1024) and memorize the one with the least
waste. (The loop(s) can end prematurely if waste 0 is
found.)

> 6_ The resulting size of the isohybrid image should better be a 
> multiple of 1MiB.
I am not sure whether this can be deduced from the
traditions. We only know that a cylinder size of exactly
1 MiB gives much hope for success.
But i understand that TAILS ISO production uses
-h 255 -s 63 because the ISO grew above 1024 MiB.
So cylinder size 1 MiB is not suitable.

> 7_ The starting absolute offset of the resulting partition should 
> better be a multiple of 4KiB.
This one is new to me.
What can possibly happen if it is not fulfilled ?

Well, isohybrid MBR partition 1 has to begin at LBA 0
in order to make it mountable, unless the ISO was
produced by xorriso with -partition_offset to create
a suitable second superblock and file tree.
The advisable -partition_offset is 16 * 2 KiB.

So the 4 KiB condition is more or less always fulfilled
for the first partition of an ISO, after isohybrid is
done with it.
But it is not fulfilled with partitions which emerge
because options --gpt and/or --apm are used.

> BTW, let's not forget that applying the isohybrid tool to an 
> already-isohybrid image will make a mess of the resulting image. 
Does it ? It should not.
What are the symptoms of mess ?

> Isohybrid tools should better inform the user and ask whether to 
> continue anyway when the input image is not a "normal" (not
isohybrid)
> ISO image.
The only condition an ISO has to fulfill for isohybrid
is that it bears as first El Torito boot image the
isolinux.bin BIOS boot image from a SYSLINUX release which
is modern enough to also offer the isohybrid tool.

The tools check this and eventually complain:

http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.in
  if ($ibsig ne "\xfb\xc0\x78\x70") {
      die "$0: $file: bootloader does not have a isolinux.bin hybrid
signature.".
          "Note that isolinux-debug.bin does not support hybrid
booting.\n";
  }

http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c
    if (memcmp(buf, "\xFB\xC0\x78\x70", 4))
        errx(1, "%s: boot loader does not have an isolinux.bin hybrid
" \
                 "signature. Note that isolinux-debug.bin does not support
" \
                 "hybrid booting", argv[0]);

isohybrid.c with combined options --gpt and --apm fails
on xorriso produced ISOs, because of inappropriate
assumptions about the structure of an El Torito boot
catalog. See this thread:
  http://www.syslinux.org/archives/2014-February/021676.html


Have a nice day :)

Thomas

> Hi,
> 
> it would be interesting to know for what solution
> TAILS decides and whether any problems arise with it.
> 
> -------------------------------------------------------
> On with the fundamental discussion of MBR cylinders:
> 
> Ady wrote:
> > 1_ The resulting ISO9660 file system size shall be a multiple of 2048 
> > (bytes_per_sector).
> 
> The known ISO production programs comply to this condition.
> (Theoretically there could be other block sizes between
>  512 byte and 64 KiB. But 2048 is the address granularity
>  of optical media and so no other block sizes have been seen
>  yet.) 
> 
> 
> > 2_ The resulting size of the isohybrid image shall be a multiple of 
> > 2048 bytes.
> > 3_ The resulting partition size should better match the end of a 
> > complete cylinder (the latter is the value to be calculated and 
> > optimized).
> 
> Would #3 allow that the ISO image file size is larger than
> the partition size ? Like the result of the truncate
> workaround ?
> 
> 
> > If the amount of cylinders would be a multiple of 2048,
> 
> You mean the size of a single cylinder, i assume.
> 
> > we would achieve #1 and #2 (and then #3),
> 
> If the ISO image file shall end exactly at the partition
> end, then #2 and #3 go together if and only if the cylinder
> size is a multiple of 2048, which is equivalent to the
> product of -h and -s being divisible by 4.
> 
> 
> > but not always #4.
> > [...hopping back ...]
> > 4_ The additional zero-padding tail (so to match the calculated 
> > complete cylinder and partition size) should be as minimal as
possible.
> 
> It is determined by the remaining space up to the next
> cylinder end. 
> 
> There are only 2 exp 14 possible cylinder sizes of wich
> only 2 exp 12 = 4096 are divisible by 2048. So this invites
> brute force. :)) 
> I.e. try all cylinder sizes which are larger than or equal
> to (ISObytesize / 1024) and memorize the one with the least
> waste. (The loop(s) can end prematurely if waste 0 is
> found.)
> 
> 
> > 6_ The resulting size of the isohybrid image should better be a 
> > multiple of 1MiB.
> 
> I am not sure whether this can be deduced from the
> traditions. We only know that a cylinder size of exactly
> 1 MiB gives much hope for success.
> But i understand that TAILS ISO production uses
> -h 255 -s 63 because the ISO grew above 1024 MiB.
> So cylinder size 1 MiB is not suitable.
> 
> 
> > 7_ The starting absolute offset of the resulting partition should 
> > better be a multiple of 4KiB.
> 
> This one is new to me.
> What can possibly happen if it is not fulfilled ?
> 
> Well, isohybrid MBR partition 1 has to begin at LBA 0
> in order to make it mountable, unless the ISO was
> produced by xorriso with -partition_offset to create
> a suitable second superblock and file tree.
> The advisable -partition_offset is 16 * 2 KiB.
> 
> So the 4 KiB condition is more or less always fulfilled
> for the first partition of an ISO, after isohybrid is
> done with it.
> But it is not fulfilled with partitions which emerge
> because options --gpt and/or --apm are used.
> 
> 
> > BTW, let's not forget that applying the isohybrid tool to an 
> > already-isohybrid image will make a mess of the resulting image. 
> 
> Does it ? It should not.
> What are the symptoms of mess ?
> 
> 
> > Isohybrid tools should better inform the user and ask whether to 
> > continue anyway when the input image is not a "normal" (not
isohybrid)
> > ISO image.
> 
> The only condition an ISO has to fulfill for isohybrid
> is that it bears as first El Torito boot image the
> isolinux.bin BIOS boot image from a SYSLINUX release which
> is modern enough to also offer the isohybrid tool.
> 
> The tools check this and eventually complain:
> 
>
http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.in
>   if ($ibsig ne "\xfb\xc0\x78\x70") {
>       die "$0: $file: bootloader does not have a isolinux.bin hybrid
signature.".
>           "Note that isolinux-debug.bin does not support hybrid
booting.\n";
>   }
> 
>
http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c
>     if (memcmp(buf, "\xFB\xC0\x78\x70", 4))
>         errx(1, "%s: boot loader does not have an isolinux.bin hybrid
" \
>                  "signature. Note that isolinux-debug.bin does not
support " \
>                  "hybrid booting", argv[0]);
> 
> isohybrid.c with combined options --gpt and --apm fails
> on xorriso produced ISOs, because of inappropriate
> assumptions about the structure of an El Torito boot
> catalog. See this thread:
>   http://www.syslinux.org/archives/2014-February/021676.html
> 
> 
> Have a nice day :)
> 
> Thomas
> 
 
Hi Thomas,

I could try to reply to all your comments -- whether my replies would 
actually help / solve / answer the issues, that's another question :) 
--, but I fear we would get slightly off-topic. Although it is all 
somewhat related to isohybrid, I'd like to focus first on improving the 
current algorithm in such a way that the resulting isohybrid image size 
will always be a multiple of 2048 bytes.

You previously suggested to:
 "Use a cylinder size in bytes which is divisible by 2048"
but that method could produce a size bigger than really necessary. 
Moreover, applying such calculation for smaller images (which might or 
might not use 255/63 geometry) would probably be considered 
unacceptable by some users.

So, instead of immediately making such a big jump (in the resulting 
size), I am suggesting to repeat (loop) the calculation (please do not 
interpret the following pseudo-code as actual functions / operations in 
C, it is not):

Cylinders  trunc(roundup( ISO_size / (Heads * Sectors_per_track * 512))

where 'ISO_size' is the size, in bytes, of the input ISO image.

 (Note: roundup(n.0) = n)

With this first-attempt value, calculate the potential resulting 
isohybrid image size. Is it a multiple of 2048 bytes?

If the potential resulting isohybrid image size is a multiple of 2048 
bytes, continue. If it is not, then add '1' to the potential Cylinders 
value.

Repeat the check for the new potential resulting isohybrid image size. 
If the input ISO image size was adequate (e.g was itself a multiple of 
2048 bytes), then this loop would be performed a maximum of 4 times.

The final value of Cylinders and the final size of the resulting 
isohybrid image would still need to comply with any other conditions 
already present in the isohybrid source code.

With this suggestion, I am not interfering with any additional checks 
or current conditions. My only intention with this suggestion is to 
make the resulting isohybrid image size a multiple of 2048 bytes, while 
minimizing the additional zero-padding (and respecting the '-h and
'-s'
parameters introduced by the user).

Now, let's assume we take your prior suggestions (instead of the 
above):

A_ "Pad up to multiple of 2048 bytes (e.g. by truncate)": this 
alternative might influence some other condition that is already 
present in the isohybrid tools.

B_ "Use a cylinder size in bytes which is divisible by 2048": this 
alternative would generate a bigger isohybrid image (comparing to my 
above suggestion) in many of the potential cases.
 

I hope I am making sense. I wish I could express my suggestion more 
accordingly to the actual code in isohybrid.c (or in the perl script). 
Since I lack the necessary knowledge, I am trying to present at least 
the corresponding logic.

If I am not mistaken, I guess the basic logic of my suggestion would 
translate to:

1_ Calculate a potential Cylinders value;
2_ Calculate 'Cylinders * Heads * Sectors_per_track';
3._ Is the result of the previous step #2 a multiple of '4'?
3.a_ If it is, continue;
3.b_ If it is not, then add '1' to the attempted Cylinders value and 
loop.
4_ Other conditions and logic already present in the isohybrid source 
code still apply (e.g. the Cylinders value for the partition table 
might be different than the one calculated after the above loop; the 
input ISO image size shall be a multiple of 2048 bytes...).

Regards,
Ady.
 
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux
>

> 
> 1_ Calculate a potential Cylinders value;
> 2_ Calculate 'Cylinders * Heads * Sectors_per_track';
> 3._ Is the result of the previous step #2 a multiple of '4'?
> 3.a_ If it is, continue;
> 3.b_ If it is not, then add '1' to the attempted Cylinders value
and
> loop.
> 4_ Other conditions and logic already present in the isohybrid source 
> code still apply (e.g. the Cylinders value for the partition table 
> might be different than the one calculated after the above loop; the 
> input ISO image size shall be a multiple of 2048 bytes...).
 
 
To be clear, the above pseudo-code is also inaccurate in the way I 
wrote it, but I assume you get the idea.
 
> 
> Regards,
> Ady.
>  
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux
>

Hi,

Ady wrote:
> I fear we would get slightly off-topic.
Yep. Since the TAILS ISO is supposed to not grow over 2 GiB
for now, it seems best to just use -h 128 -s 32 in order to
stay near to traditions.


-----------------------------------------------------------
Summary:

The alignment to cylinder size is a core feature of isohybrid.
If -h multiplied by -s is not divisible by 4, then ISO image
files may emerge from isohybrid which are not aligned to
2048 bytes. This is bad if the file shall be used as
virtual DVD-ROM.

The waste caused by a particular cylinder size depends on
the size of the ISO input image.
It is well possible to automatically determine a cylinder size
for minimum waste while keeping it divisible by 2048.
But it would probably produce non-traditional sizes.
(Whatever the importance of a traditional size is.)

It might be beneficial for the community if everybody tests
-h 252 -s 63 whether it suits as replacement of the advise
to use -h 255 -s 63 with large ISOs.

-----------------------------------------------------------

Have a nice day :)

Thomas