Syslinux - Feb 2015 - isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox

Hi,

Ady wrote:
> I am not so sure that using a different pair of '-h' and
'-s' values
> would be better than using 255/63.
But 255 * 63 is not divisible by 4 and thus can cause
trouble with virtual DVD-ROM.
> Most BIOS and most partition editors would assume this geometry.
I cannot tell for BIOSes, but partition editors have options
to set heads-per-cyl and sectors-per-head. At least fdisk
makes guesses from comparing LBA and CHS of the partition ends
under the assumption that each partition ends at a cylinder end.

> Additionally, what do you mean with "non-traditional sizes"? Or,
in
> other words, what would constitute a traditional size for the resulting 
> isohybrid image?
I mean non-traditional cylinder sizes.
Advised by hpa are 64x32 and 255x63, but i assume that he
was not aware of the problem of virtual DVD-ROMs not being
aligned to 2048-byte blocks.

So in the light of this thread it would be nice if the advise
for 255x63 could be changed to 252x63. Question is whether this
could cause any regressions with boot firmware. (Partition
editors should be happy with partition ends aligned to cylinder
ends.)

> I wonder whether my suggestion for the algorithm is inadequate
If i understand it right, then it shall look for a cylinder
size with minimal waste. That would be adequate but could produce
non-traditional cylinder sizes without the user being aware of
this. So i'd rather not advise this.

If i misunderstand your intention and you want to manipulate
the overall ISO image size, then this is inadequate because
it collides with the isohybrid feature to align the ISO image
end to the next cylinder end.
(Note well that only the image file is enlarged by isohybrid.  
 The filesystem size of the ISO stays as it was with the input
 ISO image file.)

The algorithm suffers from the wrong assumption that cylinder
size could be incremented in single block steps. It rather has
to be composed of the two factors -h {1...255} and -s {1...63}.
Some sizes are simply not possible that way. E.g. 4 MiB.
Cylinder size enumeration can be well done by two nested loops.
One for -h and one for -s.

> If different "preferred" conditions go against each other,
perhaps
> there should be some argument for the user to choose which conditions 
> are more relevant / important for his goal / needs?
But how to explain all this mess ? Even we two have problems
to get a common understanding.

The first problem is that none of the contraints discussed
here is covered by an official standard. ECMA-119 (aka ISO 9660)
simply bails out by stating that the first 32 KiB of an ISO
are designated for arbitrary system dependent data.

isohybrid chooses to put an MBR there, possibly a GPT,
and possibly an APM. All three are not designed to coexist
with the other two in the form they have:

MBR collides with APM. This is circumvented by special x86
machine commands at the start of the MBR, which cause no
side effect if executed by an x86 processor, but also look
like a Apple Partition Map Block 0 which announces block
size 2048 and a phony size of the overall image file.
It works for booting some Macs, but already Linux would
refuse to mount the HFS filesystem in the APM partition,
because Linux has block size 512 hardcoded for HFS.

GPT demands that the MBR must be a "Protective MBR".
I.e. it must show a single partition of type 0xee,
which reaches from LBA 1 to the end of the block range
that is covered by GPT.
isohybrid makes a MBR with first partition of type 0x17
and partition start at LBA 0, so that the partition is
mountable as ISO filesystem.
Further GPT/UEFI specs demand a backup GPT at the end of
the storage device. When isohybrid makes the ISO, it
cannot predict what size the final storage device will
have. So on the storage device, the backup GPT will
end up at the wrong position.

APM is not aware of MBR or GPT. Its Block 0 occupies
the same byte range as the MBR.
The isohybrid MBR exposes a minimal mock-up of an APM
Block 0 which seems to suffice for some Mac firmwares.
APM plus GPT needs an APM block size of at least 1024 to
leave room for the GPT header block at LBA 1.


It seems unfair and without much hope for success, to
put the decision on the user, who already is occupied with
the peculiarities of her/his booting operating system
and the storage devices from where it may come.

We need a simple, straight advise for the user.
Like:
"Use -h 64 -s 32 for small ISOs and -h 252 -s 63
 for ISOs over 1 GiB."

> I would tend to think that the resulting isohybrid image size should be 
> a multiple of 2048 bytes [...]
>  What currently-existing condition would not be achieved in such case?
The holy rule of cylinder alignment.

Of course this would be no show stopper:
Debian distributed several ISOs which were not aligned
to cylinder size because of a xorriso bug. No complaints or
bug reports emerged from this. Only fdisk was unhappy.

I assume the TAILS ISO with -h 255 -s 63 behaves the same
after it was padded up by program truncate.


Have a nice day :)

Thomas

> Hi,
> 
> Ady wrote:
> > I am not so sure that using a different pair of '-h' and
'-s' values
> > would be better than using 255/63.
> 
> But 255 * 63 is not divisible by 4 and thus can cause
> trouble with virtual DVD-ROM.
 
And my suggestion to modify the _amount_ of Cylinders was intended to 
overcome the "by 4" (or rather, "by 2048") issue.
 
> 
> > Most BIOS and most partition editors would assume this geometry.
> 
> I cannot tell for BIOSes, but partition editors have options
> to set heads-per-cyl and sectors-per-head. At least fdisk
> makes guesses from comparing LBA and CHS of the partition ends
> under the assumption that each partition ends at a cylinder end.
 
Hence, the algorithm should try to achieve:
 partition_size = C*H*S*512
which is equivalent to not having partial cylinders.
> 
> 
> > Additionally, what do you mean with "non-traditional sizes"?
Or, in
> > other words, what would constitute a traditional size for the
resulting
> > isohybrid image?
> 
> I mean non-traditional cylinder sizes.
> Advised by hpa are 64x32 and 255x63, but i assume that he
> was not aware of the problem of virtual DVD-ROMs not being
> aligned to 2048-byte blocks.
 
My suggestion considers whichever pair of H*S (Cylinder _size_) values 
(which are command-line parameters), so there is no contradiction 
between my suggestion and recommended Cylinder _sizes_. My suggestion 
is about the _amount_ of Cylinders.
 
> 
> So in the light of this thread it would be nice if the advise
> for 255x63 could be changed to 252x63. Question is whether this
> could cause any regressions with boot firmware. (Partition
> editors should be happy with partition ends aligned to cylinder
> ends.)
 
No, the recommendation for an isohybrid image size bigger than 1GiB is 
still a geometry of '-h 255 -s 63'. These recommended values are not 
random, and are not a "decision" that can be changed. These values are
the ones that most BIOS assume, and most partition editors assume. 
Thus, changing the recommendation to 252/63 could potentially make 
things harder for some users. The fact that some users might be able to 
overcome the situation in certain circumstances doesn't mean it is 
desirable; it is certainly not.
 
> 
> 
> > I wonder whether my suggestion for the algorithm is inadequate
> 
> If i understand it right, then it shall look for a cylinder
> size with minimal waste. That would be adequate but could produce
> non-traditional cylinder sizes without the user being aware of
> this. So i'd rather not advise this.
 
Well, then we have a (mis)communication problem. My suggestion is about 
calculating (by means of a loop) a potential (and hopefully optimal) 
Cylinder _amount_, not Cylinder size.
 
> 
> If i misunderstand your intention and you want to manipulate
> the overall ISO image size, then this is inadequate because
> it collides with the isohybrid feature to align the ISO image
> end to the next cylinder end.
> (Note well that only the image file is enlarged by isohybrid.  
>  The filesystem size of the ISO stays as it was with the input
>  ISO image file.)
 
I am having a problem understanding the conflict. My suggestion 
calculates a Cylinder _amount_, and then uses this calculated value to 
obtain the resulting isohybrid image size. The same calculated Cylinder 
amount is used for the partition table, unless other already-present 
conditions are triggered. The resulting isohybrid image size is 
calculated on the basis of a complete Cylinder (e.g. n,255,63), so 
where is the conflict?
 
> 
> The algorithm suffers from the wrong assumption that cylinder
> size could be incremented in single block steps. It rather has
> to be composed of the two factors -h {1...255} and -s {1...63}.
> Some sizes are simply not possible that way. E.g. 4 MiB.
> Cylinder size enumeration can be well done by two nested loops.
> One for -h and one for -s.
 
Except that my suggestion was not about the Cylinder _size_, but about 
the Cylinder _amount_.

As we know:
 Sector_per_track: {1...63}
            Heads: {0...m}
        Cylinders: {0...n}
where:
 'm' depends on the CHS translation method, and in this context, m=254
 'n' depends on the CHS translation method, and in this context, n=1023

so the maximum _amount_ of Cylinders *for the partition table* is 
'n+1'; 1024 in this context.

My suggestion for the calculation of the Cylinders amount is used for 
the resulting size of the isohybrid image. Although the resulting size 
of the isohybrid image is based in this calculated _amount_ of 
Cylinders, the value of Cylinders to be inserted in the partition table 
might need to be set to the maximum allowed, 1023. This is an example 
of additional conditions that should not be modified by my suggestion, 
as they are still valid.
 
> 
> 
> > If different "preferred" conditions go against each other,
perhaps
> > there should be some argument for the user to choose which conditions 
> > are more relevant / important for his goal / needs?
> 
> But how to explain all this mess ? Even we two have problems
> to get a common understanding.
 
Hopefully I am clarifying my suggestion in this email:

1_ Calculate a potential Cylinders (amount) value;
2_ Calculate 'Cylinders * Heads * Sectors_per_track';
3._ Is the result of the calculation a multiple of '4'?
3.a_ If it is, continue;
3.b_ If it is not, then add '1' to the attempted Cylinders (amount) 
value and loop to step #2.
4_ Other conditions and logic already present in the isohybrid source 
code still apply (e.g. the Cylinders value could be too-high for the 
partition table, in which case the value for the partition table is 
truncated to the maximum allowed value; the input ISO image size shall 
be a multiple of 2048 bytes...).
 
 (snip) 
 
> 
> 
> It seems unfair and without much hope for success, to
> put the decision on the user, who already is occupied with
> the peculiarities of her/his booting operating system
> and the storage devices from where it may come.
The recommended values for '-h' '-s' are still the same and are
not
affected by my suggestion (that's the whole point of it); nothing 
changes regarding users' decisions.
 
> 
> We need a simple, straight advise for the user.
> Like:
> "Use -h 64 -s 32 for small ISOs and -h 252 -s 63
>  for ISOs over 1 GiB."
 
 
There are no changes comparing to what has been recommended until now, 
255/63 for isohybrid images over 1GiB.
 
> 
> 
> > I would tend to think that the resulting isohybrid image size should
be
> > a multiple of 2048 bytes [...]
> >  What currently-existing condition would not be achieved in such case?
> 
> The holy rule of cylinder alignment.
See my comments above about Cylinders amount, not Cylinder size.
 
> 
> Of course this would be no show stopper:
> Debian distributed several ISOs which were not aligned
> to cylinder size because of a xorriso bug. No complaints or
> bug reports emerged from this. Only fdisk was unhappy.
 
Users using partition editors with strange images _are_ unhappy. Users 
not being able to boot _are_ unhappy. I have read enough complaints and 
problems about booting and failing methods. Whether users understand 
the reasons or not, most of them move on, while announcing all sorts of 
expletive poetry to everyone except where it would make a positive 
difference.

If the isohybrid algorithm can reduce problems for (some) users, it 
should.
 
> 
> I assume the TAILS ISO with -h 255 -s 63 behaves the same
> after it was padded up by program truncate.
> 
> 
> Have a nice day :)
> 
> Thomas
> 
 
Regards,
Ady.
 
> _______________________________________________
> Syslinux mailing list
> Submissions to Syslinux at zytor.com
> Unsubscribe or set options at:
> http://www.zytor.com/mailman/listinfo/syslinux
>

Hi,

Ady wrote:
> my suggestion to modify the _amount_ of Cylinders was intended to 
> overcome the "by 4" (or rather, "by 2048") issue.
Now i get your intention. You want to align the
resulting ISO to 4 full cylinders in order to avoid
the need for a cylinder size divisible by 4.

This would work. But with -h 255 -s 63 it could waste
up to 32,899,072 bytes.

> Hopefully I am clarifying my suggestion in this email:
> [...] loop to step #2.
The algorithm would not have to loop for a solution
but could compute it directly:

  if (Heads * Sectors_per_track) is divisible by 4
     # Pad up to next full cylinder
     align_factor = 1
  else if (Heads * Sectors_per_track) is divisible by 2
     # Pad up to next full double cylinder 
     align_factor = 2
  else
     # Pad up to next full quatruple cylinder (nearly 32 MB)
     align_factor = 4

In isohybrid.c the resulting align_factor would be used
like this:

-    frac = isostat.st_size % cylsize;
-    padding = (frac > 0) ? cylsize - frac : 0;
+    align_cylsize = cylsize * align_factor;
+    frac = isostat.st_size % align_cylsize;
+    padding = (frac > 0) ? align_cylsize - frac : 0;

Similarly in isohybrid.in.
There is no risk that the larger padding would exceed
1024 cylinders as long as the unpadded image does not
exceed this limit.

Nice idea, i must say. Mathematically appealing.
But up to four cylinders of padding ... ?

If anybody really wants to use this, then i would
prepare a modified copy of isohybrid.c for testing.
(I'm not its author or maintainer. Somebody else would
have to bring it into Git.)


Have a nice day :)

Thomas