Thomas Schmitt
2015-Feb-18 08:04 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
Hi, Ady wrote:> Let's not forget that the goal usually is to select CHS values so that > the resulting (optical and/or other) media would boot correctly in as > many systems as possibleAgreed. But we now have a report about failure of 255x63 on a popular pseudo-hardware, when the ISO is presented as DVD-ROM. (Probably there is no complaint when it gets presented as hard disk.) We do have statements by hpa that cylinder geometry can matter. But actually i am not aware of a particular example, from which we could judge whether 255x63 is essential for any popular hardware/firmware.> Now, if the proposed (temporal?) workaround is to use a different pair > of '-h' and '-s' values, I would suggest choosing them according to > commonly-used conventions or standards for CHS translations (e.g. > h={15,16,32,64,128,240,255}; s={16,32,63})Are there any standards ? We mainly have traditions in this aspect. C/H/S addressing is relying on an unstable concept which actually assumes that the software can detect the cylinder size from the storage device. Since decades those drum storages and early hard disks are gone. There is no cylinder any more. The number of physical read heads has decreased to at most a handful, if not a single one. Partition editors make daring assumptions from the comparison of LBA and CHS of the end addresses of partitions. If these assumptions turn out to be inconsistent, then e.g. fdisk issues complaints like: Partition 1 has different physical/logical beginnings (non-Linux?): Partition 1 has different physical/logical endings: This is the main reason, which i personaly know, why it is helpful to assume a particular CHS geometry and to aling the ISO image to a full cylinder size. I proposed -h 252 -s 63 because it yields the largest possible cylinder size which is divisible by 4 (i.e. the byte count is divisible by 512*4 = 2048). -h 240 -s 63 would fulfill this constraint, too. Size limits are 252*63*1024*512 = 8,323,596,288 resp. 240*63*1024*512 = 7,927,234,560 bytes. The workaround cannot be temporal, because the problem is fundamental. We have two candidates for a remedy: - Pad up to multiple of 2048 bytes (e.g. by truncate) - Use a cylinder size in bytes which is divisible by 2048. The pad-up method is equivalent to what a burn program would have to do when putting the ISO onto a CD or DVD. Have a nice day :) Thomas
anonym
2015-Feb-18 13:19 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
[I was dropped from cc in the initial response but I'd appreciate if you could keep it this time around since I'm not subscribed to this list. Thanks!] Ady wrote:> I do not know the exact complete procedure that the TAILS team is using > to build the original ISO image, and how it is transformed into an > isohybrid image (there are multiple tools for each, sometimes > combined).We end up generating the (non-hybrid) ISO image with something like this: genisoimage -J -l -cache-inodes -allow-multidot -A "The Amnesic Incognito Live System" -p "live-build 2.0.12-2; http://packages.qa.debian.org/live-build" -publisher "https://tails.boum.org/" -V "TAILS 1.2.2 - 20141212" -no-emul-boot -boot-load-size 4 -boot-info-table -r -b isolinux/isolinux.bin -c isolinux/boot.cat -o binary.iso binary Then we post-process it into a hybrid ISO image by running `isohybrid -h 255 -s 63 $IMAGE`.> If the isohybrid method (whichever tool you are using) is not applied > to your original ISO image, is such ISO image' size a multiple of 2048 > bytes?I would say yes. We haven't had this issue before re-introducing isohybrid in the mix. I say "re-introduced" because we used isohybrid years ago but then it was not problematic either. Back then we ran it with the `-entry 4 -type 1c` options only, so no -h or -s. Also, back then all images (no matter which compression we used) were definitely far below 1GiB in size, which may be relevant information for your later question.> What's the difference, in bytes, between the non-hybrid ISO image and > the isohybrid one? Could you please provide both exact sizes?Our current stable release [1], which wasn't a hybrid ISO, is 951764992 bytes. If I run `isohybrid -h 255 -s 63` it becomes 954132480 bytes which is a multiple of 1MiB (and hence 2048 bytes too). This seems to happen for all of our stable releases I tried, and they're all below 1GiB in size... [1] http://dl.amnesia.boum.org/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso> It is > possible that some "limit" (e.g. around the 1GiB) might be triggering > some condition (I am not saying that this is a bug in the isohybrid > tools, or that such case might trigger a problem; just assuming that > there might be some potential chance that some case might have not been > considered in the source code).... so this would explain a lot. I've actually only seen the problem with my development builds, for which I use a faster but less efficient compression for the included squashfs, so the images end up being around 1.2GiB in size. I haven't got a good sample size, but I'd say there seemed to be a 50% chance of isohybrid making the image size not divisible by 2048. If it's of any help, here are some examples of development builds's sizes before and after isohybrid post-processing: # Example 1 Before: 1263595520 After: 1266693120 which is divisible neither by 1024^2 nor 2048 # Example 2 Before: 1309853696 After: 1316044800 which is divisible by 2048 but not 1024^2 For the record, I had lot's of images that were similar to example 1's 1263595520 bytes in size, and they all ended up as 1266693120 bytes after isohybrid:ing them. Also note that the padding is 3097600 bytes and 6191104 bytes, respectively. Thomas Schmitt wrote:> I proposed -h 252 -s 63 because it yields the largest > possible cylinder size which is divisible by 4 (i.e. > the byte count is divisible by 512*4 = 2048). > -h 240 -s 63 would fulfill this constraint, too. > Size limits are 252*63*1024*512 = 8,323,596,288 > resp. 240*63*1024*512 = 7,927,234,560 bytes.Excuse my ignorance, but you say "largest possible cylinder size". We are not going to fill up to this theoretical maximum (our images are currently ~910 MiB for releases, and ~1.2 GiB for development builds) so I just want to double-check that what you propose also guarantees divisibility by 2048 for images that doesn't fill up to this maximum. Cheers!
Ady
2015-Feb-18 16:25 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
> I proposed -h 252 -s 63 because it yields the largest > possible cylinder size which is divisible by 4 (i.e. > the byte count is divisible by 512*4 = 2048). > -h 240 -s 63 would fulfill this constraint, too. > Size limits are 252*63*1024*512 = 8,323,596,288 > resp. 240*63*1024*512 = 7,927,234,560 bytes. > > > The workaround cannot be temporal, because the problem > is fundamental. > We have two candidates for a remedy: > - Pad up to multiple of 2048 bytes (e.g. by truncate) > - Use a cylinder size in bytes which is divisible by 2048. > > The pad-up method is equivalent to what a burn program > would have to do when putting the ISO onto a CD or DVD. > > > Have a nice day :) > > Thomas >The largest "'head' multiple of 4" would be as valid as using a different pair of values that would cover the total size of the ISO image. And, with "temporal" I meant that users can supply whatever they want (to the isohybrid tools included in Syslinux) as 'heads' and 'sectors_per_track' values without changing the source code (or without waiting for a new release). So, while choosing 255/63 has some intentional purpose, choosing 252/63 is as arbitrary as choosing, for instance, 240/63, or 128/32, with the difference that "252" does not coincide with any particular size limitation while "240" and "128" partially do (note that all these pairs would still cover the relevant size we are talking about in this example of TAILS, slightly over 1GiB). There are other potential advantages for smaller values, but successfully booting as many systems as possible should be the primary target. And yet, all this doesn't really matter, as different BIOS could assume different values, whether because they use different algorithms to select CHS values / translation methods, or because some bug (in the BIOS; I would be shocked :o :D). This is the reason for the recommended values (255, 63) for a size from 1GiB and up. The 'h' and 's' values are related to BIOS issues (CHS translations and size limitations). OTOH, the algorithm in the isohybrid tools being used to calculate the size of the resulting ISO image and the values for the boot code / partition table (to be inserted into the original ISO image by the isohybrid tool) are indeed related to partition editors (cylinder boundary), and to potential performance (and potential wear out) of storage media (e.g. so called "flash / pen drives" and SSDs). So let's attempt to list the minimal conditions, and the additional desired (preferred) conditions, for the algorithm: 1_ The resulting ISO9660 file system size shall be a multiple of 2048 (bytes_per_sector). 2_ The resulting size of the isohybrid image shall be a multiple of 2048 bytes. 3_ The resulting partition size should better match the end of a complete cylinder (the latter is the value to be calculated and optimized). 4_ The additional zero-padding tail (so to match the calculated complete cylinder and partition size) should be as minimal as possible. 5_ Let's not forget that the primary goal is still maximum compatibility with as many firmwares as possible (so users should choose the values of 'h' and 's' accordingly and wisely). 6_ The resulting size of the isohybrid image should better be a multiple of 1MiB. 7_ The starting absolute offset of the resulting partition should better be a multiple of 4KiB. If the amount of cylinders would be a multiple of 2048, we would achieve #1 and #2 (and then #3), but not always #4. One possibility to help with #4 would be to loop the expected calculations of cylinders and resulting image size (please avoid reading the following as real code or calculations; it is not): "cylsize = head * sector * 512*1" ... ; evaluate the expected resulting size (multiple of 2048 bytes, and complete cylinders for the partition) ; "cylsize = head * sector * 512*2" ... ; evaluate the expected resulting size (multiple of 2048 bytes, and complete cylinders for the partition) ; "cylsize = head * sector * 512*3" ... ; evaluate the expected resulting size (multiple of 2048 bytes, and complete cylinders for the partition) ; "cylsize = head * sector * 512*4" ... ; The above is not a precise and adequate pseudo-code (not even close), but my intention is just to present the general idea. My point is, if the first loop achieves the minimal (necessary) conditions, then the ISO image can be modified using such calculated values; but if not, then re-calculate with the next loop... until the calculations fulfill the expected minimal conditions and use the last calculated values to actually modify the original ISO image. BTW, let's not forget that applying the isohybrid tool to an already-isohybrid image will make a mess of the resulting image. Isohybrid tools should better inform the user and ask whether to continue anyway when the input image is not a "normal" (not isohybrid) ISO image. Regards, Ady.> _______________________________________________ > Syslinux mailing list > Submissions to Syslinux at zytor.com > Unsubscribe or set options at: > http://www.zytor.com/mailman/listinfo/syslinux >
Thomas Schmitt
2015-Feb-18 19:22 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
Hi, it would be interesting to know for what solution TAILS decides and whether any problems arise with it. ------------------------------------------------------- On with the fundamental discussion of MBR cylinders: Ady wrote:> 1_ The resulting ISO9660 file system size shall be a multiple of 2048 > (bytes_per_sector).The known ISO production programs comply to this condition. (Theoretically there could be other block sizes between 512 byte and 64 KiB. But 2048 is the address granularity of optical media and so no other block sizes have been seen yet.)> 2_ The resulting size of the isohybrid image shall be a multiple of > 2048 bytes. > 3_ The resulting partition size should better match the end of a > complete cylinder (the latter is the value to be calculated and > optimized).Would #3 allow that the ISO image file size is larger than the partition size ? Like the result of the truncate workaround ?> If the amount of cylinders would be a multiple of 2048,You mean the size of a single cylinder, i assume.> we would achieve #1 and #2 (and then #3),If the ISO image file shall end exactly at the partition end, then #2 and #3 go together if and only if the cylinder size is a multiple of 2048, which is equivalent to the product of -h and -s being divisible by 4.> but not always #4. > [...hopping back ...] > 4_ The additional zero-padding tail (so to match the calculated > complete cylinder and partition size) should be as minimal as possible.It is determined by the remaining space up to the next cylinder end. There are only 2 exp 14 possible cylinder sizes of wich only 2 exp 12 = 4096 are divisible by 2048. So this invites brute force. :)) I.e. try all cylinder sizes which are larger than or equal to (ISObytesize / 1024) and memorize the one with the least waste. (The loop(s) can end prematurely if waste 0 is found.)> 6_ The resulting size of the isohybrid image should better be a > multiple of 1MiB.I am not sure whether this can be deduced from the traditions. We only know that a cylinder size of exactly 1 MiB gives much hope for success. But i understand that TAILS ISO production uses -h 255 -s 63 because the ISO grew above 1024 MiB. So cylinder size 1 MiB is not suitable.> 7_ The starting absolute offset of the resulting partition should > better be a multiple of 4KiB.This one is new to me. What can possibly happen if it is not fulfilled ? Well, isohybrid MBR partition 1 has to begin at LBA 0 in order to make it mountable, unless the ISO was produced by xorriso with -partition_offset to create a suitable second superblock and file tree. The advisable -partition_offset is 16 * 2 KiB. So the 4 KiB condition is more or less always fulfilled for the first partition of an ISO, after isohybrid is done with it. But it is not fulfilled with partitions which emerge because options --gpt and/or --apm are used.> BTW, let's not forget that applying the isohybrid tool to an > already-isohybrid image will make a mess of the resulting image.Does it ? It should not. What are the symptoms of mess ?> Isohybrid tools should better inform the user and ask whether to > continue anyway when the input image is not a "normal" (not isohybrid) > ISO image.The only condition an ISO has to fulfill for isohybrid is that it bears as first El Torito boot image the isolinux.bin BIOS boot image from a SYSLINUX release which is modern enough to also offer the isohybrid tool. The tools check this and eventually complain: http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.in if ($ibsig ne "\xfb\xc0\x78\x70") { die "$0: $file: bootloader does not have a isolinux.bin hybrid signature.". "Note that isolinux-debug.bin does not support hybrid booting.\n"; } http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c if (memcmp(buf, "\xFB\xC0\x78\x70", 4)) errx(1, "%s: boot loader does not have an isolinux.bin hybrid " \ "signature. Note that isolinux-debug.bin does not support " \ "hybrid booting", argv[0]); isohybrid.c with combined options --gpt and --apm fails on xorriso produced ISOs, because of inappropriate assumptions about the structure of an El Torito boot catalog. See this thread: http://www.syslinux.org/archives/2014-February/021676.html Have a nice day :) Thomas
Thomas Schmitt
2015-Feb-18 20:05 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
Hi, anonym <anonym at riseup.net> wrote:> I've actually only seen the problem with my development builds, > or which I use a faster but less efficient > compression for the included squashfs, so the images end up being around > 1.2GiB in size. I haven't got a good sample size, but I'd say there > seemed to be a 50% chance of isohybrid making the image size not > divisible by 2048.It should rather be 75 % failure. Theoretically ... Since the size of the 255x63 cylinders is not divisible by 2048 but only by 512, the resulting image size is divisible by 2048 only if the number of cylinders is divisible by 4. So the failure comes and goes in cylinder size steps. 3 cylinders bad, fourth cylinder good again. In case of 255x63, the steps are 8,225,280 bytes each. I am reading in the original bug report https://labs.riseup.net/code/issues/8891> One can fix the image by running either of the following: > * isohybrid $ISO > * isohybrid $ISO -h 253 -s 63 > * isohybrid $ISO -h 255 -s 62The first one produces a cylinder size divisible by 2048. The second one obviously was lucky to cause a number of cylinders that was divisible by 4. The third one decreased the probability of failure from 75 percent to 50 percent, and was lucky to produce a number of cylinders which was divisible by 2.> Thomas Schmitt wrote: > > I proposed -h 252 -s 63 because it yields the largest > > possible cylinder size which is divisible by 4 [blocks of 512]> Excuse my ignorance, but you say "largest possible cylinder size". We > are not going to fill up to this theoretical maximum (our images are > currently ~910 MiB for releases, and ~1.2 GiB for development builds) so > I just want to double-check that what you propose also guarantees > divisibility by 2048 for images that doesn't fill up to this maximum.Yes, it guarantees byte size divisibility by 2048. That's because the ISO gets padded up to a full cylinder and each cylinder is divisible by 4 blocks = 2048 bytes. You can take into respect Ady's concerns and use -h 128 -s 32 which produces cylinders of 2 MiB size (and a maximum padding waste of 2 MiB - 2 KiB). That's enough for ISO images up to 2048 MiB. Funnily, 4 MiB is not a possible cylinder size. (One would have to add a factor of 2 to either -h or -s, but each of them would overflow. Since 2 is a prime, we cannot smear it over both options.) Have a nice day :) Thomas
Ady
2015-Feb-18 20:44 UTC
[syslinux] isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
> [I was dropped from cc in the initial response but I'd appreciate if you > could keep it this time around since I'm not subscribed to this list. > Thanks!]My guess is that the moderator allowed this email, but I do not know whether in addition your email address has been added to the mailing list. Is it? I can cc' your email address in this occasion (which I usually would avoid). I do not know whether other subscribers / senders will also do it. In the meantime, additional replies (that were already sent, and future ones) can be accessed by means of the public archive (see the links at the end of this email).> > Ady wrote: > > I do not know the exact complete procedure that the TAILS team is using > > to build the original ISO image, and how it is transformed into an > > isohybrid image (there are multiple tools for each, sometimes > > combined). > > We end up generating the (non-hybrid) ISO image with something like this: > > genisoimage -J -l -cache-inodes -allow-multidot -A "The Amnesic > Incognito Live System" -p "live-build 2.0.12-2; > http://packages.qa.debian.org/live-build" -publisher > "https://tails.boum.org/" -V "TAILS 1.2.2 - 20141212" -no-emul-boot > -boot-load-size 4 -boot-info-table -r -b isolinux/isolinux.bin -c > isolinux/boot.cat -o binary.iso binary > > Then we post-process it into a hybrid ISO image by running `isohybrid -h > 255 -s 63 $IMAGE`. >Please be aware that genisoimage is not really maintained, and that there are 2 "isohybrid" tools included in Syslinux: one is a Perl script (older, less recommended) and one program (built from isohybrid.c). Additionally, xorriso builds ISO images and has optional isohybrid options too (there is no "post-process"). There are other methods, but these are the relevant ones in this scenario. Considering the existence of alternative methods, it was important to clarify which of them we are talking about. Now we know :).> > If the isohybrid method (whichever tool you are using) is not applied > > to your original ISO image, is such ISO image' size a multiple of 2048 > > bytes? > > I would say yes. We haven't had this issue before re-introducing > isohybrid in the mix. I say "re-introduced" because we used isohybrid > years ago but then it was not problematic either. Back then we ran it > with the `-entry 4 -type 1c` options only, so no -h or -s. Also, back > then all images (no matter which compression we used) were definitely > far below 1GiB in size, which may be relevant information for your later > question. > > > What's the difference, in bytes, between the non-hybrid ISO image and > > the isohybrid one? Could you please provide both exact sizes? > > Our current stable release [1], which wasn't a hybrid ISO, is 951764992 > bytes. If I run `isohybrid -h 255 -s 63` it becomes 954132480 bytes > which is a multiple of 1MiB (and hence 2048 bytes too). This seems to > happen for all of our stable releases I tried, and they're all below > 1GiB in size...Hmm. FWIW... 951764992 / 2048 = 464729 -> correct 954132480 / (1024)^2 = 909.931... -> not a multiple of 1MiB. Perhaps you were using a 64/32 geometry when the resulting isohybrid image size was a multiple of 1MiB (which would make more sense).> > [1] > http://dl.amnesia.boum.org/tails/stable/tails-i386-1.2.3/tails-i386-1.2.3.iso > > > It is > > possible that some "limit" (e.g. around the 1GiB) might be triggering > > some condition (I am not saying that this is a bug in the isohybrid > > tools, or that such case might trigger a problem; just assuming that > > there might be some potential chance that some case might have not been > > considered in the source code). > > ... so this would explain a lot. I've actually only seen the problem > with my development builds, for which I use a faster but less efficient > compression for the included squashfs, so the images end up being around > 1.2GiB in size. I haven't got a good sample size, but I'd say there > seemed to be a 50% chance of isohybrid making the image size not > divisible by 2048.Generally speaking, I would say that the chances (as of version 6.03) of getting an isohybrid image size multiple of 2048 bytes are about %25 for the recommended values of '-h 255' and '-s 63' and size bigger than 1GiB. We should attempt to make it %100, even if this issue would be mostly relevant for virtual optical media (such as in VMs). There might be other important matters of influence for real HW (and we should try to help resolve those too), but they are not critical for _booting_.> > If it's of any help, here are some examples of development builds's > sizes before and after isohybrid post-processing: > > # Example 1 > Before: 1263595520 > After: 1266693120 which is divisible neither by 1024^2 nor 2048 > > # Example 2 > Before: 1309853696 > After: 1316044800 which is divisible by 2048 but not 1024^2 > > For the record, I had lot's of images that were similar to example 1's > 1263595520 bytes in size, and they all ended up as 1266693120 bytes > after isohybrid:ing them. Also note that the padding is 3097600 bytes > and 6191104 bytes, respectively. > > Thomas Schmitt wrote: > > I proposed -h 252 -s 63 because it yields the largest > > possible cylinder size which is divisible by 4 (i.e. > > the byte count is divisible by 512*4 = 2048). > > -h 240 -s 63 would fulfill this constraint, too. > > Size limits are 252*63*1024*512 = 8,323,596,288 > > resp. 240*63*1024*512 = 7,927,234,560 bytes. > > Excuse my ignorance, but you say "largest possible cylinder size". We > are not going to fill up to this theoretical maximum (our images are > currently ~910 MiB for releases, and ~1.2 GiB for development builds) so > I just want to double-check that what you propose also guarantees > divisibility by 2048 for images that doesn't fill up to this maximum.I think that reading (some of) the prior emails in this email thread (see the last link in this email for the mailing list archives for those that you might had missed) might help. In the meantime, for a _max_ size of: 1024 X 64 X 32 X 512 = 1GiB ( > 910MiB ) you can use '-h 64 -s 32', and most BIOS would correctly boot (whether using optical media or a flash drive). Of course, you cannot truthfully test your development images (>1GiB) with these same parameters, so this is not a solution for your case. To be clear, IM_H_O, the _user_ (in this case, TAILS' developers) padding with additional zeroes the resulting isohybrid image was a better _workaround_ than using arbitrary CHS geometries; emphasizing _user_ and _workaround_, not as a permanent solution in isohybrid.c. As I said, I think the isohybrid tools should (eventually) solve the situation anyway. Regards, Ady. PS: During the time I was writing this email, I see new emails related to this thread. I have not read them yet.> > Cheers! > _______________________________________________ > Syslinux mailing list > Submissions to Syslinux at zytor.com > Unsubscribe or set options at: > http://www.zytor.com/mailman/listinfo/syslinux >
Maybe Matching Threads
- isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
- isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
- isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
- isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox
- isohybrid and ISO images whose size is not a multiple of 2048 bytes vs. VirtualBox