On Mon, 7 Mar 2016, Pete Batard via Syslinux wrote:> I could also say something about the ISOHybrid _HACK_, which clearly is not > something that was ever intended by the people who designed ISO-9660, and > that could "either work if you're lucky or fail if you're not". Looking at > the recent history of this mailing list, it does seem to me like ISOHybrid > can be a bit of a headache...Yes I agree it's not a perfect solution. The difference is that the headache in this case is for those tech savvy people creating the hybrid iso who should be able to figure it out so the non-tech savvy end users should not have a problem with it as long as used as intended. If the end users start to modify an iso without knowing what they are doing they could easily end up with problems. If Rufus can do it automatically for them then they are lucky, if not then they just broke something they were not supposed to change. I mean, trying to do something that was not intended by those that designed it can be a valid problem but not easy to solve cleanly. If you're already rebuilding the boot loader on the iso you could also replace all of syslinux with a known working version that you have all the matching modules for and only keep the config. (This is what Ferenc has suggested.) This might save you having to keep all possible versions of syslinux ever appeared in the wild and avoid the need to detect the version of syslinux on the iso. But can break with patched syslinux versions or if they have custom modules. (But then trying to use a substitute part from a release for this customised version would likely break too.) I'm not sure if the config file is more portable between syslinux versions than modules but I guess it would be so for this keeping fewer versions could be enough. I was also thinking about storing hashes of files that you have modules for and try to idenfify the version by comparing the hashes instead of trying to extract a version number from the binary, but this may be problematic as well because building with different compilers will result in different hashes and you don't know what was used by those created the iso. I don't see a way to solve this problem in a way that is fool-proof and cannot break as this is inherently fragile because you can't know what's on the iso. It can be anything, even containing custom modules or patched, unreleased versions. Also I'm surprised that isos converted this way still work and the software on them is not confused by having its files on a different filesystem loosing case sensitivity, access rights, links, etc. For simple live CDs running entirely from an initrd this could work but for anything more complex than that I wonder how it does not break. Regards, BALATON Zoltan
> On Mon, 7 Mar 2016, Pete Batard via Syslinux wrote: > > I could also say something about the ISOHybrid _HACK_, which clearly is not > > something that was ever intended by the people who designed ISO-9660, and > > that could "either work if you're lucky or fail if you're not". Looking at > > the recent history of this mailing list, it does seem to me like ISOHybrid > > can be a bit of a headache... > > Yes I agree it's not a perfect solution. The difference is that the > headache in this case is for those tech savvy people creating the hybrid > iso who should be able to figure it out so the non-tech savvy end users > should not have a problem with it as long as used as intended. If the end > users start to modify an iso without knowing what they are doing they > could easily end up with problems. If Rufus can do it automatically for > them then they are lucky, if not then they just broke something they were > not supposed to change. I mean, trying to do something that was not > intended by those that designed it can be a valid problem but not easy to > solve cleanly. > > If you're already rebuilding the boot loader on the iso you could also > replace all of syslinux with a known working version that you have all the > matching modules for and only keep the config. (This is what Ferenc has > suggested.) This might save you having to keep all possible versions of > syslinux ever appeared in the wild and avoid the need to detect the > version of syslinux on the iso. But can break with patched syslinux > versions or if they have custom modules. (But then trying to use a > substitute part from a release for this customised version would likely > break too.) I'm not sure if the config file is more portable between > syslinux versions than modules but I guess it would be so for this keeping > fewer versions could be enough. > > I was also thinking about storing hashes of files that you have modules > for and try to idenfify the version by comparing the hashes instead of > trying to extract a version number from the binary, but this may be > problematic as well because building with different compilers will result > in different hashes and you don't know what was used by those created the > iso. > > I don't see a way to solve this problem in a way that is fool-proof and > cannot break as this is inherently fragile because you can't know what's > on the iso. It can be anything, even containing custom modules or patched, > unreleased versions. > > Also I'm surprised that isos converted this way still work and the > software on them is not confused by having its files on a different > filesystem loosing case sensitivity, access rights, links, etc. For simple > live CDs running entirely from an initrd this could work but for anything > more complex than that I wonder how it does not break. > > Regards, > BALATON ZoltanThere are several different issues / discussions / matters mentioned here. This is not the first time I read about them. Sometimes, it might just seem a matter of opinion / taste / preference / needs. Although these points can be an interesting discussion for some participants, I am going to respectfully request to go back to the main point in this email thread: improving the version identification in Syslinux's binaries, including modules. Pete's experience is valuable for the main matter in question, but it is not the only one. While testing multiple "versions" of Syslinux for comparison (regressions, consistent behavior, etc.), it is not-so-difficult to mix binaries of different "versions". Developers tend to assume (regarding other people / users) a certain level of knowledge, skills... My experience with helping other users, distros and my own beta-testing as common user suggest that the need for a better versioning identification is not limited to release and pre-release (as others have suggested in this email thread). Unless there is a clear "standardize", methodical way to add a clear "version" identification to Syslinux's binaries (including for distros, with patches), I am not going to consider the matter really solved (other might not care, or might be pleased with less than that). I should know it, as I am the one actually encountering this type of problems while testing as _common user_ and helping solve the mismatch version problems. As I mentioned before, I have seen / read / helped users with mismatch version problems countless times already, for years now. Please, improve the versioning identification (not just for official upstream (pre-)prelease). Thank you for focusing on this matter. Regards, Ady.
With apologies to Ady, I think there are important points to be made from this conversation, one which being the merging of ldlinux.sys and isolinux.bin, which I would very much like to rally people on this list into seeing as beneficial. So I will reply to this e-mail. On 2016.03.08 03:10, BALATON Zoltan via Syslinux wrote:> The difference is that the > headache in this case is for those tech savvy people creating the hybrid > iso who should be able to figure it out so the non-tech savvy end users > should not have a problem with it as long as used as intended.You're trying to reframe the context in a direction that was never there. My context was always the one of tech savvy developer trying to create a bootable USB from an ISO (be it in a process that automates this creation or something else, with the goal of making the result easy to use for non teach-savvy users). So I don't think you can go around saying "Ah, but ISOHybrid is for this context and the ldlinux.sys replacement is for that context", when, in essence, those contexts are the same.> If you're already rebuilding the boot loader on the iso you could also > replace all of syslinux with a known working version that you have all > the matching modules for and only keep the config.I already talked about this.> But can break with patched syslinux > versions or if they have custom modules.Indeed.> (But then trying to use a > substitute part from a release for this customised version would likely > break too.)How so? I would argue that, on the contrary, it's very unlikely that people will modify the Syslinux core files, as opposed to tweaking a module that does something close to what they, but not quite. My experience so far has been that replacing 'ldlinux.sys' by the closest official Syslinux version always works and I haven't seen a single case, so far, where it didn't.> I was also thinking about storing hashes of files that you have modules > for and try to idenfify the version by comparing the hashes instead of > trying to extract a version number from the binary, but this may be > problematic as well because building with different compilers will > result in different hashes and you don't know what was used by those > created the iso.I thought about that too, and came to the same conclusion.> I don't see a way to solve this problem in a way that is fool-proof and > cannot breakI very much do. And I explained exactly what that was in the PS: at the end of my long e-mail. So, once again, IMO, the Syslinux project should merge 'isolinux.bin' and 'ldlinux.sys' into a single executable, that behaves differently (through parameter, or name check, or something else) whether launched from MBR or El-Torito. Then all an ISO->USB conversion process has to do is install in 'ldlinux.sys' mode and it will work, because you are guaranteed that it will be compatible with the modules on that ISO. My current understanding is that 'isolinux.bin' and 'ldlinux.sys' are being assembled from the same set of object files, so it should be doable. Merging 'isolinux.bin' and 'ldlinux.sys' into a dual executable is something that I would have liked to see happen as soon as it became apparent that a lot of people where using ISO images as means of creating UFDs. Of course I kind of understand why this wasn't considered, but I hope that bringing forward the inherent ISO -> USB conversion problems I highlighted, which may not be have been that obvious before, can help convince people on this list that this would be a worthwhile effort. Besides, it does look to me like people on this list like dealing with "dual" solutions like ISOHybrid, so I sure wouldn't mind if other "dual" solutions were also looked into...> Also I'm surprised that isos converted this way still work and the > software on them is not confused by having its files on a different > filesystem loosing case sensitivity, access rights, links, etc. For > simple live CDs running entirely from an initrd this could work but for > anything more complex than that I wonder how it does not break.Some distro developers (IMO rightfully) do not see ISOHybrid as the panacea of ISO -> USB conversion, and, being aware that their ISOs are going to be used on a FAT32 filesystem (especially in the context of UEFI conversion, which can become effortless in this fashion), are careful enough to weed out stuff like Rock Ridge symbolic links, access rights, or anything that is not so portable. Besides, live CDs are meant to be read-only in the first place, and there isn't much in there that really warrants access rights (especially when persistence is meant to be handled separately). So the only major potential issues we have are symbolic links and the media label (when initrd boot scripts are seeking the media using a mixed case or > 11 chars label, which FAT doesn't allow), but in most circumstances, the label will be specified in the config file boot options, which Rufus can and does modify accordingly. With regards to symlinks, I've been waiting for user reports to see if it was something I needed to address in Rufus (by trying to duplicate content - of course with the caveats related to doing this), but so far, there doesn't seem to be a live distro out there that critically relies on them. Most of the usage I have seen for symlinks so far are to have a /docs link on root that points to a lower level user documentation, or something similar. In other words, as surprising as it may seem, it is my understanding that most Linux distro maintainers are very conscious that what they put on an ISO may end up on a FAT filesystem, and will therefore try to ensure that it will work there. At least I know the Debian people are (and will fix issues related to copying to FAT32 when they arise [1]) Regards, /Pete [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=789600
On Tue, 8 Mar 2016, Ady via Syslinux wrote:> Although these points can be an interesting discussion for some > participants, I am going to respectfully request to go back to the main > point in this email thread: improving the version identification in > Syslinux's binaries, including modules.Yes, sorry for distracting the thread. I just wanted to understand the problem and got many unrelated questions along the way.> level of knowledge, skills... My experience with helping other users, > distros and my own beta-testing as common user suggest that the need > for a better versioning identification is not limited to release and > pre-release (as others have suggested in this email thread). > > Unless there is a clear "standardize", methodical way to add a clear > "version" identification to Syslinux's binaries (including for distros, > with patches), I am not going to consider the matter really solved > (other might not care, or might be pleased with less than that). I > should know it, as I am the one actually encountering this type of > problems while testing as _common user_ and helping solve the mismatch > version problems.I think the problem is probably that you are trying to use the relase version as interface version which it is not. (Think about library versioning where the software package has a high level version and the library itself has an interface version that should change whenever the API changes unrelated to the software release version.) We probably don't have an interface version for modules at the moment. I don't know how to solve this best but dynamic link libraries and Linux kernel modules might have solutions for this problem that could be considered. Also to identify every change, the lowest level would be a commit hash but that's probably too low level. It's still a problem though if a distro patches a syslinux version without changing the version which we can't control so this may not be completely solved to your definition. Regards, BALATON Zoltan
On Tue, 8 Mar 2016, Pete Batard via Syslinux wrote:> On 2016.03.08 03:10, BALATON Zoltan via Syslinux wrote: >> The difference is that the >> headache in this case is for those tech savvy people creating the hybrid >> iso who should be able to figure it out so the non-tech savvy end users >> should not have a problem with it as long as used as intended. > > You're trying to reframe the context in a direction that was never there. My > context was always the one of tech savvy developer trying to create a > bootable USB from an ISO (be it in a process that automates this creation or > something else, with the goal of making the result easy to use for non > teach-savvy users). So I don't think you can go around saying "Ah, but > ISOHybrid is for this context and the ldlinux.sys replacement is for that > context", when, in essence, those contexts are the same.I really don't think these are the same context. For isohybrid tech-savvy users are creating a bootable iso (from scratch or converting an exiting non-hybrid iso) with all of their brain power available to solve problems as they encounter it and (hopefully) understanding what they are doing with non-tech savvy end users having nothing to do. While for iso to usb conversion with Rufus non-tech savvy users rely on the expertise put in that software to do this conversion for them automatically. So Rufus should be as intelligent as a tech-savvy human to do this by itself in the general case. This seems to be a bit more difficult to me. I don't think hybrid isos are a better solution just easier to program than an automatic conversion because the problems to solve by software is not the same.>> (But then trying to use a >> substitute part from a release for this customised version would likely >> break too.) > > How so? I would argue that, on the contrary, it's very unlikely that people > will modify the Syslinux core files, as opposed to tweaking a module that > does something close to what they, but not quite. My experience so far has > been that replacing 'ldlinux.sys' by the closest official Syslinux version > always works and I haven't seen a single case, so far, where it didn't.You were lucky. There's nothing that would guarantee this in my understanding.> In other words, as surprising as it may seem, it is my understanding that > most Linux distro maintainers are very conscious that what they put on an ISO > may end up on a FAT filesystem, and will therefore try to ensure that it will > work there. At least I know the Debian people are (and will fix issues > related to copying to FAT32 when they arise [1])In this case the maintainers could just include the needed ldlinux.sys on the iso as well so it can be easily converted. This is the same as including it in isolinux.bin. Although it's probably easier to do this once in syslinux then have all the packagers pick it up with the next release than having all of them make a change independently. I'm not against this change and I don't have a word on what's included in syslinux so you don't have to convince me. Although as you said, with new machines having UEFI and missing CD/DVD drives this problem could resolve itself by USB boot images replacing ISOs as the preferred boot media in the near future. (Or even boot over http can become the most used way.) Regards, BALATON Zoltan
On 3/8/2016 07:23, Pete Batard via Syslinux wrote:> [...] the merging of ldlinux.sys and isolinux.bin, which I would very > much like to rally people on this list into seeing as beneficial. [...]My understanding is that isolinux.bin and ldlinux.bin are pretty much twins[1]. If I understand correctly, you'd like the latter to be tacked onto the former because people are too stupid to put both on the ISOs that they are too stupid to anticipate might have USB disks derived from. Is that a rough understanding of the cause that you are rallying for? - Shao [1] There are obviously differences, including that the former has extra optical-disc-boot goodness