Richard W.M. Jones
2021-Mar-30 21:22 UTC
[Libguestfs] Consider to replace isoinfo by own code rather than by xorriso
On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote:> Hi, > > thank you for flying xorriso. :)) > > But as its author and looking at > > https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml > > i feel obliged to point out that using it as replacement for isoinfo > will increase the consumption of RAM and CPU cycles substantially. > > xorriso command -indev loads the whole directory tree of the ISO filesystem > with names and metadata. Depending on that tree this can be dozens of > megabytes and lots of insertion operations in a not much optimized tree > model."megabytes" may not be a problem - and in fact hasn't been in my testing. Could it grow unbounded?> On the other hand it is quite uncomplicated to implement an own ISO 9660 > PVD reader and to become independent of any external program for that > purpose. > > The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768). > Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1) > https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_edition_june_2019.pdf > or in HTML at > https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor > > E.g. the info missing in xorriso's output of -pvd_info is at offsets: > > 80 - 87 iso_volume_space_size > (unsigned 32 bit, little endian first, then again as big endian) > > 124 - 127 iso_volume_sequence_number > (unsigned 16 bit, little endian first, then again as big endian) > > 128 - 131 iso_logical_block_size > (unsigned 16 bit, little endian first, then again as big endian) > (If you don't read 2048 here, then you are in unchartered > territory.) > > The date format for e.g. iso_volume_creation_t at offset 813 - 829 > is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte > value for the time zone with 15 minutes granularity. ("hh" means > hundredths of seconds.) > See also > https://wiki.osdev.org/ISO_9660#Numerical_formats > https://wiki.osdev.org/ISO_9660#Date.2Ftime_formatOK, thanks. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
Thomas Schmitt
2021-Mar-30 22:47 UTC
[Libguestfs] Consider to replace isoinfo by own code rather than by xorriso
Hi, Richard W.M. Jones wrote:> "megabytes" may not be a problem - and in fact hasn't been in my > testing. Could it grow unbounded?It depends on the number of files, the lengths of the file names, and the amount of some other meta data in the tree. So if malice is involved: Yes, it can become nasty. In a backup ISO with 78478 files with conservative names i see about 17 MB of directory tree (because the first data content starts at LBA 8613). The memory representation of the tree in libisofs underneath xorriso will need some extra bytes per file as overhead. ps -o rsz,vsz of the process shows the numbers 47880 67136 with a dynamically linked Debian binary of xorriso, and 73836 99844 with my statically linked development version of GNU xorriso. The run time of -indev with this ISO from RAM cache is about 0.5 seconds on a not really young 3.5 GHz Xeon system. Direct reading of the PVD would just need to load 2048 bytes from LBA 16 and to pick from byte fields at well documented locations the strings (with trailing blanks) and numbers (with both endianesses). Have a nice day :) Thomas
Jiri Kucera
2021-May-26 23:17 UTC
[Libguestfs] Consider to replace isoinfo by own code rather than by xorriso
Hi Rich, if you are still interested, you can try iso-info utility from libcdio package (distributed with RHEL 9). Jiri ----- Original Message -----> From: "Richard W.M. Jones" <rjones at redhat.com> > To: "Thomas Schmitt" <scdbackup at gmx.net> > Cc: libguestfs at redhat.com, "Jiri Kucera" <jkucera at redhat.com> > Sent: Tuesday, March 30, 2021 11:22:49 PM > Subject: Re: [Libguestfs] Consider to replace isoinfo by own code rather than by xorriso > > On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote: > > Hi, > > > > thank you for flying xorriso. :)) > > > > But as its author and looking at > > > > https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml > > > > i feel obliged to point out that using it as replacement for isoinfo > > will increase the consumption of RAM and CPU cycles substantially. > > > > xorriso command -indev loads the whole directory tree of the ISO filesystem > > with names and metadata. Depending on that tree this can be dozens of > > megabytes and lots of insertion operations in a not much optimized tree > > model. > > "megabytes" may not be a problem - and in fact hasn't been in my > testing. Could it grow unbounded? > > > On the other hand it is quite uncomplicated to implement an own ISO 9660 > > PVD reader and to become independent of any external program for that > > purpose. > > > > The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768). > > Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1) > > https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_edition_june_2019.pdf > > or in HTML at > > https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor > > > > E.g. the info missing in xorriso's output of -pvd_info is at offsets: > > > > 80 - 87 iso_volume_space_size > > (unsigned 32 bit, little endian first, then again as big > > endian) > > > > 124 - 127 iso_volume_sequence_number > > (unsigned 16 bit, little endian first, then again as big > > endian) > > > > 128 - 131 iso_logical_block_size > > (unsigned 16 bit, little endian first, then again as big > > endian) > > (If you don't read 2048 here, then you are in unchartered > > territory.) > > > > The date format for e.g. iso_volume_creation_t at offset 813 - 829 > > is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte > > value for the time zone with 15 minutes granularity. ("hh" means > > hundredths of seconds.) > > See also > > https://wiki.osdev.org/ISO_9660#Numerical_formats > > https://wiki.osdev.org/ISO_9660#Date.2Ftime_format > > OK, thanks. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > Fedora Windows cross-compiler. Compile Windows programs, test, and > build Windows installers. Over 100 libraries supported. > http://fedoraproject.org/wiki/MinGW >
Richard W.M. Jones
2021-May-27 10:04 UTC
[Libguestfs] Consider to replace isoinfo by own code rather than by xorriso
On Wed, May 26, 2021 at 07:17:52PM -0400, Jiri Kucera wrote:> Hi Rich, > > if you are still interested, you can try iso-info utility from > libcdio package (distributed with RHEL 9).In the end we went with parsing the PVD directly: https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml Rich.> Jiri > > ----- Original Message ----- > > From: "Richard W.M. Jones" <rjones at redhat.com> > > To: "Thomas Schmitt" <scdbackup at gmx.net> > > Cc: libguestfs at redhat.com, "Jiri Kucera" <jkucera at redhat.com> > > Sent: Tuesday, March 30, 2021 11:22:49 PM > > Subject: Re: [Libguestfs] Consider to replace isoinfo by own code rather than by xorriso > > > > On Tue, Mar 30, 2021 at 10:57:07PM +0200, Thomas Schmitt wrote: > > > Hi, > > > > > > thank you for flying xorriso. :)) > > > > > > But as its author and looking at > > > > > > https://github.com/libguestfs/libguestfs/blob/master/daemon/isoinfo.ml > > > > > > i feel obliged to point out that using it as replacement for isoinfo > > > will increase the consumption of RAM and CPU cycles substantially. > > > > > > xorriso command -indev loads the whole directory tree of the ISO filesystem > > > with names and metadata. Depending on that tree this can be dozens of > > > megabytes and lots of insertion operations in a not much optimized tree > > > model. > > > > "megabytes" may not be a problem - and in fact hasn't been in my > > testing. Could it grow unbounded? > > > > > On the other hand it is quite uncomplicated to implement an own ISO 9660 > > > PVD reader and to become independent of any external program for that > > > purpose. > > > > > > The Primary Volume Descriptor begins at LBA 16 (= byte offset 32768). > > > Its layout is described in ECMA-119 8.4 (note: BP = byte offset + 1) > > > https://www.ecma-international.org/wp-content/uploads/ECMA-119_4th_edition_june_2019.pdf > > > or in HTML at > > > https://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor > > > > > > E.g. the info missing in xorriso's output of -pvd_info is at offsets: > > > > > > 80 - 87 iso_volume_space_size > > > (unsigned 32 bit, little endian first, then again as big > > > endian) > > > > > > 124 - 127 iso_volume_sequence_number > > > (unsigned 16 bit, little endian first, then again as big > > > endian) > > > > > > 128 - 131 iso_logical_block_size > > > (unsigned 16 bit, little endian first, then again as big > > > endian) > > > (If you don't read 2048 here, then you are in unchartered > > > territory.) > > > > > > The date format for e.g. iso_volume_creation_t at offset 813 - 829 > > > is a decimal digit string YYYMMDDhhmmsshh with a trailing binary byte > > > value for the time zone with 15 minutes granularity. ("hh" means > > > hundredths of seconds.) > > > See also > > > https://wiki.osdev.org/ISO_9660#Numerical_formats > > > https://wiki.osdev.org/ISO_9660#Date.2Ftime_format > > > > OK, thanks. > > > > Rich. > > > > -- > > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > > Read my programming and virtualization blog: http://rwmj.wordpress.com > > Fedora Windows cross-compiler. Compile Windows programs, test, and > > build Windows installers. Over 100 libraries supported. > > http://fedoraproject.org/wiki/MinGW > >-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html