Hi!
Release ISO images located at
ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/ISO-IMAGES/
were generated with mkisofs until switch to makefs. For example,
ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/ISO-IMAGES/8.2/FreeBSD-8.2-RELEASE-amd64-livefs.iso
was generated with mkisofs and has correct RockRidge extended attributes.
# isoinfo -d -R -i FreeBSD-8.2-RELEASE-amd64-livefs.iso
System id: FreeBSD
Volume id: FreeBSD_LiveFS
Volume set id:
Publisher id: The FreeBSD Project. http://www.freebsd.org/
Data preparer id:
Application id: MKISOFS ISO 9660/HFS FILESYSTEM BUILDER & CDRECORD CD-R/DVD
CREATOR (C) 1993 E.YOUNGDALE (C) 1997 J.PEARSON/J.SCHILLING
Copyright File id:
Abstract File id:
Bibliographic File id:
Rock Ridge signatures version 1 found
Rock Ridge id 'RRIP_1991A'
Eltorito validation header:
Hid 1
Bootid 88 (bootable)
This image may be unrolled correctly with respect to hardlinks using xorriso
command from ports:
# xorriso -for_backup -load volid \* -indev
../FreeBSD-8.2-RELEASE-amd64-livefs.iso -osirrox on -- -extract / livefs
-rollback_end
Newer images (8.3 and later) were generated using makefs that seem to produce
incorrect RockRidge data:
# isoinfo -d -R -i FreeBSD-8.4-BETA1-amd64-livefs.iso | grep id
System id: NetBSD
Volume id: FREEBSD_LIVEFS
Volume set id:
Publisher id:
Data preparer id:
Application id:
Copyright File id:
Abstract File id:
Bibliographic File id:
Rock Ridge signatures version 1 found
Rock Ridge id 'IEEE_P1282'
Eltorito validation header:
Hid 1
Bootid 88 (bootable)
Same xorriso command produces tons of following error messages and unrolls the
image
without respect to hardlinkg increasing size in nearly 3 times:
libisofs: WARNING : Invalid TF entry
Caused by: Wrong or damaged RR entry
bsdtar from 8.3-STABLE shows lots of errors too, while extracting
FreeBSD-8.4-BETA1-amd64-livefs.iso
mdconfig breaks hardlinks too.
Is it possible to unroll this image respecting hardlinks?
Eugene Grosbein
On Apr 9, 2013, at 7:00 AM, Eugene Grosbein <egrosbein at rdtc.ru> wrote:> Hi! > > Release ISO images located at ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/ISO-IMAGES/ > were generated with mkisofs until switch to makefs. For example, > ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/ISO-IMAGES/8.2/FreeBSD-8.2-RELEASE-amd64-livefs.iso > was generated with mkisofs and has correct RockRidge extended attributes. > > # isoinfo -d -R -i FreeBSD-8.2-RELEASE-amd64-livefs.iso > System id: FreeBSD > Volume id: FreeBSD_LiveFS > Volume set id: > Publisher id: The FreeBSD Project. http://www.freebsd.org/ > Data preparer id: > Application id: MKISOFS ISO 9660/HFS FILESYSTEM BUILDER & CDRECORD CD-R/DVD CREATOR (C) 1993 E.YOUNGDALE (C) 1997 J.PEARSON/J.SCHILLING > Copyright File id: > Abstract File id: > Bibliographic File id: > Rock Ridge signatures version 1 found > Rock Ridge id 'RRIP_1991A' > Eltorito validation header: > Hid 1 > Bootid 88 (bootable) > > This image may be unrolled correctly with respect to hardlinks using xorriso command from ports: > > # xorriso -for_backup -load volid \* -indev ../FreeBSD-8.2-RELEASE-amd64-livefs.iso -osirrox on -- -extract / livefs -rollback_end > > Newer images (8.3 and later) were generated using makefs that seem to produce incorrect RockRidge data: > > # isoinfo -d -R -i FreeBSD-8.4-BETA1-amd64-livefs.iso | grep id > System id: NetBSD > Volume id: FREEBSD_LIVEFS > Volume set id: > Publisher id: > Data preparer id: > Application id: > Copyright File id: > Abstract File id: > Bibliographic File id: > Rock Ridge signatures version 1 found > Rock Ridge id 'IEEE_P1282' > Eltorito validation header: > Hid 1 > Bootid 88 (bootable) > > Same xorriso command produces tons of following error messages and unrolls the image > without respect to hardlinkg increasing size in nearly 3 times: > > libisofs: WARNING : Invalid TF entry > Caused by: Wrong or damaged RR entry > > bsdtar from 8.3-STABLE shows lots of errors too, while extracting FreeBSD-8.4-BETA1-amd64-livefs.iso > mdconfig breaks hardlinks too. > > Is it possible to unroll this image respecting hardlinks? > > Eugene Grosbein >While not the same you can always do this mdconfig -a -t vnode -f yourfreebsd-version.iso mount -t cd9660 /dev/md0 /cdrom Then use pax, cpio , cp, rsync etc to copy the data off the image . Also if memory serves me right libarchive may be able to unpack an iso much like a tar or cpio archive . Double check that . Also makefs was imported from netbsd I would see if netbsd's isos have the same issues . Hope this helps --- Mark saad | mark.saad at longcount.org
Hi, thanks for flying xorriso. :)) I am its developer and came to this thread by googling, not by being subscribed here. Sorry for not having a message-id by which i could attach this mail to the thread. -------------------------------------------------------------------->>> libisofs: WARNING : Invalid TF entryThe image FreeBSD-8.4-BETA1-amd64-livefs.iso indeed has a Rock Ridge flaw: The TF entries announce in their FLAGS byte by value 0x07 that there are three timestamps. Each is supposed to have 7 bytes. 5 bytes of entry header and 21 bytes of timestamps would sum up to 26. But the length in LEN_TF is 0x19 = 25. The next entry "NM" begins where the last byte of the last timestamp should be: 00014370 : 00 00 00 00 54 46 19 01 07 71 03 14 05 00 1d 00 T F q 00014380 : 71 03 14 03 0d 10 00 71 03 14 04 3b 1a 4e 4d 09 q q ; N M The missing byte would tell the time zone of POSIX atime. (I should curb the number of error messages.) -------------------------------------------------------------------->>> Is it possible to unroll this image respecting hardlinks?Theoretically yes. But xorriso will not do yet. The problem sits in the PX entries: Although the ER Extension Identifier is "IEEE_P1282" and thus indicates RRIP 1.12, the size of PX entries is 0x24 = 36 as with RRIP 1.10 rather than 44 as prescribed by RRIP 1.12. The missing bytes would host a 32 bit File Serial Number (inode number) (as usual with ISO 9660 recorded in both byte sexes). The PX entries of mkisofs have 44 bytes (although ER announces RRIP 1.10 by "RRIP_1991A") and thus have inode numbers. xorriso does restore hardlinks if enabled and if the PX entries contain File Serial Numbers. xorriso could try to deduce hardlink relations from the data content addresses. Same address and non-zero length would indicate hardlink relation between two file names. (Zero length files are often mapped to the same content address without being hardlinks.) -------------------------------------------------------------------- Related specs: SUSP 1.12 (entries CE , PD , SP , ST , ER , ES) ftp://ftp.ymi.com/pub/rockridge/susp112.ps RRIP 1.12 (entries PX , PN , SL , NM , CL , PL , RE , TF , SF) ftp://ftp.ymi.com/pub/rockridge/rrip112.ps ECMA-119 aka ISO 9660 http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf -------------------------------------------------------------------- If FreeBSD is looking for a ISO 9660 + Rock Ridge generator: xorriso does this too. Bootable via El Torito, MBR, GPT, APM. I once tested https://wiki.freebsd.org/AvgLiveCD with xorriso -as mkisofs ...proposed.options... -------------------------------------------------------------------- Have a nice day :) Thomas
Hi, Warren Block wrote:> sync will support hard links with -HBut how shall rsync know that the files in the ISO image stem from hardlink siblings on the hard disk where the image was produced ? -------------------------------------------------------------------- I succeeded with this post-processing on a Linux system with bash after xorriso -for backup ... -extract / livefs : Learn the block address of content of /rescue/cat in the ISO image: lba=$(xorriso -indev ../FreeBSD-8.4-BETA1-amd64-livefs.iso \ -find /rescue/cat -exec report_lba -- 2>/dev/null | \ fgrep cat | \ awk '{print $6}') (Should yield 35682 with that image) Get a list of all files with that block address, except /rescue/cat : siblings=$(xorriso -indev ../FreeBSD-8.4-BETA1-amd64-livefs.iso \ -find / -sort_lba -lba_range "$lba" 1 \ -exec report_lba -- \ 2>/dev/null | \ grep '^File data lba' | awk '{print $12}' | \ sed -e "s/'//g" | grep -v '^/rescue/cat$' ) (This should yield 137 files: /rescue/[ ... /rescue/zpool ) Delete them in livefs/ and re-create them as links to livefs/rescue/cat: for i in $siblings do rm livefs$i ln livefs/rescue/cat livefs$i done This brings the size of livefs/rescue from 676772 KiB to 4924 KiB. ls -l livefs/rescue reports: total 676768 -r-xr-xr-x 138 thomas thomas 5007184 2013-03-20 04:14 [ -r-xr-xr-x 138 thomas thomas 5007184 2013-03-20 04:14 atacontrol ... -r-xr-xr-x 138 thomas thomas 5007184 2013-03-20 04:14 zpool -------------------------------------------------------------------- (Confessedly one could get the file list as well by ls -l and grepping for the size of livefs/rescue/cat.) -------------------------------------------------------------------- Have a nice day :) Thomas
Hi,
i am not sure whether this is the correct source code version
of makefs, but here i can see the reason for the bad Rock Ridge
TF entries
http://svnweb.freebsd.org/base/stable/8/usr.sbin/makefs/cd9660/iso9660_rrip.c?revision=224835&view=markup
Line 688 has:
p->attr.rr_entry.TF.h.length[0] = 4;
This should be 5, because after the 4 bytes of generic SUSP header
there is one byte of flags. See typedef of ISO_RRIP_TF in
.../iso9660_rrip.h line 135.
-------------------------------------------------------------------
The reason for the missing hardlink info can be seen in line 648:
/* Ignoring the serial number for now */
The program would have to detect hardlink relations among files
which get copied from the local filesystem into the ISO image.
It would then have to mark hardlink siblings by giving them the
same serial number.
This would not show them as hardlinks in mounted images on FreeBSD
or Linux. But xorriso would respect them, if -hardlinks is set to
"on".
(Done by command -for_backup.)
-------------------------------------------------------------------
There is a small bug with makefs and the Expiration Time in the
Primary Volume Descriptor.
http://svnweb.freebsd.org/base/stable/8/usr.sbin/makefs/cd9660.c?revision=225249&view=markup
Line 687:
memset(diskStructure.primaryDescriptor.expiration_date, '0'
,17);
should be something like
memset(diskStructure.primaryDescriptor.expiration_date, '0'
,16);
((char *) diskStructure.primaryDescriptor.expiration_date)[16] = 0;
ECMA-119 (ISO 9660 for cheapos) 8.4.26.1:
"If all characters in RBP 1 to 16 of this field are the digit ZERO
and the number in RBP 17 is zero, it shall mean that the date and
time are not specified."
I just fixed a contrary bug in libisofs. {:)
-------------------------------------------------------------------
A year ago FreeBSD 8 had problems with large files in mounted ISO 9660
images and with mounting multi-session images:
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038549.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038552.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038563.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038566.html
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-April/038570.html
I wonder whether these have been fixed meanwhile.
-------------------------------------------------------------------
Have a nice day :)
Thomas