Hey folks,
I'm reading up on gtar for tape archiving and it sounds kind of nasty and
not something I really want to rely on.
It looks like star from the schily tools is preferred. I'm using Centos
(and RHEL) 5.7 which seems to have star but not sdd.
Which leads me to believe that the Schily tools are maybe a bit
"rogue"
My basic requirement with what I'm doing is to use standard tools and
formats so that archives I write today can be readable in 10 years.
Is the use of Schily tools going to be contrary to my basic requirement?
Is that considered a risk for future readability?
thanks,
-Alan
--
?Don't eat anything you've ever seen advertised on TV?
- Michael Pollan, author of "In Defense of Food"
On Mon, Feb 6, 2012 at 12:04 PM, Alan McKay <alan.mckay at gmail.com> wrote:> Hey folks, > > I'm reading up on gtar for tape archiving and it sounds kind of nasty and > not something I really want to rely on. > > It looks like star from the schily tools is preferred. ?I'm using Centos > (and RHEL) 5.7 which seems to have star but not sdd.I don't think there is any such general consensus. Are you reading something that favors Solaris/*bsd over GNU based systems?> Which leads me to believe that the Schily tools are maybe a bit "rogue"I doubt if they are as well maintained in linux distros as the GNU tool set, particularly in terms of having recent fixes backported into the versions carried in enterprise distros.> My basic requirement with what I'm doing is to use standard tools and > formats so that archives I write today can be readable in 10 years.I've never had any doubts that current GNU tar would extract archives made with it 10+ years ago - in fact I'm fairly sure I've done that. Or that I'd be able to obtain a copy of it in the future.> Is the use of Schily tools going to be contrary to my basic requirement? > Is that considered a risk for future readability?It shouldn't matter if you don't use either of the version's extensions, and for archiving you probably don't need them. For example, star and GNUtar use very different concepts for incremental backups - star is sort of like dump and must work on filesystem boundaries where GNUtar's --listed incremental needs a file to hold state but will work on arbitrary directories and can span mount points. But for archiving, you probably only care about the maximum size of a file it can handle. -- Les Mikesell lesmikesell at gmail.com
> Hey folks, > > I'm reading up on gtar for tape archiving and it sounds kind of nasty and > not something I really want to rely on. > > It looks like star from the schily tools is preferred. I'm using Centos > (and RHEL) 5.7 which seems to have star but not sdd. > > Which leads me to believe that the Schily tools are maybe a bit "rogue" > > My basic requirement with what I'm doing is to use standard tools and > formats so that archives I write today can be readable in 10 years. > > Is the use of Schily tools going to be contrary to my basic requirement? > Is that considered a risk for future readability?Use star like so : star -c -p -acl -sparse -dump -fs=32m -fifostats -time \ -C /some_dir dir1 dir2 dir3 file=/dev/rmt/0mbn or something like that will work like a charm and be POSIX spec compliant. That means you can still read it in 20 years .. if the tape survives. dc -- -- http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x1D936C72FA35B44B +-------------------------+-----------------------------------+ | Dennis Clarke | Solaris and Linux and Open Source | | dclarke at blastwave.org | Respect for open standards. | +-------------------------+-----------------------------------+
> I doubt if they are as well maintained in linux distros as the GNU > tool set, particularly in terms of having recent fixes backported into > the versions carried in enterprise distros.They are updated pretty much every month.>> My basic requirement with what I'm doing is to use standard tools and >> formats so that archives I write today can be readable in 10 years. > > I've never had any doubts that current GNU tar would extract archives > made with it 10+ years ago - in fact I'm fairly sure I've done that. > Or that I'd be able to obtain a copy of it in the future.GNU tar .. has its own bugs. Is it really standards compliant?>> Is the use of Schily tools going to be contrary to my basic requirement? >> Is that considered a risk for future readability? > > It shouldn't matter if you don't use either of the version's > extensions, and for archiving you probably don't need them. For > example, star and GNUtar use very different concepts for incremental > backups - star is sort of like dump and must work on filesystem > boundaries where GNUtar's --listed incremental needs a file to hold > state but will work on arbitrary directories and can span mount > points.same sort of deal with star .. but you should go ask the author. dc -- -- http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x1D936C72FA35B44B +-------------------------+-----------------------------------+ | Dennis Clarke | Solaris and Linux and Open Source | | dclarke at blastwave.org | Respect for open standards. | +-------------------------+-----------------------------------+
> On Mon, Feb 6, 2012 at 12:48 PM, Dennis Clarke <dclarke at blastwave.org> wrote: >> >>> I doubt if they are as well maintained in linux distros as the GNU >>> tool set, particularly in terms of having recent fixes backported into >>> the versions carried in enterprise distros. >> >> They are updated pretty much every month. > > My experience is very old (probably w/CentOS 3.x), but when I found > problems with star they were fixed in its own source but I gave up > before the packaged version was updated. Maybe that has changed...It is pretty up to date : ftp://ftp.berlios.de/pub/schily/AN-2012-01-23 Course that includes a pack of goodies. Like a real cdrecord that really works with blue ray etc etc. dc -- -- http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x1D936C72FA35B44B +-------------------------+-----------------------------------+ | Dennis Clarke | Solaris and Linux and Open Source | | dclarke at blastwave.org | Respect for open standards. | +-------------------------+-----------------------------------+
Let me comment some questions in one single mail:> My basic requirement with what I'm doing is to use standard tools and > formats so that archives I write today can be readable in 10 years.Star becomes 30 in 4 months, any archive created since it's early beginning in summer 1982 can still be read back.> I don't think there is any such general consensus. Are you reading > something that favors Solaris/*bsd over GNU based systems?Schily tools (and in special star) implement support for Linux specific extensions. This is what you do not get from gtar at all. So why do Linux distros prefer gtar even though there is no Linux support?> I doubt if they are as well maintained in linux distros as the GNU > tool set, particularly in terms of having recent fixes backported into > the versions carried in enterprise distros.gtar still did not fix bugs I reported in 1993 (e.g. the bug that causes gtar to complain with "skipping to next header" even on it's own archives). I am thus sure that star not not worse than gtar....> It shouldn't matter if you don't use either of the version's > extensions, and for archiving you probably don't need them. For > example, star and GNUtar use very different concepts for incremental > backups - star is sort of like dump and must work on filesystem > boundaries where GNUtar's --listed incremental needs a file to hold > state but will work on arbitrary directories and can span mount > points. But for archiving, you probably only care about the maximum > size of a file it can handle.When I implemented incremental restores for star in September 2004, I wrote a simple script for a incremental testcase and tested the deltas with ufsdump/ufsrestore, gtar and the star version at that time. Gtar was unable to deal with my testcase, so I stopped testing it any further. If you like to discuss incrementals, you definitely need to discuss behavior at restore time and restoring incrementals definitely does not work correclty with gtar if you renamed directories. Star is used to do incremental backups/restores on a dayly base in Berlios since September 2004. Since Spring 2005, not a single problem was seen, so there are more that 2500 successful incremental restores that verify no problem even under stange conditions.> I don't think so - I'm fairly sure I've seen GNUtar complain about bad > headers, say 'skipping to next header' and then find something. It > won't do that if you used the -z option because you generally can't > recover from errors in compression. But, I've never seen a tape drive > recover from an error and continue past it anyway so in practice > that's not going to matter. If you are concerned about errors, keep > more copies.This problem is not caused by compression or not, it is a general gtar bug that I reported in 1993 already. Nobody knows why it hits and the structure of the gtar sources makes it really hard to debug this problem. The FSF was interested to throw aywa gtar and replace it by star 10 years ago for this kind of problems in gtar.> afio is an archiver (available from third-party repos, not base) which > can compress yet still recover--it basically compresses each file > individually instead of compressing the entire archive, so the file > might be unrecoverable but the rest of the archive is still intact. > I use it for my tape backups (though your point of not knowing if > it'll fit on the tape is valid).Be careful with what you believe. The CPIO archive format in general is worse with resyncing to a defective archive that TAR is. Also note that afio greates arhives that may start to be non CPIO compliant somewhere in the middle. So you can never know whether you are able to restore with anything other than afio. What if afio does not compile on your new platform because it is no longer maintained? Also note that the POSIX standard dropped CPIO as an actively supported archiveformat because (different from TAR) any extention to CPIO results in creating a new incompatible archive format. The following other problems are known with gtar: - gtar is with aprox. 5% probability unable to read it's own continuation archives from multi volume archives. This cannot be fixed as it is caused by the concept used for multi volume archives in gtar. - gtar created archives with defective sparse file until a few years ago (up to ~ 2005) in case a file was bigger than a few GB. - gtar has much less features than star - gtar does not inlcude libfind, so there is no support for the find(1) command line syntax in gtar. - gtar needs aprox. 3x more CPU time as star J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
> Let me comment some questions in one single mail: > >> My basic requirement with what I'm doing is to use standard tools and >> formats so that archives I write today can be readable in 10 years. > > Star becomes 30 in 4 months, any archive created since it's early > beginning in summer 1982 can still be read back.I have been using it for about a decade or more and anything I dumped has been a no brainer to retrieve, on any other OS or even architecture.>> I don't think there is any such general consensus. Are you reading >> something that favors Solaris/*bsd over GNU based systems? > > Schily tools (and in special star) implement support for Linux specific > extensions. This is what you do not get from gtar at all. So why do Linux > distros prefer gtar even though there is no Linux support?Is there any correlation between this and the warning message I see during a bootstrap like so : Warning: *** /usr/src/linux/include contains broken include files *** Warning: *** /usr/src/linux/include is not used this reason *** Warning: This may result in the inability to use recent Linux kernel interfaces Warning: *** linux/ext2_fs.h is not usable at all *** Warning: *** This makes it impossible to support Linux file flags *** You may try to compile using 'make COPTX=-DTRY_EXT2_FS'>> I doubt if they are as well maintained in linux distros as the GNU >> tool set, particularly in terms of having recent fixes backported into >> the versions carried in enterprise distros. > > gtar still did not fix bugs I reported in 1993 (e.g. the bug that > causes gtar to complain with "skipping to next header" even on it's > own archives). I am thus sure that star not not worse than gtar....There seems to be something missing here. The subject was "schily tools" which is a lot more than star : root at rsync:/etc/default# ls /opt/schily/bin bosh cdrecord isodebug mdigest pfsh sfind star ustar bsh change isodump mkhybrid pxupgrade sformat star_sym ved btcflash compare isoinfo mkisofs readcd sgrow suntar ved-e calc copy isovfy mt sccs sh tar ved-w calltree count jsh od scgcheck smake tartest cdda2mp3 devdump label opatch scgskeleton smt termcap cdda2ogg gnutar lndir p scpio spatch translit cdda2wav hdump match pfbsh sdd spax udiff care to comment on any of these ? Certainly bosh needs a few words. dc -- -- http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x1D936C72FA35B44B +-------------------------+-----------------------------------+ | Dennis Clarke | Solaris and Linux and Open Source | | dclarke at blastwave.org | Respect for open standards. | +-------------------------+-----------------------------------+