Are there any plans to support ZFS for write-only media such as optical storage? It seems that if mirroring or even zraid is used that ZFS would be a good basis for long term archival storage. Has this been considered? I expect that it is possible today by using files as the underlying media and then copying those individual files to optical storage. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Bob Friesenhahn wrote:> Are there any plans to support ZFS for write-only media such as > optical storage? It seems that if mirroring or even zraid is used > that ZFS would be a good basis for long term archival storage.I''m just going to assume that "write-only" here means "write-once, read-many", since it''s far too late for an April Fool''s joke. ;-) Dana
On Mon, 21 Apr 2008, Dana H. Myers wrote:> Bob Friesenhahn wrote: >> Are there any plans to support ZFS for write-only media such as optical >> storage? It seems that if mirroring or even zraid is used that ZFS would >> be a good basis for long term archival storage. > I''m just going to assume that "write-only" here means "write-once, > read-many", since it''s far too late for an April Fool''s joke.Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies such as holographic drives (300GB drives are on the market). For example, with two CD-R drives it should be possible to build a ZFS mirror on two CDs, but the I/O to these devices may need to be done in a linear sequential fashion at a rate sufficient to keep the writer happy, so temporary files (or memory-based buffering) likely need to be used. No one wants to be faced with a situation in which two copies are made to CD but both copies are deemed to be bad when they are read. ZFS could make that situation much better. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Maybe what you want is to archive files off to optical media? Perhaps ADM - http://opensolaris.org/os/project/adm ? -- mark Bob Friesenhahn wrote:> On Mon, 21 Apr 2008, Dana H. Myers wrote: > > >> Bob Friesenhahn wrote: >> >>> Are there any plans to support ZFS for write-only media such as optical >>> storage? It seems that if mirroring or even zraid is used that ZFS would >>> be a good basis for long term archival storage. >>> >> I''m just going to assume that "write-only" here means "write-once, >> read-many", since it''s far too late for an April Fool''s joke. >> > > Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies > such as holographic drives (300GB drives are on the market). For > example, with two CD-R drives it should be possible to build a ZFS > mirror on two CDs, but the I/O to these devices may need to be done in > a linear sequential fashion at a rate sufficient to keep the writer > happy, so temporary files (or memory-based buffering) likely need to > be used. > > No one wants to be faced with a situation in which two copies are made > to CD but both copies are deemed to be bad when they are read. ZFS > could make that situation much better. > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080421/4ff3a1be/attachment.html>
On Mon, 21 Apr 2008, Mark A. Carlson wrote:> Maybe what you want is to archive files off to optical media? > > Perhaps ADM - http://opensolaris.org/os/project/adm ?That looks interesting, but true archiving is needed. The level of archiving for this application is that copies would be kept thousands of feet underground in a stable salt mine on continents ''A'' and ''B''. An alternative is special temperature, humidity, and pressure controlled above-ground bunkers. It is desired that the data be preserved for hundreds or a thousand years, which would of course require copying to more modern media ever so often. The cost to create the original data is up to $200 million (today''s cost) and it can not be recreated. The size of the originals to be archived ranges from 2TB to 400TB depending on how "deep" the archiving is. The existing archive approach is in analog form but it is found that there is noticeable degredation after 50 or 100 years which is not possible to fully correct. When saw a discussion of these requirements today, ZFS immediately came to mind due to its many media-independent error detection and correction features, and the fact that it is open source. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Interesting problem. And yes you are right, there are a number of problems to solve here, see: http://blogs.sun.com/mac/en_US/entry/open_archive -- mark Bob Friesenhahn wrote:> On Mon, 21 Apr 2008, Mark A. Carlson wrote: > > >> Maybe what you want is to archive files off to optical media? >> >> Perhaps ADM - http://opensolaris.org/os/project/adm ? >> > > That looks interesting, but true archiving is needed. The level of > archiving for this application is that copies would be kept thousands > of feet underground in a stable salt mine on continents ''A'' and ''B''. > An alternative is special temperature, humidity, and pressure > controlled above-ground bunkers. It is desired that the data be > preserved for hundreds or a thousand years, which would of course > require copying to more modern media ever so often. The cost to > create the original data is up to $200 million (today''s cost) and it > can not be recreated. The size of the originals to be archived ranges > from 2TB to 400TB depending on how "deep" the archiving is. > > The existing archive approach is in analog form but it is found that > there is noticeable degredation after 50 or 100 years which is not > possible to fully correct. > > When saw a discussion of these requirements today, ZFS immediately > came to mind due to its many media-independent error detection and > correction features, and the fact that it is open source. > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080421/c9b57f01/attachment.html>
On Mon, 21 Apr 2008, Mark A. Carlson wrote:> Interesting problem. And yes you are right, there are a number > of problems to solve here, see: > > http://blogs.sun.com/mac/en_US/entry/open_archiveStandards and open source are clearly the way to go. Many open source applications have already been demonstrated to last far longer than their commercial counterparts. ZFS is open sourced but it is perhaps not mature and widespread enough yet to be seen as a stable long-term storage standard. The problem is a long term problem so there seems to be opportunity here for ZFS if it is adapted somewhat to address archiving. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Bob, ADM, with an underlying ZFS filesystem, will provide a great archiving solution when fully implemented. It allows for file replication, migration from disk to tape/optical/more disk, copies any number you may want, and allows for policy management of the archives. It''s underlying media management is provided by MMS (http://www.opensolaris.org/os/project/mms/). We follow the standards are are open sourced. - Greg Matthews, Archive Products Group, Sun Microsystems, Inc. Bob Friesenhahn wrote:> On Mon, 21 Apr 2008, Mark A. Carlson wrote: > > >> Interesting problem. And yes you are right, there are a number >> of problems to solve here, see: >> >> http://blogs.sun.com/mac/en_US/entry/open_archive >> > > Standards and open source are clearly the way to go. Many open source > applications have already been demonstrated to last far longer than > their commercial counterparts. > > ZFS is open sourced but it is perhaps not mature and widespread enough > yet to be seen as a stable long-term storage standard. The problem is > a long term problem so there seems to be opportunity here for ZFS if > it is adapted somewhat to address archiving. > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
> Are there any plans to support ZFS for write-only > media such as > optical storage? It seems that if mirroring or even > zraid is used > that ZFS would be a good basis for long term archival > storage.A simple solution would be to set up a zpool using two devices (real or ZVOL) that are the size of the optical media that you will be using, offline it when you''re done copying files to it, and then write the images to optical media. To restore later on, just reverse the process. This message posted from opensolaris.org
Hi Bob, If I was willing to do that I would simply build a pool from file- based storage being n-ISO images. It would involve the following steps 1. create blank ISO images of the size of your media 2. zpool create wormyz raidz2 image1.iso image2.iso image3.iso ... 3. Move your data to the pool 4. export the pool 5. burn the media If you need to recover, copy the data from the device using dd conv=sync,noerror The "problem" here is that by putting the data away from your machine, you loose the chance to "scrub" it on a regular basis, i.e. there is always the risk of silent corruption. I am not an expert, but the MTTDL is in tousands of years when using raidz2 with a hot-spare and regular scrubbing. If you add zpool send/receive and geographically dislocated severs, this may be better than optical media, because you detect the errors early. Hope this helps, ralf Am 21.04.2008 um 23:18 schrieb zfs-discuss-request at opensolaris.org:> Date: Mon, 21 Apr 2008 14:20:15 -0500 (CDT) > From: Bob Friesenhahn <bfriesen at simple.dallas.tx.us> > Subject: [zfs-discuss] ZFS for write-only media? > To: zfs-discuss at opensolaris.org > Message-ID: > <Pine.SOC.4.64.0804211417120.19227 at freddy.simplesystems.org> > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > > Are there any plans to support ZFS for write-only media such as > optical storage? It seems that if mirroring or even zraid is used > that ZFS would be a good basis for long term archival storage. > > Has this been considered? I expect that it is possible today by using > files as the underlying media and then copying those individual files > to optical storage. > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Tue, 22 Apr 2008, Ralf Bertling wrote:> Hi Bob, > If I was willing to do that I would simply build a pool from file- > based storage being n-ISO images. > It would involve the following steps > 1. create blank ISO images of the size of your media > 2. zpool create wormyz raidz2 image1.iso image2.iso image3.iso ... > 3. Move your data to the pool > 4. export the pool > 5. burn the media > > If you need to recover, copy the data from the device using dd > conv=sync,noerrorYes, I know that this will work and what I thought of. But I was thinking that perhaps ZFS would be able to attach to the read-only pool. At the moment it is likely not willing to attach to read-only devices since part of its function depends on writing.> The "problem" here is that by putting the data away from your machine, > you loose the chance to "scrub" > it on a regular basis, i.e. there is always the risk of silent > corruption.Running a scrub is pointless since the media is not writeable. :-)> I am not an expert, but the MTTDL is in tousands of years when using > raidz2 with a hot-spare and regular scrubbing.A thousand years ago, knights were storming castle calls. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Bob Friesenhahn wrote:>> The "problem" here is that by putting the data away from your machine, >> you loose the chance to "scrub" >> it on a regular basis, i.e. there is always the risk of silent >> corruption. >> > > Running a scrub is pointless since the media is not writeable. :-) > >But that''s the point. You can''t correct silent errors on write once media because you can''t write the repair. I think it makes more sense to save a checksum of the entire CD/DVD/etc. media separately, so you can check the validity of your data that way, instead of using ZFS on WORM media. Jon
On Tue, 22 Apr 2008, Jonathan Loran wrote:>> > But that''s the point. You can''t correct silent errors on write once > media because you can''t write the repair.Yes, you can correct the error (at time of read) due to having both redundant media, and redundant blocks. That is a normal function of ZFS. It just not possible to correct the failed block on the media by re-writing it or moving its data to a new location. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Bob Friesenhahn wrote:> On Tue, 22 Apr 2008, Jonathan Loran wrote: >>> >> But that''s the point. You can''t correct silent errors on write once >> media because you can''t write the repair. > > Yes, you can correct the error (at time of read) due to having both > redundant media, and redundant blocks. That is a normal function of > ZFS. It just not possible to correct the failed block on the media by > re-writing it or moving its data to a new location. >I suppose with ditto blocks, this has some merrit. Someone needs to characterize how errors probigate on different types of WORM media. perhaps this has already been done. In my experience, when DVD-R go south, they really go bad at once. Not a lot of small bit errors. But a full analysis would be good. Probably it would make the most sence to write mirrored WORM disks with different technology to hedge your bets. Jon
Peter Brouwer, Principal Storage Architect, Office of the Chief Technologist, Sun MicroSystems
2008-Apr-22 19:46 UTC
[zfs-discuss] ZFS for write-only media?
An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080422/d1280251/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3303 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080422/d1280251/attachment.bin>
On Tue, 22 Apr 2008, Jonathan Loran wrote:>> > I suppose with ditto blocks, this has some merrit. Someone needs to > characterize how errors probigate on different types of WORM media. perhaps > this has already been done. In my experience, when DVD-R go south, they > really go bad at once. Not a lot of small bit errors. But a full analysis > would be good. Probably it would make the most sence to write mirrored WORM > disks with different technology to hedge your bets.It does not really matter since ZFS supports various forms of RAID, including arbitrary mirroring. If possible, the media can be purchased from different vendors so there is less chance of similar bit-rot across the lot. With $40 to $200 million spent per project, a few extra copies is in the noise. :-) Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
"Dana H. Myers" <Dana.Myers at Sun.COM> wrote:> Bob Friesenhahn wrote: > > Are there any plans to support ZFS for write-only media such as > > optical storage? It seems that if mirroring or even zraid is used > > that ZFS would be a good basis for long term archival storage. > I''m just going to assume that "write-only" here means "write-once, > read-many", since it''s far too late for an April Fool''s joke.I know two write-only device types: WOM Write-only media WORN Write-once read never (this one is often used for backups ;-) J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Bob Friesenhahn <bfriesen at simple.dallas.tx.us> wrote:> On Mon, 21 Apr 2008, Dana H. Myers wrote: > > > Bob Friesenhahn wrote: > >> Are there any plans to support ZFS for write-only media such as optical > >> storage? It seems that if mirroring or even zraid is used that ZFS would > >> be a good basis for long term archival storage. > > I''m just going to assume that "write-only" here means "write-once, > > read-many", since it''s far too late for an April Fool''s joke. > > Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies > such as holographic drives (300GB drives are on the market). For > example, with two CD-R drives it should be possible to build a ZFS > mirror on two CDs, but the I/O to these devices may need to be done in > a linear sequential fashion at a rate sufficient to keep the writer > happy, so temporary files (or memory-based buffering) likely need to > be used.CD-R media is not really WORM (write once read many) as CD-R does not allow to write _every_ sector exactly once. Due to the way scrambeled error correction is implemented, there is always unusable 7 sectors between two areas on the medium that have been written independently. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
> "Dana H. Myers" <Dana.Myers at Sun.COM> wrote: > > > Bob Friesenhahn wrote: > > > Are there any plans to support ZFS for write-only > media such as > > > optical storage? It seems that if mirroring or > even zraid is used > > > that ZFS would be a good basis for long term > archival storage. > > I''m just going to assume that "write-only" here > means "write-once, > > read-many", since it''s far too late for an April > Fool''s joke. > > I know two write-only device types: > > WOM Write-only media > WORN Write-once read never (this one is often used > for backups ;-) > > J?rgSave $$ (or ??) - use /dev/null instead. This message posted from opensolaris.org
Joerg Schilling schrieb:> WOM Write-only mediahttp://www.national.com/rap/files/datasheet.pdf Daniel
On Thu, 24 Apr 2008, Daniel Rock wrote:> Joerg Schilling schrieb: >> WOM Write-only media > > http://www.national.com/rap/files/datasheet.pdfI love this part of the specification: Cooling The 25120 is easily cooled by employment of a six-foot fan, 1/2" from the package. If the device fails, you have exceeded the ratings. In such cases, more air is recommended. There was an article in German c''t magazine''s issue exactly 13 years ago this month that benchmarked various operating system''s Null devices. They tested an unnamed "hardware null device prototype", now I finally know what that one actually was ! :-) FrankH.
"Richard L. Hamilton" <rlhamil at smart.net> wrote:> > I know two write-only device types: > > > > WOM Write-only media > > WORN Write-once read never (this one is often used > > for backups ;-) > > > > J?rg > > Save $$ (or ??????) - use /dev/null instead.See: ftp://ftp.berlios.de/pub/star/testscripts/zwicky/ The Zwicky test claims that that most popular backup program is called "no backup" ;-) J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily