zfs discuss - Sep 2006 - Archiving on ZFS

Hi

We are thinking about moving away from our Magneto-Optical based archive system
(WORM technology). At the moment, we use a volume manager, which virtualizes the
WORM''s in the jukebox and presents them as UFS Filesystems. The volume
manager automatically does asynchronous replication to an identical system in
another datacenter. To speed up slow WORM access, the volume manager has a read
cache.

Because of this cache, we did not find out until we checked the WORMs directly,
that we have a silent data corruption on some WORMs (Surprise! Surprise!).

Mainly because of this, I was thinking about replacing to whole bunch with
something more robust and modern... (guess what :-)

Anyway, there are still some points, that came to into my mind:

-The mechanism to asynchronously replicate to another host could be simulated
using zfs send/receive. Still, I would prefer having a replication, that is
automatically triggered, like Sun''s StorEdge Network Data Replicator
does this for UFS. This could be easily implemented in ZFS, I guess.

-We have a lot of small files (about 7 millions ~4-32k Files). Like everyone, we
want to be SOX compliant. So I tried to run BART over those files, to get a
fingerprint. I remember it took a couple of hours to complete. At least it was
much faster than on UFS. How can this be speeded up? Maybe we have to split
those files to seperate filesystems. This leads to my next point:

-I want to be sure, that nobody (maybe not even root) changes my filesystems for
the next couple of years.. I know there is a read-only property, but it might
not be enough. On our Hitachi array, we have a WORM functionality, which blocks
write access to a LUN until a specified date. While this works, it is not as
flexible as we want it, as the LUNs are too big for our use. Every day we are
archiving documents. At the end of the day we want to freeze the filesystem.
Would it be possible to add a time-lock property to ZFS? Could this be extended
to still allow new files to be added to the locked file system , but not
allowing to add/modify files (ZFS ACL''s could handle this)? Would
something like this make sense?

Thanks for your thoughts...
 
 
This message posted from opensolaris.org

Mika Borner wrote:
> -The mechanism to asynchronously replicate to another host could be
> simulated using zfs send/receive. Still, I would prefer having a
> replication, that is automatically triggered, like Sun''s StorEdge
> Network Data Replicator does this for UFS. This could be easily
> implemented in ZFS, I guess.
Yep, you can roll your own remote replication using zfs send/receive. 
Eventually ZFS will provide a more automated way of setting this up.
> -I want to be sure, that nobody (maybe not even root) changes my
> filesystems for the next couple of years..  I know there is a
> read-only property, but it might not be enough. On our Hitachi array,
> we have a WORM functionality, which blocks write access to a LUN
> until a specified date. While this works, it is not as flexible as we
> want it, as the LUNs are too big for our use. Every day we are
> archiving documents. At the end of the day we want to freeze the
> filesystem. 
Sounds like you want to take a snapshot at the end of each day.
Snapshots can never be changed.
> Would it be possible to add a time-lock property to ZFS?
> Could this be extended to still allow new files to be added to the
> locked file system , but not allowing to add/modify files (ZFS
ACL''s
> could handle this)? Would something like this make sense?
Yep, we''d like to implement a mode where you can''t remove any 
information from a filesystem, but you can add more data to it.  This is 
related to 4890717 "want append-only files".

--matt