zfs discuss - Jan 2007 - Multiple Read one Writer Filesystem

Hi, I am considering using ZFS so that multiple clients can "share"
the
same filesystem. I know that ZFS does not support a distributed filesystem.

The alternative I am considering is to have a single filesystem 
available to many clients using a SAN (iSCSI in this case). However only 
one client would mount the ZFS filesystem as read/write while the others 
would mount it read-only. For my application, all files are written once 
then only ever read or deleted.

Is it the case that when a new file is added (by the writer machine) 
that the clients that are mounted read only would see and be able to 
read this new file?

Does this apply to soft-link files as well?

Does anyone have experience with such a configuration?

Mike

Mike Papper wrote:
>
> The alternative I am considering is to have a single filesystem 
> available to many clients using a SAN (iSCSI in this case). However 
> only one client would mount the ZFS filesystem as read/write while the 
> others would mount it read-only. For my application, all files are 
> written once then only ever read or deleted. 
I know this doesn''t work for UFS. I''m fairly certain this
won''t work for
ZFS for many of the same reasons. (Writes in any hosts buffer cache, 
inflight transactions, metadata updates, etc.) Why not use NFS?

On 1/15/07, Torrey McMahon <tmcmahon2 at yahoo.com>
wrote:
> Mike Papper wrote:
> >
> > The alternative I am considering is to have a single filesystem
> > available to many clients using a SAN (iSCSI in this case). However
> > only one client would mount the ZFS filesystem as read/write while the
> > others would mount it read-only. For my application, all files are
> > written once then only ever read or deleted.
>
> I know this doesn''t work for UFS. I''m fairly certain this
won''t work for
> ZFS for many of the same reasons. (Writes in any hosts buffer cache,
> inflight transactions, metadata updates, etc.) Why not use NFS?
The reason why it wouldn''t work is because of the system caching
reads.  The FS assumes that it knows everything that is going into/out
of the file system but this is not true in a parallel file system.

What you will find is that while you can mount the UFS (ZFS should
prevent mounting but that''s another story), any updates on previously
read files will not be visible.

I agree that you should look at NFS.


-- 
Just me,
Wire ...

Thanks for the feedback, it seems caching is the main concern and if I 
always only write any given file once (then perhaps do a flush and a 
close after the write to empty the cache) and from then only ever read 
the file, will the scheme I had in mind work?

Also, will ZFS really prevent my mounting the same FS by several 
different machines? Is there a way around this?

Mike

Wee Yeh Tan wrote:
> On 1/15/07, Torrey McMahon <tmcmahon2 at yahoo.com> wrote:
>> Mike Papper wrote:
>> >
>> > The alternative I am considering is to have a single filesystem
>> > available to many clients using a SAN (iSCSI in this case).
However
>> > only one client would mount the ZFS filesystem as read/write while
the
>> > others would mount it read-only. For my application, all files are
>> > written once then only ever read or deleted.
>>
>> I know this doesn''t work for UFS. I''m fairly certain
this won''t work for
>> ZFS for many of the same reasons. (Writes in any hosts buffer cache,
>> inflight transactions, metadata updates, etc.) Why not use NFS?
> 
> The reason why it wouldn''t work is because of the system caching
> reads.  The FS assumes that it knows everything that is going into/out
> of the file system but this is not true in a parallel file system.
> 
> What you will find is that while you can mount the UFS (ZFS should
> prevent mounting but that''s another story), any updates on
previously
> read files will not be visible.
> 
> I agree that you should look at NFS.
> 
>

On Sun, Jan 14, 2007 at 07:06:20PM -0800, Mike Papper
wrote:
> Thanks for the feedback, it seems caching is the main concern and if I 
> always only write any given file once (then perhaps do a flush and a 
> close after the write to empty the cache) and from then only ever read 
> the file, will the scheme I had in mind work?
> 
> Also, will ZFS really prevent my mounting the same FS by several 
> different machines? Is there a way around this?
No, ZFS will not prevent you from doing this, but you will corrupt your
pool (there is a bug filed to track the hostid in the label to prevent
this).  There is also no way to open a pool in read-only mode, and ZFS
will die when it cannot write to the device (both with appropriate bugs
filed).

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Jan 14, 2007, at 21:37, Wee Yeh Tan wrote:
> On 1/15/07, Torrey McMahon <tmcmahon2 at yahoo.com> wrote:
>> Mike Papper wrote:
>> >
>> > The alternative I am considering is to have a single filesystem
>> > available to many clients using a SAN (iSCSI in this case).
However
>> > only one client would mount the ZFS filesystem as read/write  
>> while the
>> > others would mount it read-only. For my application, all files are
>> > written once then only ever read or deleted.
>>
>> I know this doesn''t work for UFS. I''m fairly certain
this won''t
>> work for
>> ZFS for many of the same reasons. (Writes in any hosts buffer cache,
>> inflight transactions, metadata updates, etc.) Why not use NFS?
>
> The reason why it wouldn''t work is because of the system caching
> reads.  The FS assumes that it knows everything that is going into/out
> of the file system but this is not true in a parallel file system.
>
> What you will find is that while you can mount the UFS (ZFS should
> prevent mounting but that''s another story), any updates on
previously
> read files will not be visible.
>
> I agree that you should look at NFS.
Or just use QFS - this will do exactly what you want, plus you''ll  
also have
the option to go multi-writer if you so wish.  As for the issue of  
local buffer
cache synchronization you can set the invalidation rate, or just  
completely
defeat the buffer cache if you wish with per file and per directory  
semantics
that can be set on the fly.  DIRECTIO might suck for certain types of  
random
IO that may not align on a backend storage stripe, but it sure does help
with multi-node synchronization .. (cache on the storage end and build
your volumes there instead) .. i think you''ll find it works  
wonderfully with
iSCSI and if you want to integrate with ZFS - i''d think about using  
zvols
aligned with the QFS DAU and directio (so you don''t double buffer  
between
QFS and the ZFS COW which will always buffer)

In my mind QFS would have been a much better base for integration of the
ZFS feature set -- the flexibility, performance, and scalability are  
wonderful
.. but alas - project funding, politics, and pride can be a cruel  
mistress
particularly when you have differences of opinion in FS design  
philosophy
(older Cray programmers vs newer VM programmers)

oh .. in case you''ve never heard of QFS for whatever marketing reason:
http://www.sun.com/storagetek/management_software/data_management/qfs/

give it a try .. free download with no lockdown key ..

.je

Not to be a ninny, but a one-write/many read is pretty much the design 
scenario for using CacheFS in conjunction with NFS on the client side.

That is, assuming that only one client is doing the writing. If all your 
clients are doing writing (just maybe not to the same file), then you 
DON''T have what you describe. You have a true distributed filesystem, 
and you have to look elsewhere than ZFS.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

(It''s perhaps worth noting that cachefs won''t work with NFSv4,
so if you want to try this, manually force your server and/or clients into v3.)

This will, of course, limit your scalability to whatever your NFS server can
push through the network (modulo caching).  QFS is a better choice if you need
scalability beyond that.
 
 
This message posted from opensolaris.org

Hi Torrey,
> On 1/15/07, Torrey McMahon <tmcmahon2 at yahoo.com>
> wrote:
> What you will find is that while you can mount the
> UFS (ZFS should
> prevent mounting but that''s another story), any
> updates on previously
> read files will not be visible.
Actually it''s quite worth with UFS: mounting a filesystem twice can
lead to panics on the read-only node due to missing cache coherency. See Infodoc
77146 ("Why ufs Filesystems Should Not be Mounted Concurrently on Multiple
Systems", sorry, support contract needed) for details.

Best regards

  -- Dagobert
 
 
This message posted from opensolaris.org