Hi, I''ve just started using ZFS + NFS, and i was wondering if there is anything i can do to optimise it for being used as a mailstore ? ( small files, lots of them, with lots of directory''s and high concurrent access ) So any ideas guys? P
On Tue, Jun 27, 2006 at 10:14:06AM +0200, Patrick wrote:> Hi, > > I''ve just started using ZFS + NFS, and i was wondering if there is > anything i can do to optimise it for being used as a mailstore ? ( > small files, lots of them, with lots of directory''s and high > concurrent access ) > > So any ideas guys?check out this thread, which may answer some of your questions: http://www.opensolaris.org/jive/thread.jspa?messageID=40617 sounds like your workload is very similar to mine. is all public access via NFS? also, check out this blog entry from Roch: http://blogs.sun.com/roller/page/roch?entry=the_dynamics_of_zfs for small file workloads, setting recordsize to a value lower than the default (128k) may prove useful. grant.
grant beattie wrote:>On Tue, Jun 27, 2006 at 10:14:06AM +0200, Patrick wrote: > > > >>Hi, >> >>I''ve just started using ZFS + NFS, and i was wondering if there is >>anything i can do to optimise it for being used as a mailstore ? ( >>small files, lots of them, with lots of directory''s and high >>concurrent access ) >> >>So any ideas guys? >> >> > >check out this thread, which may answer some of your questions: > > http://www.opensolaris.org/jive/thread.jspa?messageID=40617 > >sounds like your workload is very similar to mine. is all public >access via NFS? > >also, check out this blog entry from Roch: > > http://blogs.sun.com/roller/page/roch?entry=the_dynamics_of_zfs > >for small file workloads, setting recordsize to a value lower than the >default (128k) may prove useful. > >So what about for software development (like Solaris :-) where we''ve got lots of small files that we might be editting (biggest might be 128k) but when it comes time to compile, we can be writing out megabytes of data. Has anyone done a build of OpenSolaris over NFS served by ZFS and compared with a local ZFS build? How do both of those compared with UFS ? Darren
Hi,> sounds like your workload is very similar to mine. is all public > access via NFS?Well it''s not ''public directly'', courier-imap/pop3/postfix/etc... but the maildirs are accessed directly by some programs for certain things.> for small file workloads, setting recordsize to a value lower than the > default (128k) may prove useful.When changing things like recordsize, can i do it on the fly on a volume ? ( and then if i can what happens to the data already on the volume ? ) Also, another question, when turning compression on does the data already on the volume become compressed in the background ? or is in writes from then on ? P
On Tue, Jun 27, 2006 at 11:16:40AM +0200, Patrick wrote:> >sounds like your workload is very similar to mine. is all public > >access via NFS? > > Well it''s not ''public directly'', courier-imap/pop3/postfix/etc... but > the maildirs are accessed directly by some programs for certain > things.yes, that''s what I meant. a notable characteristic of most MTAs is that they are fsync() intensive, which can have an impact on ZFS performance. you will probably want to benchmark your IO pattern with various different configurations.> >for small file workloads, setting recordsize to a value lower than the > >default (128k) may prove useful. > > When changing things like recordsize, can i do it on the fly on a > volume ? ( and then if i can what happens to the data already on the > volume ? )yes, as with most (all?) ZFS properties, the recordsize can be changed on the fly. existing data is unchanged - the modified settings only affect new writes.> Also, another question, when turning compression on does the data > already on the volume become compressed in the background ? or is in > writes from then on ?as above, existing data remains unchanged. it may be desirable to do such things in the background, because it might be impractical or impossible to do it using regular filesystem access without interrupting applications. the same applies to adding new disks to an existing pool. I think there''s an RFE for these sort of operations. grant.
Darren Reed wrote:> Has anyone done a build of OpenSolaris over NFS served by > ZFS and compared with a local ZFS build?Yep, it sucks but builds over NFS always suck anyway. These to machines are on the same Gigabit switch. The NFS server is a V880 with 16G RAM, the NFS client is a V40z (4 proc) with 8G RAM. (borg.sfbay and ardbeg.sfbay for Sun people''s info).> How do both of those compared with UFS ?It didn''t appear to be any worse that UFS but I didn''t do any serious analysis of this. -- Darren J Moffat
Darren Reed writes: > grant beattie wrote: > > >On Tue, Jun 27, 2006 at 10:14:06AM +0200, Patrick wrote: > > > > > > > >>Hi, > >> > >>I''ve just started using ZFS + NFS, and i was wondering if there is > >>anything i can do to optimise it for being used as a mailstore ? ( > >>small files, lots of them, with lots of directory''s and high > >>concurrent access ) > >> > >>So any ideas guys? > >> > >> > > > >check out this thread, which may answer some of your questions: > > > > http://www.opensolaris.org/jive/thread.jspa?messageID=40617 > > > >sounds like your workload is very similar to mine. is all public > >access via NFS? > > > >also, check out this blog entry from Roch: > > > > http://blogs.sun.com/roller/page/roch?entry=the_dynamics_of_zfs > > > >for small file workloads, setting recordsize to a value lower than the > >default (128k) may prove useful. > > > > > > So what about for software development (like Solaris :-) where > we''ve got lots of small files that we might be editting (biggest > might be 128k) but when it comes time to compile, we can be > writing out megabytes of data. Small files are stored in small records so we should be fine for that. The recordsize (would be better named max_recordsize) is the size of records (I call them Filesystem blocks) when files gets to be larger than recordsize. Smaller files are stored as single records (in increment of disk sectors). -r
Patrick writes: > Hi, > > > sounds like your workload is very similar to mine. is all public > > access via NFS? > > Well it''s not ''public directly'', courier-imap/pop3/postfix/etc... but > the maildirs are accessed directly by some programs for certain > things. > > > for small file workloads, setting recordsize to a value lower than the > > default (128k) may prove useful. > > When changing things like recordsize, can i do it on the fly on a > volume ? ( and then if i can what happens to the data already on the > volume ? ) You do it On the fly for a given FS (recordsize it''s not a property of ZVOL). Files that were largers than the previous recordsize will not change. Files that we smaller and thus were stored as a single record, will continue to be stored as single record until a write makes the file bigger than the current value of recordsize. At which point they are store as multiple records of the new recordsize. Performance wise I don''t worry too much about these things. > > Also, another question, when turning compression on does the data > already on the volume become compressed in the background ? or is in > writes from then on ? My understanding is : From then on. -r > > P > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Tue, Jun 27, 2006 at 12:07:47PM +0200, Roch wrote:> > > for small file workloads, setting recordsize to a value lower than the > > > default (128k) may prove useful. > > > > When changing things like recordsize, can i do it on the fly on a > > volume ? ( and then if i can what happens to the data already on the > > volume ? ) > > You do it On the fly for a given FS (recordsize it''s not a > property of ZVOL). Files that were largers than the previous > recordsize will not change. Files that we smaller and thus > were stored as a single record, will continue to be stored > as single record until a write makes the file bigger than > the current value of recordsize. At which point they are > store as multiple records of the new recordsize. Performance > wise I don''t worry too much about these things.ah, yes. the key here is "until a write makes the file bigger", which would ~never happen given Maildir format mail as the files are not modified after they are written. they may be unlinked, renamed, or rewritten with a new name - but not modified. grant.
grant beattie writes: > On Tue, Jun 27, 2006 at 12:07:47PM +0200, Roch wrote: > > > > > for small file workloads, setting recordsize to a value lower than the > > > > default (128k) may prove useful. > > > > > > When changing things like recordsize, can i do it on the fly on a > > > volume ? ( and then if i can what happens to the data already on the > > > volume ? ) > > > > You do it On the fly for a given FS (recordsize it''s not a > > property of ZVOL). Files that were largers than the previous > > recordsize will not change. Files that we smaller and thus > > were stored as a single record, will continue to be stored > > as single record until a write makes the file bigger than > > the current value of recordsize. At which point they are > > store as multiple records of the new recordsize. Performance > > wise I don''t worry too much about these things. > > ah, yes. the key here is "until a write makes the file bigger", > which would ~never happen given Maildir format mail as the files > are not modified after they are written. > > they may be unlinked, renamed, or rewritten with a new name - but not > modified. > > grant. > I''d figure that there would be no need to tune the recordsize for such usage. And if the associated mailboxes are themselve rather small there again no need to tune. -r