Hi, I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan). Our setup is Dovecot, Maildir for email storage and NFS to share mailboxes (more than 30k users) across POP/IMAP and MX servers. NetApp NFS works fine also under high load but have some limitation for inode numbers per Volume and is expensive (but recently their prices have dropped). ZFS, I read, suggest to create many small Raid Group to increase IOPS, but this configuration (N Raid instead of one RAID-DP like NetApp) is more complex to manage, or not? Someone has experiences with ZFS and NFS(v3) in high load environments? Thanks -- Alessio Cecchi Postmaster @ http://www.qboxmail.it https://www.linkedin.com/in/alessice
El 11/03/2016 a las 11:22 a.m., Alessio Cecchi escribi?:> Hi, > > I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan). Our > setup is Dovecot, Maildir for email storage and NFS to share mailboxes > (more than 30k users) across POP/IMAP and MX servers. > > NetApp NFS works fine also under high load but have some limitation for > inode numbers per Volume and is expensive (but recently their prices > have dropped). > > ZFS, I read, suggest to create many small Raid Group to increase IOPS, > but this configuration (N Raid instead of one RAID-DP like NetApp) is > more complex to manage, or not? > > Someone has experiences with ZFS and NFS(v3) in high load environments? > > ThanksBe careful to no do any synchronous writes under ZFS. Every sync write can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in linux). Im using it in a 3k user environment and works great with a 4TB raid 10, and dovecot cache files in a SSD disk. Saludos, Juan.
On 3/11/2016 9:58 AM, Juan Bernhard <juan at inti.gob.ar> wrote:> Be careful to no do any synchronous writes under ZFS. Every sync write > can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in > linux). Im using it in a 3k user environment and works great with a 4TB > raid 10, and dovecot cache files in a SSD disk.>From what I've heard you should not use hardware based RAID (ie, RAID10)setups with ZFS, you should let ZFS handle it. Maybe that is the source of your latency issues?
On Fri, 11 Mar 2016 11:58:00 -0300 Juan Bernhard <juan at inti.gob.ar> wrote:> > El 11/03/2016 a las 11:22 a.m., Alessio Cecchi escribi?: > > Hi, > > > > I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan). Our > > setup is Dovecot, Maildir for email storage and NFS to share mailboxes > > (more than 30k users) across POP/IMAP and MX servers. > > > > NetApp NFS works fine also under high load but have some limitation for > > inode numbers per Volume and is expensive (but recently their prices > > have dropped). > > > > ZFS, I read, suggest to create many small Raid Group to increase IOPS, > > but this configuration (N Raid instead of one RAID-DP like NetApp) is > > more complex to manage, or not? > > > > Someone has experiences with ZFS and NFS(v3) in high load environments? > > > > Thanks > > Be careful to no do any synchronous writes under ZFS. Every sync write > can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in > linux). Im using it in a 3k user environment and works great with a 4TB > raid 10, and dovecot cache files in a SSD disk. > > Saludos, Juan.zfs set sync=disabled ? -- Regards, Stephan
On 11/03/2016 14:58, Juan Bernhard wrote:>> Someone has experiences with ZFS and NFS(v3) in high load environments? >> >> Thanks > > Be careful to no do any synchronous writes under ZFS.By default all NFS writes are synchronous but I assume dovcot sync writes all data anyway so in this case the NFS sync doesn't matter.> Every sync write can take up to 3 seconds of latency (under freebsd, > I didnt test ZFS in linux).sync writes should take a few ms (they do for me). If you have enough load for them to be a problem you should have enough revenue to afford an SSD as a ZFS write cache / SLOG and then they will no longer be a problem.