dovecot - Mar 2016 - NetApp NFS vs. ZFS and NFS for Maildir

Hi,

I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan). Our 
setup is Dovecot, Maildir for email storage and NFS to share mailboxes 
(more than 30k users) across POP/IMAP and MX servers.

NetApp NFS works fine also under high load but have some limitation for 
inode numbers per Volume and is expensive (but recently their prices 
have dropped).

ZFS, I read, suggest to create many small Raid Group to increase IOPS, 
but this configuration (N Raid instead of one RAID-DP like NetApp) is 
more complex to manage, or not?

Someone has experiences with ZFS and NFS(v3) in high load environments?

Thanks
-- 
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

El 11/03/2016 a las 11:22 a.m., Alessio Cecchi escribi?:
> Hi,
>
> I'm evaluating to switch from NetApp to a ZFS appliance (like Qsan).
Our
> setup is Dovecot, Maildir for email storage and NFS to share mailboxes
> (more than 30k users) across POP/IMAP and MX servers.
>
> NetApp NFS works fine also under high load but have some limitation for
> inode numbers per Volume and is expensive (but recently their prices
> have dropped).
>
> ZFS, I read, suggest to create many small Raid Group to increase IOPS,
> but this configuration (N Raid instead of one RAID-DP like NetApp) is
> more complex to manage, or not?
>
> Someone has experiences with ZFS and NFS(v3) in high load environments?
>
> Thanks
Be careful to no do any synchronous writes under ZFS. Every sync write 
can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in 
linux). Im using it in a 3k user environment and works great with a 4TB 
raid 10, and dovecot cache files in a SSD disk.

Saludos, Juan.

On 3/11/2016 9:58 AM, Juan Bernhard <juan at inti.gob.ar>
wrote:
> Be careful to no do any synchronous writes under ZFS. Every sync write 
> can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in 
> linux). Im using it in a 3k user environment and works great with a 4TB 
> raid 10, and dovecot cache files in a SSD disk.
>From what I've heard you should not use hardware based RAID (ie, RAID10)
setups with ZFS, you should let ZFS handle it.

Maybe that is the source of your latency issues?

On Fri, 11 Mar 2016 11:58:00 -0300
Juan Bernhard <juan at inti.gob.ar> wrote:
> 
> El 11/03/2016 a las 11:22 a.m., Alessio Cecchi escribi?:
> > Hi,
> >
> > I'm evaluating to switch from NetApp to a ZFS appliance (like
Qsan). Our
> > setup is Dovecot, Maildir for email storage and NFS to share mailboxes
> > (more than 30k users) across POP/IMAP and MX servers.
> >
> > NetApp NFS works fine also under high load but have some limitation
for
> > inode numbers per Volume and is expensive (but recently their prices
> > have dropped).
> >
> > ZFS, I read, suggest to create many small Raid Group to increase IOPS,
> > but this configuration (N Raid instead of one RAID-DP like NetApp) is
> > more complex to manage, or not?
> >
> > Someone has experiences with ZFS and NFS(v3) in high load
environments?
> >
> > Thanks
> 
> Be careful to no do any synchronous writes under ZFS. Every sync write 
> can take up to 3 seconds of latency (under freebsd, I didnt test ZFS in 
> linux). Im using it in a 3k user environment and works great with a 4TB 
> raid 10, and dovecot cache files in a SSD disk.
> 
> Saludos, Juan.
zfs set sync=disabled ?


-- 
Regards,
Stephan

On 11/03/2016 14:58, Juan Bernhard wrote:
>> Someone has experiences with ZFS and NFS(v3) in high load environments?
>>
>> Thanks
>
> Be careful to no do any synchronous writes under ZFS.
By default all NFS writes are synchronous but I assume dovcot sync 
writes all data anyway so in this case the NFS sync doesn't matter.

> Every sync write can take up to 3 seconds of latency (under freebsd,
> I didnt test ZFS in linux).
sync writes should take a few ms (they do for me). If you have enough 
load for them to be a problem you should have enough revenue to afford 
an SSD as a ZFS write cache / SLOG and then they will no longer be a 
problem.