zfs discuss - Jan 2007 - zfs / nfs issue (not performance :-) with courier-imap

I am not sure if this is a zfs issue, and nfs issue, or a combination  
of the two, or not an issue with them per se (caching or whatever),  
or a courier-imap issue, or even a mail client issue.

However, the issue happens in at least two different unrelated mail  
clients, so I don''t think it is client related, and I have spoken to  
someone who uses courier-imap on nfs mounted directories for maildir  
mailstore using FreeBSD 6.x  to NetApp nfs servers without issue (my  
nfs client if FreeBSD 6.x while the server is Solaris 10 x86 serving  
ZFS backed filesystems over nfs), so maybe it is something to do with  
ZFS and NFS interaction.

Basically, I have a few maildir mailstores that are mounted on my  
FreeBSD imap server from a Solaris 10 sever that serves them using  
NFSv3 from ZFS filesystems (each maildir has its own ZFS  
filesystem).  Most of my maildirs are on a local disk and do not have  
a problem and a few on the nfz/zfs do not have the problem and a few  
have the problem that appeared right after they were migrated from  
the local disk to the zfs/nfs filesystem for testing (we would  
eventually like to move over all mail to this nfz/zfs setup).

Basically, in the affected accounts (under Apple Mail.app and Windows  
Thunderbird), you can delete 1 or more messages, (mark for delete),  
expunge, and then mail starting some place in the list after the  
deleted messages starts to show the wrong mail content for the given  
message as shown in the list view.

say I have messages A B C D E F G etc

A
B
C
D
E
F
G

I delete C and expunge

Now it looks like this

A
B
D
E
F
G

but if I click, say E, it has F''s contents, F has Gs contents, and no
mail has D''s contents that I can see.  But the list in the mail
client list view is correct.

--

Some feedback from the courier mail list, from a guy who runs the  
FreeBSD nfs clients to NetApp nfs servers with courier without issue,  
thought it might be an nfs caching issue or something on the client  
or server.

Since this is ZFS backed nfs, I thought to ask here to see if there  
were any gotchas or anything that might be causing this.

ATIME is off (but was on earlier and the problem still happened  
before I switched it)
CHECKSOM COMPRESS DEVICES EXEC SETUID are ON and RDONLY and ZONED are  
OFF.  ACLMODE is groupmask and ACLINHEREIT is secure.

I have not messed around with the ZIL business to improve performance.

Thanks for any insight on how I might have set this up wrong.

Thanks
Chad

---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2411 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070125/1ac9fe8c/attachment.bin>

Hello Chad,

Thursday, January 25, 2007, 9:14:24 AM, you wrote:

CLSNL> I am not sure if this is a zfs issue, and nfs issue, or a combination
CLSNL> of the two, or not an issue with them per se (caching or whatever),  
CLSNL> or a courier-imap issue, or even a mail client issue.

CLSNL> However, the issue happens in at least two different unrelated mail  
CLSNL> clients, so I don''t think it is client related, and I have
spoken to
CLSNL> someone who uses courier-imap on nfs mounted directories for maildir  
CLSNL> mailstore using FreeBSD 6.x  to NetApp nfs servers without issue (my  
CLSNL> nfs client if FreeBSD 6.x while the server is Solaris 10 x86 serving  
CLSNL> ZFS backed filesystems over nfs), so maybe it is something to do with
CLSNL> ZFS and NFS interaction.

CLSNL> Basically, I have a few maildir mailstores that are mounted on my  
CLSNL> FreeBSD imap server from a Solaris 10 sever that serves them using  
CLSNL> NFSv3 from ZFS filesystems (each maildir has its own ZFS  
CLSNL> filesystem).  Most of my maildirs are on a local disk and do not have
CLSNL> a problem and a few on the nfz/zfs do not have the problem and a few  
CLSNL> have the problem that appeared right after they were migrated from  
CLSNL> the local disk to the zfs/nfs filesystem for testing (we would  
CLSNL> eventually like to move over all mail to this nfz/zfs setup).

CLSNL> Basically, in the affected accounts (under Apple Mail.app and Windows
CLSNL> Thunderbird), you can delete 1 or more messages, (mark for delete),  
CLSNL> expunge, and then mail starting some place in the list after the  
CLSNL> deleted messages starts to show the wrong mail content for the given  
CLSNL> message as shown in the list view.

CLSNL> say I have messages A B C D E F G etc

CLSNL> A
CLSNL> B
CLSNL> C
CLSNL> D
CLSNL> E
CLSNL> F
CLSNL> G

CLSNL> I delete C and expunge

CLSNL> Now it looks like this

CLSNL> A
CLSNL> B
CLSNL> D
CLSNL> E
CLSNL> F
CLSNL> G

CLSNL> but if I click, say E, it has F''s contents, F has Gs
contents, and no
CLSNL> mail has D''s contents that I can see.  But the list in the
mail
CLSNL> client list view is correct.

I don''t belive it''s a problem with nfs/zfs server.

Please try with simple dtrace script to see (or even truss) what files
your imapd actually opens when you click E - I don''t belive it opens E
and you get F contents, I would bet it opens F.



-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Robert Milkowski wrote:
> CLSNL> but if I click, say E, it has F''s contents, F has Gs
contents, and no
> CLSNL> mail has D''s contents that I can see.  But the list in
the mail
> CLSNL> client list view is correct.
>
> I don''t belive it''s a problem with nfs/zfs server.
>
> Please try with simple dtrace script to see (or even truss) what files
> your imapd actually opens when you click E - I don''t belive it
opens E
> and you get F contents, I would bet it opens F.
>   
I completely agree with Robert.  I''d personally suggest
''truss'' to start
because its trivial to use, then start using DTrace to further hone down 
the problem.

In the case of Courier-IMAP the best way to go about it would be to 
truss the parent (courierlogger, which calls courierlogin and ultimately 
imapd) using ''truss -f -p <PID>''.   Then open the
mailbox and watch
those stat''s and open''s closely.

I''ll be very interested in your findings.  We use Courier on NFS/ZFS 
heavily and I''m thankful to report having no such problems.

benr.

On Jan 25, 2007, at 1:02 PM, Ben Rockwood wrote:
> Robert Milkowski wrote:
>> CLSNL> but if I click, say E, it has F''s contents, F has Gs
>> contents, and no
>> CLSNL> mail has D''s contents that I can see.  But the list
in the
>> mail
>> CLSNL> client list view is correct.
>>
>> I don''t belive it''s a problem with nfs/zfs server.
>>
>> Please try with simple dtrace script to see (or even truss) what  
>> files
>> your imapd actually opens when you click E - I don''t belive it
>> opens E
>> and you get F contents, I would bet it opens F.
>>
>
> I completely agree with Robert.  I''d personally suggest
''truss'' to
> start because its trivial to use, then start using DTrace to  
> further hone down the problem.
>
> In the case of Courier-IMAP the best way to go about it would be to  
> truss the parent (courierlogger, which calls courierlogin and  
> ultimately imapd) using ''truss -f -p <PID>''.   Then
open the
> mailbox and watch those stat''s and open''s closely.
>
> I''ll be very interested in your findings.  We use Courier on NFS/ 
> ZFS heavily and I''m thankful to report having no such problems.
Thanks for all the help. It turned out to only be tangentially  
related :-) to nfs and not at all really to zfs.  And not a caching  
issue.

courier-imap keeps a uid database file for helping it and in the  
affected maildirs it couldn''t update that due to bad permissions on  
the main maildir (subdirs where maildir stores stuff were correct).    
Since those are inherited on the nfs client from the server I updated  
the permissions on the nfs server and all is well.

Thanks
Chad



>
> benr.
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net



-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2411 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070125/53ed84ec/attachment.bin>

Hello Ben,

Thursday, January 25, 2007, 9:02:53 PM, you wrote:

BR> I completely agree with Robert.  I''d personally suggest
''truss'' to start
BR> because its trivial to use, then start using DTrace to further hone down
BR> the problem.


I would argue that truss is simpler than dtrace in that case, but it''s
me I guess :)))))))))))


dtrace -ask ''is it zfs/nfs?''
  dtrace: definitely not
dtrace -ask ''what is it then?''
  dtrace: permissions

:))



-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com