dovecot - May 2012 - Error in dovecot 2.0.13: "Too many levels of symbolic links"

Hi all,

I've run into an issue we have with dovecot versions 2.0.12 through 2.0.14
on
latest CentOS 6.2 when dealing with Maildirs located on NFS partitions.
Although I saw Mr. Egbert's thread, his resolution does not work on our
end.

When dealing with users with over 500 messages in any of their Maildir
folders, they occasionally receive an error from the server:

Internal error occurred. Refer to server log for more information.

The corresponding log entries:

dovecot: imap(user1): Error: readdir(/u1/user1/Maildir/.AUTO-DELETED-SPAM/new)
failed: Too many levels of symbolic links
dovecot: imap(user1): Error: readdir(/u1/user1/Maildir/.logwatch/new) failed:
Too many levels of symbolic links
dovecot: imap(user2): Error: readdir(/u2/user2/Maildir/cur) failed: Too many
levels of symbolic links
dovecot: imap(user2): Error: readdir(/u2/user2/Maildir/.Trash/new) failed: Too
many levels of symbolic links
dovecot: imap(user3): Error: readdir(/u1/user3/Maildir/.Trash/new) failed: Too
many levels of symbolic links
dovecot: imap(user3): Error: readdir(/u4/user3/Maildir/.AUTO-DELETED-SPAM/new)
failed: Too many levels of symbolic links
dovecot: imap(user4): Error: readdir(/u1/user4/Maildir/.AUTO-DELETED-SPAM/new)
failed: Too many levels of symbolic links
dovecot: imap(user4): Error: readdir(/u1/user4/Maildir/new) failed: Too many
levels of symbolic links

A 'find . -type l' shows there are absolutely no symbolic links to be
found in those directories.

Running readdir.c that Timo provided always returns errno == 0 on {cur,new} of
all the directories reported in the logs.

It does not occur consistently in production, but it does tend to occur
to a fuzzy set of the same users on different partitions. We can
replicate it every time in testing.

The file count in these directories are all over the place, from less than a
thousand to multiple thousands. However, it's hard to calculate what may
have been in new prior to the clients processing/sorting that mail. It
may have been more on the higher side than lower.

The error tends to favor new over cur, but there are exceptions.

The tests we have been trying:

Create a new folder using alpine, save anywhere from 1000 to 10000 identical
emails to the new folder. 

Change to that new folder, thus prompting the move from new to cur. The client
receives the message and aborts the opening. Anywhere from 400-520 files are
successfully moved each time. It takes numerous attempts to finally move
all the files over--which I suspect some IMAP clients do, hence the
inconsistent errors in production.

We then create a brand new folder with the same amount of files and
verify that the error is reproduceable--which it is.

I'm leaning towards some sort of bug on the NFS client level, as Timo
mentioned, but the odd thing is we were running this setup since early
April and did not begin seeing these errors until early May--with user
load being relatively consistent during that time. I would think the bug
would have been evident before then. I'm also curious as to why dovecot
is getting an ELOOP error code from it.

Any ideas or pointers would be appreciated, sorry for the length. :)

-- 
 Tom Pawlowski
 OIT-CSS System Administrator           office: Hill 145
 email: tompru at jla.rutgers.edu          phone:  (732) 445-2634

On 17.5.2012, at 0.20, Tom Pawlowski wrote:
> dovecot: imap(user1): Error:
readdir(/u1/user1/Maildir/.AUTO-DELETED-SPAM/new) failed: Too many levels of
symbolic links
..
> Create a new folder using alpine, save anywhere from 1000 to 10000
identical
> emails to the new folder. 
> 
> Change to that new folder, thus prompting the move from new to cur. The
client
> receives the message and aborts the opening. Anywhere from 400-520 files
are
> successfully moved each time. It takes numerous attempts to finally move
> all the files over--which I suspect some IMAP clients do, hence the
> inconsistent errors in production.

See if the attached patch makes any difference?
> I'm leaning towards some sort of bug on the NFS client level, as Timo
What NFS server are you using? Perhaps it's related to that.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: diff
Type: application/octet-stream
Size: 486 bytes
Desc: not available
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20120517/f092e615/attachment-0004.obj>

Well, the good news is this didn't end up being a dovecot or NetApp
issue. RHEL 6.2 has a kernel bug:

https://bugzilla.redhat.com/show_bug.cgi?id=770250

Solution is to upgrade to the latest kernel released on 5/15:

kernel-2.6.32-220.17.1.el6

We no longer get this error after the update. Thanks for taking a look!

* Timo Sirainen <tss at iki.fi> [2012-05-17 00:35:49 +0300]:
> On 17.5.2012, at 0.20, Tom Pawlowski wrote:
> 
> > dovecot: imap(user1): Error:
readdir(/u1/user1/Maildir/.AUTO-DELETED-SPAM/new) failed: Too many levels of
symbolic links
> ..
> > Create a new folder using alpine, save anywhere from 1000 to 10000
identical
> > emails to the new folder. 
> > 
> > Change to that new folder, thus prompting the move from new to cur.
The client
> > receives the message and aborts the opening. Anywhere from 400-520
files are
> > successfully moved each time. It takes numerous attempts to finally
move
> > all the files over--which I suspect some IMAP clients do, hence the
> > inconsistent errors in production.
> 
> 
> See if the attached patch makes any difference?
> 
> > I'm leaning towards some sort of bug on the NFS client level, as
Timo
> 
> What NFS server are you using? Perhaps it's related to that.
> 


-- 
 Tom Pawlowski
 OIT-CSS System Administrator           office: Hill 145
 email: tompru at jla.rutgers.edu          phone:  (732) 445-2634