thr3ads.net - dovecot - [Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear [Jan 2006]

If this information is useful, please help other people find it:
Share via:

Tom Alsberg

2006-Jan-22 14:05 UTC

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

Hi there...

Having updated to 1.0.beta1, I for a while thought a bad bug with IDLE 
handling that caused new mail in INBOX not to appear when using Mozilla 
Thunderbird has been solved.  Although I can not reproduce one kind of 
problem anymore, after a short while it again appears that one does not 
always see one's new mail with Thunderbird.

I can not consistently reproduce the bug, but from some network captures 
it indeed appears that sometimes Thunderbird has (in one of its 
connections to the server) INBOX SELECTed and IDLE on it, new mail 
arrives, but 10 minutes afterwards it was still not sent an EXISTS message.

Initially the symptoms appeared to suggest that something really bad 
happened to the index files, but I tried INDEX=MEMORY and that did not 
help much.
Thunderbird using multiple connections to the server does of course only 
make debugging more difficult.  But it appears that without multiple 
connections, one has to constantly cycle around SELECTing to get 
notified on more than one mailbox, so that may justified.

I am a bit lost on this now, and would appreciate some ideas.  Although 
there were several related bugs I noticed in Thunderbird 1.0, it appears 
that they are gone in 1.5, and yet now it really seems Dovecot is doing 
something wrong here.

Attached is the configuration file I use for the testing installation of 
Dovecot 1.0.beta1 (somebody tell me if you see something wrong there). 
The mailbox arrangement is as follows, right now (until we have a better 
way):

* INBOX in /var/mail/%u
* most mailboxes in ~/imap.maildir/... as maildirs.
* mboxes in ~/imap.mbox/... accessible as #imap.mbox/...
* the mbox:~/mail/ is for when namespaces are not used - now there are 
so it should not be relevant

Except for myself, and one test account I have, I have two guinea-pigs 
using it (who are well aware that it is still under development and that 
  it will break once in a while) and reporting problems to me.

Vaguely (since I can't figure the exact reasons), the symptom can be 
described as follows:  When using Mozilla Thunderbird to access mail, 
new mail arrives to INBOX (mbox:/var/mail/%u), but is sometimes not seen 
for a long while after it arrives.  Sometimes the first (or first few) 
message arriving appears, but then no new messages appear, and sometimes 
everything works right and immediately new mail is seen.  However 
whenever (according to Thunderbird) something changes in the mailbox, 
the mailbox is seen up-to-date as it really was at that time.

   Thanks, any help appreciated,
   -- Tom

-- 
   Tom Alsberg - hacker (being the best description fitting this space)
   Web page:	http://www.cs.huji.ac.il/~alsbergt/
DISCLAIMER:  The above message does not even necessarily represent what
my fingers have typed on the keyboard, save anything further.
-------------- next part --------------
# Dovecot IMAP server configuration at CS.HUJI
##############################################
#
# $Id$

# GENERAL PARAMETERS:
#####################

# Base directory where to store runtime data:
base_dir	= /var/run/dovecot

# Protocols we want to be serving:
protocols	= imap pop3

# IP or host address where to listen in for connections:
listen		= *

ssl_disable	= yes


# LOGIN PROCESSES:
##################

# Directory for authentication sockets:
#login_dir	= /var/run/dovecot/login

# chroot login process to the login_dir:
#login_chroot	= yes

# User to use for the login process:
login_user	= nobody

# Maximum process size in megabytes:
#login_process_size	= 16

# Have each login be processed in its own process:
#login_process_per_connection	= yes

# Number of login processes to create:
#login_processes_count	= 3

# Maximum number of extra login processes to create:
#login_max_processes_count	= 128

# Maximum number of connections allowed in login state:
#login_max_logging_users	= 256

# Greeting message for clients:
login_greeting	= Dovecot mail access server at CS.HUJI ready.


# MAIL PROCESSES:
#################

# Maximum number of running mail processes:
#max_mail_processes	= 1024

# Show more verbose process title (in ps):
verbose_proctitle	= yes

# Valid UID range for users:
first_valid_uid		= 10
#last_valid_uid		= 0

# Valid GID range for users:
#first_valid_gid	= 1
#last_valid_gid		= 0

# Extra groups for mail processes:
#mail_extra_groups	
# ':' separated list of directories under which chrooting is allowed:
#valid_chroot_dirs	
# Default chroot directory for mail processes:
#mail_chroot	
# Default MAIL environment to use when it's not set:
default_mail_env	= mbox:~/mail/:INBOX=/var/mail/%u


# NAMESPACES:
#############

namespace	private {
	separator	= /
	prefix		= ""
	location	= maildir:~/imap.maildir/
}


namespace	private {
	separator	= /
	prefix		= "#imap.mbox/"
	location	= mbox:~/imap.mbox/:INBOX=/var/mail/%u
	inbox		= yes
}


# MAIL STORE:
#############

# Space-separated list of fields to initially save into cache file:
#mail_cache_fields	
# Space-separated list of fields to never save into cache file:
#mail_never_cache_fields	
# Like mailbox_check_interval, but used for IDLE command:
#mailbox_idle_check_interval	= 30

# Whether to allow full filesystem access to clients:
#mail_full_filesystem_access	= no

# Maximum allowed length for mail keyword name:
#mail_max_keyword_length	= 50

# Whether to save mails with CR+LF instead of plain LF:
#mail_save_crlf	= no

# Whether to use mmap to access mail files:
#mail_read_mmaped	= no

# Whether to disable mmap at all:
mmap_disable	= yes

# Whether to disable write to mmap:
#mmap_no_write	= no

# Locking method for index files:
lock_method	= dotlock

# Whether to list only directories in LIST command:
maildir_stat_dirs	= yes

# Whether to copy mail to other folders using hard links:
#maildir_copy_with_hardlinks	= no

# Locking methods to use for locking mbox:
mbox_read_locks		= dotlock
mbox_write_locks	= dotlock

# Maximum time in seconds to wait for lock (all of them):
mbox_lock_timeout	= 300

# Timeout to overwrite dotlock files if mailbox isn't modified:
mbox_dotlock_change_timeout	= 30

# Whether to read only appended data on change (unless corrupt):
#mbox_dirty_syncs	= yes

# Like mbox_dirty_syncs, but no full syncs with SELECT, EXAMINE,
# EXPUNGE, or CHECK:
#mbox_very_dirty_syncs	= no

# Whether to delay writing mbox headers until doing a full write sync:
#mbox_lazy_writes	= yes

# umask to use for mail files and directories:
umask			= 0077

# Whether drop all privileges before exec of mail process:
#mail_drop_priv_before_exec	= no

# Maximum mail process size in megabytes:
#mail_process_size	= 256

# Log prefix for mail processes:
mail_log_prefix		= "%Us(%u): "


# IMAP SPECIFIC SETTINGS:
#########################

protocol imap {
	# Login executable location:
	#login_executable	= /usr/local/dovecot/libexec/dovecot/imap-login

	# IMAP executable location:
	#mail_executable	= /usr/local/dovecot/libexec/dovecot/imap

	# Maximum IMAP command line length in bytes:
	#imap_max_line_length	= 65536

	# Suypport for dynamically loadable modules:
	#mail_use_modules	= no
	#mail_modules		= /usr/local/dovecot/lib/dovecot/imap

	# Whether to send IMAP capabilities in greeting message:
	login_greeting_capability	= yes

	# Workarounds for various client bugs:
	imap_client_workarounds	= outlook-idle netscape-eoh
}


# POP3 SPECIFIC SETTINGS:
#########################

protocol pop3 {
	# Login executable location:
	#login_executable	= /usr/local/dovecot/libexec/dovecot/pop3-login

	# POP3 executable location:
	#mail_executable	= /usr/local/dovecot/libexec/dovecot/pop3

	# Whether to not update recent/seen flags in POP3 sessions:
	#pop3_no_flag_updates	= no

	# Whether to support deprecated LAST command:
	#pop3_enable_last	= no

	# POP3 UIDL format to use:
	#pop3_uidl_format	= %08Xu%08Xv

	# Support for dynamically loadable modules
	#mail_user_modules	= no
	#mail_modules		= /usr/local/dovecot/lib/dovecot/pop3

	# Workarounds for various client bugs:
	pop3_client_workarounds	= outlook-no-nuls oe-ns-eoh
}


# AUTHENTICATION PROCESSES:
###########################

# Authentication executable location:
#auth_executable	= /usr/local/dovecot/libexec/dovecot/dovecot-auth

# Maximum authentication process size in megabytes:
#auth_process_size	= 256

# Authentication cache size in kilobytes:
#auth_cache_size	= 0

# Time to live in seconds for cached data:
#auth_cache_ttl		= 3600

# Space separated list of realms for SASL authentication mechanisms:
#auth_realms	
# Default realm/domain to use if none was specified:
#auth_default_realm	
# List of allowed characters in username:
#auth_username_chars	=
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789.-_@

# Username character translation before it's looked up in database:
#auth_username_translation	
# Whether to disable plaintext authentication
disable_plaintext_auth	= no

# More verbose authentication logging:
auth_verbose	= yes

# Even more verbose authentication logging for debugging purposes:
auth_debug	= yes

auth	default {
	# Space separated list of wanted authentication mechanisms
	mechanisms	= plain


	# Password database specification:
	passdb pam {
	}

	# User database specification:
	userdb passwd {
	}

	# User to use for the authentication process:
	user	= root

	# Directory where to chroot the process:
	#chroot	
	# Number of authentication processes to create:
	#count	= 1

	# Whether to require a valid SSL client certificate for
	# authentication:
	#ssl_require_client_cert	= no
}

Reuben Farrelly

2006-Jan-22 14:16 UTC

head link

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

Hi Tom,

On 23/01/2006 1:05 a.m., Tom Alsberg wrote:> Hi there...
> 
> Having updated to 1.0.beta1, I for a while thought a bad bug with IDLE 
> handling that caused new mail in INBOX not to appear when using Mozilla 
> Thunderbird has been solved.  Although I can not reproduce one kind of 
> problem anymore, after a short while it again appears that one does not 
> always see one's new mail with Thunderbird.
> 
> I can not consistently reproduce the bug, but from some network captures 
> it indeed appears that sometimes Thunderbird has (in one of its 
> connections to the server) INBOX SELECTed and IDLE on it, new mail 
> arrives, but 10 minutes afterwards it was still not sent an EXISTS message.
I am seeing something like this too and am using Thunderbird, but I'm not
sure
that this is actually a Thunderbird problem.  I'm unsure when it first
appeared
but it's pre -beta1 anyway.

What seems to happen for me is that if the new mail is delivered into 
Maildir/new/ or a file copied in there, I don't often see it until TB does
it's
2 minute "check-for-new-mail-in-all-folders" scan.  Then it is picked
up as a
new mail.  There is no network activity from the server to the client.

However if I put a file by hand, say for example "cp /etc/hosts 
/home/reuben/Maildir/cur" into Maildir/cur/ then Thunderbird sees it 
immediately.  Dovecot sends this to the client and breaks the IDLE straight
away:

367  53.927740  192.168.0.5 -> 192.168.0.8  IMAP Response: * 21 EXISTS

It's a bit strange but I can reproduce it on demand, Tom can you reproduce
it
that way too or is this a different bug?  [I think it's the same one..]

reuben

Tom Alsberg

2006-Jan-22 14:30 UTC

head link

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

Previously I wrote:> * INBOX in /var/mail/%u
> * most mailboxes in ~/imap.maildir/... as maildirs.
> * mboxes in ~/imap.mbox/... accessible as #imap.mbox/...
> * the mbox:~/mail/ is for when namespaces are not used - now there are 
>   so it should not be relevant
A few related details which are probably highly relevant I forgot to add:

* Dovecot is running on FreeBSD 5.4

* both home directories and /var/mail are on NFS - stored on another 
server (NetApp)
   - This means that the only way to get new mail is to do continuously 
check (poll/busy-wait) whether there are any changes in the mailbox, 
which the IMAP process does.  No epoll/inotify or anything like that.
   - We have and had many different operating systems, and generally the 
situation here is that NFS locking (fcntl/flock) always breaks some way 
or another since we can't get all hosts to work the same way with it, so 
we don't try at all.  All mail accesses (deliveries using Exim, and 
access using all mail clients we have around here) are using dotlocks 
(creating a file with the same name ending with ".lock") to avoid 
concurrency issues.  Dotlocking is configured in dovecot.conf (attached 
in the referred message) as follows:

lock_method     = dotlock
mbox_read_locks         = dotlock
mbox_write_locks        = dotlock
# Timeout to overwrite dotlock files if mailbox isn't modified:
mbox_dotlock_change_timeout     = 30

   -- Tom

-- 
   Tom Alsberg - hacker (being the best description fitting this space)
   Web page:	http://www.cs.huji.ac.il/~alsbergt/
DISCLAIMER:  The above message does not even necessarily represent what
my fingers have typed on the keyboard, save anything further.

Seemingly Similar Threads

Search for more apparently analagous threads

dovecot - Jan 2006 - Dovecot-1.0.beta1: New mail sometimes fails to appear

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

[Dovecot] Dovecot-1.0.beta1: New mail sometimes fails to appear

Seemingly Similar Threads