dovecot - Feb 2009 - "dovecot-uidlist: Duplicate file entry at line" error

This is with respect to an error that I am facing in dovecot.

The error is that is seen in logs is "Feb 23 00:04:46 mailblade1
dovecot: IMAP(USERNAME): /indexes/USERNAME/.INBOX/dovecot-uidlist:
Duplicate file entry at line 7:
1234776125.M559298P3988.s2o.qlc.co.in,S=13111,W=13470:2, (uid 94277 ->
97805)"

This error is seen for multiple users.

Once this error occurs for a user, the user downloads mails all over
again in POP3.

Please help! And let us know what can be done.

Details of our setup
----------------------

The setup has deployed 2 servers with Dovecot using the same storage
over NFS with the following options:

* Dovecot Version: v1.1.8 on both servers
* Server1 for retrieving mails using POP and imap Access
* Server2 for Mail Delivery via postfix and dovecot as LDA
* Number of users: 20000
* Number of mails Handled: 500K per day

--Note: Based on recommendation, kernel was upgraded from 2.6.18 (-
see below for details) and yet the issue persists.


Specifications of the server
-------------------------------
1) Server spec used for mail Access
??? OS: Linux 2.6.28 i686 CentOS release 5.2 (Final)
??? kernel : 2.6.28
??? /proc/sys/fs/inotify/max_user_instances = 1024
??? /proc/sys/fs/epoll/max_user_instances? = 1024
??? Memory : 2GB

2) Server used for Dovecot Delivery
??? OS: Linux 2.6.21-1.3194.fc7 i686 Fedora release 7 (Moonshine)
??? kernel : 2.6.21
??? /proc/sys/fs/inotify/max_user_instances = 128
??? /proc/sys/fs/epoll/max_user_instances? = NA
??? Memory : 2 GB


Conf on both the system as follows --------------------------------------
# 1.1.8: /usr/local/etc/dovecot.conf
#NFS Settings
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes

#Other special settings
pop3_no_flag_updates(pop3): yes
pop3_lock_session(pop3): yes



# all the settings
base_dir: /usr/local/var/run/dovecot
log_path:
info_log_path:
log_timestamp: %b %d %H:%M:%S
syslog_facility: mail
protocols: imap pop3
listen(default): *:143
listen(imap): *:143
listen(pop3): *:110
ssl_listen:
ssl_disable: yes
ssl_ca_file:
ssl_cert_file: /etc/ssl/certs/dovecot.pem
ssl_key_file: /etc/ssl/private/dovecot.pem
ssl_key_password:
ssl_parameters_regenerate: 168
ssl_cipher_list:
ssl_cert_username_field: commonName
ssl_verify_client_cert: no
disable_plaintext_auth: no
verbose_ssl: no
shutdown_clients: yes
nfs_check: yes
version_ignore: no
login_dir: /usr/local/var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
login_user: dovecot
login_greeting: Welcome to MailServe Popserver.
login_log_format_elements: user=<%u> method=%m rip=%r lip=%l %c
login_log_format: %$: %s
login_process_per_connection: no
login_chroot: yes
login_greeting_capability: no
login_process_size: 64
login_processes_count: 3
login_max_processes_count: 128
login_max_connections: 256
valid_chroot_dirs:
mail_chroot:
max_mail_processes: 600
mail_max_userip_connections(default): 50
mail_max_userip_connections(imap): 50
mail_max_userip_connections(pop3): 8
verbose_proctitle: yes
first_valid_uid: 99
last_valid_uid: 0
first_valid_gid: 99
last_valid_gid: 0
mail_extra_groups:
mail_access_groups:
mail_privileged_group:
mail_uid:
mail_gid:
mail_location: maildir:~/Maildir:INDEX=/indexes/%u:CONTROL=/indexes/%u
mail_cache_fields:
mail_never_cache_fields: imap.envelope
mail_cache_min_mail_count: 0
mailbox_idle_check_interval: 30
mail_debug: no
mail_full_filesystem_access: no
mail_max_keyword_length: 50
mail_save_crlf: no
mmap_disable: yes
dotlock_use_excl: yes
fsync_disable: no
mail_nfs_storage: yes
mail_nfs_index: yes
mailbox_list_index_disable: yes
lock_method: fcntl
maildir_stat_dirs: no
maildir_copy_with_hardlinks: yes
maildir_copy_preserve_filename: no
mbox_read_locks: fcntl
mbox_write_locks: dotlock fcntl
mbox_lock_timeout: 300
mbox_dotlock_change_timeout: 120
mbox_min_index_size: 0
mbox_dirty_syncs: yes
mbox_very_dirty_syncs: no
mbox_lazy_writes: yes
dbox_rotate_size: 2048
dbox_rotate_min_size: 16
dbox_rotate_days: 1
umask: 63
mail_drop_priv_before_exec: no
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_process_size: 256
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
mail_log_prefix: %Us(%u):
mail_log_max_lines_per_sec: 10
imap_max_line_length: 65536
imap_capability:
imap_client_workarounds:
imap_logout_format: bytes=%i/%o
pop3_no_flag_updates(default): no
pop3_no_flag_updates(imap): no
pop3_no_flag_updates(pop3): yes
pop3_enable_last: no
pop3_reuse_xuidl: no
pop3_lock_session(default): no
pop3_lock_session(imap): no
pop3_lock_session(pop3): yes
pop3_uidl_format: %08Xu%08Xv
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls
pop3_logout_format: top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
dict_db_config:
auth default:
? mechanisms: plain
? realms:
? default_realm:
? cache_size: 0
? cache_ttl: 3600
? cache_negative_ttl: 3600
? executable: /usr/local/libexec/dovecot/dovecot-auth
? user: root
? chroot:
? username_chars:
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@
? username_translation:
? username_format:
? master_user_separator:
? anonymous_username: anonymous
? krb5_keytab:
? gssapi_hostname:
? winbind_helper_path: /usr/bin/ntlm_auth
? failure_delay: 2
? verbose: no
? debug: no
? debug_passwords: no
? ssl_require_client_cert: no
? ssl_username_from_cert: no
? ntlm_use_winbind: no
? count: 1
? worker_max_count: 5
? worker_max_request_count: 0
? process_size: 256
? passdb:
??? driver: sql
??? args: /usr/local/etc/dovecot-mysql.conf
??? deny: no
??? pass: no
??? master: no
? userdb:
??? driver: sql
??? args: /usr/local/etc/dovecot-mysql.conf
? userdb:
??? driver: prefetch
??? args:
? socket:
??? type: listen
??? client:
????? path: /var/run/dovecot/auth-client
????? mode: 432
????? user: nobody
????? group: nobody
??? master:
????? path: /var/run/dovecot/auth-master
????? mode: 384
????? user: nobody
????? group: nobody
plugin:
? quota: maildir:storage=1024
-------------------------------------

On 2/24/2009, Dovecot List (dovecot at jigs.in) wrote:
> Conf on both the system as follows 
Please just post output of dovecot -n instead of copy/paste from config
file.

-- 

Best regards,

Charles

Please just post output of dovecot -n instead of copy/paste from config
file.

Output of dovecot -n
protocols: imap pop3
listen(default): *:143
listen(imap): *:143
listen(pop3): *:110
ssl_disable: yes
disable_plaintext_auth: no
login_dir: /usr/local/var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
login_greeting: Welcome to MailServe Popserver.
login_process_per_connection: no
max_mail_processes: 600
mail_max_userip_connections(default): 50
mail_max_userip_connections(imap): 50
mail_max_userip_connections(pop3): 8
verbose_proctitle: yes
first_valid_uid: 99
first_valid_gid: 99
mail_location: maildir:~/Maildir:INDEX=/indexes/%u:CONTROL=/indexes/%u
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
pop3_no_flag_updates(default): no
pop3_no_flag_updates(imap): no
pop3_no_flag_updates(pop3): yes
pop3_lock_session(default): no
pop3_lock_session(imap): no
pop3_lock_session(pop3): yes
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls
auth default:
  worker_max_count: 5
  passdb:
    driver: sql
    args: /usr/local/etc/dovecot-mysql.conf
  userdb:
    driver: sql
    args: /usr/local/etc/dovecot-mysql.conf
  userdb:
    driver: prefetch
  socket:
    type: listen
    client:
      path: /var/run/dovecot/auth-client
      mode: 432
      user: nobody
      group: nobody
    master:
      path: /var/run/dovecot/auth-master
      mode: 384
      user: nobody
      group: nobody
plugin:
  quota: maildir:storage=1024

-- 
View this message in context:
http://www.nabble.com/%22dovecot-uidlist%3A-Duplicate-file-entry-at-line%22-error-tp22176104p22180350.html
Sent from the Dovecot mailing list archive at Nabble.com.

On Tue, 2009-02-24 at 10:54 +0530, Dovecot List wrote:
> --Note: Based on recommendation, kernel was upgraded from 2.6.18 (-
> see below for details) and yet the issue persists.
..
> 1) Server spec used for mail Access
>     OS: Linux 2.6.28 i686 CentOS release 5.2 (Final)
>     kernel : 2.6.28
> 2) Server used for Dovecot Delivery
>     OS: Linux 2.6.21-1.3194.fc7 i686 Fedora release 7 (Moonshine)
>     kernel : 2.6.21
But you didn't upgrade both servers. The problem could be caused just as
well by the 2) server.

What do you use as NFS server?

Anyway since you've already set all the Dovecot's mail_nfs*=yes
settings, there's really nothing else you can do on Dovecot's side.
Dovecot really wants a perfectly working NFS setup, and it's not always
easy to do that..
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20090224/e96b82db/attachment-0002.bin>

Timo Sirainen wrote:
> 
> But you didn't upgrade both servers. The problem could be caused just
as
> well by the 2) server.
> 
Is it that we should have had both kernel equal?


Timo Sirainen wrote:
> 
> What do you use as NFS server?
> 
We use Netapp's 2050c NAS box.


Timo Sirainen wrote:
> 
> Anyway since you've already set all the Dovecot's mail_nfs*=yes
> settings
> 
Would setting up any different lock mechanism, work? Currently we have gone
with defaults.


Timo Sirainen wrote:
> 
> Dovecot really wants a perfectly working NFS setup, and it's not always
> easy to do that.
> 
Even though it might be difficult, is there any ideal, sure shot setup. We
can try that as well.

Thank you!



-----
--- jigs
-- 
View this message in context:
http://www.nabble.com/%22dovecot-uidlist%3A-Duplicate-file-entry-at-line%22-error-tp22176104p22198903.html
Sent from the Dovecot mailing list archive at Nabble.com.