dovecot - Aug 2010 - getdents() with 4KB buffer - seems slow (Maildir, large inbox)

Hi,

I have a very large inbox (~146K mails) in Maildir format and dovecot
seems to spend a lot of time rescanning the directory, especially when
the server is loaded. I'm not sure whether this is triggered by
Thunderbird or done regularly, but it takes longer when the server is
loaded, so sometimes it seems that it is scanning continuously. Since
it takes around 2000 getdents64() syscalls to scan my inbox, I'm
wondering whether the buffer size is a bit small with 4KB and might
benefit from being configurable:

...
getdents64(8, /* 73 entries */, 4096)   = 4088
getdents64(8, /* 73 entries */, 4096)   = 4088
getdents64(8, /* 73 entries */, 4096)   = 4088
getdents64(8, /* 73 entries */, 4096)   = 4088
getdents64(8, /* 73 entries */, 4096)   = 4088
...

(strace output of an imap process belonging to me)

The behaviour is identical on 1.0.15 and 1.2.13 (I just upgraded).

Regards,
 Marinos

PS. I know that the main reason for slowness is my large inbox and the
server being loaded (disk I/O), but still, so many syscalls just to
rescan the inbox seems wasteful.

# dovecot -n
# 1.2.13: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-6-xen-686 i686 Debian 4.0
log_timestamp: %Y-%m-%d %H:%M:%S
ssl_listen: *
ssl_cert_file: XXXXX (censored)
ssl_key_file: XXXXX
login_dir: /var/run/dovecot/login
login_executable: /usr/lib/dovecot/imap-login
login_process_size: 32
login_processes_count: 8
verbose_proctitle: yes
mail_privileged_group: mail
mail_location: maildir:~/Maildir
mbox_write_locks: fcntl dotlock
mail_process_size: 2048
imap_client_workarounds: outlook-idle
auth default:
  passdb:
    driver: pam
  userdb:
    driver: passwd
  socket:
    type: listen
    client:
      path: /var/spool/postfix/private/auth
      mode: 432
      user: postfix
      group: postfix

On Mon, 2010-08-30 at 17:38 +0200, Marinos Yannikos wrote:
> I have a very large inbox (~146K mails) in Maildir format and dovecot
> seems to spend a lot of time rescanning the directory, especially when
> the server is loaded. I'm not sure whether this is triggered by
> Thunderbird or done regularly, but it takes longer when the server is
> loaded, so sometimes it seems that it is scanning continuously. 
You could see if maildir_very_dirty_syncs=yes is helpful.
> Since
> it takes around 2000 getdents64() syscalls to scan my inbox, I'm
> wondering whether the buffer size is a bit small with 4KB and might
> benefit from being configurable:
getdents64() is an internal kernel function and shouldn't be called
directly (Dovecot uses readdir(), which is implemented with
getdents64()). Also I doubt the buffer size is configurable and it
probably wouldn't even help if a larger one was used, 2000 isn't yet
that many syscalls.