Gregory K. Ruiz-Ade
2006-Jan-30 19:44 UTC
[Dovecot] extremely slow delete/move operations?
I've been working to migrate several hundered users on to our new mail servers, which run dovecot 0.99.11 (RHEL4). The problem I'm seeing is that deleting or moving a message seems to take a very long time. It's not uncommon for Thundirbird (1.0.7) to have delays of 5-10 seconds to delete a single message from my Inbox (<500k) and place it in my Trash folder (~100MB right now). The systems are Dell PowerEdge 1850 (dual 3.6GHz Xeon) with two 300GB SCSI drives in a hardware mirror. Filesystems are ext3 on LVM. Inboxes are on /var/spool/mail (/export/mail, symlinked into /var). User mail folders are in their home directories (/export/homes1/<user>, automounted to /home/<user>; some home directories are NFS automounted from a NetApp). Both the mail spool and home directories are available over NFS for compatibility reasons with the former setup. What's really bugging me is that I've used the same software (RHEL4 and dovecot 0.9x.x) at my last workplace, and the performance was outstanding; it beat UWIMAPd hands-down, especially with large mailboxes. Here, the opposite seems to be true. The primary difference seems to be that we're using NFS here, but I wasn't using NFS at my last job. I'm getting a lot of grief over the delays when users delete mail or move mail from their inbox to one of their mail folders. Enough so that they're now resisting moving to the new server. Where do I look to find out what's causing the slowness? What can I do to make it faster? Mailbox formats are mbox. Using these options in /etc/dovecot.conf: ----- protocols = imaps pop3s imap_listen = [::] pop3_listen = [::] ssl_cert_file = /usr/share/ssl/certs/csefast.ucsd.edu_cert.pem ssl_key_file = /usr/share/ssl/private/csefast.ucsd.edu_key.pem login_dir = /var/run/dovecot-login login = imap login = pop3 verbose_proctitle = yes first_valid_uid = 100 default_mail_env = mbox:~/:INBOX=/var/mail/%u mail_full_filesystem_access = yes mbox_locks = dotlock fcntl auth = default auth_mechanisms = plain auth_userdb = passwd auth_passdb = pam auth_user = root ----- Thanks in advance, Gregory -- Gregory K. Ruiz-Ade Sr. Systems Administrator Computer Science and Engineering University of California, San Diego Office: EBU3b 1216 Phone: (858) 822-2625 E-mail: gkra@ucsd.edu
Gregory K. Ruiz-Ade wrote:> The primary difference seems to be that we're using NFS here<snip> 8<> Mailbox formats are mbox.Those are the two most important pieces of information in your message (IMHO). mbox is always slower than maildir when the filesize exceeds the operating memory for the server (i.e. it cannot be mmapped). NFS will always be slower than "local" access, even on NetApp's supposed 'high-performance' NFS. The combination is deadly, since every message stored in Trash requires copying the entire mbox file and appending the new message, then deleting the original and renaming the new file in its place. Then there is the additional performance drag caused by Dovecot rewriting the index, as well as Thunderbird (which you should really upgrade to 1.5) keeping its local index up to date. You can confirm this by emptying your trash folder and seeing if the performance changes (which I am confident it will). There are three things you can do to improve performance: 1) Upgrade to Dovecot 1.0 stable (or even Beta2); 2) Switch to Maildir (instead of mbox); 3) Install a SAN (instead of the NFS mounted NAS). The last is the only one that costs you hard capital; the other two only involve your time, which as we all know, doesn't count. ;-) HTH John -- John Peacock Director of Information Research and Technology Rowman & Littlefield Publishing Group 4501 Forbes Boulevard Suite H Lanham, MD 20706 301-459-3366 x.5010 fax 301-429-5748
Gregory K. Ruiz-Ade wrote:> mailboxes. Here, the opposite seems to be true. The primary difference > seems to be that we're using NFS here, but I wasn't using NFS at my last > job.- put the indexes in a local directory (or use INDEX=memory, but that's not as good) - upgrade to 1.0.beta2 - migrate to maildir when possible
On Mon, Jan 30, 2006 at 09:44:30AM -0800, Gregory K. Ruiz-Ade wrote:> my Inbox (<500k) and place it in my Trash folder (~100MB right now)....> The primary difference seems to be that we're using NFS here, but I > wasn't using NFS at my last job....> Mailbox formats are mbox.Big mboxes on NFS, brrrrrr. Off the cuff, I'd say that your problem is right there. Anyway, it's a recipe for problems. Why not maildirs? Preferably with a filesystem that handles large directories well. HTH
On Jan 30, 2006, at 11:12 AM, Gregory K. Ruiz-Ade wrote:> Message: 10 > The thing is, these mail servers _are_ the local storage for the > mail spools > and home directories. There is no network involved when dovecot is > accessing the INBOX or saved mail folders. Yes, they're "NFS" > mounted, but > this is RHEL4 (linux 2.6.x), and when an NFS share is mounted from > the local > host, it's done as a bind mount, which means local disk access.I don't understand this at all. Mounting anything via NFS should only be done from a remote server, where when you type "mount" at a commandline you see something like: 10.10.10.2:/data on /var/mail type nfs (rw,nfsvers=3,addr=10.10.10.2) If your /etc/fstab or your automounts on your RHEL4 box are mouting LOCAL disks as type NFS, you have set up something very very wrong. Local disks should be mounted at boot like so: /dev/sda3 on / type ext3 (rw) Let's be certain we're clear about this. The disks where dovecot reads and writes your mboxes are local to the RHEL4 machine? If so, you're not using NFS at all. Roger Weeks
Gregory K. Ruiz-Ade <gkra <at> ucsd.edu> writes:> > I've been working to migrate several hundered users on to our new mail > servers, which run dovecot 0.99.11 (RHEL4). The problem I'm seeing is that > deleting or moving a message seems to take a very long time. It's not > uncommon for Thundirbird (1.0.7) to have delays of 5-10 seconds to delete a > single message from my Inbox (<500k) and place it in my Trash folder (~100MB > right now).The size of the directory as a whole doesn't matter, the amount of files can cause problems (with certain filesystems).> What's really bugging me is that I've used the same software (RHEL4 and > dovecot 0.9x.x) at my last workplace, and the performance was outstanding; > it beat UWIMAPd hands-down, especially with large mailboxes. Here, the > opposite seems to be true. The primary difference seems to be that we're > using NFS here, but I wasn't using NFS at my last job.NFS can be a real performance killer, but to be sure I'd strace (with -tt) a dovecot process and check where it spends most of it's time. best regards, Michael Renner