Hi
Okay so we've been having this issue since forever and I figured why
the heck not ask it here since it's mostly related to dovecot. The
thing is, we have a huge amount of public folders (at the moment,
around 1100). Now, with dovecot indexing and caching we're mostly
okay, also being careful with things like Thunderbird 3's default
'Download everything' option ("Keep messages for this account on
this
computer") and such. However once in a while, someone goes rogue, we
install a new version of thunderbird, someone accidentally sets up an
email account in thunderbird mistakingly leaving the download
everything option on. This causes high IO on our server, and a single
user can quickly more or less kill the whole server in mere minutes,
load average quickly spiking to 30-40-50 and everything becomes
sloooooow (obviously).
Is there any way I can limit the amount of IO a certain user can use?
Thanks and best regards,
KT
P.S.
my dovecot -n is:
# 2.0.9: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.26-2-xen-amd64 x86_64 Debian 5.0.8
auth_mechanisms = plain login
disable_plaintext_auth = no
listen = *
mail_location = maildir:~/Maildir
mail_plugins = acl
maildir_very_dirty_syncs = yes
namespace {
inbox = yes
location prefix separator = .
type = private
}
namespace {
location = maildir:/home/_shared/projects:INDEX=~/Maildir/_shared
prefix = shared.
separator = .
subscriptions = no
type = public
}
namespace {
location = maildir:/home/_shared/sys:INDEX=~/Maildir/sys
prefix = sys.
separator = .
subscriptions = no
type = public
}
passdb {
driver = pam
}
plugin {
acl = vfile
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename flag_change append
mail_log_fields = uid box msgid from subject size vsize flags
mail_log_group_events = no
}
protocols = imap pop3
service imap {
executable = imap
}
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol imap {
mail_max_userip_connections = 40
mail_plugins = acl imap_acl
}
K?d?r Tam?s (KTamas) put forth on 4/7/2011 9:33 AM:> Hi > > Okay so we've been having this issue since forever and I figured why > the heck not ask it here since it's mostly related to dovecot. The > thing is, we have a huge amount of public folders (at the moment, > around 1100). Now, with dovecot indexing and caching we're mostly > okay, also being careful with things like Thunderbird 3's default > 'Download everything' option ("Keep messages for this account on this > computer") and such. However once in a while, someone goes rogue, we > install a new version of thunderbird, someone accidentally sets up an > email account in thunderbird mistakingly leaving the download > everything option on. This causes high IO on our server, and a single > user can quickly more or less kill the whole server in mere minutes, > load average quickly spiking to 30-40-50 and everything becomes > sloooooow (obviously). > > Is there any way I can limit the amount of IO a certain user can use?TTBOMK Dovecot has no data rate limiting controls, neither globally nor per user. As I see it you have a few of options: 1. Switch to mdbox or mbox storage format to decrease IOs per email 2. Beef up the server with many more RAID spindles 3. Eliminate all unnecessary filesystem metadata and logging activity for instance, atime, if not done already 4. Switch to a filesystem matched to your workload EXT3/4/Reiser are not optimal for high concurrency multi-user server workloads switch to XFS or JFS if currently using EXT3/4 5. Install a traffic shaper in front of, or on, the Dovecot server Configure it to clamp any TCP sessions that exceed, say, 10 Mbit/s for more than 10 seconds down to 1 Mbit/s for a 1 minute duration. Tune to taste until you get the desired results. Option 5 is the most sure fire way to solve the problem you describe. It is the most direct solution, and likely free (not including your time to set it up), assuming netfilter etc scripts are available to accomplish this. -- Stan
On Thu, 2011-04-07 at 16:33 +0200, K?d?r Tam?s (KTamas) wrote:> However once in a while, someone goes rogue, we > install a new version of thunderbird, someone accidentally sets up an > email account in thunderbird mistakingly leaving the download > everything option on. This causes high IO on our server, and a single > user can quickly more or less kill the whole server in mere minutes, > load average quickly spiking to 30-40-50 and everything becomes > sloooooow (obviously).So a single process that is reading files fast enough from disk can cause disk IO to spike in a way that makes all other processes wait for available disk IO? I don't think that's a common problem. Typically only that one process would be waiting for disk IO and the load would increase by 1 (or maybe a little bit more, but definitely not by 50).. Maybe you should look into Linux I/O schedulers and see if you're using the wrong one.> Is there any way I can limit the amount of IO a certain user can use?Not in Dovecot.
From: Timo Sirainen <tss at iki.fi> Subject: Re: [Dovecot] IO rate quotas?> So a single process that is reading files fast enough from disk can > cause disk IO to spike in a way that makes all other processes wait for > available disk IO?It's not a single process. Thunderbird's newest defaults are to syncronize every folder and it will open one connectrion per folder in order to do that. Appropriate disk/swappiness/readahead/cache tuning is required - along with enough ram to cope with several users trying this at once. The underlaying filesystem is also important. as a for-instance, this kind of load with maildir format will effectively _kill_ a GFS box. Our fix was to install systemwide defaults on managed systems to not synchronize (not needed, the home directories are on network storage anyway) and to ask people to check/unset this on laptops/etc