Why would I be seeing a million of these?
dovecot: 2008-10-27 09:12:33 Warning:
chdir(/mail/mail/domain/u/username/) blocked for 18 secs
Two servers were working fine and then all of a sudden this morning
bam 500 load and theres a million dovecots and tons of these lines in
the logs.
It is like everyone gets up at exactly 8:30 and hits the pop3 server at once.
Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64,
2.6.26-1-amd64 i had the performance mode options for login
configured, i just set back to the single process per login though, I
think that hasnt fallen over that way but I really dont remember
anymore. I included the dovecot -n using the suggest performance login
settings? Do I have something totally out of whack?
# 1.1.5: /etc/dovecot/dovecot.conf
base_dir: /var/run/dovecot/
log_path: /var/log/dovecot/dovecot.log
info_log_path: /var/log/dovecot/info.log
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap pop3
listen(default): 208.200.248.2 208.200.248.7
listen(imap): 208.200.248.2 208.200.248.7
listen(pop3): 208.200.248.2
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
login_greeting: Hello there.
login_log_format_elements: <%u> <%r>
login_log_format: %s
login_process_per_connection: no
login_process_size: 128
login_processes_count: 4
login_max_processes_count: 8
login_max_connections: 128
valid_chroot_dirs: /mail/mail
mail_max_userip_connections(default): 15
mail_max_userip_connections(imap): 15
mail_max_userip_connections(pop3): 4
verbose_proctitle: yes
first_valid_uid: 8
last_valid_uid: 8
first_valid_gid: 8
last_valid_gid: 8
mail_access_groups: mail
mail_privileged_group: mail
mail_uid: 8
mail_gid: 8
mail_location: maildir:~/:INDEX=/dovecot-index
mailbox_idle_check_interval: 60
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mailbox_list_index_disable: no
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3):
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
mail_log_prefix: %Ls <%u>:
imap_client_workarounds(default): outlook-idle delay-newmail
imap_client_workarounds(imap): outlook-idle delay-newmail
imap_client_workarounds(pop3):
pop3_no_flag_updates(default): no
pop3_no_flag_updates(imap): no
pop3_no_flag_updates(pop3): yes
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s
pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s
auth default:
mechanisms: plain login
default_realm: olypen.com
cache_size: 8192
cache_ttl: 300
worker_max_count: 60
passdb:
driver: sql
args: /etc/dovecot/dovecot-sql.conf
userdb:
driver: prefetch
userdb:
driver: sql
args: /etc/dovecot/dovecot-sql.conf
socket:
type: listen
client:
path: /var/run/dovecot/auth-client
mode: 438
user: mail
group: mail
master:
path: /var/run/dovecot/auth-master
mode: 438
user: mail
group: mail
plugin:
quota: maildir
quota_rule: Trash:ignore
quota_rule2: Sent:ignore
quota_rule3: Spam:ignore
quota_rule4: Archive:ignore
quota_rule5: Sent Items:ignore
quota_rule6: Deleted Items:ignore
auth_socket_path: /var/run/dovecot/auth-master
On Mon, Oct 27, 2008 at 9:20 AM, Brandon Lamb <brandonlamb at gmail.com> wrote:> Why would I be seeing a million of these? > > dovecot: 2008-10-27 09:12:33 Warning: > chdir(/mail/mail/domain/u/username/) blocked for 18 secs > > Two servers were working fine and then all of a sudden this morning > bam 500 load and theres a million dovecots and tons of these lines in > the logs. > > It is like everyone gets up at exactly 8:30 and hits the pop3 server at once. > > Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, > 2.6.26-1-amd64 i had the performance mode options for login > configured, i just set back to the single process per login though, I > think that hasnt fallen over that way but I really dont remember > anymore. I included the dovecot -n using the suggest performance login > settings? Do I have something totally out of whack? > > # 1.1.5: /etc/dovecot/dovecot.conf > base_dir: /var/run/dovecot/ > log_path: /var/log/dovecot/dovecot.log > info_log_path: /var/log/dovecot/info.log > log_timestamp: %Y-%m-%d %H:%M:%S > protocols: imap pop3 > listen(default): 208.200.248.2 208.200.248.7 > listen(imap): 208.200.248.2 208.200.248.7 > listen(pop3): 208.200.248.2 > disable_plaintext_auth: no > login_dir: /var/run/dovecot/login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > login_greeting: Hello there. > login_log_format_elements: <%u> <%r> > login_log_format: %s > login_process_per_connection: no > login_process_size: 128 > login_processes_count: 4 > login_max_processes_count: 8 > login_max_connections: 128 > valid_chroot_dirs: /mail/mail > mail_max_userip_connections(default): 15 > mail_max_userip_connections(imap): 15 > mail_max_userip_connections(pop3): 4 > verbose_proctitle: yes > first_valid_uid: 8 > last_valid_uid: 8 > first_valid_gid: 8 > last_valid_gid: 8 > mail_access_groups: mail > mail_privileged_group: mail > mail_uid: 8 > mail_gid: 8 > mail_location: maildir:~/:INDEX=/dovecot-index > mailbox_idle_check_interval: 60 > mmap_disable: yes > mail_nfs_storage: yes > mail_nfs_index: yes > mailbox_list_index_disable: no > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota > mail_plugins(imap): quota imap_quota > mail_plugins(pop3): > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > mail_log_prefix: %Ls <%u>: > imap_client_workarounds(default): outlook-idle delay-newmail > imap_client_workarounds(imap): outlook-idle delay-newmail > imap_client_workarounds(pop3): > pop3_no_flag_updates(default): no > pop3_no_flag_updates(imap): no > pop3_no_flag_updates(pop3): yes > pop3_client_workarounds(default): > pop3_client_workarounds(imap): > pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh > pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s > pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s > pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s > auth default: > mechanisms: plain login > default_realm: olypen.com > cache_size: 8192 > cache_ttl: 300 > worker_max_count: 60 > passdb: > driver: sql > args: /etc/dovecot/dovecot-sql.conf > userdb: > driver: prefetch > userdb: > driver: sql > args: /etc/dovecot/dovecot-sql.conf > socket: > type: listen > client: > path: /var/run/dovecot/auth-client > mode: 438 > user: mail > group: mail > master: > path: /var/run/dovecot/auth-master > mode: 438 > user: mail > group: mail > plugin: > quota: maildir > quota_rule: Trash:ignore > quota_rule2: Sent:ignore > quota_rule3: Spam:ignore > quota_rule4: Archive:ignore > quota_rule5: Sent Items:ignore > quota_rule6: Deleted Items:ignore > auth_socket_path: /var/run/dovecot/auth-masterWell, I just rebooted all 4 servers and now the load is a cool 0.50 and under.... What would make them all be around 10 (well 40 to 100++) but then back down to 5 to 10 after restarting dovecot. But then a reboot and they come back and stay at under 1 for a day.
On Mon, Oct 27, 2008 at 9:20 AM, Brandon Lamb <brandonlamb at gmail.com> wrote:> Why would I be seeing a million of these? > > dovecot: 2008-10-27 09:12:33 Warning: > chdir(/mail/mail/domain/u/username/) blocked for 18 secs > > Two servers were working fine and then all of a sudden this morning > bam 500 load and theres a million dovecots and tons of these lines in > the logs. > > It is like everyone gets up at exactly 8:30 and hits the pop3 server at once. > > Anyway, dovecot 1.1.5, nfs mount, debian lenny/sid amd64, > 2.6.26-1-amd64 i had the performance mode options for login > configured, i just set back to the single process per login though, I > think that hasnt fallen over that way but I really dont remember > anymore. I included the dovecot -n using the suggest performance login > settings? Do I have something totally out of whack? > > # 1.1.5: /etc/dovecot/dovecot.conf > base_dir: /var/run/dovecot/ > log_path: /var/log/dovecot/dovecot.log > info_log_path: /var/log/dovecot/info.log > log_timestamp: %Y-%m-%d %H:%M:%S > protocols: imap pop3 > listen(default): 208.200.248.2 208.200.248.7 > listen(imap): 208.200.248.2 208.200.248.7 > listen(pop3): 208.200.248.2 > disable_plaintext_auth: no > login_dir: /var/run/dovecot/login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > login_greeting: Hello there. > login_log_format_elements: <%u> <%r> > login_log_format: %s > login_process_per_connection: no > login_process_size: 128 > login_processes_count: 4 > login_max_processes_count: 8 > login_max_connections: 128 > valid_chroot_dirs: /mail/mail > mail_max_userip_connections(default): 15 > mail_max_userip_connections(imap): 15 > mail_max_userip_connections(pop3): 4 > verbose_proctitle: yes > first_valid_uid: 8 > last_valid_uid: 8 > first_valid_gid: 8 > last_valid_gid: 8 > mail_access_groups: mail > mail_privileged_group: mail > mail_uid: 8 > mail_gid: 8 > mail_location: maildir:~/:INDEX=/dovecot-index > mailbox_idle_check_interval: 60 > mmap_disable: yes > mail_nfs_storage: yes > mail_nfs_index: yes > mailbox_list_index_disable: no > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota > mail_plugins(imap): quota imap_quota > mail_plugins(pop3): > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > mail_log_prefix: %Ls <%u>: > imap_client_workarounds(default): outlook-idle delay-newmail > imap_client_workarounds(imap): outlook-idle delay-newmail > imap_client_workarounds(pop3): > pop3_no_flag_updates(default): no > pop3_no_flag_updates(imap): no > pop3_no_flag_updates(pop3): yes > pop3_client_workarounds(default): > pop3_client_workarounds(imap): > pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh > pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s > pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s > pop3_logout_format(pop3): retr=%r/%b, del=%d/%m, size=%s > auth default: > mechanisms: plain login > default_realm: olypen.com > cache_size: 8192 > cache_ttl: 300 > worker_max_count: 60 > passdb: > driver: sql > args: /etc/dovecot/dovecot-sql.conf > userdb: > driver: prefetch > userdb: > driver: sql > args: /etc/dovecot/dovecot-sql.conf > socket: > type: listen > client: > path: /var/run/dovecot/auth-client > mode: 438 > user: mail > group: mail > master: > path: /var/run/dovecot/auth-master > mode: 438 > user: mail > group: mail > plugin: > quota: maildir > quota_rule: Trash:ignore > quota_rule2: Sent:ignore > quota_rule3: Spam:ignore > quota_rule4: Archive:ignore > quota_rule5: Sent Items:ignore > quota_rule6: Deleted Items:ignore > auth_socket_path: /var/run/dovecot/auth-masterdotlock_use_excl = yes fsync_disable = no lock_method = fcntl Could I be running into some nfs locking issue? Wouldnt a reboot clear any nfs um, handles or whatnot that the mail server(s) might have? Do these look like the right values for using NFS? Should I try using lock_method = dotlock instead?