Hello All,
We are struggling to find a solution to a problem we are encountering
with a load-balanced email setup. Currently, we have a Coyote loadbalancer,
and 3 Postfix/Dovecot nodes that then get their information from a mysql
database. The problem is that after a couple weeks, we start seeing NFS
locking issues occur, which then takes email completely down, requiring a
site visit to basically do an interactive mode reboot to shut off NFS so
that we can remount the filesystem again. We have our mailstore on an
Aberdeen NAS which is accessed over NFS v. 4.
Dovecot version 1.1.2
Here's the relevant mysql string:
grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf
driver = mysql
connect = host=X.X.X.X dbname=Something user=Something password=Something
password_query = SELECT username as user, password,
concat('/NFS1MAILDIR/mailSysV2/', maildir) as userdb_home,
concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as userdb_mail, 143
as
userdb_uid, 143 as userdb_gid, concat('*:bytes=', floor(quota*1024)) AS
quota_rule FROM mailbox WHERE username = '%u' AND active = '1'
user_query = SELECT concat('/NFS1MAILDIR/mailSysV2/', maildir) as home,
concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as mail,
concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as maildir, 143 AS
uid,
143 AS gid, concat('*:bytes=', floor(quota*1024)) AS quota_rule FROM
mailbox
WHERE username = '%u' AND active = '1'
And here's what our configuration is on each of the nodes:
dovecot -n
# 1.1.2: /usr/local/etc/dovecot.conf
protocols: imap pop3
ssl_ca_file: /usr/local/etc/dovecot/certs/tdpserver.crt
ssl_cert_file: /usr/local/etc/dovecot/certs/tdpserver.crt
ssl_key_file: /usr/local/etc/dovecot/certs/tdpserver.key
ssl_parameters_regenerate: 0
ssl_cipher_list: ALL:!LOW:!SSLv2
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
login_user: postfix
login_greeting: Something witty here..
login_process_per_connection: no
login_max_connections: 512
max_mail_processes: 2048
mail_max_userip_connections: 50
verbose_proctitle: yes
first_valid_uid: 69
last_valid_uid: 500
first_valid_gid: 69
last_valid_gid: 500
mail_access_groups: postfix
mail_privileged_group: postfix
mail_location:
mbox:/NFS1MAILDIR/mailSysV2/%d/%u:INDEX=/usr/local/mail/indexes/%d/%1n/%n
mmap_disable: yes
dotlock_use_excl: no
mail_nfs_storage: yes
mail_nfs_index: yes
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot/imap
mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3
imap_client_workarounds(default): delay-newmail netscape-eoh
tb-extra-mailbox-sep
imap_client_workarounds(imap): delay-newmail netscape-eoh
tb-extra-mailbox-sep
imap_client_workarounds(pop3):
pop3_client_workarounds(default):
pop3_client_workarounds(imap):
pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh
auth default:
mechanisms: login plain
user: dovecot
passdb:
driver: sql
args: /usr/local/etc/dovecot-mysql.conf
userdb:
driver: sql
args: /usr/local/etc/dovecot-mysql.conf
socket:
type: listen
client:
path: /var/spool/postfix/private/auth
mode: 432
user: postfix
group: postfix
master:
path: /var/run/dovecot/auth-master
mode: 384
user: dovecot
group: dovecot
plugin:
quota: dict:user::proxy::quota
imap_quota: dict:user::proxy::quota
quota_rule: *:storage=528576
dict:
quota: mysql:/usr/local/etc/dovecot/dovecot-dict-quota.conf
Any help you can offer is greatly, greatly appreciated, as well as opinions
on any other 'validated' postfix/dovecot email enterprise design
solutions.
Thanks Very Much In Advance,
R.T
Sounds like an issue with nfs 4 to me? We're running nfs3 very happily no issues like this have happened. I don't see that running nfs4 offers many advantages over v3 for the email workload. Mark -- Mark Zealey -- Shared Hosting Team Leader Product Development * Webfusion 123-reg.co.uk, webfusion.co.uk, donhost.co.uk, supanames.co.uk This mail is subject to http://www.gxn.net/disclaimer> -----Original Message----- > From: dovecot-bounces+mark.zealey=webfusion.com at dovecot.org > [mailto:dovecot-bounces+mark.zealey=webfusion.com at dovecot.org] > On Behalf Of Rod Treweek > Sent: 11 November 2008 00:07 > To: dovecot at dovecot.org > Cc: Tait Grove > Subject: [Dovecot] Unbreakable NFS locking issues... > > Hello All, > > We are struggling to find a solution to a problem we are > encountering > with a load-balanced email setup. Currently, we have a Coyote > loadbalancer, > and 3 Postfix/Dovecot nodes that then get their information > from a mysql > database. The problem is that after a couple weeks, we start > seeing NFS > locking issues occur, which then takes email completely down, > requiring a > site visit to basically do an interactive mode reboot to shut > off NFS so > that we can remount the filesystem again. We have our mailstore on an > Aberdeen NAS which is accessed over NFS v. 4. > > Dovecot version 1.1.2 > > Here's the relevant mysql string: > > grep -v '^ *\(#.*\)\?$' dovecot-mysql.conf > driver = mysql > connect = host=X.X.X.X dbname=Something user=Something > password=Something > password_query = SELECT username as user, password, > concat('/NFS1MAILDIR/mailSysV2/', maildir) as userdb_home, > concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as > userdb_mail, 143 as > userdb_uid, 143 as userdb_gid, concat('*:bytes=', > floor(quota*1024)) AS > quota_rule FROM mailbox WHERE username = '%u' AND active = '1' > user_query = SELECT concat('/NFS1MAILDIR/mailSysV2/', > maildir) as home, > concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as mail, > concat('maildir:/NFS1MAILDIR/mailSysV2/', maildir) as > maildir, 143 AS uid, > 143 AS gid, concat('*:bytes=', floor(quota*1024)) AS > quota_rule FROM mailbox > WHERE username = '%u' AND active = '1' > > > And here's what our configuration is on each of the nodes: > > dovecot -n > # 1.1.2: /usr/local/etc/dovecot.conf > protocols: imap pop3 > ssl_ca_file: /usr/local/etc/dovecot/certs/tdpserver.crt > ssl_cert_file: /usr/local/etc/dovecot/certs/tdpserver.crt > ssl_key_file: /usr/local/etc/dovecot/certs/tdpserver.key > ssl_parameters_regenerate: 0 > ssl_cipher_list: ALL:!LOW:!SSLv2 > disable_plaintext_auth: no > login_dir: /var/run/dovecot/login > login_executable(default): /usr/local/libexec/dovecot/imap-login > login_executable(imap): /usr/local/libexec/dovecot/imap-login > login_executable(pop3): /usr/local/libexec/dovecot/pop3-login > login_user: postfix > login_greeting: Something witty here.. > login_process_per_connection: no > login_max_connections: 512 > max_mail_processes: 2048 > mail_max_userip_connections: 50 > verbose_proctitle: yes > first_valid_uid: 69 > last_valid_uid: 500 > first_valid_gid: 69 > last_valid_gid: 500 > mail_access_groups: postfix > mail_privileged_group: postfix > mail_location: > mbox:/NFS1MAILDIR/mailSysV2/%d/%u:INDEX=/usr/local/mail/indexe > s/%d/%1n/%n > mmap_disable: yes > dotlock_use_excl: no > mail_nfs_storage: yes > mail_nfs_index: yes > mail_executable(default): /usr/local/libexec/dovecot/imap > mail_executable(imap): /usr/local/libexec/dovecot/imap > mail_executable(pop3): /usr/local/libexec/dovecot/pop3 > mail_plugins(default): quota imap_quota > mail_plugins(imap): quota imap_quota > mail_plugins(pop3): quota > mail_plugin_dir(default): /usr/local/lib/dovecot/imap > mail_plugin_dir(imap): /usr/local/lib/dovecot/imap > mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 > imap_client_workarounds(default): delay-newmail netscape-eoh > tb-extra-mailbox-sep > imap_client_workarounds(imap): delay-newmail netscape-eoh > tb-extra-mailbox-sep > imap_client_workarounds(pop3): > pop3_client_workarounds(default): > pop3_client_workarounds(imap): > pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh > auth default: > mechanisms: login plain > user: dovecot > passdb: > driver: sql > args: /usr/local/etc/dovecot-mysql.conf > userdb: > driver: sql > args: /usr/local/etc/dovecot-mysql.conf > socket: > type: listen > client: > path: /var/spool/postfix/private/auth > mode: 432 > user: postfix > group: postfix > master: > path: /var/run/dovecot/auth-master > mode: 384 > user: dovecot > group: dovecot > plugin: > quota: dict:user::proxy::quota > imap_quota: dict:user::proxy::quota > quota_rule: *:storage=528576 > dict: > quota: mysql:/usr/local/etc/dovecot/dovecot-dict-quota.conf > > > > Any help you can offer is greatly, greatly appreciated, as > well as opinions > on any other 'validated' postfix/dovecot email enterprise > design solutions. > > Thanks Very Much In Advance, > > R.T >
Firstly, posting the same message with a slightly modified subject wont get you any answers any sooner, please refrain from doing this, it only gives you a bad name and may have the opposite affect to one you desire. On Tue, 2008-11-11 at 10:06, Rod Treweek wrote:> Hello All, > > We are struggling to find a solution to a problem we are encountering > with a load-balanced email setup. Currently, we have a Coyote loadbalancer, > and 3 Postfix/Dovecot nodes that then get their information from a mysql > database. The problem is that after a couple weeks, we start seeing NFS > locking issues occur, which then takes email completely down, requiring a > site visit to basically do an interactive mode reboot to shut off NFS so > that we can remount the filesystem again. We have our mailstore on an > Aberdeen NAS which is accessed over NFS v. 4. >Not seen this before, but we use Foundry LB's and NetApp filers. Coyote Is BSD based isn't it... How is your setup, you are not running the NFS mounts through the LB as well are you? I would hope they are on a separate private LAN> Dovecot version 1.1.2Generally when lodging requests for help, it pays to use the latest version of software, 1.1.2 is old.> Here's the relevant mysql string:If the queries work at all, there's nothing wrong with them.> mbox:/NFS1MAILDIR/mailSysV2/%d/%u:INDEX=/usr/local/mail/indexes/%d/%1n/%nmbox over NFS ? That is asking for trouble, mbox is *not* NFS safe, you are best changing to maildir which is. Cheers