dovecot - Aug 2004 - MySQL passdb, auth_verbose, and documentation

I've been playing around with Dovecot for the past few months for 
testing (on a Debian 3.0 system), and initially it was easy to setup 
and worked well.

Recently I upgraded to 0.99.10.9, and when I rebuilt it I enabled 
MySQL and SSL.

The first problem I ran into was upon startup it complained that there 
was no certificate, which makes sense, except that I hadn't enabled 
imaps in the config file. It would make more sense not to require SSL 
support files if SSL isn't being used. But this is minor.

I encountered more serious problems when I tried migrating from real 
user accounts to virtual accounts via MySQL. Has anyone written a 
howto on setting up Dovecot with MySQL? (Just pointing to 
dovecot-mysql.conf leaves out a lot.) If not, I'll volunteer to write 
something up. 

I set up dovecot-mysql.conf as I thought it should be, created a 
database, added a record with a digest-md5 password, and changed some 
dovecot.conf directives as follows:

default_mail_env = maildir:/var/mail/%d/%n/
auth_mechanisms = digest-md5
auth_default_realm = example.com
auth_userdb = static uid=5000 gid=5000 home=/var/mail/%d/%n/
auth_passdb = mysql /etc/dovecot-mysql.conf


For the static auth_userdb I wasn't quite sure what to put after home= 
as I wasn't really defining a home directory location. I initially 
tried making it identical to default_mail_env, but later learned that 
it clearly didn't like the "maildir:" prefix.

Some questions the documentation left unanswered: Is home= required if 
default_mail_env will suffice for virtual users? Can mail= be used 
instead, as implied by the database documentation? Practically 
speaking, is there any difference between the two in the case of 
virtual user accounts?

I tried to access Dovecot from a mail client and repeatedly received a 
login failure. Unfortunately, this is typical of what I saw in the log 
file:

Aug  8 03:53:50 lex dovecot: Dovecot starting up
Aug  8 03:53:52 lex dovecot-auth: MySQL: connected to localhost
Aug  8 03:55:13 lex imap-login: Disconnected: Inactivity [192.168.0.200]

I enabled auth_verbose:
auth_verbose = yes

and checked my syslog.conf to make sure all priorities were being 
logged:
mail.*                          /var/log/syslog/mail.log

but still no additional verbosity. How should things look different 
when auth_verbose is enabled?

Checking MySQL's log I could see that Dovecot was successfully 
connecting to the database server, but never executing any queries. I 
was going to next turn on logging of all IMAP traffic, but noticed the 
Wiki suggested trying a manual session, which I did:

% telnet localhost imap
* OK dovecot ready.
1 LOGIN tmetro at example.com password
1 NO Login failed: Unsupported authentication mechanism

Isn't this the kind of thing that should appear in the log when 
auth_verbose is enabled?

That seemed odd, as 'digest-md5' is mentioned right in dovecot.conf as 
a valid choice.

In retrospect, I gather what the message above is trying to tell me is 
that Dovecot wasn't configured to accept a 'plain' authentication, 
which is apparently what I was using in my manual session. It would be 
nice if the error message could express that a bit better.

I did have some suspicion that it might be the that plain text 
authentication was being prevented, as at that point I didn't know 
what auth_mechanisms applied to, but I looked at dovecot.conf and 
found disable_plaintext_auth = no, which seemed to imply that plain 
text at the IMAP protocol level should have been permitted. Might it 
be clearer if that attribute was incorporated into the auth_mechanisms 
settings or at least moved into the auth section?

I switched it to 'plain-md5', as mentioned in auth.txt, and tried 
again. That resulted in:

Aug  8 21:11:53 lex dovecot-auth: Unknown authentication mechanism
'plain-md5'
Aug  8 21:11:53 lex dovecot: Auth process died too early - shutting down
Aug  8 21:11:53 lex dovecot: child 10293 (auth) returned error 89
Aug  8 21:11:53 lex imap-login: fd_send(-1) failed: Broken pipe


That got me wondering whether there were two different areas where 
authentication terms apply - one being the password exchange in 
the IMAP protocol, and the other being how the password is stored on 
the server - yet the discussion of the two seems to be mixed together 
in the documentation.

I tried switching to:
auth_mechanisms = plain

and finally got something more expected:

Aug  8 21:20:44 lex dovecot-auth: mysql(tmetro at example.com): Password
mismatch

I tried putting the password into the database unencrypted, but that 
didn't work (probably because of my default_pass_scheme setting?). 
Generating a PLAIN-MD5 per auth.txt finally worked.

This leads to some questions: auth_mechanisms doesn't seem to be 
describing the way in which the password is stored, so what does it 
describe? In dovecot-mysql.conf I had:
default_pass_scheme = DIGEST-MD5

and didn't change it. What purpose does it serve? Is it only to 
identify the encryption used when there isn't a {SCHEME} tag present?

For DIGEST-MD5, which encrypts "user:realm:pass", does realm include 
the @? Does user include @realm? I assumed no in both cases, yet it 
didn't work.


In other matters...during one of my login attempts I got:

* OK dovecot ready.
1 LOGIN tmetro at example.com password
1 OK Logged in.
Connection closed by foreign host.

The abrupt disconnect after an apparently successful login seemed odd. 
The log file showed:

Aug  8 21:23:47 lex dovecot: chdir(maildir:/var/mail/example.com/tmetro/) failed
with uid 5000: Permission denied
Aug  8 21:23:47 lex dovecot: child 10315 (imap) returned error 89

which is the problem I discussed above, but is it reasonable for 
Dovecot to disconnect the IMAP socket and not send an error message to 
the client?


At the end of the dovecot-example.conf is:

#auth = digest_md5
#auth_methods = digest-md5
#auth_realms = 
#auth_userdb = passwd-file /etc/passwd.imap
#auth_passdb = passwd-file /etc/passwd.imap
#auth_user = imapauth
#auth_chroot = 

Is auth_methods an alias for auth_mechanisms?


Aside from the insufficient detail in the log, most of the difficulty 
I encountered could be avoided with better documentation and comments 
in the config files. I'd be happy to submit clarified comments once I 
get some answers to the above questions and confirm my understanding 
of how things work.

 -Tom

Hi Tom,

I can't answer all your questions but maybe some of them:

Tom wrote:
> I encountered more serious problems when I tried migrating from real
> user accounts to virtual accounts via MySQL. Has anyone written a 
> howto on setting up Dovecot with MySQL? (Just pointing to 
> dovecot-mysql.conf leaves out a lot.)
Yes, I did setup a new server with dovecot today. Here is my
dovecot-mysql.conf:
db_host = localhost
db_port = 3306
db = xams
db_user = dovecot
db_passwd = hallo
db_client_flags = 0
default_pass_scheme = PLAIN-MD5
#password_query = SELECT password FROM users WHERE username = '%u'
password_query = SELECT u.password FROM pm_sites s INNER JOIN pm_domains d ON
s.id = d.siteid INNER JOIN pm_users u ON s.id = u.siteid WHERE s.sitestate =
'default' AND d.name = '%d' AND u.name = '%n' AND
u.accountstate = 'default'
#user_query = SELECT home, uid, gid FROM users WHERE username = '%u'
db_unix_socket = 

Password query is a bit more complex as I am using XAMS
(www.xams.org). As default_pass_scheme says the password in the
database is md5 encrypted.
> I set up dovecot-mysql.conf as I thought it should be, created a
> database, added a record with a digest-md5 password, and changed some 
> dovecot.conf directives as follows:
> default_mail_env = maildir:/var/mail/%d/%n/
> auth_mechanisms = digest-md5
> auth_default_realm = example.com
> auth_userdb = static uid=5000 gid=5000 home=/var/mail/%d/%n/
> auth_passdb = mysql /etc/dovecot-mysql.conf
That is very similar to my setup except that I'm using
     auth_mechanisms = plain
because Mozilla doesn't support other mechanisms.
> I tried putting the password into the database unencrypted, but that
> didn't work (probably because of my default_pass_scheme setting?). 
Yes, I think so.
> This leads to some questions: auth_mechanisms doesn't seem to be
> describing the way in which the password is stored, so what does it 
> describe?
I think it descrips the format in which the password is delivered by
the mail client.
> Aside from the insufficient detail in the log,
I want to second that one. More and detailled error messages can
definitely help. To me it would have been very helpful if there were
a debugging log mode where all communication with the client and the
database is shown in the logs (including passwords etc!).

HTH

-- 
Felix

On 9.8.2004, at 09:31, Tom Metro wrote:
> The first problem I ran into was upon startup it complained that there
> was no certificate, which makes sense, except that I hadn't enabled
> imaps in the config file. It would make more sense not to require SSL
> support files if SSL isn't being used. But this is minor.
imaps in protocol line only means that it's listening in imaps-port. 
SSL can still be used by giving STARTTLS command. So SSL is really 
disabled only when you set ssl_disable = yes, and then it doesn't ask 
about certificates.
> I encountered more serious problems when I tried migrating from real
> user accounts to virtual accounts via MySQL. Has anyone written a
> howto on setting up Dovecot with MySQL? (Just pointing to
> dovecot-mysql.conf leaves out a lot.) If not, I'll volunteer to write
> something up.
I've written howto for Postgresql in wiki. Maybe it could be used as a 
base for Mysql-specific howto.
> Some questions the documentation left unanswered: Is home= required if
> default_mail_env will suffice for virtual users? Can mail= be used
> instead, as implied by the database documentation? Practically
> speaking, is there any difference between the two in the case of
> virtual user accounts?
Dovecot chdir()s into user's home directory. Currently there's no other 
difference. Maybe later I add some other things such as possibility to 
read user-specific settings from ~/.dovecotrc. Also if you enable 
rawlogs they're written under home directory.
> Aug  8 03:53:50 lex dovecot: Dovecot starting up
> Aug  8 03:53:52 lex dovecot-auth: MySQL: connected to localhost
> Aug  8 03:55:13 lex imap-login: Disconnected: Inactivity 
> [192.168.0.200]
..
> but still no additional verbosity. How should things look different
> when auth_verbose is enabled?
auth_verbose logs each authentication attempt and reason if it failed.
> 1 LOGIN tmetro at example.com password
> 1 NO Login failed: Unsupported authentication mechanism
>
> Isn't this the kind of thing that should appear in the log when
> auth_verbose is enabled?
Hmm. auth_verbose is handled by dovecot-auth process completely. In 
this case imap-login process never sent anything to dovecot-auth 
because it knew itself that it won't work.

Perhaps it should be logged, but it'd be kind of annoying to fix 
without ugly kludging.
> That seemed odd, as 'digest-md5' is mentioned right in dovecot.conf
as
> a valid choice.
It enables digest-md5 authentication mechanism then which works only if 
client supports and uses it.
> In retrospect, I gather what the message above is trying to tell me is
> that Dovecot wasn't configured to accept a 'plain'
authentication,
> which is apparently what I was using in my manual session. It would be
> nice if the error message could express that a bit better.
Yes, could be. The error message is fine for AUTHENTICATE command I 
think, but for LOGIN it could be something like: "Plaintext 
authentication mechanism is disabled"
> I did have some suspicion that it might be the that plain text
> authentication was being prevented, as at that point I didn't know
> what auth_mechanisms applied to, but I looked at dovecot.conf and
> found disable_plaintext_auth = no, which seemed to imply that plain
> text at the IMAP protocol level should have been permitted. Might it
> be clearer if that attribute was incorporated into the auth_mechanisms
> settings or at least moved into the auth section?
Not really. Plaintext auth mechanisms can still be used if SSL is 
enabled. As for moving into auth section.. Hm.. Maybe. But it's also 
somewhat SSL-specific setting and it's currently grouped with SSL 
settings. How about:

# Space separated list of wanted authentication mechanisms:
#   plain digest-md5 anonymous
#
# All IMAP/POP3 clients support plain-authentication. Digest-MD5 is more
# secure, but it's not widely supported by clients. Note that by default
# plaintext authentication is disabled unless SSL is used - see
# disable_plaintext_auth setting.
auth_mechanisms = plain
> That got me wondering whether there were two different areas where
> authentication terms apply - one being the password exchange in
> the IMAP protocol, and the other being how the password is stored on
> the server - yet the discussion of the two seems to be mixed together
> in the documentation.
Where? I don't see any obvious mixes, except that DIGEST-MD5 is both an 
authentication mechanism name and password scheme name (because they're 
supposed to be used together).
> I tried putting the password into the database unencrypted, but that
> didn't work (probably because of my default_pass_scheme setting?).
Yep.
> This leads to some questions: auth_mechanisms doesn't seem to be
> describing the way in which the password is stored, so what does it
> describe?
The way client and server perform the authentication. With plain client 
sends the password directly, others are more advanced.
>  In dovecot-mysql.conf I had:
> default_pass_scheme = DIGEST-MD5
>
> and didn't change it.
Default is PLAIN-MD5. DIGEST-MD5 scheme isn't very useful unless you're 
using DIGEST-MD5 authentication.
>  What purpose does it serve? Is it only to
> identify the encryption used when there isn't a {SCHEME} tag present?
Right.
> For DIGEST-MD5, which encrypts "user:realm:pass", does realm
include
> the @? Does user include @realm? I assumed no in both cases, yet it
> didn't work.
Clients that support DIGEST-MD5 usually ask you for username, realm and 
password separately. So there's no @ anywhere. Although I think 
currently if realm isn't given but username contains @ character, the 
realm is taken from after the @ char. Realms are a bit confusing and 
I'm not sure how well they even work.
> Aug  8 21:23:47 lex dovecot: 
> chdir(maildir:/var/mail/example.com/tmetro/) failed with uid 5000: 
> Permission denied
> Aug  8 21:23:47 lex dovecot: child 10315 (imap) returned error 89
>
> which is the problem I discussed above, but is it reasonable for
> Dovecot to disconnect the IMAP socket and not send an error message to
> the client?
In case of a initial server misconfiguration, I think it's good enough. 
And I'm not sure if there's any easy and non-horribly-kludgy way to do 
it with current design.
> Is auth_methods an alias for auth_mechanisms?
It's an old name for it that I had forgotten to change. Fixed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20040809/048144ad/attachment-0001.bin>

Timo Sirainen wrote:
>> 1 LOGIN tmetro at example.com password
>> 1 NO Login failed: Unsupported authentication mechanism
> 
> The error message is fine for AUTHENTICATE command I 
> think, but for LOGIN it could be something like: "Plaintext 
> authentication mechanism is disabled"
Yes, that would help clarify.

>> That got me wondering whether there were two different areas where
>> authentication terms apply - one being the password exchange in
>> the IMAP protocol, and the other being how the password is stored 
>> on the server - yet the discussion of the two seems to be mixed 
>> together in the documentation.
> 
> Where? I don't see any obvious mixes, except that DIGEST-MD5...
http://dovecot.org/doc/auth.txt says:
> Authentication is split into three parts: authentication mechanism,
> password database and user database.
Which is good, except it never defines what "authentication mechanism"
really means, and how it is distinct from how the password is stored.

Password database is quite self explanatory, but it wouldn't hurt to 
explain the purpose of the user database here.

The information in auth.txt is of course technically correct, but the 
listing of mechanisms at the top of the file looks enough like the list 
of schemes under "Generating passwords" to cause confusion.

> ...DIGEST-MD5 is both an authentication mechanism name and password 
> scheme name (because they're supposed to be used together).
That's enough to start raising confusion. There are PLAIN-MD5 and MD5 
schemes that look similar to PLAIN and CRAM-MD5 mechanisms. When you're 
looking at something as a new user, you never know if the differences 
are significant or just inconsistencies.

> DIGEST-MD5 scheme isn't very useful unless you're using DIGEST-MD5 
> authentication.
A pretty important point that isn't made in auth.txt.


Quoting from auth.txt again:
> Most password databases support only plaintext authentication. passwd-file
> and LDAP exceptions since they support multiple password schemes.
The reasoning behind this should be explained. (Though that would be 
more obvious once "authentication mechanism" is well defined.) Also
the
wording is confusing as it is, and also out of date because it doesn't 
mention pgsql and mysql.

The point you want to make is that if the client authenticates using 
plain text in the IMAP protocol, then Dovecot can encrypt the password 
to match a variety of encryption schemes used in password databases.

If the client authenticates using a mechanism like DIGEST-MD5, which is 
a one-way hashing algorithm, Dovecot can't "translate" it to some
other
scheme, so the password databases must use a password encryption scheme 
that matches the IMAP protocol authentication mechanism.

> How about:
> 
> # Space separated list of wanted authentication mechanisms:
> #   plain digest-md5 anonymous
> #
> # All IMAP/POP3 clients support plain-authentication. Digest-MD5 is more
> # secure, but it's not widely supported by clients. Note that by
default
> # plaintext authentication is disabled unless SSL is used - see
> # disable_plaintext_auth setting.
> auth_mechanisms = plain
Here too, I think it would be useful to define "authentication 
mechanisms", but you should probably first provide a better definition 
for "Authentication processes." Though the term "process"
makes me think
  that we're talking about an executable. But I guess it is defining the 
behavior of a child process - right?

Perhaps:

##
## Authentication processes
##

# An Authentication process is a child process used by Dovecot that
# handles the authentication steps. The steps cover an authentication
# mechanism (auth_mechanisms, how the client authenticates in the IMAP
# protocol), which password database should be queried (auth_passdb),
# and which user database should be queried (auth_userdb, to obtain
# UID, GID, and location of the user's mailbox/home directory).
#
# You can have multiple processes, though a typical configuration will
# have only one. Each time "auth = xx" is seen, a new process
# definition is started. The point of multiple processes is to be able
# to set stricter permissions. (See auth_user below.)
#
# Just remember that only one Authentication process is asked for the
# password, so you can't have different passwords accessible through
# different process definitions (unless they have different
# auth_mechanisms, and you're ok with having different password for
# each mechanisms).

(What is the order in which Authentication processes are chosen, if say, 
you have multiple defined for the 'plain' auth_mechanisms? Or is that 
considered a configuration error?

The above paragraph might be describing an obscure enough condition that 
it would be better moved to auth.txt. If anything, the point you might 
want to get across is that each auth mechanism should uniquely map to a 
single password database.)


And...

# Specifies how the client authenticates in the IMAP protocol.
# Space separated list of permitted authentication mechanisms:
#   anonymous plain digest-md5 cram-md5
#
# anonymous - No authentication required.
# plain - The password is sent as plain text. All IMAP/POP3 clients
#  support this, and the password can be encrypted by Dovecot to match
#  any of the encryption schemes used in password databases.
# digest-md5 and cram-md5 - both encrypt the password so it is more
#  secure in transit, but are not well supported by clients, and
#  require that the password database use a matching encryption
#  scheme.
#
# See auth.txt for more details.
#
# If you are using SSL there is less benefit to digest-md5 and
# cram-md5 as the communication is already encrypted. Note that by
# default plain text authentication is disabled unless SSL is used -
# see disable_plaintext_auth setting.
auth_mechanisms = plain

(I thought by default disable_plaintext_auth=no. Has that changed?

My assertion, "the password can be encrypted by Dovecot to match any of 
the encryption schemes used in password databases," may not be accurate. 
Your comments seem to imply that Dovecot won't translate a plain 
password to a digest-md5 storage scheme (perhaps also cram-md5), though 
it seems this should be possible.)

  -Tom

Timo Sirainen wrote:
>> auth_userdb = static uid=5000 gid=5000 home=/var/mail/%d/%n/
 >>
>> Some questions the documentation left unanswered: 
>> Is home= required if default_mail_env will suffice for virtual users?
>> Can mail= be used instead, as implied by the database documentation?
These questions weren't answered in your earlier reply.

>> Practically speaking, is there any difference between the two in the
case of
>> virtual user accounts?
> 
> Dovecot chdir()s into user's home directory. Currently there's no
other
> difference.
Though if only home= is supported by static, then it seems there isn't a 
mechanism to specify the mailbox type (other than as implied by the 
trailing slash) or the INBOX location.

It seems a little confusing that in a real account scenario home= 
defines the user's actual home directory (right?), but not necessarily 
their mail directory, yet under a virtual user scenario the two are the 
same thing. Or maybe it just seems that way because my default_mail_env 
is set to the identical path in my case, as would be typical in a 
virtual user setup.


Also, would it be practical to check during startup that the UID and GID 
specified by the static db don't fall outside the range of:

first_valid_uid = 1000
last_valid_uid = 5001
first_valid_gid = 1000
last_valid_gid = 5001

I initially had that problem with my setup, but fairly quickly saw a run 
time error in the log that led me to the cause.

  -Tom