dovecot - Apr 2013 - Dovecot has no understandable configuration any more

Hello,

for many years i used an mailserver with dovecot V1 very satisfied!
http://mailserver.decotrain.de/
One of the reasons to use dovecot was a simply configuration.
Dovecot was the best choice for IMAP for me!

Now i updated the server to Debian testing with dovecot 2.1.7.
It's not possible for me to get a running authentication for virtual 
mail users!

I recently found this mailing and have the same problem:
http://www.dovecot.org/list/dovecot/2012-March/064181.html

The documentation says *nothing* about my problem how to authenticate 
with mysql!
http://wiki2.dovecot.org/VirtualUsers

I can solve the problem only with try and error without a chance in 
this complex system.
It's nice to have all this possibilites but they are not useful if not 
documented.
There should be some clear examples for different scenarios to be found 
in your wiki.


My problem at this time:

I don't know how to configure the userdb to work with mysql 
authentification?
All mail is running under the special user mail.

dovecot.conf
===========
disable_plaintext_auth = yes

passdb {
   driver = sql
   # Path for SQL configuration file, see 
example-config/dovecot-sql.conf.ext
   args = /etc/dovecot/dovecot-sql.conf

}

userdb {
   driver = prefetch
}

first_valid_uid = 8
last_valid_uid = 8

first_valid_gid = 8
last_valid_gid = 8

mail_location = maildir:/srv/mail/%d/%n


dovecot-sql.conf
===============
driver = mysql

connect = host=/var/run/mysqld/mysqld.sock dbname=... => WORKS!

default_pass_scheme = SHA1

password_query = SELECT password, '/srv/mail/%d/%n' AS home,
'mail' AS
uid, 'mail' AS gid FROM user WHERE username = '%n' AND domain =
'%d' AND
IMAP_allowed = 'YES'


Errors
=====
Apr 30 10:28:22 auth: Debug: client out: OK     1       user=xxxx at yyyy  
     home=/srv/mail/yyyy/xxxx      uid=mail
         gid=mail
Apr 30 10:28:22 auth: Debug: master in: REQUEST 3163291649      5347    
1       87066c907eef8f4e5ed71319b560dd2a
Apr 30 10:28:22 auth: Error: 
prefetch(xxxx at yyyy,::1,<doTKyI/brgAAAAAAAAAAAAAAAAAAAAAB>): userdb 
lookup not possible with only userdb prefetch
Apr 30 10:28:22 auth: Error: 
userdb(xxxx at yyyy,::1,<doTKyI/brgAAAAAAAAAAAAAAAAAAAAAB>): user not
found
from userdb prefetch
Apr 30 10:28:22 auth: Debug: master out: NOTFOUND       3163291649
Apr 30 10:28:22 imap: Error: Authenticated user not found from userdb, 
auth lookup id=3163291649 (client-pid=5347 client-id=1)
Apr 30 10:28:22 imap-login: Info: Internal login failure (pid=5347 
id=1) (internal failure, 1 succesful auths): user=<karsten at
dct.mine.nu>,
method=PLAIN, rip=::1, lip=::1, mpid=5352, secured, 
session=<doTKyI/brgAAAAAAAAAAAAAAAAAAAAAB>


I have no idea how to solve the problem and up to know i could not find 
an example with google.
Please help.

Best regards
Karsten

Hello,

my last problem could be solved now with

userdb {
   driver = static
   args = uid=mail gid=mail home=/srv/mail/%d/%n
}


I can connect to dovecot locally with an webmailer.
But there is no listener process to be seen outside of the server?

Locally this ports are working, but not outside:

# netstat -tulpn | grep dove
tcp6       0      0 :::110                  :::*                    
LISTEN      5741/dovecot
tcp6       0      0 :::143                  :::*                    
LISTEN      5741/dovecot
tcp6       0      0 :::993                  :::*                    
LISTEN      5741/dovecot
tcp6       0      0 :::995                  :::*                    
LISTEN      5741/dovecot

Is this a configuration problem inside dovecot?


Another problem is the migration from the existant emails from V1 to 
V2:
In the log i can find:

Apr 30 11:14:57 imap(xxxx at yyyy): Error: Broken file 
/srv/mail/yyyy/xxxx/.INBOX.dir.09/dovecot-uidlist line 1: Broken header 
(uidvalidity = 0, next_uid=697)

How can i recover this?
Delete all the index files?

Has someone an idea how to migrate the emails from charset ISO8859-1 to 
UTF-8?


Best regards
Karsten

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 30 Apr 2013, forum wrote:
> for many years i used an mailserver with dovecot V1 very satisfied!
> http://mailserver.decotrain.de/
> One of the reasons to use dovecot was a simply configuration.
> Dovecot was the best choice for IMAP for me!
Hmm, there not many changes in how to configure Dovecot.
> The documentation says *nothing* about my problem how to authenticate with 
> mysql!
> http://wiki2.dovecot.org/VirtualUsers
Funny, that the link had the answer.
Did you configured Dovecot v1 differently then?

The static userdb had not changed since v1.

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUX/WaV3r2wJMiz2NAQK36Af9EjydgVte2ChWWV7sXc0UCiP0o+3VgcGG
89uMAWdimcuZRWAs5JyvSFuh1FSbXBm6LR0MSVIq60oGJtkpNeCb+YKaEgtz74vs
KGEnJxyBfmrF3RxKyvjwWcfQ4nifzDLOCTLfEakQSXRhW2+8OuTfE9pzhOq2DwLP
p4D8Q5Jc1wtidm65DkKp3UFBubAdmU+PTrRMmTNMkjENKiXM4h1HuklCJ0/HtERZ
gfi8QXRXdTTi6tT5PQnC821IM/PUVLU+m4aCObztiu2Fgr982ruz7kCQjyaaLio7
1OZo2nbg3lRoAtFcTSCFqliDa+rFDxYA31xmfhppG203n0xOwhRxpw==viiU
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 30 Apr 2013, forum wrote:
> I can connect to dovecot locally with an webmailer.
> But there is no listener process to be seen outside of the server?
>
> Locally this ports are working, but not outside:
>
> # netstat -tulpn | grep dove
> tcp6       0      0 :::110                  :::*                    LISTEN 
> 5741/dovecot
> tcp6       0      0 :::143                  :::*                    LISTEN 
> 5741/dovecot
> tcp6       0      0 :::993                  :::*                    LISTEN 
> 5741/dovecot
> tcp6       0      0 :::995                  :::*                    LISTEN 
> 5741/dovecot
Where is:

a) the listener on IPv4

b) the output of dovecot -n ?
> Is this a configuration problem inside dovecot?
That knows your doveconf -n output.
> Apr 30 11:14:57 imap(xxxx at yyyy): Error: Broken file 
> /srv/mail/yyyy/xxxx/.INBOX.dir.09/dovecot-uidlist line 1: Broken header 
> (uidvalidity = 0, next_uid=697)
>
> How can i recover this?
Does this error persists or do you see it just once per
mailbox?
>
> Has someone an idea how to migrate the emails from charset ISO8859-1 to 
> UTF-8?
The mailbody? recode Latin1..utf8 < in > out, but IMAP defines, that you 
cannot change messages inplace. The client need to redownload them.

The directory name? Probably there is some tool. Otherwise:

for dir in ...; do
  mv "$dir" "$(echo "$dir"| recode latin1..utf8)"
done

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUX/XbV3r2wJMiz2NAQJLZQgAp/ty1I+h7lIBuUOM8+qKcSs8GjVTjrIX
lrNEyJAh3FNL9ARlXqfGsu2bP+jOr/4nHde1H/9E215zBSzTk0XVXBr1xXSCMFv7
kWxkyUPug+YGOTkKipaaLIhwD0R5DcPx4F3oYXwLj5lybbzo8/sKrn1Hb4rW098W
swJZT5QScHjCN9S9RfqAiaCghv1Nfl8pSAXWU0I2lj1B2mMFJjYrpr4HFvNPq3vQ
a9Wdq4dXdeZmugBTSC8tQH2O3+WViLDWsbMdZSPnV6yuBPLoKwpXQxJnVOusY/HO
0iin4sypQrM703v9y/Cg6CVG7vyxBD3sUduZFhdSUlXO/aOYtktFgg==2HR1
-----END PGP SIGNATURE-----

Hello Steffen,

Am 02.05.2013 10:17, schrieb Steffen Kaiser:
> On Tue, 30 Apr 2013, forum wrote:
>
>> Now i have found the solution.
>> It seems to help to ask you. ;)
>>
>> The solution was to change from
>> listen = [::]
>> to
>> listen = *, [::]
>>
>> There is no explanation for it - just try and error ...
>
> Hmm, [::] -> in6addr_any (any in IPv6 protocol), therefore you had no
listener (er, Dovecot listener process) on IPv4,
> but IPv6 only. Maybe Dovecot v1 handled [::] differently and added *.
I can only say that it was working afterwards.
>
>> I lost many hours just to find out that i have to define e.g.
>> service imap-login {
>> Before there was no error, dovecot running and no ports listening.
>
> http://wiki2.dovecot.org/Tools/Doveconf
> has the example:
>
> "
> doveconf can be also used to convert v1.x configuration files into v2.x
format.
>
> doveconf -n -c /oldpath/dovecot.conf > /etc/dovecot/dovecot.conf.new
> "
Good to know - but to late for me.
The utilites are a good idea!

I think there are (to many) details handled slightly different.
Of course this is no problem - but not easy to find out.

I had some problems like this upgrading from exim 4.6 to 4.8 - but not so hard.
Dovecot steps into the same problems as exim.
It becomes so mighty that everything is possible - but it is hard to understand
how to master it.
The wiki is a good solution, but i miss a description of the dependencies that
are not clear.
You only understand it when you have mastered it.
>>>> Has someone an idea how to migrate the emails from charset
ISO8859-1 to UTF-8?
>>>
>>> The mailbody? recode Latin1..utf8 < in > out, but IMAP
defines, that
>>> you cannot change messages inplace. The client need to redownload
>>> them.
>>
>> The client should left the message on the server with no local copy.
>
> No, I mean: IMAP defines that messages are immutable, you must not change a
message file on the server. Therefore you
> cannot recode a message file on the server and hope that the clients pick
up the change. You need to cause that the MUAs
> think that the modified message is new. E.g. with Maildir backend you
rename each message file:
Yes - i understand.
The encoding is defined in the email - so this should be no problem.

I had the problem that Thunderbird deletes some Emails because he thought they
should be deleted.
Is this marked somewhere?
>
> 1367436834.P516Q0M961879.server:2,Sa
> -> 1367436834.P516Q0M961879.server.a:2,Sa
Ahh - good to know.
>
> see change before ":". Then each MUA thinks that it had not seen
the message, because Dovecot assigns a new UID to it.
At least i deleted all the index files and stepped with the client through the
directories.
Now everything is working fine.

Thanks for your help and tips!

Cheers
Karsten