dovecot - Sep 2013 - Bug report: "doveadm rename" encodes special characters wrongly in mUTF-7 (in fs)

Hello there,

I'm pretty sure I found a bug in latest Dovecot 2.2.5.5 (EE) when playing
with the doveadm command.

Naturally renaming a users mail folder should be done via "doveadm es
consistent.

So when using "doveadm rename" trying to rename a folder with special
characters e.g. german umlauts like "??" it seems to create the folder
wrongly in the filesystem.

Example: When I want to rename a users folder 'geschaftliches' to
'gesch&AOQ-ftliches' ("&AOQ-" is the correct mUTF-7
encoding for "??") using the command

doveadm mailbox rename -u user 'geschaftliches'
'gesch&AOQ-ftliches'

it ends up in the filesystem renamed as "gesch&-AOQ-ftliches"
which has an falsely added "-" set after the "&" and
therefore the folder is also wrongly shown in Mailclients like Thunderbird or
web-mail systems like Horde.
(Apparently "doveadm rename" supports only UTF-8 which shows up when
trying to rename the folder with
"doveadm mailbox rename -u user 'gesch??ftliches'" I get the
error message "Fatal: Mailbox name not valid UTF-8: gesch??ftliches")

To get things worse, when renaming the mailfolder in Thunderbird to
"gesch??ftliches" it gets correctly written in the filesystem as
"gesch&AOQ-ftliches" - but afterwards it's NOT found anymore
in the dovecot index, e.g. when trying
doveadm force-resync -u user
it is not listed anymore (until back-renaming in fs to the name it was indexed
before).

This problem can be reproduced. Can you please fix this? 
(Not sure if this might change - maybe add an option to support UTF-7 similar to
doveadm mailbox list "-7" option)

Thanks in advance,
Megodin



My (relevant part) dovecot configuration (dovecot -n): 

mailstorage @ NFS with nfsvers=3 (NetApp)

# 2.2.5.5 (e5350245ad9b): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.22.1.el6.x86_64 x86_64 CentOS release 6.4 (Final) nfs
auth_cache_size = 10 M
auth_cache_ttl = 10 mins
auth_debug = yes
auth_master_user_separator = *
auth_username_format = %n
auth_verbose = yes
disable_plaintext_auth = no
listen = *
lock_method = dotlock
mail_debug = yes
mail_fsync = always
mail_gid = vmail
mail_home = /mnt/placeholder/%u
mail_location = sdbox:/mnt/placeholder/%u/mailstore
mail_plugins = " quota fts fts_lucene"
mail_privileged_group = mail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date ihave
mmap_disable = yes
passdb {
  args = /etc/dovecot/users.blocked
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/passwd.masterusers
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  fts = lucene
  fts_lucene = whitespace_chars=@.
  quota = dict:User quota::file:%h/dovecot-quota
  quota_rule = *:storage=100M
  quota_rule2 = Trash:storage=+50M
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=98%% quota-warning 98 %u
  sieve = /mnt/placeholder/%u/.dovecot.sieve
  sieve_dir = /mnt/placeholder/%u/sieve
  sieve_global_dir = /var/lib/dovecot/sieve/global/
  sieve_global_path = /var/lib/dovecot/sieve/default.sieve
}
postmaster_address = postmaster at example.com
protocols = imap pop3 sieve
service auth {
  unix_listener auth-userdb {
    group = vmail
    user = vmail
  }
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  unix_listener quota-warning {
    user = vmail
  }
  user = vmail
}
ssl_cert = </example.pem
ssl_key = </example2.pem
userdb {
  driver = prefetch
}
userdb {
  args = /etc/dovecot/dovecot-ldap-userdb.conf.ext
  driver = ldap
}
verbose_proctitle = yes
verbose_ssl = yes
protocol lda {
  mail_plugins = " quota fts fts_lucene sieve"
}
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
  mail_plugins = " quota fts fts_lucene imap_quota"
}




---
Alle Postf??cher an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen!
http://email.freenet.de/basic/Informationen

_________________________________________________________________
Send and receive anonymous emails to your inbox with InboxAlias.
http://www.inboxalias.com

Sorry, it seems like some part of the sh*tty freenet webmailer omitted half of
my second sentence of the message I just posted before...


. ..so please replace the weird sentence:
"Naturally renaming a users mail folder should be done via "doveadm es
consistent."

with:
"Naturally renaming a users mail folder should be done via "doveadm es
consistent."


Sorry for confusion,
Megodin




---
Alle Postf??cher an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen!
http://email.freenet.de/basic/Informationen

_________________________________________________________________
Send and receive anonymous emails to your inbox with InboxAlias.
http://www.inboxalias.com

What the heck, I don't get it... seems like it's bug's day for me
today... the d*mn freenet web-mailclient omitted half of the sentence on exact
the same part *again* - so I split it to several lines and omit some quotes...

Hope this is the last try to get the _whole_ message out:


Naturally renaming a users mail folder should be done via 
doveadm rename
- and not via filesystems "mv" command - to keep dovecot indexes
consistent.



So sorry again for all that additional reading garbage (forum admins, please
feel free to fill in the correct sentence to my original post and delete the 2
correction posts).

Seems like a have to write another bug report to freenet admins... (sigh)

Sorry again,
Megodin




---
Alle Postf??cher an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen!
http://email.freenet.de/basic/Informationen

_________________________________________________________________
Send and receive anonymous emails to your inbox with InboxAlias.
http://www.inboxalias.com

On 20.9.2013, at 17.42, megodin at inboxalias.com wrote:
> So when using "doveadm rename" trying to rename a folder with
special characters e.g. german umlauts like "??" it seems to create
the folder wrongly in the filesystem.
> 
> Example: When I want to rename a users folder 'geschaftliches' to
'gesch&AOQ-ftliches' ("&AOQ-" is the correct mUTF-7
encoding for "??") using the command
Forget about mUTF-7 with Dovecot. It's an IMAP-protocol relic, and Dovecot
is trying to get rid of it in as many places as possible. Although by default
the filesystem users mUTF-7, but that can also be changed with a setting.
> (Apparently "doveadm rename" supports only UTF-8 which shows up
when trying to rename the folder with
> "doveadm mailbox rename -u user 'gesch??ftliches'" I get
the error message "Fatal: Mailbox name not valid UTF-8:
gesch??ftliches")
I suspect your shell is breaking the 8bit chars into invalid UTF-8. I just tried
and it works fine with me.
> To get things worse, when renaming the mailfolder in Thunderbird to
"gesch??ftliches" it gets correctly written in the filesystem as
"gesch&AOQ-ftliches" - but afterwards it's NOT found anymore
in the dovecot index, e.g. when trying
> doveadm force-resync -u user
> it is not listed anymore (until back-renaming in fs to the name it was
indexed before).
Folders aren't indexed by default (mailbox_list_index=no) and force-resync
doesn't affect them. Are you saying "doveadm mailbox list -u user"
doesn't show some folder? I'm not aware of any bugs related to this.
> This problem can be reproduced. Can you please fix this? 
I'd need to get some more specifics. Show show exists in filesystems and
what commands don't produce expected results (and what they produce
instead).
> (Not sure if this might change - maybe add an option to support UTF-7
similar to doveadm mailbox list "-7" option)
Nah, mUTF-7 hopefully will die some day.

Hi there Timo,
first of all thank you for taking your time for support!
> Forget about mUTF-7 with Dovecot. It's an IMAP-protocol relic, and
> Dovecot is trying to get rid of it in as many places as possible. 
> Although by default the filesystem users mUTF-7, but that can also 
> be changed with a setting.
Thanks for clarifying this.

> I suspect your shell is breaking the 8bit chars into invalid UTF-8. I
> just tried and it works fine with me.
Sorry, we got confusion here... apparently my (lousy) freemail provider 
which I use to post on this mailinglist doesn't send some special 
characters etc. unaltered exactly as I typed  - so I made a console 
screenshot of the doveadm rename problem to make things clear.
Please have a look here, it will reveal the doveadm rename problem 
(and that my shell is working fine): 
http://i.imgur.com/08wKji4.png


> > To get things worse, when renaming the mailfolder in Thunderbird to 
> > "gesch????ftliches" it gets correctly written in the
filesystem as
> > "gesch&AOQ-ftliches"- but afterwards it's NOT found
anymore in the
> > dovecot index, e.g. when trying doveadm force-resync -u user it is 
> > not listed anymore (untilback-renaming in fs to the name it was 
> > indexed before).
> Folders aren't indexed by default (mailbox_list_index=no) and
> force-resync doesn't affect them. Are you saying
> "doveadm mailbox list -u user" doesn't show some folder?
I'm not
> aware of any bugs related to this.
No, the "doveadm mailbox list -u user" works just fine.

BUT - I'm still wondering about the resync issue - As I understand the 
"doveadm force-resync -u user *" command rebuilds the index for the 
MAILS within the mailfolders? If so, how can the resync be done if the 
mailfolder itself is not found by doveadm anymore?
> I'd need to get some more specifics. Show show exists in filesystems
> and what commands don't produce expected results (and what they 
> produce instead).
For clearing up my point, I made another console screenshot with 
red markings to point out the problem (bug). 
Please have another look here:
http://i.imgur.com/icPfys9.png


Thanks for your support,
Megodin





---
Alle Postf??cher an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen!
http://email.freenet.de/basic/Informationen

_________________________________________________________________
Send and receive anonymous emails to your inbox with InboxAlias.
http://www.inboxalias.com

> If doveadm says it's invalid UTF-8, it's invalid UTF-8. I guess
your
> terminal isn't actually using UTF-8 then, but something else.
("locale"
> output should say something about UTF-8.) I guess doveadm could also 
> automatically translate parameters to UTF-8, but that's a bit annoying 
> to implement.
You were absolutely right. After thorough testing I could make up the 
problem to the SSH-Client PuTTY I was using on a Windows machine while 
testing. The default "remote character setting" is 
"ISO-8859-1:1998 (Latin-1, West Europe)", when re-setting to
"UTF-8",
opening a new shell and testing the "doveadm mailbox rename ..." with 
german umlauts just works fine then.
(Just for the sake of completeness, the "locale" settings were set to 
(LANG=de_DE.utf-8) globally in /etc/sysconfig/i18n per default.)

> The problem here is that * is expanded by your shell, not doveadm. 
> And it expands into Tr&-AOQ-sh as it's in the filesystem, but
that's
> only the mUTF-7 encoding of it. The UTF-8 version of the name is 
> Tr&AOQ-sh. So doveadm only sees that you attempted to resync a 
> nonexistent mailbox. Using '*' with quotes would work, since
doveadm
> would do the expansion then.
Thanks for pointing that * / '*' issue out.

I now unterstand that the "doevadm mailbox rename" converts the input 
to UTF _before_ applying it in the filesystem.

Now it makes sense that 
doveadm mailbox rename -u user 'Trash' 'Tr&AOQ-sh'
must be expanded to Tr&-AOQ-sh. The "-" character directly after
the
"&"in Tr&-AOQ-sh comes from a special mUTF-Specification (as
stated in
RFC 3501, section 5.1.3):

"In modified UTF-7, printable US-ASCII characters, except for
"&",
represent themselves; that is, characters with octet values 0x20-0x25
and 0x27-0x7e.  The character "&" (0x26) is represented by the
two-octet sequence "&-"."

So e.g. if I wanted a german umlaut to be encoded in the filesystem, 
I must enter it directly into dovedm instead of the UTF encoded value.

One small point left...
> The UTF-8 version of the name is Tr&AOQ-sh
Just for understanding - "Tr&AOQ-sh"  is IMHO UTF-7, not UTF-8. 
Accordingly to what stated before, "Tr&-AOQ-sh" and
"Tr&AOQ-sh" are
encoded both the same (UTF-7), the first seen in clients as 
"Tr&AOQ-sh" and the second as "Tr??sh".

Thanks for all your help!
Megodin



---
Alle Postf??cher an einem Ort. Jetzt wechseln und E-Mail-Adresse mitnehmen!
http://email.freenet.de/basic/Informationen

_________________________________________________________________
Send and receive anonymous emails to your inbox with InboxAlias.
http://www.inboxalias.com