Hi folks.
Here's our situation: Migrating from UW-IMAP. Have lots (as in, tens of
thousands) of clients set up using '~/mail' as the IMAP root, and using
subscriptions.
Dovecot mail_location is using '~' for mbox storage. We've disabled
full
filesystem access in Dovecot in order to use ACLs for shared folders. Hence
we have a hidden namespace for backwards compatibility with those clients
using '~/mail' (dovecot -n output is at the end of this message so you
can
see exact settings). Here's the problem. If the client is not taking
advantage of this hidden namespace--say it's configured to use plain old
'mail' as the root directory--LSUB and SUBSCRIBE work as I'd expect.
Subscribing to a folder called 'foo' puts this in the user's
subscriptions
file:
mail/foo
And this (from Thunderbird's log) is the LSUB exchange:
48203776[ed8ebe0]: 2e03600:server.umn.edu:A:SendData: 6 lsub ""
"mail/*"
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=26 needmore=0]
48203776[ed8ebe0]: 2e03600:serverl.umn.edu:A:CreateNewLineFromSocket: *
LSUB () "/" "mail/foo"
48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=38 needmore=0]
(trimmed...)
48203776[ed8ebe0]: 2e03600:bhayden.email.umn.edu:A:CreateNewLineFromSocket:
6 OK Lsub completed.
All is well. However, when the client is configured with '~/mail' as the
root and hence triggers the hidden namespace for such, SUBSCRIBE only
writes a path relative to that namespace prefix. So as opposed the
'mail/foo' in the first example, we get:
foo
written to the subs folder.
This leads to three problems:
1. Folders subscribed to when using this namespace will either disappear or
break in a client using a different IMAP root setting that actually refers
to the same place. For example, if this user moves to a client (such as
both of the webmail offerings on campus) that use 'mail/' as the prefix,
these folders will not be visible.
This is a fairly sizeable population of our users, whom we aren't
necessarily in the position to say "too bad" to. :)
2. Users moved over from UW-IMAP, who were using '~/mail' for the IMAP
root, have a subscriptions file full of:
~/mail/bar
Dovecot, with the same client configuration (since it's expecting a bare
folder name to which it will prepend the namespace prefix), interprets this
as so:
49036288[fa7e740]: 2eabc00:server.umn.edu:A:CreateNewLineFromSocket: * LSUB
() "/" "~/mail/~/mail/foobar"
Leading to having the folder displayed in the client's list, but giving an
error on SELECT. Now, this one we could theoretically "solve" by
scrubbing
the prefixes out of subs files as we convert; however, that leaves us with
#1 above, where the bare folder name in the subs file depends on (to the
average user) an obscure client config setting, which in practice in a
large heterogeneous environment is unreliable.
3. Users who have been using an IMAP root of 'mail' in some client and
subscribed to folders, who then use a client with the root set to
'~/mail'
see what appears to be "duplicates" of the same folder at the top
level and
then within a bogus 'mail/' subdirectory. They (quite reasonably, from
their perspective), delete one of these, only to find they've deleted the
only copy of said folder. Not good.
Also, users can't solve some of these problems by unsubbing and re-subbing
on their own, as trying to unsubscribe from a folder that doesn't exist
(such as ~/mail/~/mail/foobar in the example above) fails.
So, I'm wondering if there's any way to address this. I looked through
the
list archives and code pretty thoroughly. Only thing I noticed was a CVS
log mentioning a "subscriptions=yes" setting for namespaces, which
sounded
like it might address this problem. However, putting that setting into
dovecot.conf causes it to fail to start with a "Unrecognized config
option"
error, and using an imap_executable wrapper that sets the env variable as
specified in the code (NAMESPACE_<number>_SUBSCRIPTIONS) sets it, as I can
see from the dovecot log output:
Namespace: type=private, prefix=~/mail/, sep=/, inbox=no, hidden=yes,
subscriptions=yes
But doesn't seem to have any effect. These two things combined lead me to
believe this option was deprecated at some point?
Any thoughts or suggestions? While we're moving away from UW for a reason
;), it does seem to handle this one thing more coherently as it recognizes,
for example, that if the server equivalent of mail_location's storage is
'~', then subscription file entries: '~/mail/foo' and
'mail/foo' are the
same thing, while 'foo' is '~/foo'--regardless of client
settings or
namespaces. Whereas in the same situation Dovecot handles a client IMAP
root of '~/mail' vs 'mail' differently, even though they are in
fact the
same thing.
This is also internally inconsistent, because (as noted above) when a
namespace is not in effect Dovecot SUBSCRIBEs using a path relative to the
mail_location storage--including the client's IMAP root portion of the
path--whereas when a namespace is in effect it SUBSCRIBEs with only the
bare mailbox name without the current client's IMAP root to give it context
and meaning. Wouldn't it seem better to have these be consistent in this
respect?
We need to figure this out somehow before going any further with our
migration. We'll modify the source if necessary, but are hoping for a
solution that doesn't require maintaining local mods to the source code
unless we absolutely have to. :) And this seems like a basic enough issue
that it'd be good to handle it in the real codebase. Thanks for any help or
insight,
-Brian Hayden
Internet Services
University of Minnesota
dovecot -n output: (The shell script mail_executable doesn't come into play
here--it sets some namespace settings only for special shared accounts
using Maildir, and doesn't touch anything for the normal mbox accounts
discussed above.)
# /etc/opt/dovecot/dovecot.conf syslog_facility: local6 protocols: imap
imaps pop3 pop3s ssl_cert_file: /etc/opt/openssl/certs/dovecot.pem
ssl_key_file: /etc/opt/openssl/private/dovecot.pem shutdown_clients: no
login_dir: /var/opt/dovecot/run/dovecot/login login_executable(default):
/opt/dovecot/libexec/dovecot/imap-login login_executable(imap):
/opt/dovecot/libexec/dovecot/imap-login login_executable(pop3):
/opt/dovecot/libexec/dovecot/pop3-login max_mail_processes: 8192
first_valid_uid: 99 mail_location:
mbox:~:INBOX=/var/mail/%u:INDEX=/var/mail/.dovecot-index/%1u/%u/
mail_debug: yes mail_executable(default):
/opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(imap):
/opt/dovecot/libexec/dovecot/mail_path.sh mail_executable(pop3):
/opt/dovecot/libexec/dovecot/pop3 mail_plugins(default): zlib
mail_plugins(imap): zlib mail_plugins(pop3): mail_plugin_dir(default):
/opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap):
/opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3):
/opt/dovecot/lib/dovecot/pop3 imap_client_workarounds(default):
delay-newmail outlook-idle tb-extra-mailbox-sep
imap_client_workarounds(imap): delay-newmail outlook-idle
tb-extra-mailbox-sep imap_client_workarounds(pop3): outlook-idle
pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3):
%08Xu%08Xv namespace:
type: private
separator: /
inbox: yes
namespace:
type: private
separator: /
prefix: ~/mail/
location: mbox:%h/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/
hidden: yes
namespace:
type: private
separator: /
prefix: ~%u/mail/
location: mbox:~%u/mail/:INDEX=/var/mail/.dovecot-index/%1u/%u/
hidden: yes
namespace:
type: private
separator: /
prefix: Shared/
location:
maildir:~/Maildir/Shared:INDEX=/var/mail/.dovecot-index/%1u/u/shared auth
default:
username_format: %Lu
passdb:
driver: pam
args: session=yes *
userdb:
driver: passwd
plugin:
acl: vfile:
On Jun 18 2007, bhayden at umn.edu wrote:>Hi folks. > >Here's our situation: Migrating from UW-IMAP. Have lots (as in, tens of >thousands) of clients set up using '~/mail' as the IMAP root, and using >subscriptions.The rest of my original message is at the end for context. :) Brief: Subscriptions in general are given to inconsistent behavior if there is any difference in client settings from one session to the next. Dovecot writes subscriptions without the namespace prefix and then prepends the current ns prefix to every item in the subscription list for LSUB response. So if the namespace prefix changes session-to-session; or if there are legacy entries with more "full" paths in the subs file, LSUB returns duplicates and paths that do not exist. This leads to very confused users and in some cases data loss when they (reasonably) try to delete a "duplicate" and find they've deleted the only copy of a folder. In order to avoid this, I made a (very) simple two-part mod. The first bit simply writes subscriptions with the ns prefix included: diff cmd-subscribe.c cmd-subscribe.c.orig : 35c35 < if (mail_storage_set_subscribed(storage, verify_name, subscribe) < 0) ---> if (mail_storage_set_subscribed(storage, mailbox, subscribe) < 0)The second changes LSUB so that it only returns entries from the subscription list which match the ns prefix, and avoids prepending the ns prefix to LSUB responses: diff cmd-list.c cmd-list.c.orig : 138,141c138 < } else if (ctx->lsub && < strncmp(ctx->ns->prefix, list->name, strlen(ctx->ns->prefix))) { < continue; < } else if (!ctx->lsub) { ---> } else {Now we err on the side of not showing some previously subscribed folders if the client config changes, or if there is random junk in the subscriptions file--rather than trying to show everything and returning duplicate and non-existent mailbox paths in LSUB. Hopefully Timo (or anyone else), you could offer some thoughts on whether this is an acceptable strategy, or if I've overlooked something. Thanks very much, -Brian Hayden Internet Services University of Minnesota> > Dovecot mail_location is using '~' for mbox storage. We've disabled full > filesystem access in Dovecot in order to use ACLs for shared folders. > Hence we have a hidden namespace for backwards compatibility with those > clients using '~/mail' (dovecot -n output is at the end of this message > so you can see exact settings). Here's the problem. If the client is not > taking advantage of this hidden namespace--say it's configured to use > plain old 'mail' as the root directory--LSUB and SUBSCRIBE work as I'd > expect. Subscribing to a folder called 'foo' puts this in the user's > subscriptions file: > >mail/foo > >And this (from Thunderbird's log) is the LSUB exchange: > >48203776[ed8ebe0]: 2e03600:server.umn.edu:A:SendData: 6 lsub "" "mail/*" >48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=26 needmore=0] >48203776[ed8ebe0]: 2e03600:serverl.umn.edu:A:CreateNewLineFromSocket: * >LSUB () "/" "mail/foo" >48203776[ed8ebe0]: ReadNextLine [stream=ed8ef88 nb=38 needmore=0] > (trimmed...) > 48203776[ed8ebe0]: > 2e03600:bhayden.email.umn.edu:A:CreateNewLineFromSocket: >6 OK Lsub completed. > >All is well. However, when the client is configured with '~/mail' as the >root and hence triggers the hidden namespace for such, SUBSCRIBE only >writes a path relative to that namespace prefix. So as opposed the >'mail/foo' in the first example, we get: > >foo > >written to the subs folder. > >This leads to three problems: > > 1. Folders subscribed to when using this namespace will either disappear > or break in a client using a different IMAP root setting that actually > refers to the same place. For example, if this user moves to a client > (such as both of the webmail offerings on campus) that use 'mail/' as the > prefix, these folders will not be visible. > >This is a fairly sizeable population of our users, whom we aren't >necessarily in the position to say "too bad" to. :) > >2. Users moved over from UW-IMAP, who were using '~/mail' for the IMAP >root, have a subscriptions file full of: > >~/mail/bar > > Dovecot, with the same client configuration (since it's expecting a bare > folder name to which it will prepend the namespace prefix), interprets > this as so: > > 49036288[fa7e740]: 2eabc00:server.umn.edu:A:CreateNewLineFromSocket: * > LSUB () "/" "~/mail/~/mail/foobar" > >Leading to having the folder displayed in the client's list, but giving an >error on SELECT. Now, this one we could theoretically "solve" by scrubbing >the prefixes out of subs files as we convert; however, that leaves us with >#1 above, where the bare folder name in the subs file depends on (to the >average user) an obscure client config setting, which in practice in a >large heterogeneous environment is unreliable. > > 3. Users who have been using an IMAP root of 'mail' in some client and > subscribed to folders, who then use a client with the root set to > '~/mail' see what appears to be "duplicates" of the same folder at the > top level and then within a bogus 'mail/' subdirectory. They (quite > reasonably, from their perspective), delete one of these, only to find > they've deleted the only copy of said folder. Not good. > >Also, users can't solve some of these problems by unsubbing and re-subbing >on their own, as trying to unsubscribe from a folder that doesn't exist >(such as ~/mail/~/mail/foobar in the example above) fails. > > So, I'm wondering if there's any way to address this. I looked through > the list archives and code pretty thoroughly. Only thing I noticed was a > CVS log mentioning a "subscriptions=yes" setting for namespaces, which > sounded like it might address this problem. However, putting that setting > into dovecot.conf causes it to fail to start with a "Unrecognized config > option" error, and using an imap_executable wrapper that sets the env > variable as specified in the code (NAMESPACE_<number>_SUBSCRIPTIONS) sets > it, as I can see from the dovecot log output: > >Namespace: type=private, prefix=~/mail/, sep=/, inbox=no, hidden=yes, >subscriptions=yes > >But doesn't seem to have any effect. These two things combined lead me to >believe this option was deprecated at some point? > > Any thoughts or suggestions? While we're moving away from UW for a reason > ;), it does seem to handle this one thing more coherently as it > recognizes, for example, that if the server equivalent of mail_location's > storage is '~', then subscription file entries: '~/mail/foo' and > 'mail/foo' are the same thing, while 'foo' is '~/foo'--regardless of > client settings or namespaces. Whereas in the same situation Dovecot > handles a client IMAP root of '~/mail' vs 'mail' differently, even though > they are in fact the same thing. > > This is also internally inconsistent, because (as noted above) when a > namespace is not in effect Dovecot SUBSCRIBEs using a path relative to > the mail_location storage--including the client's IMAP root portion of > the path--whereas when a namespace is in effect it SUBSCRIBEs with only > the bare mailbox name without the current client's IMAP root to give it > context and meaning. Wouldn't it seem better to have these be consistent > in this respect? > > We need to figure this out somehow before going any further with our > migration. We'll modify the source if necessary, but are hoping for a > solution that doesn't require maintaining local mods to the source code > unless we absolutely have to. :) And this seems like a basic enough issue > that it'd be good to handle it in the real codebase. Thanks for any help > or insight, > >-Brian Hayden >Internet Services >University of Minnesota
On Mon, 2007-06-18 at 10:59 -0500, bhayden at umn.edu wrote:> Here's our situation: Migrating from UW-IMAP. Have lots (as in, tens of > thousands) of clients set up using '~/mail' as the IMAP root, and using > subscriptions.See if the attached patch fixes everything. I'll add it to v1.1. -------------- next part -------------- A non-text attachment was scrubbed... Name: subscriptions-skip-hidden-ns.diff Type: text/x-patch Size: 2073 bytes Desc: not available URL: <http://dovecot.org/pipermail/dovecot/attachments/20070627/86ebda38/attachment-0004.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://dovecot.org/pipermail/dovecot/attachments/20070627/86ebda38/attachment-0005.bin>