dovecot - Apr 2011 - 2.0.12 (fa2e92526716): Deleting mails and folders from public namespaces

Hello,

we basically run a dovecot imap setup where users' mailboxes are in
private namespaces (as usual), but there is a public namespace with
several thousand deeply nested folders also.

The client we use is Thunderbird 3. The public namespace shows up in
each user's account at the top level (i.e. same level as Inbox, Sent and
so on), and users can make new subfolders in the public namespace, copy
mails to it and so on. So, our setup basically works.

But whenever a user tries to delete a folder from the public namespace,
dovecot gives the following error: "Renaming not supported across
non-private namespaces".

I think I have a grasp why this happens (Thunderbird tries to move the
folder from the public namespace into the trash folder (private
namespace) of the user who does the deletion), and I also have read
Timo's statement that he does not plan to support such renaming due to
big problems he is seeing with it.

Nevertheless, since deleting a folder from a public archive is not a too
uncommon thing, I wanted to ask how other people have solved that.

It came to my mind to configure Thunderbird to immediately delete
folders and messages (instead of moving them to trash folder). I have
tested that, and it worked, but is somehow dangerous and unsatisfying.

Furthermore, I haven't completely understood the reason for the
different behaviour of folders in the public namespace and a single
message therein.

If I configure Thunderbird to use the trash folder, users can delete
messages from the public namespace, i.e. Thunderbird can move messages
from the public namespace to the private trash folder of the respective
account.

This is obviously not true for folders (even if they are empty). What's
the difference between an empty folder and a single message in this
respect? Is it really the ACL thing which makes folders so difficult to
handle opposed to single messages?

Thank you very much for any ideas!

Regards,

Peter

************
dovecot -n:

# 2.0.12 (fa2e92526716): /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.1
auth_verbose = yes
auth_verbose_passwords = sha1
first_valid_uid = 103
maildir_copy_with_hardlinks = no
namespace {
  hidden = no
  inbox = yes
  list = yes
  location maildir:/home/vmail/content/users/%u:INDEX=/home/vmail/index/users/%u
  prefix   separator = /
  subscriptions = yes
  type = private
}
namespace {
  hidden = no
  inbox = no
  list = yes
  location maildir:/home/vmail/content/archive:INDEX=/home/vmail/index/archive
  prefix = Archive/
  separator = /
  subscriptions = yes
  type = public
}
passdb {
  args = scheme=SSHA username_format=%u /etc/dovecot/users
  driver = passwd-file
}
plugin {
  acl = vfile
}
protocols = " imap"
service auth {
  user = root
}
service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    address = 192.168.20.140
    port = 993
    ssl = yes
  }
}
ssl_cert = </etc/dovecot/imap-ssl.home.omeganet.de.crt
ssl_key = </etc/dovecot/imap-ssl.home.omeganet.de.key
userdb {
  args = uid=vmail gid=vmail home=/home/vmail/content/users/%u
  driver = static
}
protocol imap {
  mail_plugins = acl imap_acl
}

On 26.4.2011, at 23.48, Binarus wrote:
> But whenever a user tries to delete a folder from the public namespace,
> dovecot gives the following error: "Renaming not supported across
> non-private namespaces".
..
> Furthermore, I haven't completely understood the reason for the
> different behaviour of folders in the public namespace and a single
> message therein.

Talking only about ACLs I guess it would be ok to allow renaming from public
namespace to a private namespace. Renaming one of your own private mailboxes to
public namespace is a bit trickier, because the default ACL is "no
permissions". So when renaming, a new explicit ACL should be added to the
user who did the renaming, because otherwise user could no longer access that
mailbox at all. Still doable I guess.

Although it is possible to give user permission to delete foo/bar but not give
permission to create children to foo. Then if user renames foo/bar to
Trash/foo/bar, it's not possible to rename it back. Probably not a real
problem.

Renaming from/to shared namespace needs to update the shared users acl dict.

Quota also needs to be updated. I'm not sure if lazy-expunge has any trouble
with this. Multi-dbox backend can't do a fast rename at all, because it
would have to physically copy all of the messages from one mdbox storage to
another mdbox storage. Maybe there are other problems I can't remember now.

So looks like it should be possible to do all this, with the exception of adding
an inefficiency to mdbox, but it's some work..

On 26.04.2011 23:25, Timo Sirainen wrote:
> On 26.4.2011, at 23.48, Binarus wrote:
> 
>> But whenever a user tries to delete a folder from the public namespace,
>> dovecot gives the following error: "Renaming not supported across
>> non-private namespaces".
> ..
>> Furthermore, I haven't completely understood the reason for the
>> different behaviour of folders in the public namespace and a single
>> message therein.
> 
> 
> Talking only about ACLs I guess it would be ok to allow renaming from
public namespace to a private namespace. Renaming one of your own private
mailboxes to public namespace is a bit trickier, because the default ACL is
"no permissions". So when renaming, a new explicit ACL should be added
to the user who did the renaming, because otherwise user could no longer access
that mailbox at all. Still doable I guess.
> 
> Although it is possible to give user permission to delete foo/bar but not
give permission to create children to foo. Then if user renames foo/bar to
Trash/foo/bar, it's not possible to rename it back. Probably not a real
problem.
> 
> Renaming from/to shared namespace needs to update the shared users acl
dict.
> 
> Quota also needs to be updated. I'm not sure if lazy-expunge has any
trouble with this. Multi-dbox backend can't do a fast rename at all, because
it would have to physically copy all of the messages from one mdbox storage to
another mdbox storage. Maybe there are other problems I can't remember now.
> 
> So looks like it should be possible to do all this, with the exception of
adding an inefficiency to mdbox, but it's some work..
Thank you very much for your answer which shows that -as always-
problems are more complicated than they seem first.

Nevertheless, I don't want to give up, have a few thoughts and just
would like to know I am right and what others are thinking:

- I have used Cyrus 2.2 for a long time before I switched to Dovecot; in
Cyrus 2.2, I did not have the problem. So I am wondering how the problem
has been solved there.

- Obviously, users can create new folders in public namespace (if the
ACL allows) and their private mailboxes; furthermore, they can delete
folders from public namespace and private mailboxes (if the ACL allows)
(I am talking of a pure delete here, not of moving the folder to Trash).

Thus, we could emulate moving folders between any namespaces (for
example, from public namespace to private Trash) by first copying them
to the new location and then deleting them from the original location.

Let's assume we have a folder in a public namespace and are allowed to
do everything with it. Then, we could delete this folder (this time, I
am meaning moving it to private Trash) by doing the following steps
manually:

a) Create a new folder with the same name in Trash (which may be in the
private mailbox); recursively repeat this with all subfolders.

b) Move all messages from the folder in the public namespace to the
folder in Trash; recursively repeat this with all subfolders (remark:
moving messages does not seem to cause problems, as opposed to moving
folders).

c) Recursively delete the folder with all subfolders (which are
completely empty now) from the public namespace (of course, this again
is a "real" delete).

I have tested this procedure with multiple IMAP clients against Dovecot,
and there were no problems. Automation of the procedure would not be
expected to lead to problems, would it?.

- From a user's point of view, it is not important how messages get
deleted or moved behind the scenes. A users which wants his stuff moved
doesn't care if moving is accomplished by a rename or by another
mechanism. In fact, he doesn't even know about renaming. He just wants
the stuff which was in some folder to show up in some other folder.

It seems that the problems come from the fact that deletion is done by
renaming. I am not in IMAP but suppose that there is some rename command
which is used by email clients in such cases. Maybe there is also a
delete command which dovecot is "emulating" by doing a rename - I
really
don't know...

So why not implement the rename / delete command in the way that I have
described above? I know that a rename probably (for example on a file
system or Dovecot's backends) is by far much more efficient than the
copy-move-delete mechanism, but we could do the following:

If the renaming is problematic due to reasons Timo has mentioned in his
first reply, then emulate the renaming by the copy-move-delete
mechanism; if renaming would be no problem, then use the normal method
(the method which is already implemented).

The inefficiency of the copy-move-delete mechanism would be no argument
against using it: the alternative would be having the user to do these
steps by hand, for every single subfolder (there may be thousands of
them...). This is just not acceptable.

One could argue that this functionality should be in the client, and
indeed, I suspect that some clients are working that way in some
situations. Nevertheless, I think such clients are bad, by using
inefficient operations where it would not be necessary and putting the
servers under load.

IMHO, such functionality belongs into the server side since only the
server can reliably decide if there will be problems before trying the
operation.

I am absolutely sure users would prefer waiting for inefficient
copy-move-delete operations instead of not being able to move their
stuff at all.

Perhaps implementing such a behaviour would not be too difficult?

At the end, I would like to stress that this is not just an academic
question. The need to move folders, especially between public and
private namespaces, is very common in small and large companies, and
although my original post was about deleting folders, the problem in
fact is a real big one (although some clients seem to circumvent it by
wasting server resources).

I am curiously looking forward to any comments.

Regards,

Peter