dovecot - Jan 2008 - More efficient Deleting of a whole folder (or Purging Trash)

Hi,

I've straced a few dovecot processes after hitting purge on a large 
Trash folder (35,000 messages). It looks like it is going through each 
message, one by one, and removing (unlinking) each one.

Is there not a more efficient way to do this? If Dovecot knows the whole 
folder is being deleted (ie a Trash purge), could it do something clever 
with the filesystem to just remove the whole folder?

Incidently, the command that initially deleted those 35k messages to 
Trash took about 1 hour to complete on my server and effectively locked 
my account out for all that time. There were also an awlful lot of these 
output from my strace:

nanosleep({0, 131475000}, NULL)         = 0
lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock",
{st_mode=S_IFREG|0600, st_size=0, ...}) = 0

Not sure if there's much one can do about speeding up deleting a lot of 
messages though. My webmail client has a 'delete all messages' button 
for each folder. If there was an efficient IMAP command for deleting a 
whole folder that would be useful in those circumstances.

Interested to know if there is anything that can be improved.

Thanks,
Daniel


Dovecot 1.0.9
XFS Filesystem
Maildir Directories

At 4:03 PM +0000 1/29/08, Daniel Watts wrote:
>Hi,
>
>I've straced a few dovecot processes after hitting purge on a large 
>Trash folder (35,000 messages). It looks like it is going through 
>each message, one by one, and removing (unlinking) each one.
>
>Is there not a more efficient way to do this? If Dovecot knows the 
>whole folder is being deleted (ie a Trash purge), could it do 
>something clever with the filesystem to just remove the whole folder?
Not with most filesystems. Generally you need to unlink each file in 
a directory first before removing the directory. I'm not familiar 
with the details of XFS, but with most filesystems the directory is 
just a list of the names and inodes of the items in the directory, so 
deleting the directory node itself leaves all of those inodes and all 
of the data blocks they point to untouched: all you've done is lost 
track of them.

For Dovecot, the mailbox index also has to be updated for each 
deletion, and while that in itself is not terribly expensive per 
message, it has to be done one message at a time to maintain 
consistency between the index and the real mail.
>Incidently, the command that initially deleted those 35k messages to 
>Trash took about 1 hour to complete on my server and effectively 
>locked my account out for all that time. There were also an awlful 
>lot of these output from my strace:
>
>nanosleep({0, 131475000}, NULL)         = 0
>lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock",
>{st_mode=S_IFREG|0600, st_size=0, ...}) = 0
>
>Not sure if there's much one can do about speeding up deleting a lot 
>of messages though. My webmail client has a 'delete all messages' 
>button for each folder. If there was an efficient IMAP command for 
>deleting a whole folder that would be useful in those circumstances.
>
>Interested to know if there is anything that can be improved.
As a general issue, I doubt it.

However:
>Dovecot 1.0.9
>XFS Filesystem
>Maildir Directories
XFS historically has had very poor delete performance unless specially tuned.
-- 
Bill Cole                                  
bill at scconsult.com

On Jan 29, 2008, at 6:03 PM, Daniel Watts wrote:
> Is there not a more efficient way to do this? If Dovecot knows the  
> whole folder is being deleted (ie a Trash purge), could it do  
> something clever with the filesystem to just remove the whole folder?
If you use IMAP DELETE command, it renames the directory to ..DOVECOT- 
TRASH (or something like it) and only after that it starts unlinking  
the files. It wouldn't be too difficult to have this return success  
immediately and then keep deleting the files on the background. I'm  
not sure if this is worth the trouble though. It would also be  
possible to do this as a plugin.
> Incidently, the command that initially deleted those 35k messages to  
> Trash took about 1 hour to complete on my server and effectively  
> locked my account out for all that time. There were also an awlful  
> lot of these output from my strace:
>
> nanosleep({0, 131475000}, NULL)         = 0
> lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/ 
> dovecot-uidlist.lock", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
You mean this was in the deleting process, or the other processes? The  
deleting process should have kept the maildir locked while it was  
deleting the files, so other processes were locked out. This is by  
design and can't be changed, at least until I some day add inotify  
support for maildir synchronization.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20080130/e709bc49/attachment-0002.bin>

Hi Timo,
> On Jan 29, 2008, at 6:03 PM, Daniel Watts wrote:
>
>> Is there not a more efficient way to do this? If Dovecot knows the 
>> whole folder is being deleted (ie a Trash purge), could it do 
>> something clever with the filesystem to just remove the whole folder?
>
> If you use IMAP DELETE command, it renames the directory to 
> ..DOVECOT-TRASH (or something like it) and only after that it starts 
> unlinking the files. It wouldn't be too difficult to have this return 
> success immediately and then keep deleting the files on the 
> background. I'm not sure if this is worth the trouble though. It would 
> also be possible to do this as a plugin.
Something like that would be great. Would this actually then avoid the 
locking problem described below?
>> Incidently, the command that initially deleted those 35k messages to 
>> Trash took about 1 hour to complete on my server and effectively 
>> locked my account out for all that time. There were also an awlful 
>> lot of these output from my strace:
>>
>> nanosleep({0, 131475000}, NULL)         = 0
>>
lstat("/home/virtual/xxx.com/home/admin/Maildir/.Folders.Porter/dovecot-uidlist.lock",
>> {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
>
> You mean this was in the deleting process, or the other processes? The 
> deleting process should have kept the maildir locked while it was 
> deleting the files, so other processes were locked out. This is by 
> design and can't be changed, at least until I some day add inotify 
> support for maildir synchronization.
You're probably right - another process is probably trying to access the 
folder that is currently being deleted and being prevented from doing 
so. There was an awlful lot of polling going on though - probably man 
tens per second. Not sure if this causes the system to thrash a bit and 
raise load unnecessarily.

If you renamed the folder to *.DOVECOT-DELETED or something you won't 
have another process trying to access the same folder and locking the 
account up.

IMHO this benefit is worth the effort. =)

Daniel