Hi, I do store mail in mdbox format of 150m in size (dovecot 2.3.9). Once in a while I do experience mdbox files of smaller size, even after applying 'doveadm purge' and previous expunges by the users. like: -rw------- 1 vmail dovecot 104854595 Feb 9 2019 /var/mail/.maildirs/userX/storage/m.22 -rw------- 1 vmail dovecot 29088478 Mar 8 2019 /var/mail/.maildirs/userX/storage/m.31 -rw------- 1 vmail dovecot 98210890 Mar 20 2019 /var/mail/.maildirs/userX/storage/m.39 (Currently the counter is at file number 129.) Well, I never experienced missing mail or alike, but these "holes" in filesize irritates me, and yes, it is more or less a cosmetic issue. Nevertheless, I do sometimes want to get rid of these "holes" by backing up all mail and re-injecting the backup into a vanilla account of that user. And I used this approach when I wanted to store all mail messages in larger mdbox files; again, rather a cosmetic issue. BUT that takes a very, very long time contrary to the speed of 'doveadm purge'. Unfortunately, that command starts somewhere with more recent mdbox files and never from scratch (oldest mdbox file). Whishlist: Would it be much of an effort to implement an option like: 'doveadm purge -f' and '-f' standing for 'force' or 'from scratch' or 'from the very first message found' or 'you name it'? Thanks in advance and thanks for Dovecot and with kind regards, Michael
On 8.12.2019 22.10, Michael Grimm via dovecot wrote:> Hi, > > I do store mail in mdbox format of 150m in size (dovecot 2.3.9). > > Once in a while I do experience mdbox files of smaller size, even after applying 'doveadm purge' and previous expunges by the users. like: > > -rw------- 1 vmail dovecot 104854595 Feb 9 2019 /var/mail/.maildirs/userX/storage/m.22 > -rw------- 1 vmail dovecot 29088478 Mar 8 2019 /var/mail/.maildirs/userX/storage/m.31 > -rw------- 1 vmail dovecot 98210890 Mar 20 2019 /var/mail/.maildirs/userX/storage/m.39 > > (Currently the counter is at file number 129.) > > Well, I never experienced missing mail or alike, but these "holes" in filesize irritates me, and yes, it is more or less a cosmetic issue. > > Nevertheless, I do sometimes want to get rid of these "holes" by backing up all mail and re-injecting the backup into a vanilla account of that user. And I used this approach when I wanted to store all mail messages in larger mdbox files; again, rather a cosmetic issue. > > BUT that takes a very, very long time contrary to the speed of 'doveadm purge'. Unfortunately, that command starts somewhere with more recent mdbox files and never from scratch (oldest mdbox file). > > Whishlist: Would it be much of an effort to implement an option like: > > 'doveadm purge -f' > and '-f' standing for 'force' or 'from scratch' or 'from the very first message found' or 'you name it'? > > Thanks in advance and thanks for Dovecot and with kind regards, > Michael >What purge does is that it removes mails that have refcount=0, so "from scratch" makes no sense. Renumbering the files "for neatness" is rather heavy operation, as you'd need to move mails around quite a lot. This is very little benefit to just cater for holes. Aki
Aki Tuomi via dovecot <dovecot at dovecot.org> wrote:> On 8.12.2019 22.10, Michael Grimm via dovecot wrote:>> I do store mail in mdbox format of 150m in size (dovecot 2.3.9). >> >> Once in a while I do experience mdbox files of smaller size, even after applying 'doveadm purge' and previous expunges by the users. like: >> >> -rw------- 1 vmail dovecot 104854595 Feb 9 2019 /var/mail/.maildirs/userX/storage/m.22 >> -rw------- 1 vmail dovecot 29088478 Mar 8 2019 /var/mail/.maildirs/userX/storage/m.31 >> -rw------- 1 vmail dovecot 98210890 Mar 20 2019 /var/mail/.maildirs/userX/storage/m.39 >> >> (Currently the counter is at file number 129.) >> >> Well, I never experienced missing mail or alike, but these "holes" in filesize irritates me, and yes, it is more or less a cosmetic issue. >> >> Nevertheless, I do sometimes want to get rid of these "holes" by backing up all mail and re-injecting the backup into a vanilla account of that user. And I used this approach when I wanted to store all mail messages in larger mdbox files; again, rather a cosmetic issue. >> >> BUT that takes a very, very long time contrary to the speed of 'doveadm purge'. Unfortunately, that command starts somewhere with more recent mdbox files and never from scratch (oldest mdbox file). >> >> Whishlist: Would it be much of an effort to implement an option like: >> >> 'doveadm purge -f' >> and '-f' standing for 'force' or 'from scratch' or 'from the very first message found' or 'you name it'?> What purge does is that it removes mails that have refcount=0, so "from scratch" makes no sense.My observations are that 'doveadm purge' starts with the very first mdbox file found with an refcount=0, and then shuffles all subsequent mail with refcount!=0 in all subsequent mdbox files into newly created mdbox files starting right after the last mdbox file number.> Renumbering the files "for neatness" is rather heavy operation, as you'd need to move mails around quite a lot.In the scenario described above, 'doveadm purge' already moves quite a lot of mails around, here a couple of 100m in volume.> This is very little benefit to just cater for holes.Agreed, as I stated above, it is more or less a cosmetic issue. But I do see some value in such an option as well, e.g.: Given you want to decrease your mdbox size from 100m to 10m because you decided, that it is too risky to store that many mails into one file. You may modify 'mdbox_rotate_size=10m' accordingly and all subsequent mail will be stored in smaller mdbox files, but old mail will remain being stored in the larger files. I do not know how a Dovecot user may accomplish this task without backing up all mail and re-injecting the backup into a vanilla account. Correct, or do I miss some functionality? With kind regards, Michael