dovecot - Nov 2020 - Recovering expunged but not purged e-mails from mdbox with zlib compression

> On 10/11/2020 20:07 Daniel Sch?tze <dms at cwa.uk.com> wrote:
> 
> 
> Dear Aki
> Thank you. Unfortunately I'm struggling to get the right syntax for
this as it looks like someone else was here too
> https://dovecot.org/pipermail/dovecot/2018-July/112441.html
> The location in my dovecot.conf is 
> 
> mdbox:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/%%n
This syntax is for accessing shared folders.

You probably should try

doveadm -Dv -o mail_location=mdbox_deleted:/path/to/mdbox fetch -u victim text
ALL
> I'm trying to fetch a message (for testing purposes now) based on
it's guid as that is available from doveadm dump.
> So my syntax is (based on the previous person who didn't get it to
work).
> 
> doveadm -o
"mail_location=mdbox_deleted:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/%%n"
fetch "body" guid (msg.guid from doveadm dump)
> 
Doveadm does not support var expand, so this will not work. See above for
syntax.
> But that's getting no response. I've tried putting in the hard path
in case there is any trouble with the %%h etc but that doesn't help either.
> I've also tried the fetch with the subject of a known deleted message
and also adding mailbox Drafts (as I know that's where it is or rather was)
but to no avail.
> Given doveadm dump gives msg.uid and no subject I'd prefer to use that
in any case.
> There are no error messages, I do with doveadm was a little more verbose to
help over these bumps!
> 
> Any chance of a helping hand?
> 
> 
> 
> 
Aki
> 
> 
> 
> Daniel Sch?tze 
>  Director 
>  
>  ------------------ 
>  
>  CWA International Ltd 
>  5th Floor, 42 Trinity Square 
>  London 
>  EC3N 4DJ 
>  
>  (t) + 44 (0)20 7242 8444 
>  (e) dms at cwa.uk.com 
>  (w) http://www.cwa.international/
> On 10/11/20 10:53, Aki Tuomi wrote:
> 
> > You can use mdbox_deleted driver to access mails with refcount 0. See
> > https://wiki2.dovecot.org/MailboxFormat/dbox
> > 
> > Aki
> > 
> > On 10.11.2020 12.42, Daniel Sch?tze wrote:
> > 
> > > Yesterday I had to recover an e-mail which a user had deleted.?
If I
> > > understand this correctly the message was expunged but not purged
> > > (doveadm purge had not been run).
> > > 
> > > This e-mail was clearly still in the mdbox stored with zlib
> > > compression as I could tell using the doveadm dump command
(doveadm
> > > dump -t dbox filename).
> > > 
> > > I could however not reveal the e-mail with the normal doveadm
fetch -u
> > > username "body" guid (guid from dump output)
> > > 
> > > In the end I was able to recover the e-mail by cutting it out
from the
> > > mdbox and running gunzip over it, but this method was very fiddly
and
> > > would not have worked for a bulk job.? I appreciate I could have
just
> > > deleted the index files and gotten all the deleted messages back
by
> > > having the index rebuilt but that would have been the proverbial
> > > "hammer to crack a nut".
> > > 
> > > Fortunately this e-mail had no attachment for me to worry about,
as
> > > those are also detached for sis by dovecot.
> > > 
> > > Can anyone tell me of a quick and easy way to recover one or
multiple
> > > e-mails marked as expunged but not purged which are stored in a
mdbox
> > > with zlib compression?? I'm sure I'll have to do this
again the future
> > > and my method wouldn't work with a folder!
> > > 
> > > Clearly if I was not using zlib compression I could just have
read the
> > > contents of the mdbox without any complication.
> > > 
> > > 
> > > -- 
> > > Daniel Sch?tze
> > > Director
> > > 
> > > ------------------
> > > 
> > > CWA International Ltd
> > > 5th Floor, 42 Trinity Square
> > > London
> > > EC3N 4DJ
> > > 
> > > (t) + 44 (0)20 7242 8444
> > > (e) dms at cwa.uk.com
> > > (w) http://www.cwa.international/
> > >

Dear Aki

Thank you again.

Running

doveadm -Dv -o 
maillocation=mdbox_deleted:/usr/home/vmail/mail/username/mdbox/storage/file 
fetch -u username text all

Does dump messages out.

However the messages do not include two messages which are present in 
the mdbox file and which I recovered by hand and so am using as a test 
to see if I can extract them effectively.? These were in the Drafts 
mailbox so I've also run the command with mailbox Drafts at the end.

The output seems to be just dumping messages which are not expunged!

Daniel Sch?tze
Director

------------------

CWA International Ltd
5th Floor, 42 Trinity Square
London
EC3N 4DJ

(t) + 44 (0)20 7242 8444
(e) dms at cwa.uk.com
(w) http://www.cwa.international/
On 10/11/20 18:16, Aki Tuomi wrote:
>> On 10/11/2020 20:07 Daniel Sch?tze <dms at cwa.uk.com> wrote:
>>
>>
>> Dear Aki
>> Thank you. Unfortunately I'm struggling to get the right syntax for
this as it looks like someone else was here too
>> https://dovecot.org/pipermail/dovecot/2018-July/112441.html
>> The location in my dovecot.conf is
>>
>>
mdbox:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/%%n
> This syntax is for accessing shared folders.
>
> You probably should try
>
> doveadm -Dv -o mail_location=mdbox_deleted:/path/to/mdbox fetch -u victim
text ALL
>
>> I'm trying to fetch a message (for testing purposes now) based on
it's guid as that is available from doveadm dump.
>> So my syntax is (based on the previous person who didn't get it to
work).
>>
>> doveadm -o
"mail_location=mdbox_deleted:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/%%n"
fetch "body" guid (msg.guid from doveadm dump)
>>
> Doveadm does not support var expand, so this will not work. See above for
syntax.
>
>> But that's getting no response. I've tried putting in the hard
path in case there is any trouble with the %%h etc but that doesn't help
either.
>> I've also tried the fetch with the subject of a known deleted
message and also adding mailbox Drafts (as I know that's where it is or
rather was) but to no avail.
>> Given doveadm dump gives msg.uid and no subject I'd prefer to use
that in any case.
>> There are no error messages, I do with doveadm was a little more
verbose to help over these bumps!
>>
>> Any chance of a helping hand?
>>
>>
>>
>>
> Aki
>
>>
>>
>> Daniel Sch?tze
>>   Director
>>   
>>   ------------------
>>   
>>   CWA International Ltd
>>   5th Floor, 42 Trinity Square
>>   London
>>   EC3N 4DJ
>>   
>>   (t) + 44 (0)20 7242 8444
>>   (e) dms at cwa.uk.com
>>   (w) http://www.cwa.international/
>> On 10/11/20 10:53, Aki Tuomi wrote:
>>
>>> You can use mdbox_deleted driver to access mails with refcount 0.
See
>>> https://wiki2.dovecot.org/MailboxFormat/dbox
>>>
>>> Aki
>>>
>>> On 10.11.2020 12.42, Daniel Sch?tze wrote:
>>>
>>>> Yesterday I had to recover an e-mail which a user had deleted.?
If I
>>>> understand this correctly the message was expunged but not
purged
>>>> (doveadm purge had not been run).
>>>>
>>>> This e-mail was clearly still in the mdbox stored with zlib
>>>> compression as I could tell using the doveadm dump command
(doveadm
>>>> dump -t dbox filename).
>>>>
>>>> I could however not reveal the e-mail with the normal doveadm
fetch -u
>>>> username "body" guid (guid from dump output)
>>>>
>>>> In the end I was able to recover the e-mail by cutting it out
from the
>>>> mdbox and running gunzip over it, but this method was very
fiddly and
>>>> would not have worked for a bulk job.? I appreciate I could
have just
>>>> deleted the index files and gotten all the deleted messages
back by
>>>> having the index rebuilt but that would have been the
proverbial
>>>> "hammer to crack a nut".
>>>>
>>>> Fortunately this e-mail had no attachment for me to worry
about, as
>>>> those are also detached for sis by dovecot.
>>>>
>>>> Can anyone tell me of a quick and easy way to recover one or
multiple
>>>> e-mails marked as expunged but not purged which are stored in a
mdbox
>>>> with zlib compression?? I'm sure I'll have to do this
again the future
>>>> and my method wouldn't work with a folder!
>>>>
>>>> Clearly if I was not using zlib compression I could just have
read the
>>>> contents of the mdbox without any complication.
>>>>
>>>>
>>>> -- 
>>>> Daniel Sch?tze
>>>> Director
>>>>
>>>> ------------------
>>>>
>>>> CWA International Ltd
>>>> 5th Floor, 42 Trinity Square
>>>> London
>>>> EC3N 4DJ
>>>>
>>>> (t) + 44 (0)20 7242 8444
>>>> (e) dms at cwa.uk.com
>>>> (w) http://www.cwa.international/
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20201110/01d7fd16/attachment-0001.html>

Could one use this "mail_location=mdbox_deleted:/path/to/mdbox" for
lets
say a 'recovery' namespace. Where users can copy their deleted messages 
from?




-----Original Message-----
From: Aki Tuomi [mailto:aki.tuomi at open-xchange.com] 
Sent: Tuesday, November 10, 2020 7:16 PM
To: Daniel Sch?tze; dovecot at dovecot.org
Subject: Re: Recovering expunged but not purged e-mails from mdbox with 
zlib compression

> On 10/11/2020 20:07 Daniel Sch?tze <dms at cwa.uk.com> wrote:
> 
> 
> Dear Aki
> Thank you. Unfortunately I'm struggling to get the right syntax for 
> this as it looks like someone else was here too 
> https://dovecot.org/pipermail/dovecot/2018-July/112441.html
> The location in my dovecot.conf is
> 
> mdbox:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:INDEXPVT=~/mdbox/shared/%
> %n
This syntax is for accessing shared folders.

You probably should try

doveadm -Dv -o mail_location=mdbox_deleted:/path/to/mdbox fetch -u 
victim text ALL
> I'm trying to fetch a message (for testing purposes now) based on
it's
guid as that is available from doveadm dump.
> So my syntax is (based on the previous person who didn't get it to 
work).
> 
> doveadm -o 
> "mail_location=mdbox_deleted:%%h/mdbox:INDEX=/indexdisk/indexes/%%n:IN
> DEXPVT=~/mdbox/shared/%%n" fetch "body" guid (msg.guid from
doveadm
> dump)
> 
Doveadm does not support var expand, so this will not work. See above 
for syntax.
> But that's getting no response. I've tried putting in the hard path
in
case there is any trouble with the %%h etc but that doesn't help
either.
> I've also tried the fetch with the subject of a known deleted message 
and also adding mailbox Drafts (as I know that's where it is or rather 
was) but to no avail.
> Given doveadm dump gives msg.uid and no subject I'd prefer to use that 
in any case.
> There are no error messages, I do with doveadm was a little more 
verbose to help over these bumps!
> 
> Any chance of a helping hand?
> 
> 
> 
> 
Aki
> 
> 
> 
> Daniel Sch?tze
>  Director
>  
>  ------------------
>  
>  CWA International Ltd
>  5th Floor, 42 Trinity Square
>  London
>  EC3N 4DJ
>  
>  (t) + 44 (0)20 7242 8444
>  (e) dms at cwa.uk.com
>  (w) http://www.cwa.international/
> On 10/11/20 10:53, Aki Tuomi wrote:
> 
> > You can use mdbox_deleted driver to access mails with refcount 0. 
> > See https://wiki2.dovecot.org/MailboxFormat/dbox
> > 
> > Aki
> > 
> > On 10.11.2020 12.42, Daniel Sch?tze wrote:
> > 
> > > Yesterday I had to recover an e-mail which a user had deleted.?
If
> > > I understand this correctly the message was expunged but not 
> > > purged (doveadm purge had not been run).
> > > 
> > > This e-mail was clearly still in the mdbox stored with zlib 
> > > compression as I could tell using the doveadm dump command 
> > > (doveadm dump -t dbox filename).
> > > 
> > > I could however not reveal the e-mail with the normal doveadm 
> > > fetch -u username "body" guid (guid from dump output)
> > > 
> > > In the end I was able to recover the e-mail by cutting it out
from
> > > the mdbox and running gunzip over it, but this method was very 
> > > fiddly and would not have worked for a bulk job.? I appreciate I 
> > > could have just deleted the index files and gotten all the
deleted
> > > messages back by having the index rebuilt but that would have
been
> > > the proverbial "hammer to crack a nut".
> > > 
> > > Fortunately this e-mail had no attachment for me to worry about, 
> > > as those are also detached for sis by dovecot.
> > > 
> > > Can anyone tell me of a quick and easy way to recover one or 
> > > multiple e-mails marked as expunged but not purged which are 
> > > stored in a mdbox with zlib compression?? I'm sure I'll
have to do
> > > this again the future and my method wouldn't work with a
folder!
> > > 
> > > Clearly if I was not using zlib compression I could just have
read
> > > the contents of the mdbox without any complication.
> > > 
> > > 
> > > --
> > > Daniel Sch?tze
> > > Director
> > > 
> > > ------------------
> > > 
> > > CWA International Ltd
> > > 5th Floor, 42 Trinity Square
> > > London
> > > EC3N 4DJ
> > > 
> > > (t) + 44 (0)20 7242 8444
> > > (e) dms at cwa.uk.com
> > > (w) http://www.cwa.international/
> > >