Mark Zealey
2011-Nov-26 16:33 UTC
[Dovecot] using dsync to convert mailboxes looses caching options
Hi there,
We're trying to convert users from Maildir to sdbox at present; I'm
using dsync to achieve this (2.0.16) however when the user's have been
converted we only get minimal information in the caching files. Is there
some way to preserve all the caching decisions that were previously made
so that when the user logs in to the new mailbox we don't have to cause
an io storm rebuilding the cache that we know was good? Dovecot seems to
be partially doing this - if i remove the logs/cache from the source
mailbox no cache files are built in the conversion; if i put them back
then we get a cache file built but it only contains a few bits of
information (guid, date.save). Looking into this a bit further i find
that when the caches are present at source the fields are preserved but
the 'last used' date and caching decisions are not which I suspect means
dsync doesn't bother caching on import - only fields with a yes decision
in the source are copied (but their decision is only copied as a tmp
with the date of import). For example:
Source idxview cache:
-- Cache fields --
# Name Type Size Dec Last used
0: flags bit 4 tmp
2011-11-25 16:09
1: date.received fix 4 yes
2011-11-26 16:17
2: size.virtual fix 8 tmp
2011-11-25 16:09
3: imap.bodystructure str - tmp
2011-11-25 16:09
4: mime.parts var - tmp
2011-11-25 16:09
5: hdr.IMPORTANCE hdr - tmp
2011-11-25 16:09
6: hdr.X-PRIORITY hdr - tmp
2011-11-25 16:09
7: hdr.CONTENT-TYPE hdr - tmp
2011-11-25 16:09
...
18: date.sent fix 8 no
1970-01-01 01:00
19: date.save fix 4 yes
2011-11-26 16:17
20: size.physical fix 8 no
1970-01-01 01:00
21: imap.body str - no
1970-01-01 01:00
... (24 total entries. stripped ones are just hdr. with a tmp decision)
dst cache:
# Name Type Size Dec Last used
0: flags bit 4 tmp
1970-01-01 01:00
1: date.sent fix 8 no
1970-01-01 01:00
2: date.received fix 4 tmp
1970-01-01 01:00
3: date.save fix 4 tmp
2011-11-26 16:19
4: size.virtual fix 8 tmp
1970-01-01 01:00
5: size.physical fix 8 no
1970-01-01 01:00
6: imap.body str - no
1970-01-01 01:00
7: imap.bodystructure str - tmp
1970-01-01 01:00
8: imap.envelope str - no!
1970-01-01 01:00
9: pop3.uidl str - no
1970-01-01 01:00
10: guid str - tmp
2011-11-26 16:19
11: mime.parts var - tmp
1970-01-01 01:00
12: hdr.IMPORTANCE hdr - tmp
1970-01-01 01:00
13: hdr.X-PRIORITY hdr - tmp
1970-01-01 01:00
...
and dst cached message for example:
RECORD: seq=57, uid=12207, flags=0x09 (Seen Answered)
- ext 1 modseq : 1 (0100000000000000)
- ext 3 cache : 5516 (8c150000)
- cache offset=5516 size=92, prev_offset = 0
- guid: 1321895282.XXXt,S=7399760,W=7496225
- date.save: 1322324286 (3e11d14e)
src idx record for this message contains pretty much all fields.
Thanks,
Mark
Timo Sirainen
2011-Dec-08 05:53 UTC
[Dovecot] using dsync to convert mailboxes looses caching options
On Sat, 2011-11-26 at 18:33 +0200, Mark Zealey wrote:> We're trying to convert users from Maildir to sdbox at present; I'm > using dsync to achieve this (2.0.16) however when the user's have been > converted we only get minimal information in the caching files. Is there > some way to preserve all the caching decisions that were previously made > so that when the user logs in to the new mailbox we don't have to cause > an io storm rebuilding the cache that we know was good? Dovecot seems to > be partially doing this - if i remove the logs/cache from the source > mailbox no cache files are built in the conversion; if i put them back > then we get a cache file built but it only contains a few bits of > information (guid, date.save). Looking into this a bit further i find > that when the caches are present at source the fields are preserved but > the 'last used' date and caching decisions are not which I suspect means > dsync doesn't bother caching on import - only fields with a yes decision > in the source are copied (but their decision is only copied as a tmp > with the date of import). For example:How are you calling dsync? Does the destination already exist? I tried with: rm -rf /tmp/foo; dsync -u tss -m INBOX mirror sdbox:/tmp/foo It sets all of the cache fields with "yes" or "tmp" decision, as it should. But yes, the "last used" field should probably be copied as well. Perhaps the problem with you is that dsync actually writes all of the cache fields, but then it does a "cache compression" at the end, which sees that the "last used" fields are so old, so it deletes them. But yes, it is a problem that dsync doesn't update caching decisions.. Hmm. I guess I'll have to fix that for v2.1.
Mark Zealey
2011-Dec-08 14:45 UTC
[Dovecot] using dsync to convert mailboxes looses caching options
With 2.0.16 hdr.xxx fields get copied fine (but of course without timestamp). With the patch you provided they don't get copied whether using mirror or backup & starting from scratch. I'm doing a Maildir to sdbox migration otherwise don't think I'm doing anytihng strange. Mark ________________________________________ From: Mark Zealey Sent: 08 December 2011 09:35 To: Timo Sirainen Cc: Dovecot Mailing List Subject: RE: [Dovecot] using dsync to convert mailboxes looses caching options OK I'll test the header copying more fully. The reason we want to preserve caching decisions is to avoid an IO storm when users log in to their mailboxes after an sdbox upgrade so it would be great to be able to have some way to warm caches. Mark