dovecot - Dec 2013 - Thunderbird message cache out of sync after repetitive rsyncs...

Ok, hopefully there is a solution to this.

I've been experimenting with multiple rsyncs in preparation for pulling 
the trigger on the mail server switch, but have a problem that I really 
want to fox before doing so.

Apparently something causes Thunderbirds local message cache to get out 
of sync with dovecot after a sync.

Here is the series of commands I'm running:

1. stop postfix, stop dovecot on new server

2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt' 
/path/to/vmail/example.com/ /var/vmail/example.com/

3. chown vmail:vmail /var/vmail

4. start dovecot, start postfix

ls -al /var/vmail/example.com/user/cur

shows all of the messages that should be there, and all perms are correct.

Go to my account that is pointed to this mail server/account, and none 
of the new messages show up. Also, some messages are still showing up 
that shouldn't.

I've tried compacting the folders, closing/relaunching Thunderbird, but 
nothing helps.

The only way to get them to show up is to go to the local Thunderbird 
cache for the account, and delete the files associated with the folder 
having the problem.

The problem is, ALL folders will have this problem, which means that 
everyone will need to delete ALL of their local cahced folders.

This will be a major support problem.

Anyone have any ideas?

-- 

Best regards,

*/Charles/*

Am 24.12.2013 18:02, schrieb Charles Marcus:
> Ok, hopefully there is a solution to this.
> 
> I've been experimenting with multiple rsyncs in preparation for pulling
the trigger on the mail server switch, but
> have a problem that I really want to fox before doing so.
> 
> Apparently something causes Thunderbirds local message cache to get out of
sync with dovecot after a sync
that is not dovecot specific and a thunderbird problem
right click on the folder -> properties -> repair

it happens from time to time that after that messages re-appear
and this happens on any mailserver, not only dovecot

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 246 bytes
Desc: OpenPGP digital signature
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20131224/649d5ecc/attachment.bin>

On 12/24/2013 11:02 AM, Charles Marcus wrote:
> Ok, hopefully there is a solution to this.
> 
> I've been experimenting with multiple rsyncs in preparation for pulling
> the trigger on the mail server switch, but have a problem that I really
> want to fox before doing so.
> 
> Apparently something causes Thunderbirds local message cache to get out
> of sync with dovecot after a sync.
> 
> Here is the series of commands I'm running:
> 
> 1. stop postfix, stop dovecot on new server
> 
> 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt'
> /path/to/vmail/example.com/ /var/vmail/example.com/
> 
> 3. chown vmail:vmail /var/vmail
> 
> 4. start dovecot, start postfix
> 
> ls -al /var/vmail/example.com/user/cur
> 
> shows all of the messages that should be there, and all perms are correct.
> 
> Go to my account that is pointed to this mail server/account, and none
> of the new messages show up. Also, some messages are still showing up
> that shouldn't.
> 
> I've tried compacting the folders, closing/relaunching Thunderbird, but
> nothing helps.
> 
> The only way to get them to show up is to go to the local Thunderbird
> cache for the account, and delete the files associated with the folder
> having the problem.
> 
> The problem is, ALL folders will have this problem, which means that
> everyone will need to delete ALL of their local cahced folders.
> 
> This will be a major support problem.
> 
> Anyone have any ideas?
The source of the problem is almost certainly out of sync Dovecot index
files between the old and new servers, and thus TBird.  After an rsync
copy of the mails the new server must create the indexes on-the-fly when
TBird connects, and the resulting new indexes are likely not identical
to the old server.  Thus TBird is seeing a different mailbox view.

TBird keeps its own indexes for all IMAP folders.  It has nothing little
or nothing to do with local cached copies of folders.  I don't use GLODA
and I don't cache locally, but I still have a .msf file for each Dovecot
IMAP folder, some of them multiple MBs in size.  These are strictly
indexes.  It's these local indexes not being in sync with your new
Dovecot server indexes that I'm pretty sure is the cause of your problem.

If the mailbox contents are identical before/after the copy, you might
try copying the indexes over from the old mail server, preserving
permissions, creation time, atime, etc.  If the server indexes are
identical before/after the rsync you should avoid this problem, assuming
everything else is identical, including server hostnames, IP addresses,
encryption key, etc, etc.  TBird tracks mailboxes by server name in
Account Settings after all.  If the server name changes that'll cause
TBird to create an alternate local folder hierarchy in the profile
directory.  And that'll wreak havoc on your indexes, mailbox view, etc.

-- 
Stan

On 2013-12-24 12:02 PM, Charles Marcus <CMarcus at Media-Brokers.com>
wrote:
> Apparently something causes Thunderbirds local message cache to get 
> out of sync with dovecot after a sync. 
Ok, I had a new thought about the problem of invalid local client caches...

There is this on the dovecot wiki about the control files:

http://wiki2.dovecot.org/MailLocation/Maildir

"Dovecot stores some Maildir metadata into two control files:

     dovecot-uidlist file contains IMAP UID <-> Maildir filename mapping
...
If the messages get new UIDs, the IMAP clients will invalidate their 
local cache and download the messages all over again. If you do this for 
all the users, you could cause huge disk I/O bursts to your server. "

This sounds like exactly what I want (IMAP clients will INVALIDATE THEIR 
LOCAL CACHE...) to happen the first time users access their mail on the 
new server.

So, I'm thinking what I could do is simply exclude the dovecot-uid* 
files during the transfer (and delete existing ones using 
--delete-excluded) during the rsync, and this would cause everyone's 
Thunderbird to regenerate their local caches, thus eliminating the need 
to rebuild the .msf files (either manually or by scripting their deletion)?

The new server is a VM on a pretty hefty box, with 16GB allocated, and 
if I do this before people come in, their initial logins would be spread 
out, not exactly at the same time, so the 'huge disk IO bursts' 
shouldn't be a big problem.

Obviously this would only happen the for the client they first access 
their mail with, so I'd need to block access to the new mail server from 
outside (ie, so mobile clients, which are polling the new server, 
wouldn't cause the regeneration of the dovecot-uid* files before 
Thunderbird gets a chance to, but...

Hmmm... what will happen for mobile clients... crap... will those get 
confused to, but leaving me with no easy way to delete their local cache 
(like I could the .msf files)?

Have to first test and make sure this will actually solve the problem 
though...

-- 

Best regards,

*/Charles/*

None of that should be happening. The client shouldn't be able to become
confused, because it should see identical mailboxes in the old and in the new
server. You can check with IMAP protocol if the rsync actually preserved
everything correctly (what's in excludes.txt?) :

doveadm exec imap -u user at domain
a select inbox
b uid search all

Do this in both old and new server. Make sure that UIDVALIDITY and UIDNEXT
replies to SELECT are the same, and also that the SEARCH reply is the same.

Did you stop Dovecot on the old server during rsync? That could explain if it
was modified while rsync was running.

On 24.12.2013, at 19.02, Charles Marcus <CMarcus at Media-Brokers.com>
wrote:
> Ok, hopefully there is a solution to this.
> 
> I've been experimenting with multiple rsyncs in preparation for pulling
the trigger on the mail server switch, but have a problem that I really want to
fox before doing so.
> 
> Apparently something causes Thunderbirds local message cache to get out of
sync with dovecot after a sync.
> 
> Here is the series of commands I'm running:
> 
> 1. stop postfix, stop dovecot on new server
> 
> 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt'
/path/to/vmail/example.com/ /var/vmail/example.com/
> 
> 3. chown vmail:vmail /var/vmail
> 
> 4. start dovecot, start postfix
> 
> ls -al /var/vmail/example.com/user/cur
> 
> shows all of the messages that should be there, and all perms are correct.
> 
> Go to my account that is pointed to this mail server/account, and none of
the new messages show up. Also, some messages are still showing up that
shouldn't.
> 
> I've tried compacting the folders, closing/relaunching Thunderbird, but
nothing helps.
> 
> The only way to get them to show up is to go to the local Thunderbird cache
for the account, and delete the files associated with the folder having the
problem.
> 
> The problem is, ALL folders will have this problem, which means that
everyone will need to delete ALL of their local cahced folders.
> 
> This will be a major support problem.
> 
> Anyone have any ideas?
> 
> -- 
> 
> Best regards,
> 
> */Charles/*