Charles Marcus
2013-Dec-24 17:02 UTC
[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
Ok, hopefully there is a solution to this. I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so. Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync. Here is the series of commands I'm running: 1. stop postfix, stop dovecot on new server 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/ 3. chown vmail:vmail /var/vmail 4. start dovecot, start postfix ls -al /var/vmail/example.com/user/cur shows all of the messages that should be there, and all perms are correct. Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't. I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps. The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem. The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders. This will be a major support problem. Anyone have any ideas? -- Best regards, */Charles/*
Reindl Harald
2013-Dec-24 17:04 UTC
[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
Am 24.12.2013 18:02, schrieb Charles Marcus:> Ok, hopefully there is a solution to this. > > I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but > have a problem that I really want to fox before doing so. > > Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a syncthat is not dovecot specific and a thunderbird problem right click on the folder -> properties -> repair it happens from time to time that after that messages re-appear and this happens on any mailserver, not only dovecot -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 246 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20131224/649d5ecc/attachment.bin>
Stan Hoeppner
2013-Dec-24 18:11 UTC
[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
On 12/24/2013 11:02 AM, Charles Marcus wrote:> Ok, hopefully there is a solution to this. > > I've been experimenting with multiple rsyncs in preparation for pulling > the trigger on the mail server switch, but have a problem that I really > want to fox before doing so. > > Apparently something causes Thunderbirds local message cache to get out > of sync with dovecot after a sync. > > Here is the series of commands I'm running: > > 1. stop postfix, stop dovecot on new server > > 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt' > /path/to/vmail/example.com/ /var/vmail/example.com/ > > 3. chown vmail:vmail /var/vmail > > 4. start dovecot, start postfix > > ls -al /var/vmail/example.com/user/cur > > shows all of the messages that should be there, and all perms are correct. > > Go to my account that is pointed to this mail server/account, and none > of the new messages show up. Also, some messages are still showing up > that shouldn't. > > I've tried compacting the folders, closing/relaunching Thunderbird, but > nothing helps. > > The only way to get them to show up is to go to the local Thunderbird > cache for the account, and delete the files associated with the folder > having the problem. > > The problem is, ALL folders will have this problem, which means that > everyone will need to delete ALL of their local cahced folders. > > This will be a major support problem. > > Anyone have any ideas?The source of the problem is almost certainly out of sync Dovecot index files between the old and new servers, and thus TBird. After an rsync copy of the mails the new server must create the indexes on-the-fly when TBird connects, and the resulting new indexes are likely not identical to the old server. Thus TBird is seeing a different mailbox view. TBird keeps its own indexes for all IMAP folders. It has nothing little or nothing to do with local cached copies of folders. I don't use GLODA and I don't cache locally, but I still have a .msf file for each Dovecot IMAP folder, some of them multiple MBs in size. These are strictly indexes. It's these local indexes not being in sync with your new Dovecot server indexes that I'm pretty sure is the cause of your problem. If the mailbox contents are identical before/after the copy, you might try copying the indexes over from the old mail server, preserving permissions, creation time, atime, etc. If the server indexes are identical before/after the rsync you should avoid this problem, assuming everything else is identical, including server hostnames, IP addresses, encryption key, etc, etc. TBird tracks mailboxes by server name in Account Settings after all. If the server name changes that'll cause TBird to create an alternate local folder hierarchy in the profile directory. And that'll wreak havoc on your indexes, mailbox view, etc. -- Stan
Charles Marcus
2013-Dec-26 15:07 UTC
[Dovecot] Dovecot control files - WAS: Re: Thunderbird message cache out of sync after repetitive rsyncs...
On 2013-12-24 12:02 PM, Charles Marcus <CMarcus at Media-Brokers.com> wrote:> Apparently something causes Thunderbirds local message cache to get > out of sync with dovecot after a sync.Ok, I had a new thought about the problem of invalid local client caches... There is this on the dovecot wiki about the control files: http://wiki2.dovecot.org/MailLocation/Maildir "Dovecot stores some Maildir metadata into two control files: dovecot-uidlist file contains IMAP UID <-> Maildir filename mapping ... If the messages get new UIDs, the IMAP clients will invalidate their local cache and download the messages all over again. If you do this for all the users, you could cause huge disk I/O bursts to your server. " This sounds like exactly what I want (IMAP clients will INVALIDATE THEIR LOCAL CACHE...) to happen the first time users access their mail on the new server. So, I'm thinking what I could do is simply exclude the dovecot-uid* files during the transfer (and delete existing ones using --delete-excluded) during the rsync, and this would cause everyone's Thunderbird to regenerate their local caches, thus eliminating the need to rebuild the .msf files (either manually or by scripting their deletion)? The new server is a VM on a pretty hefty box, with 16GB allocated, and if I do this before people come in, their initial logins would be spread out, not exactly at the same time, so the 'huge disk IO bursts' shouldn't be a big problem. Obviously this would only happen the for the client they first access their mail with, so I'd need to block access to the new mail server from outside (ie, so mobile clients, which are polling the new server, wouldn't cause the regeneration of the dovecot-uid* files before Thunderbird gets a chance to, but... Hmmm... what will happen for mobile clients... crap... will those get confused to, but leaving me with no easy way to delete their local cache (like I could the .msf files)? Have to first test and make sure this will actually solve the problem though... -- Best regards, */Charles/*
Timo Sirainen
2013-Dec-26 16:05 UTC
[Dovecot] Thunderbird message cache out of sync after repetitive rsyncs...
None of that should be happening. The client shouldn't be able to become confused, because it should see identical mailboxes in the old and in the new server. You can check with IMAP protocol if the rsync actually preserved everything correctly (what's in excludes.txt?) : doveadm exec imap -u user at domain a select inbox b uid search all Do this in both old and new server. Make sure that UIDVALIDITY and UIDNEXT replies to SELECT are the same, and also that the SEARCH reply is the same. Did you stop Dovecot on the old server during rsync? That could explain if it was modified while rsync was running. On 24.12.2013, at 19.02, Charles Marcus <CMarcus at Media-Brokers.com> wrote:> Ok, hopefully there is a solution to this. > > I've been experimenting with multiple rsyncs in preparation for pulling the trigger on the mail server switch, but have a problem that I really want to fox before doing so. > > Apparently something causes Thunderbirds local message cache to get out of sync with dovecot after a sync. > > Here is the series of commands I'm running: > > 1. stop postfix, stop dovecot on new server > > 2. rsync -rltgovDHP --delete --exclude-from 'excludes.txt' /path/to/vmail/example.com/ /var/vmail/example.com/ > > 3. chown vmail:vmail /var/vmail > > 4. start dovecot, start postfix > > ls -al /var/vmail/example.com/user/cur > > shows all of the messages that should be there, and all perms are correct. > > Go to my account that is pointed to this mail server/account, and none of the new messages show up. Also, some messages are still showing up that shouldn't. > > I've tried compacting the folders, closing/relaunching Thunderbird, but nothing helps. > > The only way to get them to show up is to go to the local Thunderbird cache for the account, and delete the files associated with the folder having the problem. > > The problem is, ALL folders will have this problem, which means that everyone will need to delete ALL of their local cahced folders. > > This will be a major support problem. > > Anyone have any ideas? > > -- > > Best regards, > > */Charles/*
Reasonably Related Threads
- Server Migration Attempt - new messages DELETED after secondary rsyncs
- Ignoring permission differences...
- [OT] preferred clients
- Dovecot, Thunderbird and messages in wrong folder
- Thunderbird 24.0 for Windows seems to ignore Samba4.0.9 permissions settings