Hi, I have recently migrated (under emergency conditions) a dovecot imap/pop based server to a new instance. The mailboxes used mdbox format and due to various screwups I had corrupt indexes. I thought I'd cleaned this up but then I found that this new instance hadn't been set up correctly for nfs. Long story short, I still get users with new cases of corrupt indexes. The symptom is imap either showing NO mail in their inbox, or, not showing any recently delivered mail in the box, until I rm -f dovecot.map.index / doveadm force-resync -u user. It would be a huge help if there could be some method to detect if this is the case for any given user and to proactively do the force-resync process for them instead of waiting for that support call (or service cancellation...). I have looked around and have not found any tool capable of 'linting' an mdbox format inbox, and it seems like something like this should have been or would be an extremely useful debugging tool both during development as well as to troubleshoot stuff in the field. I would love to know if anyone either has such a tool, or any general suggestion how I could go about finding these cases and addressing them. I believe I have the nfs issue resolved and will not be creating new cases, so I just want to fix the ~3000 boxes I have now and move forward. Thank you. Mike-
On 25 Jul 2019, at 20.55, Mike via dovecot <dovecot at dovecot.org> wrote:> > Hi, > > > I have recently migrated (under emergency conditions) a dovecot imap/pop > based server to a new instance. The mailboxes used mdbox format and due > to various screwups I had corrupt indexes. I thought I'd cleaned this up > but then I found that this new instance hadn't been set up correctly for > nfs. Long story short, I still get users with new cases of corrupt > indexes. The symptom is imap either showing NO mail in their inbox, or, > not showing any recently delivered mail in the box, until I rm -f > dovecot.map.index / doveadm force-resync -u user. > > It would be a huge help if there could be some method to detect if this > is the case for any given user and to proactively do the force-resync > process for them instead of waiting for that support call (or service > cancellation...). I have looked around and have not found any tool > capable of 'linting' an mdbox format inbox, and it seems like something > like this should have been or would be an extremely useful debugging > tool both during development as well as to troubleshoot stuff in the > field. I would love to know if anyone either has such a tool, or any > general suggestion how I could go about finding these cases and > addressing them. I believe I have the nfs issue resolved and will not be > creating new cases, so I just want to fix the ~3000 boxes I have now and > move forward.I think you could do something with using "doveadm dump" command. I'm not sure in what ways your mdboxes are corrupted, so there might be an easier way to do it, but for a generic checker I think this would work: * doveadm dump dovecot.map.index and remember all the "uid" numbers * doveadm dump dovecot.index for each folder and remember all the "map_uid" numbers. * See if any map_uid is missing in dovecot.map.index's uid numbers. yes -> force-resync You can also use doveadm dump for the storage/m.* files to see what they contain, but this likely won't be useful for this case.
On 27 Jul 2019, at 14.13, Timo Sirainen <timo at sirainen.com> wrote:> > On 25 Jul 2019, at 20.55, Mike via dovecot <dovecot at dovecot.org> wrote: >> >> Hi, >> >> >> I have recently migrated (under emergency conditions) a dovecot imap/pop >> based server to a new instance. The mailboxes used mdbox format and due >> to various screwups I had corrupt indexes. I thought I'd cleaned this up >> but then I found that this new instance hadn't been set up correctly for >> nfs. Long story short, I still get users with new cases of corrupt >> indexes. The symptom is imap either showing NO mail in their inbox, or, >> not showing any recently delivered mail in the box, until I rm -f >> dovecot.map.index / doveadm force-resync -u user. >> >> It would be a huge help if there could be some method to detect if this >> is the case for any given user and to proactively do the force-resync >> process for them instead of waiting for that support call (or service >> cancellation...). I have looked around and have not found any tool >> capable of 'linting' an mdbox format inbox, and it seems like something >> like this should have been or would be an extremely useful debugging >> tool both during development as well as to troubleshoot stuff in the >> field. I would love to know if anyone either has such a tool, or any >> general suggestion how I could go about finding these cases and >> addressing them. I believe I have the nfs issue resolved and will not be >> creating new cases, so I just want to fix the ~3000 boxes I have now and >> move forward. > > > I think you could do something with using "doveadm dump" command. I'm not sure in what ways your mdboxes are corrupted, so there might be an easier way to do it, but for a generic checker I think this would work: > > * doveadm dump dovecot.map.index and remember all the "uid" numbers > * doveadm dump dovecot.index for each folder and remember all the "map_uid" numbers. > * See if any map_uid is missing in dovecot.map.index's uid numbers. yes -> force-resync > > You can also use doveadm dump for the storage/m.* files to see what they contain, but this likely won't be useful for this case.Or actually reading further: It looks like all your indexes are gone, even the folders' dovecot.index* files? Wouldn't a simple solution then be just to check if "doveadm mailbox status" if the user looks empty (or mostly empty) you'd run the force-resync? Or another thought: If INBOX's dovecot.index was completely lost and was recreated, you can see the index's creation timestamp in the "indexid" field in "doveadm dump dovecot.index" - if it's new enough do the force-resync. BTW. If all the indexes are gone, force-resync adds all storage/m.* mails back to their original folder. So if user had moved mails to another folder they come back to e.g. INBOX. It also loses message flags, and brings back mails that were already expunged but not yet doveadm purged.