Hello, I was forced to do an emergency server migration with about 3000 mailboxes and I botched it somewhat. Hoping to find some help here. My symptoms are, users showing apparently deleted mail messages, some replicated mail, and a time period of mail that is just 'missing'. The source server is centos 6.10 with dovecot 2.2.5, and the destination is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did an rsync from the old to the new. My screw up however was that my rsync was not --delete files, and had to be run multiple times over the course of a few weeks due to to bandwidth disparities and the sheer size of the mail spool approaching 700GB (source server had 10mbps. Should have used sneakernet instead). The bottom line however is that I wound up having many extra m.* files in each user storage dir that were not actually present on the source when the final, final sync was made and I went live with the migration server.? I know this is my fsckup but now I have to deal with it. I first tried to fix the symptoms with a force-resync on the boxes not realizing the above, and that seemed to make things worse by only showing new mail as of the server migration date. Ugh. New mail is being delivered and this does appear in user mail boxes. But as I said, there seems to be some users with a problem with old mail presumably deleted on the prior server suddenly re-appearing (probbly due to the extraneous m.* mdbox files present). A more pressing problem is that there is simply a time period of mail that is in user mailboxes (on the old server) from before the migration, which is not displaying now for these users on the new server. I have investigated this and hacked on "mdbox-recover.pl" somewhat so that it iterates over all mdbox files for a user and I can extract ALL messages, and see that, yes, the mail really is present in the mdbox files, but pop/imap never shows it. This is puzzling to me. I have a test user that when I check imap, it shows only 21 message in inbox and 1 in sent. But if I extract the messages from the mdbox files, then I actually have 1117 messages.? I know I can't just rsync the source server again with the right --delete flag, there has been lots of mail deliveries to the migrated boxes and so I don't want to lose all of that and create yet more problems. But I need to find some way to make these present mail messages appear for my users again. Im thinking it's got to be a flag of some kind since the data is there it's just ignored. Wondering what you would reccomend? Thanks. Mike-
<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div>
<br>
</div>
<blockquote type="cite">
<div>
On 15/07/2019 09:00 Mike via dovecot <
<a
href="mailto:dovecot@dovecot.org">dovecot@dovecot.org</a>>
wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Hello,
</div>
<div>
<br>
</div>
<div>
I was forced to do an emergency server migration with about 3000
</div>
<div>
mailboxes and I botched it somewhat. Hoping to find some help here.
</div>
<div>
<br>
</div>
<div>
My symptoms are, users showing apparently deleted mail messages, some
</div>
<div>
replicated mail, and a time period of mail that is just 'missing'.
</div>
<div>
<br>
</div>
<div>
The source server is centos 6.10 with dovecot 2.2.5, and the destination
</div>
<div>
is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did
</div>
<div>
an rsync from the old to the new. My screw up however was that my rsync
</div>
<div>
was not --delete files, and had to be run multiple times over the course
</div>
<div>
of a few weeks due to to bandwidth disparities and the sheer size of the
</div>
<div>
mail spool approaching 700GB (source server had 10mbps. Should have used
</div>
<div>
sneakernet instead). The bottom line however is that I wound up having
</div>
<div>
many extra m.* files in each user storage dir that were not actually
</div>
<div>
present on the source when the final, final sync was made and I went
</div>
<div>
live with the migration server. I know this is my fsckup but now I have
</div>
<div>
to deal with it.
</div>
<div>
<br>
</div>
<div>
I first tried to fix the symptoms with a force-resync on the boxes not
</div>
<div>
realizing the above, and that seemed to make things worse by only
</div>
<div>
showing new mail as of the server migration date. Ugh.
</div>
<div>
<br>
</div>
<div>
New mail is being delivered and this does appear in user mail boxes. But
</div>
<div>
as I said, there seems to be some users with a problem with old mail
</div>
<div>
presumably deleted on the prior server suddenly re-appearing (probbly
</div>
<div>
due to the extraneous m.* mdbox files present). A more pressing problem
</div>
<div>
is that there is simply a time period of mail that is in user mailboxes
</div>
<div>
(on the old server) from before the migration, which is not displaying
</div>
<div>
now for these users on the new server.
</div>
<div>
<br>
</div>
<div>
I have investigated this and hacked on "mdbox-recover.pl" somewhat
so
</div>
<div>
that it iterates over all mdbox files for a user and I can extract ALL
</div>
<div>
messages, and see that, yes, the mail really is present in the mdbox
</div>
<div>
files, but pop/imap never shows it. This is puzzling to me. I have a
</div>
<div>
test user that when I check imap, it shows only 21 message in inbox and
</div>
<div>
1 in sent. But if I extract the messages from the mdbox files, then I
</div>
<div>
actually have 1117 messages.
</div>
<div>
<br>
</div>
<div>
I know I can't just rsync the source server again with the right
</div>
<div>
--delete flag, there has been lots of mail deliveries to the migrated
</div>
<div>
boxes and so I don't want to lose all of that and create yet more
</div>
<div>
problems. But I need to find some way to make these present mail
</div>
<div>
messages appear for my users again. Im thinking it's got to be a flag of
</div>
<div>
some kind since the data is there it's just ignored. Wondering what you
</div>
<div>
would reccomend?
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Thanks.
</div>
<div>
<br>
</div>
<div>
Mike-
</div>
</blockquote>
<div>
<br>
</div>
<div>
Try with one mailbox first.
</div>
<div>
<br>
</div>
<div>
Delete dovecot.index.map files and run force-resync.
</div>
<div>
<br>
</div>
<div class="io-ox-signature">
<pre>---
Aki Tuomi</pre>
</div>
</body>
</html>
<!doctype html>
<html>
<head>
<meta charset="UTF-8">
</head>
<body>
<div>
<br>
</div>
<blockquote type="cite">
<div>
On 15/07/2019 09:22 Aki Tuomi via dovecot <dovecot@dovecot.org> wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<blockquote type="cite">
<div>
On 15/07/2019 09:00 Mike via dovecot <
<a
href="mailto:dovecot@dovecot.org">dovecot@dovecot.org</a>>
wrote:
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Hello,
</div>
<div>
<br>
</div>
<div>
I was forced to do an emergency server migration with about 3000
</div>
<div>
mailboxes and I botched it somewhat. Hoping to find some help here.
</div>
<div>
<br>
</div>
<div>
My symptoms are, users showing apparently deleted mail messages, some
</div>
<div>
replicated mail, and a time period of mail that is just 'missing'.
</div>
<div>
<br>
</div>
<div>
The source server is centos 6.10 with dovecot 2.2.5, and the destination
</div>
<div>
is ubuntu 16 / dovecot 2.2.22. User mailboxes are mdbox format and I did
</div>
<div>
an rsync from the old to the new. My screw up however was that my rsync
</div>
<div>
was not --delete files, and had to be run multiple times over the course
</div>
<div>
of a few weeks due to to bandwidth disparities and the sheer size of the
</div>
<div>
mail spool approaching 700GB (source server had 10mbps. Should have used
</div>
<div>
sneakernet instead). The bottom line however is that I wound up having
</div>
<div>
many extra m.* files in each user storage dir that were not actually
</div>
<div>
present on the source when the final, final sync was made and I went
</div>
<div>
live with the migration server. I know this is my fsckup but now I have
</div>
<div>
to deal with it.
</div>
<div>
<br>
</div>
<div>
I first tried to fix the symptoms with a force-resync on the boxes not
</div>
<div>
realizing the above, and that seemed to make things worse by only
</div>
<div>
showing new mail as of the server migration date. Ugh.
</div>
<div>
<br>
</div>
<div>
New mail is being delivered and this does appear in user mail boxes. But
</div>
<div>
as I said, there seems to be some users with a problem with old mail
</div>
<div>
presumably deleted on the prior server suddenly re-appearing (probbly
</div>
<div>
due to the extraneous m.* mdbox files present). A more pressing problem
</div>
<div>
is that there is simply a time period of mail that is in user mailboxes
</div>
<div>
(on the old server) from before the migration, which is not displaying
</div>
<div>
now for these users on the new server.
</div>
<div>
<br>
</div>
<div>
I have investigated this and hacked on "mdbox-recover.pl"
somewhat so
</div>
<div>
that it iterates over all mdbox files for a user and I can extract ALL
</div>
<div>
messages, and see that, yes, the mail really is present in the mdbox
</div>
<div>
files, but pop/imap never shows it. This is puzzling to me. I have a
</div>
<div>
test user that when I check imap, it shows only 21 message in inbox and
</div>
<div>
1 in sent. But if I extract the messages from the mdbox files, then I
</div>
<div>
actually have 1117 messages.
</div>
<div>
<br>
</div>
<div>
I know I can't just rsync the source server again with the right
</div>
<div>
--delete flag, there has been lots of mail deliveries to the migrated
</div>
<div>
boxes and so I don't want to lose all of that and create yet more
</div>
<div>
problems. But I need to find some way to make these present mail
</div>
<div>
messages appear for my users again. Im thinking it's got to be a flag
of
</div>
<div>
some kind since the data is there it's just ignored. Wondering what you
</div>
<div>
would reccomend?
</div>
<div>
<br>
</div>
<div>
<br>
</div>
<div>
Thanks.
</div>
<div>
<br>
</div>
<div>
Mike-
</div>
</blockquote>
<div>
<br>
</div>
<div>
Try with one mailbox first.
</div>
<div>
<br>
</div>
<div>
Delete dovecot.index.map files and run force-resync.
</div>
<div>
<br>
</div>
<div class="io-ox-signature">
<pre>---
Aki Tuomi</pre>
</div>
</blockquote>
<div>
<br>
</div>
<div>
Sorry I ment dovecot.map.index there
</div>
<div class="io-ox-signature">
<pre>---
Aki Tuomi</pre>
</div>
</body>
</html>