Charles Marcus
2013-Dec-27 13:41 UTC
[Dovecot] Server Migration Attempt - new messages DELETED after secondary rsyncs
Starting a new thread, as I've got a lot more details now...
First, I know this may be (or is even likely) something weird happening
with Thunderbird, but the bottom line is, I have to find a resolution or
a workaround or something, or I will be unable to pull the trigger on
the server migration. It is simply unacceptable to lose ANY mails doing
this.
Goal: migrate mailstore from old server to new server using rsync
Thunderbird (24.2) is set up to access the new account for testing
Problem: after rsync, ls shows new messages (11 for the 27th), but when
Thunderbird accesses the maildir again after another rsync, MOST (but
not all) of the new messages are deleted (only one for the 27th remains)
Timo - I'd be happy to do anything necessary to track this down if
you'll just tell me what you need...
Note: if you are pretty confident that upgrading dovecot on the old
server and using dsync to do the migration will solve the problem,
obviously I'm willing to do that if there is no other choice, but... I
hope you'll want to get to the bottom of this regardless, as, according
to you and everything I've read, it should work without losing any emails...
Ok, as to details...
Source (ol/active server): dovecot 2.1.17, postfix virtual for delivery,
vmail owned by postfix:postfix
Note: using prefix=INBOX. courier compatibility namespace
Target: dovecot 2.2.10, dovecot LDA, vmail owned by vmail:vmail, nothing
being actively delivered (yet)
Note: NOT using prefix=INBOX. courier compatiblity namespace
Both doveconf -n outputs at end of this message
*** Could the different namespaces, or differing dovecot versions cause
this?
Current /snapshot/ being used was taken at about 3:48am this morning
rsync command (dovecot is STOPPED before performing, source is the
latest snapshot of the active server):
rsync -rltgovDHP --delete --delete-excluded --exclude-from
'/path/to/excludes.txt' /snapshot/.../vmail/example.com/user/
/.../vmail/example.com/user/
I'm only syncing the one user until I get this problem resolved as it is
faster.
excludes.txt contains only (just to eliminate these which are still on
the source):
courierimap*
Right after the rsync:
ls -al /.../user
(changes depending on what is in includes.txt)
-rw------- 1 vmail vmail 51 Dec 18 10:13 dovecot-keywords
-rw------- 1 vmail vmail 221638 Dec 27 02:42 dovecot-uidlist
-rw------- 1 vmail vmail 8 Dec 18 09:56 dovecot-uidvalidity
-r--r--r-- 1 vmail vmail 0 Jun 9 2012 dovecot-uidvalidity.4fd32805
ls -al /.../user/cur
(copy/pasted newest thru 12/25 - this list is always the same after a sync)
-rw------- 1 vmail vmail 24904 Dec 25 15:47
1388004452.Vfe02Ic72976M281736.oldhost.example.com:2,
-rw------- 1 vmail vmail 5825360 Dec 25 22:52
1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2,
-rw------- 1 vmail vmail 27718 Dec 26 02:19
1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2,
-rw------- 1 vmail vmail 28083 Dec 26 07:11
1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S
-rw------- 1 vmail vmail 100651 Dec 26 08:02
1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2,
-rw------- 1 vmail vmail 5286 Dec 26 09:41
1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S
-rw------- 1 vmail vmail 53203 Dec 26 09:55
1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S
-rw------- 1 vmail vmail 12039 Dec 26 12:48
1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS
-rw------- 1 vmail vmail 13531 Dec 26 13:14
1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S
-rw------- 1 vmail vmail 25161 Dec 26 15:47
1388090846.Vfe02Ic768acM489505.oldhost.example.com:2,
-rw------- 1 vmail vmail 5037 Dec 26 16:27
1388093257.Vfe02Ic76918M766778.oldhost.example.com:2,
-rw------- 1 vmail vmail 3157 Dec 26 17:01
1388095312.Vfe02Ic76997M703744.oldhost.example.com:2,
-rw------- 1 vmail vmail 22480 Dec 26 17:50
1388098239.Vfe02Ic769deM225078.oldhost.example.com:2,
-rw------- 1 vmail vmail 10043 Dec 26 19:59
1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2,
-rw------- 1 vmail vmail 10635 Dec 26 21:11
1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2,
-rw------- 1 vmail vmail 8695 Dec 26 21:11
1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2,
-rw------- 1 vmail vmail 11045 Dec 26 21:56
1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2,
-rw------- 1 vmail vmail 823 Dec 26 22:40
1388115600.Vfe02Ib61cf5M913598.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 815 Dec 26 22:40
1388115627.Vfe02Ib63443M811945.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 785 Dec 26 22:40
1388115628.Vfe02Ib659c9M617314.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 6156 Dec 27 00:05
1388120705.Vfe02Ib7aa31M58188.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 869 Dec 27 00:05
1388120713.Vfe02Ib8506aM185504.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 925 Dec 27 00:29
1388122155.Vfe02Ibb2a1fM764275.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 823 Dec 27 00:38
1388122687.Vfe02Ic2f69aM168075.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 801 Dec 27 00:38
1388122696.Vfe02Ic31befM507147.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 801 Dec 27 00:38
1388122709.Vfe02Ic3ac2bM981625.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 4576 Dec 27 00:57
1388123820.Vfe02Ic769e8M270744.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 8379 Dec 27 01:33
1388126003.Vfe02Ib7fd87M510118.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 27647 Dec 27 02:20
1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2,
-rw------- 1 vmail vmail 6373 Dec 27 02:42
1388130144.Vfe02Ic76a99M631316.oldhost.example.com:2,ST
-rw------- 1 vmail vmail 3906 Dec 27 03:01
1388131281.Vfe02Ic76a9cM599830.oldhost.example.com:2,ST
Start dovecot, select INBOX in Thunderbird for this account, it only
displays these (I got this ls output from AFTER the messages in the cur
directory are deleted per below - before doing any of the 3 aqctions
below, ALL of the messages show in the ls output):
ls -al /.../user/cur
-rw------- 1 vmail vmail 24904 Dec 25 15:47
1388004452.Vfe02Ic72976M281736.oldhost.example.com:2,
-rw------- 1 vmail vmail 5825360 Dec 25 22:52
1388029948.Vfe02Ic7306fM556006.oldhost.example.com:2,
-rw------- 1 vmail vmail 27718 Dec 26 02:19
1388042391.Vfe02Ic33b40M543019.oldhost.example.com:2,
-rw------- 1 vmail vmail 28083 Dec 26 07:11
1388059893.Vfe02Ic76388M193423.oldhost.example.com:2,S
-rw------- 1 vmail vmail 100651 Dec 26 08:02
1388062961.Vfe02Ic763c2M415005.oldhost.example.com:2,
-rw------- 1 vmail vmail 5286 Dec 26 09:41
1388068882.Vfe02Ic7642dM83715.oldhost.example.com:2,S
-rw------- 1 vmail vmail 53203 Dec 26 09:55
1388069721.Vfe02Ic7646fM716189.oldhost.example.com:2,S
-rw------- 1 vmail vmail 12039 Dec 26 12:48
1388080111.Vfe02Ic76742M972942.oldhost.example.com:2,RS
-rw------- 1 vmail vmail 13531 Dec 26 13:14
1388081648.Vfe02Ic7674cM68343.oldhost.example.com:2,S
-rw------- 1 vmail vmail 25161 Dec 26 15:47
1388090846.Vfe02Ic768acM489505.oldhost.example.com:2,
-rw------- 1 vmail vmail 5037 Dec 26 16:27
1388093257.Vfe02Ic76918M766778.oldhost.example.com:2,
-rw------- 1 vmail vmail 3157 Dec 26 17:01
1388095312.Vfe02Ic76997M703744.oldhost.example.com:2,
-rw------- 1 vmail vmail 22480 Dec 26 17:50
1388098239.Vfe02Ic769deM225078.oldhost.example.com:2,
-rw------- 1 vmail vmail 10043 Dec 26 19:59
1388105984.Vfe02Ic76a25M12505.oldhost.example.com:2,
-rw------- 1 vmail vmail 10635 Dec 26 21:11
1388110298.Vfe02Ic76a65M649895.oldhost.example.com:2,
-rw------- 1 vmail vmail 8695 Dec 26 21:11
1388110319.Vfe02Ic76a6fM696948.oldhost.example.com:2,
-rw------- 1 vmail vmail 11045 Dec 26 21:56
1388112965.Vfe02Ic76a57M185504.oldhost.example.com:2,
-rw------- 1 vmail vmail 27647 Dec 27 02:20
1388128838.Vfe02Ic7610fM483481.oldhost.example.com:2,
This is weird, because it does show ONE new message from the 27th.
And again, ls -al on the cur dir at this point still shows ALL of the
messages (11 for the 27th, and 18 for the 26th) shown in the FIRST ls
output, ABOVE this second listing (which is shown only to show you what
Thunderbird is seeing, and what shows in the ls output AFTER doing any
of the 3 actions below).
Now...
Doing ANY of the following results in MOST (BUT NOT ALL) OF THE NEW
MESSAGES BEING DELETED ON THE SERVER - ie, ls -al output changes to the
match what is immediately above (what was showing in Thunderbird right
after selecting the Inbox right after the rsync).
1. Right-click on the Inbox > Properties > Repair Folder (in Thunderbird)
2. Right-click on INBOX > Compact (in Thunderbird)
3. Manually deleting the .msf file for the INBOX
Obviously, I must find a resolution for this before I can pull the
trigger on this migration.
************************************************************************
old server dovecon -n:
oldhost : Fri Dec 27, 05:59:12 : ~
# doveconf -n
# 2.1.17: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.7-gentoo-r1 x86_64 Gentoo Base System release 2.2
auth_default_realm = example.com
auth_mechanisms = plain login
first_valid_uid = 207
last_valid_uid = 207
listen = *
login_log_format_elements = user=<%u> method=%m rip=%r lport=%{lport}
mpid=%e %c session=<%{session}>
mail_gid = 207
mail_location = maildir:~/:INDEX=MEMORY
mail_uid = 207
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix = INBOX.
separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap lmtp
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = postfix
mode = 0600
user = postfix
}
}
service imap-login {
service_count = 0
}
service imap {
process_limit = 2048
}
ssl = required
ssl_cert = </etc/ssl/myCerts/imap.pem
ssl_key = </etc/ssl/myCerts/imap_key.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol smtp {
auth_verbose = yes
}
protocol lda {
mail_plugins = sieve
}
protocol imap {
mail_max_userip_connections = 20
}
oldhost : Fri Dec 27, 08:22:04 : ~
**********************************************************
new server dovecot -n:
host : Fri Dec 27, 08:13:34 : ~
# doveconf -n
# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.17-gentoo x86_64 Gentoo Base System release 2.2
auth_default_realm = example.com
auth_mechanisms = plain login
first_valid_uid = 200
last_valid_uid = 200
listen = *
login_log_format_elements = user=<%u> method=%m %c rip=%r lport=%{lport}
mail_gid = vmail
mail_location = maildir:~
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace inbox {
inbox = yes
location mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix separator = .
}
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
}
protocols = imap lmtp
service auth-worker {
user = $default_internal_user
}
service auth {
unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
}
unix_listener auth-userdb {
group = vmail
mode = 0600
user = vmail
}
}
service imap-login {
service_count = 0
}
service imap {
process_limit = 2048
}
ssl = required
ssl_cert = </etc/ssl/myCerts/imap.pem
ssl_key = </etc/ssl/myCerts/imap_key.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
protocol smtp {
auth_verbose = yes
}
protocol lda {
auth_verbose = yes
mail_plugins = sieve
}
protocol imap {
mail_max_userip_connections = 20
}
host : Fri Dec 27, 08:19:59 : ~
--
Best regards,
*/Charles/*
Reindl Harald
2013-Dec-27 14:12 UTC
[Dovecot] Server Migration Attempt - new messages DELETED after secondary rsyncs
Am 27.12.2013 14:41, schrieb Charles Marcus:> Thunderbird (24.2) is set up to access the new account for testing > Problem: after rsync, ls shows new messages (11 for the 27th), but when Thunderbird accesses the maildir again > after another rsync, MOST (but not all) of the new messages are deleted (only one for the 27th remains)why the hell do you do a *second* rsync? rsync in case of mailservers is for *the first one and only* 1:1 migration "man imapsync" for futher sane options using IMAP and not mangle files -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 246 bytes Desc: OpenPGP digital signature URL: <http://dovecot.org/pipermail/dovecot/attachments/20131227/c12625e4/attachment.bin>
Rick Romero
2013-Dec-27 14:28 UTC
[Dovecot] Server Migration Attempt - new messages DELETED after secondary rsyncs
Quoting Charles Marcus <CMarcus at media-brokers.com>:> Starting a new thread, as I've got a lot more details now... > > rsync command (dovecot is STOPPED before performing, source is the > latest snapshot of the active server): > > rsync -rltgovDHP --delete --delete-excluded --exclude-from > '/path/to/excludes.txt' /snapshot/.../vmail/example.com/user/ > /.../vmail/example.com/user/I have done it this way: After the 1st rsync, the client should be accessing the new location - the server should be delivering to the new location, and you do one last rsync to catch any straggling deliveries. For your issue appears on first blush that Thunderbird is just removing the email marked as Trash (T flag).> I'm only syncing the one user until I get this problem resolved as > it 1388115600.Vfe02Ib61cf5M913598.oldhost.example.com:2,ST> Start dovecot, select INBOX in Thunderbird for this account, it only > displays these (I got this ls output from AFTER the messages in the > cur directory are deleted per below - before doing any of the 3 > aqctions below, ALL of the messages show in the ls output): > ls -al /.../user/curNo ,ST files, just ,S (Seen). Rick
Possibly Parallel Threads
- Duplicated (but only for unread) messages after second rsync pass...
- [PATCH] mllib: hostname: replace the hostname on Debian also in /etc/hosts (RHBZ#953907).
- Re: [PATCH] mllib: hostname: replace the hostname on Debian also in /etc/hosts (RHBZ#953907).
- [PATCH 1/2] mllib: hostname: replace the hostname on Debian also in /etc/hosts (RHBZ#953907).
- Getting duplicated mails while using imapc and dsync