> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote: > > > Hi Aki, > > Am 03.11.22 um 10:29 schrieb Aki Tuomi: > >> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote: > >> > >> > >> Hi Aki, > >> > >> Am 03.11.22 um 09:12 schrieb Aki Tuomi: > >>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote: > >>>> > >>>> > >>>> Hi Aki, > >>>> > >>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: > >>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote: > >>>>>> > >>>>>> > >>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using > >>>>>> doveadm backup -R, which works for all folders but the INBOX itself, > >>>>>> which always stays empty. > >>>>>> > >>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used > >>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX > >>>>>> > >>>>>> Dovecot uses the following namespace to migrate into: > >>>>>> > >>>>>> namespace inboxes { > >>>>>> ? inbox = yes > >>>>>> ? location > >>>>>> ? mailbox Sent { > >>>>>> ??? auto = subscribe > >>>>>> ??? special_use = \Sent > >>>>>> ? } > >>>>>> ? ### some more folders omitted ### > >>>>>> ? prefix = INBOX/ > >>>>>> ? separator = / > >>>>>> ? subscriptions = no > >>>>>> } > >>>>> Hi! > >>>>> > >>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. > >>>>> > >>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. > >>>> Does a migration config file specified with doveadm -c <file> add to and > >>>> overwrite the existing Dovecot configuration for the time the command > >>>> runs, like the -o options, or do I need to start a separate server with > >>>> a full configuration to e.g. have my authentication and mailbox location > >>>> available? > >>>> > >>>> Ralf > >>>> > >>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. > >>> > >>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. > >> I created now a separate instance with a modified configuration file > >> with no auto=subscribe (or create), no replication and an empty storage. > >> doveadm config -n is attached. > >> > >> Unfortunately the result is identical to my previous tries: > >> > >> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o > >> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R > >> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log > >> > >> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes > >> caused a desync. You may want to run dsync again: Remote lost mailbox > >> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) > >> > >> doveadm mailbox status -u someuser at somedomain all INBOX > >> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 > >> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c > >> firstsaved=never > >> > >> Any ideas what else to try or how to debug that further? > >> > >> I can send you the full log to your personal address, if that helps ... > >> > >> Ralf > > You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords. > > The mailbox directory did NOT exist before, therefore no need to rm -rf it. > > I send the logs to your private address only, I feel not comfortable to > post them on the list. > > Ralf >1. You did not delete the mailbox. 2. You are using **mbox** for subscription namespace, please don't. Also Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0 This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again? I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system. Aki
hopefully this is NOT off topic I too migrated from cyrus imap server, it was to say at least fun you need to get the dovecot server up and running and testing with making a new account and get the folder structures working dependant on your required setups, as what seems to be indicated elsewhere in these posts (namespace, virtual users, dbpassword etc) if you intend to run replication then both servers need to be setup and running fully before migrating. Don't put that off it just adds work afterwards. at the end of the day moving from cyrus to dovecot i used imapsync which is a perl script to migrate, copy etc see : https://imapsync.lamiral.info/ the one major note is the seperator, if you used '.' on cyrus you need to convert it to '-' or something else as those folders will not migrate into a default dovecot configuration and will get skipped. ie : folders can not have a '.' in them on default dovecot. Otherwise imapsync pretty much does the trick. basically : imapsync --host1 69.49.101.233 --user1 ap at unitedelevatorltd.com --password1 Password \ --host2 mail18.scom.ca --user2 ap at unitedelevatorltd.com --password2 Password \ --regextrans2 "s,\.,_,g" note the regextrans2 expresion above handles the '.' to '-' there is extensive documentation on the site i have just listed what i had to use to get an account to work and move over with out any data loss. basically if you write a script to do the copy's on a users account per mailbox basis then things should migrate well. ie : get user list / passwords from old cyrus then create mailbox on new dovecot server (ie update password database etc) then run imapsync like above. repeat for every user you want to migrate. I did not have any issues with the INBOX but after a month of testing found this was the best way to go. Happy Thursday !!! Thanks - paul Paul Kudla Scom.ca Internet Services <http://www.scom.ca> 004-1009 Byron Street South Whitby, Ontario - Canada L1N 4S3 Toronto 416.642.7266 Main?1.866.411.7266 Fax?1.888.892.7266 Email?paul at scom.ca On 11/3/2022 5:54 AM, Aki Tuomi wrote:> > >> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote: >> >> >> Hi Aki, >> >> Am 03.11.22 um 10:29 schrieb Aki Tuomi: >>>> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote: >>>> >>>> >>>> Hi Aki, >>>> >>>> Am 03.11.22 um 09:12 schrieb Aki Tuomi: >>>>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote: >>>>>> >>>>>> >>>>>> Hi Aki, >>>>>> >>>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote: >>>>>>>> >>>>>>>> >>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>>>>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>>>>>> which always stays empty. >>>>>>>> >>>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>>>>>> >>>>>>>> Dovecot uses the following namespace to migrate into: >>>>>>>> >>>>>>>> namespace inboxes { >>>>>>>> ? inbox = yes >>>>>>>> ? location >>>>>>>> ? mailbox Sent { >>>>>>>> ??? auto = subscribe >>>>>>>> ??? special_use = \Sent >>>>>>>> ? } >>>>>>>> ? ### some more folders omitted ### >>>>>>>> ? prefix = INBOX/ >>>>>>>> ? separator = / >>>>>>>> ? subscriptions = no >>>>>>>> } >>>>>>> Hi! >>>>>>> >>>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>>>>>> >>>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >>>>>> Does a migration config file specified with doveadm -c <file> add to and >>>>>> overwrite the existing Dovecot configuration for the time the command >>>>>> runs, like the -o options, or do I need to start a separate server with >>>>>> a full configuration to e.g. have my authentication and mailbox location >>>>>> available? >>>>>> >>>>>> Ralf >>>>>> >>>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. >>>>> >>>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. >>>> I created now a separate instance with a modified configuration file >>>> with no auto=subscribe (or create), no replication and an empty storage. >>>> doveadm config -n is attached. >>>> >>>> Unfortunately the result is identical to my previous tries: >>>> >>>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o >>>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R >>>> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log >>>> >>>> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes >>>> caused a desync. You may want to run dsync again: Remote lost mailbox >>>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) >>>> >>>> doveadm mailbox status -u someuser at somedomain all INBOX >>>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 >>>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c >>>> firstsaved=never >>>> >>>> Any ideas what else to try or how to debug that further? >>>> >>>> I can send you the full log to your personal address, if that helps ... >>>> >>>> Ralf >>> You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords. >> >> The mailbox directory did NOT exist before, therefore no need to rm -rf it. >> >> I send the logs to your private address only, I feel not comfortable to >> post them on the list. >> >> Ralf >> > > 1. You did not delete the mailbox. > > 2. You are using **mbox** for subscription namespace, please don't. > > Also > > Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 > Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0 > > This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again? > > I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system. > > Aki >
Hi Aki, Am 03.11.22 um 10:54 schrieb Aki Tuomi:>> On 03/11/2022 11:46 EET Ralf Becker <rb at egroupware.org> wrote: >> >> >> Hi Aki, >> >> Am 03.11.22 um 10:29 schrieb Aki Tuomi: >>>> On 03/11/2022 11:27 EET Ralf Becker <rb at egroupware.org> wrote: >>>> >>>> >>>> Hi Aki, >>>> >>>> Am 03.11.22 um 09:12 schrieb Aki Tuomi: >>>>>> On 03/11/2022 10:09 EET Ralf Becker <rb at egroupware.org> wrote: >>>>>> >>>>>> >>>>>> Hi Aki, >>>>>> >>>>>> Am 03.11.22 um 08:50 schrieb Aki Tuomi: >>>>>>>> On 03/11/2022 09:46 EET Ralf Becker <rb at egroupware.org> wrote: >>>>>>>> >>>>>>>> >>>>>>>> I'm trying to migrate an old Cyrus 2.5 server to Dovecot 2.3.19 using >>>>>>>> doveadm backup -R, which works for all folders but the INBOX itself, >>>>>>>> which always stays empty. >>>>>>>> >>>>>>>> The Cyrus side uses altnamespace:no and unixhierarchysep:no, it's used >>>>>>>> as imapc: remote in doveadm backup -R with imapc_list_prefix=INBOX >>>>>>>> >>>>>>>> Dovecot uses the following namespace to migrate into: >>>>>>>> >>>>>>>> namespace inboxes { >>>>>>>> ? inbox = yes >>>>>>>> ? location >>>>>>>> ? mailbox Sent { >>>>>>>> ??? auto = subscribe >>>>>>>> ??? special_use = \Sent >>>>>>>> ? } >>>>>>>> ? ### some more folders omitted ### >>>>>>>> ? prefix = INBOX/ >>>>>>>> ? separator = / >>>>>>>> ? subscriptions = no >>>>>>>> } >>>>>>> Hi! >>>>>>> >>>>>>> When syncing mailboxes from other server, you should use migration config file, which has **no** auto=subscribe or auto=create folders, as these can mess up with synchronization. >>>>>>> >>>>>>> Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/ for more details. >>>>>> Does a migration config file specified with doveadm -c <file> add to and >>>>>> overwrite the existing Dovecot configuration for the time the command >>>>>> runs, like the -o options, or do I need to start a separate server with >>>>>> a full configuration to e.g. have my authentication and mailbox location >>>>>> available? >>>>>> >>>>>> Ralf >>>>>> >>>>> It does not add/replace/overwrite configuration, you provide a fresh config file which is used *instead of* the default dovecot.conf. >>>>> >>>>> You don't need to run a separate instance necessarely, although in some larger migrations this has been used as well. >>>> I created now a separate instance with a modified configuration file >>>> with no auto=subscribe (or create), no replication and an empty storage. >>>> doveadm config -n is attached. >>>> >>>> Unfortunately the result is identical to my previous tries: >>>> >>>> doveadm -o namespace/subs/location=mbox:/var/dovecot/subs -o >>>> imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R >>>> -u someuser at somedomain imapc: 2>&1 | tee /tmp/doveadm-backup.log >>>> >>>> Nov 03 09:06:35 dsync(someuser at somedomain): Warning: Mailbox changes >>>> caused a desync. You may want to run dsync again: Remote lost mailbox >>>> GUID c92f64f79f0d1ed01e6d5b314f04886c (maybe it was just deleted?) >>>> >>>> doveadm mailbox status -u someuser at somedomain all INBOX >>>> INBOX messages=0 recent=0 uidnext=1 uidvalidity=1577952633 unseen=0 >>>> highestmodseq=1 vsize=0 guid=c92f64f79f0d1ed01e6d5b314f04886c >>>> firstsaved=never >>>> >>>> Any ideas what else to try or how to debug that further? >>>> >>>> I can send you the full log to your personal address, if that helps ... >>>> >>>> Ralf >>> You should rm -rf the target folder first. Can you attach `doveadm -D backup` logs? Check that it won't contain passwords. >> The mailbox directory did NOT exist before, therefore no need to rm -rf it. >> >> I send the logs to your private address only, I feel not comfortable to >> post them on the list. >> >> Ralf >> > 1. You did not delete the mailbox. > > 2. You are using **mbox** for subscription namespace, please don't. > > Also > > Nov 03 09:05:33 dsync(): Debug: brain M: Local mailbox tree: INBOX guid=c8adef115c84636335000000effb6190 uid_validity=1667466332 uid_next=1 subs=no last_change=0 last_subs=0 > Nov 03 09:05:33 dsync(): Debug: brain S: Local mailbox tree: INBOX guid=c92f64f79f0d1ed01e6d5b314f04886c uid_validity=1577952633 uid_next=32746 subs=no last_change=0 last_subs=0 > > This clearly shows that you did not, in fact, rm -rf the user's mailboxes prir running backup. Can you please try again, and clean up the user from t target mailsystem before running backup again? > > I would also strongly recommend not using ACL plugin while doing backup, unless you are backing up ACLs from source system.I removed all ACL plugin and config from my Dovecot config and removed the subscription namespace from my doveadm backup command, but now it fails with an error: doveadm -o imapc_user='someuser' -o imapc_password='secret' -D backup -n INBOX/ -R -u someuser at somedomain imapc: Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Effective uid=90, gid=101, home=/var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser at somedomain 156): Debug: Home dir not found: /var/dovecot/imap/somedomain/someuser Nov 03 11:12:15 doveadm(someuser at somedomain): Error: namespace configuration error: subscriptions=yes namespace missing doveadm config -n is attached. Ralf -- Ralf Becker EGroupware GmbH [www.egroupware.org] Handelsregister HRB Kaiserslautern 3587 Gesch?ftsf?hrer Birgit und Ralf Becker Leibnizstr. 17, 67663 Kaiserslautern, Germany Telefon +49 631 31657-0 -------------- next part -------------- # 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf # Pigeonhole version 0.5.19 (4eae2f79) # OS: Linux 4.15.0-140-generic x86_64 # Hostname: 6cc0d9aef9ad auth_cache_negative_ttl = 2 mins auth_cache_size = 10 M auth_cache_ttl = 5 mins auth_master_user_separator = * auth_mechanisms = plain login auth_username_chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@#" default_client_limit = 3500 default_process_limit = 512 disable_plaintext_auth = no doveadm_port = 26 first_valid_uid = 90 imapc_features = rfc822.size fetch-headers imapc_host = 10.44.88.3 imapc_list_prefix = INBOX listen = * log_path = /dev/stderr login_greeting = Dovecot FRA.khs ready mail_access_groups = dovecot mail_attribute_dict = file:%h/dovecot-metadata mail_fsync = never mail_gid = dovecot mail_location = mdbox:~/mdbox mail_log_prefix = "%s(%u %p): " mail_max_userip_connections = 200 mail_plugins = quota notify replication mail_log mail_prefetch_count = 20 mail_uid = dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave vnd.dovecot.debug mbox_min_index_size = 1000 B mdbox_rotate_size = 50 M namespace inboxes { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = INBOX/ separator = / subscriptions = no } passdb { args = /etc/dovecot/dovecot-dict-master-auth.conf driver = dict master = yes } passdb { args = /etc/dovecot/dovecot-dict-auth.conf driver = dict } plugin { mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = uid box msgid size quota = dict:User quota::ns=INBOX/:file:%h/dovecot-quota quota_rule = *:storage=200GB sieve = ~/sieve/dovecot.sieve sieve_after = /var/dovecot/sieve/after.d/ sieve_before = /var/dovecot/sieve/before.d/ sieve_dir = ~/sieve sieve_extensions = +editheader sieve_user_log = ~/.sieve.log } postmaster_address = admins at egroupware.org protocols = imap pop3 lmtp sieve quota_full_tempfail = yes service auth-worker { user = $default_internal_user } service auth { drop_priv_before_exec = no inet_listener { port = 113 } } service doveadm { inet_listener { port = 26 } vsz_limit = 640 M } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } process_min_avail = 5 service_count = 1 vsz_limit = 64 M } service imap { executable = imap process_limit = 2048 vsz_limit = 640 M } service lmtp { inet_listener lmtp { port = 24 } unix_listener lmtp { mode = 0666 } vsz_limit = 512 M } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service pop3 { executable = pop3 } service postlogin { executable = script-login -d rawlog -b -t } ssl_cert = </etc/certs/mail.egroupware.org.pem ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-sql.conf driver = sql } userdb { args = /etc/dovecot/dovecot-dict-auth.conf driver = dict } verbose_proctitle = yes protocol lda { mail_plugins = quota notify replication mail_log sieve quota } protocol imap { imap_metadata = yes mail_max_userip_connections = 200 mail_plugins = quota notify replication mail_log quota imap_quota } protocol lmtp { mail_max_lock_timeout = 25 secs mail_plugins = quota notify replication mail_log sieve quota }