Le 13/02/2015 16:19, Casey Stone a ?crit :> On Feb 5, 2015, at 10:39 PM, Casey Stone <tcstone at caseystone.com> wrote: > >> Hello: >> >> I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. >> >> I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. >> >> Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: >> >> doveadm sync -R tcps:mainserver.example.com:12345 >> >> Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. >> >> The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? >> >> Thanks for your help. > No repsonses :-( > > Here is what it looks like when it crashes with an out of memory error: > > (start of the run) > Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone > Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ > Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ > Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ > Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1 > > <<<many, many more brain messages>>> > > (end of the run) > Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory > Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] > Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump) > > I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied.To use dsync, use Dovecot 2.2.15 to avoid any trouble. Use apt.dovecot.fi repository. Emmanuel.
On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel <emmanuel.fuste at thalesgroup.com> wrote:> Le 13/02/2015 16:19, Casey Stone a ?crit : >> On Feb 5, 2015, at 10:39 PM, Casey Stone <tcstone at caseystone.com> wrote: >> >>> Hello: >>> >>> I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. >>> >>> I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. >>> >>> Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: >>> >>> doveadm sync -R tcps:mainserver.example.com:12345 >>> >>> Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. >>> >>> The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? >>> >>> Thanks for your help. >> No repsonses :-( >> >> Here is what it looks like when it crashes with an out of memory error: >> >> (start of the run) >> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone >> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ >> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ >> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ >> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1 >> >> <<<many, many more brain messages>>> >> >> (end of the run) >> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory >> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] >> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump) >> >> I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. > To use dsync, use Dovecot 2.2.15 to avoid any trouble. > Use apt.dovecot.fi repository. > > Emmanuel.At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running. I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo? By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying). Thanks.
Le 16/02/2015 20:40, Casey Stone a ?crit :> On Feb 13, 2015, at 3:42 PM, FUSTE Emmanuel <emmanuel.fuste at thalesgroup.com> wrote: > >> Le 13/02/2015 16:19, Casey Stone a ?crit : >>> On Feb 5, 2015, at 10:39 PM, Casey Stone <tcstone at caseystone.com> wrote: >>> >>>> Hello: >>>> >>>> I've been looking forward to getting my mail server up to Dovecot 2.2+ to be able to use the sync mechanism. I run my own mail server just for myself, with a few different accounts, and want to keep a master and backup server in sync. >>>> >>>> I'm running the Ubuntu server 14.04.1 mail stack which features Dovecot 2.2.9 (and Postfix). My setup is to use system users (userdb passwd / passdb pam) with ~/Maildir. I'll post full sanitized output of dovecot -n if it seems necessary. I have not enabled any plugins (do I need the replicator plugin active?) I have in my conf a doveadm_password defined. >>>> >>>> Anyway, after setting up an ssl listener on the main machine and after considerable struggles with SSL, I was able to run doveadm sync from the backup server successfully for a small mailbox (around 78 MB) with this command: >>>> >>>> doveadm sync -R tcps:mainserver.example.com:12345 >>>> >>>> Since I run this command as the system user on the backup server (same system users as main server) it 'just works' for the correct single user with no further options required. My plan is to run a daily cron job to sync once daily for each user. >>>> >>>> The problem is when I try to sync a larger mailbox, say 1 GB, dsync-server on the remote (master) machine throws fatal error 83 Out of Memory. I already raised vsz_limit to 512 MB. Problems probably arise with mailboxes around 200 MB though I haven't tested specifically. So my question is, is this expected and I will need to give my VM much more memory to be able to use dovecot sync, or do I have something set wrong, or is it a bug? >>>> >>>> Thanks for your help. >>> No repsonses :-( >>> >>> Here is what it looks like when it crashes with an out of memory error: >>> >>> (start of the run) >>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Effective uid=1002, gid=1002, home=/home/tcstone >>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: Namespace inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list$ >>> Feb 13 14:02:38 thepost dovecot: doveadm(10.0.1.22,tcstone): Debug: maildir++: root=/data/tcstone/Maildir, index=, indexpvt=, control=, inbo$ >>> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: Namespace : Using permissions from /data/tcstone/Maildir: mode=0700 gid=defau$ >>> Feb 13 14:02:39 thepost dovecot: dsync-server(tcstone): Debug: brain S: out state=send_mailbox_tree changed=1 >>> >>> <<<many, many more brain messages>>> >>> >>> (end of the run) >>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: pool_system_realloc(536870912): Out of memory >>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x5e271) [0x7f9d2056b271] -> /usr/lib/dovecot/libdovecot.so.0(+0x5e34e) [0x7f9d2056b34e] -> /usr/lib/dovecot/libdovecot.so.0(i_error+0) [0x7f9d20526bf8] -> /usr/lib/dovecot/libdovecot.so.0(+0x72d53) [0x7f9d2057fd53] -> /usr/lib/dovecot/libdovecot.so.0(+0x7792a) [0x7f9d2058492a] -> /usr/lib/dovecot/libdovecot.so.0(+0x77be6) [0x7f9d20584be6] -> /usr/lib/dovecot/libdovecot.so.0(+0x78748) [0x7f9d20585748] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_send+0x1a) [0x7f9d20583e1a] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x4c05) [0x7f9d1f6a0c05] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(openssl_iostream_bio_sync+0x21) [0x7f9d1f6a1881] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7a4d) [0x7f9d1f6a3a4d] -> /usr/lib/dovecot/modules/libssl_iostream_openssl.so(+0x7d69) [0x7f9d1f6a3d69] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_sendv+0x8d) [0x7f9d20583d7d] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsendv+0xf) [0x7f9d20583e5f] -> /usr/lib/dovecot/libdovecot.so.0(o_stream_nsend+0x1a) [0x7f9d20583e8a] -> dovecot/doveadm-server(+0x2b03f) [0x7f9d20d3003f] -> dovecot/doveadm-server(+0x2c768) [0x7f9d20d31768] -> dovecot/doveadm-server(dsync_ibc_send_mail+0x29) [0x7f9d20d2f309] -> dovecot/doveadm-server(dsync_brain_sync_mails+0x5fc) [0x7f9d20d24a1c] -> dovecot/doveadm-server(dsync_brain_run+0x523) [0x7f9d20d20f93] -> dovecot/doveadm-server(+0x1c270) [0x7f9d20d21270] -> dovecot/doveadm-server(+0x2de60) [0x7f9d20d32e60] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x27) [0x7f9d2057b247] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0xd7) [0x7f9d2057bfd7] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f9d2057ade8] -> dovecot/doveadm-server(+0x1a189) [0x7f9d20d1f189] -> dovecot/doveadm-server(+0xebeb) [0x7f9d20d13beb] >>> Feb 13 14:02:52 thepost dovecot: dsync-server(tcstone): Fatal: master: service(doveadm): child 13232 returned error 83 (Out of memory (service doveadm { vsz_limit=512 MB }, you may need to increase it) - set DEBUG_OUTOFMEM=1 environment to get core dump) >>> >>> I haven't tested whether it is simply the effect of having DEBUG active that kills it... A run requiring moving just over 100 MB of mail is enough to cause the error. I have tested manually copying the Maildir to the backup server, then running the sync or backup command -- this works. Thus is seems related not to the size of the mailbox but the size of the data that needs to be copied. >> To use dsync, use Dovecot 2.2.15 to avoid any trouble. >> Use apt.dovecot.fi repository. >> >> Emmanuel. > At first I scoffed at this notion, preferring to stick with my distributions own packaged versions, but then I realised that there is this 'enterprise version' called dovecot-ee available, and now it even is free (previously $99/year I guess). I registered for this and have a license and username/password now, but it seems Ubuntu is only supported as of 12.04 not 14.04 that I am running. > > I tried to add the repo changing 'precise' for 'trusty' but the repo did not respond to that. Anyone know if it's safe to use 'precise' on a trusty server, or if there is progress to create packages for trusty? Also, should I apt-get remove dovecot-core etc before installing from the new repo? > > By the way, the backup operations seem to be working now using doveadm backup -u username -R tcps:mainserver.example.com:12345 if I do the 'seeding' of the backup manually (by copying). > > Thanks.Yes, it is safe. You dont need to remove dovecot-core etc before installing from the new repo. Apt will do it for you automaticaly. Emmanuel.