thr3ads.net - dovecot - Assertion during dsync receive [Feb 2018]

If this information is useful, please help other people find it:
Share via:

Ian Bobbitt

2018-Feb-23 18:33 UTC

Assertion during dsync receive

Thanks. I've had the user clear out that mailbox, and replication is working
fine for them again.

Is there a better way to catch this than watch for crashes and read the
backtrace to find what mailbox needs to be shrunk?

Where is the threshold for "too big"?

-- Ian

On 2/23/18 11:33 AM, Aki Tuomi wrote:> The mailbox is too big.
>
>
>
> ---
> Aki Tuomi
> Dovecot oy
>
> -------- Original message --------
> From: Ian Bobbitt <ibobbitt at globalnoc.iu.edu>
> Date: 23/02/2018 17:52 (GMT+02:00)
> To: dovecot at dovecot.org
> Subject: Assertion during dsync receive
>
> Hi,
>
> I'm getting an assertion failed on the receiving side, causing syncs to
fail for one user. The servers are setup so that
> only one is receiving any traffic other than replication at any time. The
one that's only receiving replications is the
> one that's failing.
>
> I've tried deleting the user's home on the receiving server, but it
still crashes during the sync. Oddly, the user's
> home is 7.4G on the sending server, but ends up at 42G on the receiving
side, even after deleting and trying a fresh sync.
>
> The mailbox implicated in the backtrace ("Spam") does have a very
large number of messages in it.
> On sender:
> Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105
unseen=16 highestmodseq=744588 vsize=34468460093
> guid=090ed93a7a055559abf10200fdf6807a firstsaved=1498744186
> On receiver:
> Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105
unseen=16 highestmodseq=744589 vsize=34468496809
> guid=090ed93a7a055559abf10200fdf6807a firstsaved=1519396172
>
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file
> /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
/gnoc/mail/home/bgeels/mail/storage: rebuilding indexes
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file
mail-index-util.c: line 10 (mail_index_uint32_to_offset):
> assertion failed: (offset < 0x40000000)
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)
> [0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be)
[0x7feb584144be] ->
> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] ->
>
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0)
[0x7feb587906d0] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854)
[0x7feb58774f34] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f)
[0x7feb587884ff] ->
>
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
[0x7feb5870b3ae] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
[0x7feb5870ccd8] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c)
[0x7feb5870ce7c] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b)
[0x7feb5870cf3b] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
[0x7feb586f2834] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
[0x7feb586f28d7] ->
> dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] ->
dovecot/doveadm-server() [0x43edc0] ->
> dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] ->
dovecot/doveadm-server(dsync_brain_run+0x541)
> [0x43acf1] -> dovecot/doveadm-server() [0x43b070] ->
dovecot/doveadm-server() [0x44fe5f] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2]
->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
[0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
[0x7feb58429d6c] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] ->
dovecot/doveadm-server() [0x4209c5] ->
> dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server()
[0x4377f4] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2]
->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
[0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
[0x7feb58429d6c]
> Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child
82098 killed with signal 6 (core dumped)
>
> I've attached the output of `doveconf -n` and the full backtrace from a
core dump.
>
> Dovecot 2.2.33.2 (GhettoForge package)
> CentOS 7 x86_64
> XFS, no NFS.
>
>
> -- Ian
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180223/98231781/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4090 bytes
Desc: S/MIME Cryptographic Signature
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180223/98231781/attachment-0001.p7s>

Aki Tuomi

2018-Feb-23 18:53 UTC

head link

Assertion during dsync receive

Once you cache grows bigger than 0x4000000 you have problems


---Aki TuomiDovecot oy
-------- Original message --------From: Ian Bobbitt <ibobbitt at
globalnoc.iu.edu> Date: 23/02/2018  20:33  (GMT+02:00) To: dovecot at
dovecot.org Subject: Re: Assertion during dsync receive

    Thanks. I've had the user clear out that mailbox, and replication is
    working fine for them again.

    

    Is there a better way to catch this than watch for crashes and read
    the backtrace to find what mailbox needs to be shrunk?

    

    Where is the threshold for "too big"?

    

    -- Ian

    

    On 2/23/18 11:33 AM, Aki Tuomi wrote:

    
      
      The mailbox is too big.
      

      
      

      
      

      
      
        
        ---
        Aki Tuomi
        Dovecot oy
      
      

      
      
        -------- Original message --------
        From: Ian Bobbitt <ibobbitt at globalnoc.iu.edu> 
        Date: 23/02/2018 17:52 (GMT+02:00) 
        To: dovecot at dovecot.org 
        Subject: Assertion during dsync receive 
        

        
      
      Hi,

      

      I'm getting an assertion failed on the receiving side, causing
      syncs to fail for one user. The servers are setup so that

      only one is receiving any traffic other than replication at any
      time. The one that's only receiving replications is the

      one that's failing.

      

      I've tried deleting the user's home on the receiving server, but
      it still crashes during the sync. Oddly, the user's

      home is 7.4G on the sending server, but ends up at 42G on the
      receiving side, even after deleting and trying a fresh sync.

      

      The mailbox implicated in the backtrace ("Spam") does have a
very
      large number of messages in it.

      On sender:

      Spam messages=1217764 recent=0 uidnext=1218103
      uidvalidity=1379509105 unseen=16 highestmodseq=744588
      vsize=34468460093

      guid=090ed93a7a055559abf10200fdf6807a firstsaved=1498744186

      On receiver:

      Spam messages=1217766 recent=352 uidnext=1218105
      uidvalidity=1379509105 unseen=16 highestmodseq=744589
      vsize=34468496809

      guid=090ed93a7a055559abf10200fdf6807a firstsaved=1519396172

      

      Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking
      index file

      /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index

      Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
      /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes

      Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file
      mail-index-util.c: line 10 (mail_index_uint32_to_offset):

      assertion failed: (offset < 0x40000000)

      Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw
      backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)

      [0x7feb584143de] ->
      /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) [0x7feb584144be]
      ->

      /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c]
      ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0)
      [0x7feb587906d0] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854)
      [0x7feb58774f34] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f)
      [0x7feb587884ff] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
      [0x7feb5870b3ae] ->

      /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
      [0x7feb5870ccd8] ->

      /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c)
      [0x7feb5870ce7c] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b)
      [0x7feb5870cf3b] ->

      /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
      [0x7feb586f2834] ->

      /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
      [0x7feb586f28d7] ->

      dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475)
      [0x445495] -> dovecot/doveadm-server() [0x43edc0] ->

      dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653]
      -> dovecot/doveadm-server(dsync_brain_run+0x541)

      [0x43acf1] -> dovecot/doveadm-server() [0x43b070] ->
      dovecot/doveadm-server() [0x44fe5f] ->

      /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
      [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
      [0x7feb5842b3bf] ->

      /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
      [0x7feb58429d6c] ->

      /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
      [0x7feb58429f28] -> dovecot/doveadm-server() [0x4209c5] ->

      dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server()
      [0x4377f4] ->

      /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
      [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
      [0x7feb5842b3bf] ->

      /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
      [0x7feb58429d6c]

      Feb 23 14:57:33 dsync-local(bgeels): Fatal: master:
      service(doveadm): child 82098 killed with signal 6 (core dumped)

      

      I've attached the output of `doveconf -n` and the full backtrace
      from a core dump.

      

      Dovecot 2.2.33.2 (GhettoForge package)

      CentOS 7 x86_64

      XFS, no NFS.

      

      

      -- Ian

    
    

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180223/a9c59235/attachment.html>

Tanstaafl

2018-Feb-23 19:36 UTC

head link

Assertion during dsync receive

On Fri Feb 23 2018 13:53:27 GMT-0500 (Eastern Standard Time), Aki Tuomi
<aki.tuomi at dovecot.fi> wrote:> Once you cache grows bigger than 0x4000000 you have problems
This is for a single mailbox? IS this only a problem for mbox and maybe
sdbox?

Apparently Analagous Threads

Search for more possibly parallel threads

dovecot - Feb 2018 - Assertion during dsync receive

Assertion during dsync receive

Assertion during dsync receive

Assertion during dsync receive

Apparently Analagous Threads