dovecot - Oct 2017 - Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)

We'll take a look


---Aki TuomiDovecot oy
-------- Original message --------From: Reuben Farrelly <reuben-dovecot at
reub.net> Date: 12/10/2017  07:50  (GMT+02:00) To: Aki Tuomi <aki.tuomi at
dovecot.fi> Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel
(eaf276b33)
Hi Aki,

I think I posted a bt full of this in the original email I sent to the list:

https://www.dovecot.org/pipermail/dovecot/2017-October/109583.html

I've got heaps of coredumps if we need to pick apart one specific one 
now though.

Reuben


On 12/10/2017 3:36 PM, Aki Tuomi wrote:
> can you provide bt full output from core?
> 
> 
> 
> ---
> Aki Tuomi
> Dovecot oy
> 
> -------- Original message --------
> From: Reuben Farrelly <reuben-dovecot at reub.net>
> Date: 12/10/2017 06:22 (GMT+02:00)
> To: Aki Tuomi <aki.tuomi at dovecot.fi>, dovecot at dovecot.org
> Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)
> 
> Hi,
> 
> Just to confirm this is still a major problem for me with the 2.2.33.1 
> release (did not exist in previous releases).
> 
> Reuben
> 
> 
> On 8/10/2017 6:30 AM, Aki Tuomi wrote:
>> Thanks! We'll look into it.
>>
>>
>>
>> ---
>> Aki Tuomi
>> Dovecot oy
>>
>> -------- Original message --------
>> From: Reuben Farrelly <reuben-dovecot at reub.net>
>> Date: 07/10/2017 13:08 (GMT+02:00)
>> To: dovecot at dovecot.org
>> Subject: Core Dumps with dovecot-2.2 -git as of 2.2.devel (eaf276b33)
>>
>> This appears to be breaking dsync at the moment and is caused by
>> something committed in the last week or so.? This is with the current
>> -git of dovecot-2.2 (eaf276b33).
>>
>> The crash happens all the time - the logs are full of this message.
>>
>> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Warning: I/O
>> leak: 0x7f8bb6581b4c (ostream-file.c:350, fd 14)
>> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Panic: file
>> ioloop.c: line 126 (io_remove_full): assertion failed: (io->callback
!>> NULL)
>> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Error: Raw
>> backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xd48a6)
[0x7f8bb654a8a6]
>> -> /usr/lib64/dovecot/li
>> bdovecot.so.0(+0xd5e26) [0x7f8bb654be26] ->
>> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] ->
>> /usr/lib64/dovecot/libdovecot.so.0(+0xf5cb0) [0x7f8bb6
>> 56bcb0] -> /usr/lib64/dovecot/libdovecot.so.0(io_remove+0x1d)
>> [0x7f8bb656be31] -> /usr/lib64/dovecot/libdovecot.so.0(+0x10b5f8)
>> [0x7f8bb65815f8] -> /usr/lib64/dovec
>> ot/libdovecot.so.0(o_stream_cork+0x5a) [0x7f8bb657e289] ->
>> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x6406) [0x7f8bb12d6406]
>> -> /usr/lib64/dovecot/libssl_iost
>> ream_openssl.so(openssl_iostream_bio_sync+0x18) [0x7f8bb12d6af6] ->
>> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa52e) 
>> [0x7f8bb12da52e] -> /
>> usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa6ef) [0x7f8bb12da6ef]
>> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7)
>> [0x7f8bb657e765] ->
>> /usr/lib64/dovecot/libdovecot.so.0(o_stream_send+0x61) [0x7f8bb657e688]
>> -> /usr/lib64/dovecot/libdovecot.so.0(+0x10dc2c) [0x7f8bb6583c2c]
->
>> /usr/lib64/dovecot/libdovecot.so.0(+0x10ddd4) [0x7f8bb6583dd4] ->
>> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7)
[0x7f8bb657e765]
>> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsendv+0x55)
>> [0x7f8bb657e93f] ->
>> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsend+0x61)
[0x7f8bb657e8d3]
>> -> dovecot/doveadm-server() [0x444a35] ->
>> /usr/lib64/dovecot/libdovecot.so.0(i_warning+0xce) [0x7f8bb654afdc]
->
>> /usr/lib64/dovecot/libdovecot.so.0(io_loop_destroy+0x2ca)
>> [0x7f8bb656d8cd] -> dovecot/doveadm-server() [0x4454d1] ->
>> dovecot/doveadm-server() [0x445793] -> dovecot/doveadm-server()
>> [0x445dd6] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0xd6)
>> [0x7f8bb656d241] ->
>> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x225)
>> [0x7f8bb656fe58] ->
>> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3e)
>> [0x7f8bb656d422] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0xbe)
>> [0x7f8bb656d345]
>> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Fatal: master:
>> service(doveadm): child 27186 killed with signal 6 (core dumped)
>>
>>
>> thunderstorm reuben # file core
>> core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style,
>> from 'dovecot/doveadm-server', real uid: 0, effective uid: 0,
real gid:
>> 0, effective gid: 0, execfn:
'/usr/libexec/dovecot/doveadm-server',
>> platform: 'x86_64'
>>
>> thunderstorm reuben # gdb /usr/libexec/dovecot/doveadm-server core
>> GNU gdb (Gentoo 8.0.1 vanilla) 8.0.1
>> Copyright (C) 2017 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.? Type "show
copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-pc-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <https://bugs.gentoo.org/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to
"word"...
>> Reading symbols from /usr/libexec/dovecot/doveadm-server...done.
>> [New LWP 27186]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `dovecot/doveadm-server'.
>> Program terminated with signal SIGABRT, Aborted.
>> #0? 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6
>> (gdb) bt full
>> #0? 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6
>> No symbol table info available.
>> #1? 0x00007f8bb60e9aa7 in abort () from /lib64/libc.so.6
>> No symbol table info available.
>> #2? 0x00007f8bb654a8f5 in default_fatal_finish (type=LOG_TYPE_PANIC,
>> status=0) at failures.c:201
>> ???????? backtrace = 0x228e4d8
>> "/usr/lib64/dovecot/libdovecot.so.0(+0xd48a6) [0x7f8bb654a8a6]
->
>> /usr/lib64/dovecot/libdovecot.so.0(+0xd5e26) [0x7f8bb654be26] ->
>> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39] ->
/usr"...
>> #3? 0x00007f8bb654be26 in i_internal_fatal_handler (ctx=0x7ffef81daa50,
>> format=0x7f8bb65ba898 "file %s: line %d (%s): assertion failed:
(%s)",
>> args=0x7ffef81daa70)
>> ???? at failures.c:670
>> ???????? status = 0
>> #4? 0x00007f8bb654ac39 in i_panic (format=0x7f8bb65ba898 "file %s:
line
>> %d (%s): assertion failed: (%s)") at failures.c:275
>> ???????? ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp =
0x0,
>> timestamp_usecs = 0}
>> ???????? args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area
>> 0x7ffef81dab50, reg_save_area = 0x7ffef81daa90}}
>

Are you using the xi packages? If yes, can you please send me one fresh core?

Aki
> On October 12, 2017 at 8:43 AM Aki Tuomi <aki.tuomi at dovecot.fi>
wrote:
> 
> 
> We'll take a look
> 
> 
> ---Aki TuomiDovecot oy
> -------- Original message --------From: Reuben Farrelly <reuben-dovecot
at reub.net> Date: 12/10/2017  07:50  (GMT+02:00) To: Aki Tuomi <aki.tuomi
at dovecot.fi> Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel
(eaf276b33)
> Hi Aki,
> 
> I think I posted a bt full of this in the original email I sent to the
list:
> 
> https://www.dovecot.org/pipermail/dovecot/2017-October/109583.html
> 
> I've got heaps of coredumps if we need to pick apart one specific one 
> now though.
> 
> Reuben
> 
> 
> On 12/10/2017 3:36 PM, Aki Tuomi wrote:
> > can you provide bt full output from core?
> > 
> > 
> > 
> > ---
> > Aki Tuomi
> > Dovecot oy
> > 
> > -------- Original message --------
> > From: Reuben Farrelly <reuben-dovecot at reub.net>
> > Date: 12/10/2017 06:22 (GMT+02:00)
> > To: Aki Tuomi <aki.tuomi at dovecot.fi>, dovecot at dovecot.org
> > Subject: Re: Core Dumps with dovecot-2.2 -git as of 2.2.devel
(eaf276b33)
> > 
> > Hi,
> > 
> > Just to confirm this is still a major problem for me with the 2.2.33.1
> > release (did not exist in previous releases).
> > 
> > Reuben
> > 
> > 
> > On 8/10/2017 6:30 AM, Aki Tuomi wrote:
> >> Thanks! We'll look into it.
> >>
> >>
> >>
> >> ---
> >> Aki Tuomi
> >> Dovecot oy
> >>
> >> -------- Original message --------
> >> From: Reuben Farrelly <reuben-dovecot at reub.net>
> >> Date: 07/10/2017 13:08 (GMT+02:00)
> >> To: dovecot at dovecot.org
> >> Subject: Core Dumps with dovecot-2.2 -git as of 2.2.devel
(eaf276b33)
> >>
> >> This appears to be breaking dsync at the moment and is caused by
> >> something committed in the last week or so.? This is with the
current
> >> -git of dovecot-2.2 (eaf276b33).
> >>
> >> The crash happens all the time - the logs are full of this
message.
> >>
> >> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Warning:
I/O
> >> leak: 0x7f8bb6581b4c (ostream-file.c:350, fd 14)
> >> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Panic:
file
> >> ioloop.c: line 126 (io_remove_full): assertion failed:
(io->callback !> >> NULL)
> >> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Error: Raw
> >> backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xd48a6)
[0x7f8bb654a8a6]
> >> -> /usr/lib64/dovecot/li
> >> bdovecot.so.0(+0xd5e26) [0x7f8bb654be26] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39]
->
> >> /usr/lib64/dovecot/libdovecot.so.0(+0xf5cb0) [0x7f8bb6
> >> 56bcb0] -> /usr/lib64/dovecot/libdovecot.so.0(io_remove+0x1d)
> >> [0x7f8bb656be31] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x10b5f8)
> >> [0x7f8bb65815f8] -> /usr/lib64/dovec
> >> ot/libdovecot.so.0(o_stream_cork+0x5a) [0x7f8bb657e289] ->
> >> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0x6406)
[0x7f8bb12d6406]
> >> -> /usr/lib64/dovecot/libssl_iost
> >> ream_openssl.so(openssl_iostream_bio_sync+0x18) [0x7f8bb12d6af6]
->
> >> /usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa52e) 
> >> [0x7f8bb12da52e] -> /
> >> usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa6ef)
[0x7f8bb12da6ef]
> >> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7)
> >> [0x7f8bb657e765] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_send+0x61)
[0x7f8bb657e688]
> >> -> /usr/lib64/dovecot/libdovecot.so.0(+0x10dc2c)
[0x7f8bb6583c2c] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(+0x10ddd4) [0x7f8bb6583dd4]
->
> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0xc7)
[0x7f8bb657e765]
> >> -> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsendv+0x55)
> >> [0x7f8bb657e93f] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(o_stream_nsend+0x61)
[0x7f8bb657e8d3]
> >> -> dovecot/doveadm-server() [0x444a35] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(i_warning+0xce)
[0x7f8bb654afdc] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(io_loop_destroy+0x2ca)
> >> [0x7f8bb656d8cd] -> dovecot/doveadm-server() [0x4454d1] ->
> >> dovecot/doveadm-server() [0x445793] -> dovecot/doveadm-server()
> >> [0x445dd6] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0xd6)
> >> [0x7f8bb656d241] ->
> >>
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x225)
> >> [0x7f8bb656fe58] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3e)
> >> [0x7f8bb656d422] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0xbe)
> >> [0x7f8bb656d345]
> >> Oct? 7 20:31:24 thunderstorm.reub.net dovecot: doveadm: Fatal:
master:
> >> service(doveadm): child 27186 killed with signal 6 (core dumped)
> >>
> >>
> >> thunderstorm reuben # file core
> >> core: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
SVR4-style,
> >> from 'dovecot/doveadm-server', real uid: 0, effective uid:
0, real gid:
> >> 0, effective gid: 0, execfn:
'/usr/libexec/dovecot/doveadm-server',
> >> platform: 'x86_64'
> >>
> >> thunderstorm reuben # gdb /usr/libexec/dovecot/doveadm-server core
> >> GNU gdb (Gentoo 8.0.1 vanilla) 8.0.1
> >> Copyright (C) 2017 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later
> >> <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.? Type
"show copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "x86_64-pc-linux-gnu".
> >> Type "show configuration" for configuration details.
> >> For bug reporting instructions, please see:
> >> <https://bugs.gentoo.org/>.
> >> Find the GDB manual and other documentation resources online at:
> >> <http://www.gnu.org/software/gdb/documentation/>.
> >> For help, type "help".
> >> Type "apropos word" to search for commands related to
"word"...
> >> Reading symbols from /usr/libexec/dovecot/doveadm-server...done.
> >> [New LWP 27186]
> >> [Thread debugging using libthread_db enabled]
> >> Using host libthread_db library
"/lib64/libthread_db.so.1".
> >> Core was generated by `dovecot/doveadm-server'.
> >> Program terminated with signal SIGABRT, Aborted.
> >> #0? 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6
> >> (gdb) bt full
> >> #0? 0x00007f8bb60e7f70 in raise () from /lib64/libc.so.6
> >> No symbol table info available.
> >> #1? 0x00007f8bb60e9aa7 in abort () from /lib64/libc.so.6
> >> No symbol table info available.
> >> #2? 0x00007f8bb654a8f5 in default_fatal_finish
(type=LOG_TYPE_PANIC,
> >> status=0) at failures.c:201
> >> ???????? backtrace = 0x228e4d8
> >> "/usr/lib64/dovecot/libdovecot.so.0(+0xd48a6)
[0x7f8bb654a8a6] ->
> >> /usr/lib64/dovecot/libdovecot.so.0(+0xd5e26) [0x7f8bb654be26]
->
> >> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f8bb654ac39]
-> /usr"...
> >> #3? 0x00007f8bb654be26 in i_internal_fatal_handler
(ctx=0x7ffef81daa50,
> >> format=0x7f8bb65ba898 "file %s: line %d (%s): assertion
failed: (%s)",
> >> args=0x7ffef81daa70)
> >> ???? at failures.c:670
> >> ???????? status = 0
> >> #4? 0x00007f8bb654ac39 in i_panic (format=0x7f8bb65ba898
"file %s: line
> >> %d (%s): assertion failed: (%s)") at failures.c:275
> >> ???????? ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp
= 0x0,
> >> timestamp_usecs = 0}
> >> ???????? args = {{gp_offset = 40, fp_offset = 48,
overflow_arg_area > >> 0x7ffef81dab50, reg_save_area = 0x7ffef81daa90}}
> >

No.  Compiling entirely from -git source (on one system) and a Gentoo 
ebuild on another two.

Reuben

On 12/10/2017 10:38 PM, Aki Tuomi wrote:
> Are you using the xi packages? If yes, can you please send me one fresh
core?
> 
> Aki
> 
>> On October 12, 2017 at 8:43 AM Aki Tuomi <aki.tuomi at
dovecot.fi> wrote:
>>
>>
>> We'll take a look
>>
>>
>> ---Aki TuomiDovecot oy
>> -------- Original message --------From: Reuben Farrelly
<reuben-dovecot at reub.net> Date: 12/10/2017  07:50  (GMT+02:00) To: Aki
Tuomi <aki.tuomi at dovecot.fi> Subject: Re: Core Dumps with dovecot-2.2
-git as of 2.2.devel (eaf276b33)
>> Hi Aki,
>>
>> I think I posted a bt full of this in the original email I sent to the
list:
>>
>> https://www.dovecot.org/pipermail/dovecot/2017-October/109583.html
>>
>> I've got heaps of coredumps if we need to pick apart one specific
one
>> now though.
>>
>> Reuben