thr3ads.net - dovecot - Auth process sometimes stop responding after upgrade [Sep 2018]

If this information is useful, please help other people find it:
Share via:

Simone Lazzaris

2018-Sep-18 12:15 UTC

Auth process sometimes stop responding after upgrade

In data marted? 18 settembre 2018 14:07:26 CEST, Aki Tuomi ha
scritto:> If you are using systemd, create
> /etc/systemd/system/dovecot.service.d/limits.conf and put
> [Service]LimitCORE=infinity
> 
> and run 
> systemctl daemon-reloadsystemctl restart dovecot
Nope, I'm on a debian 7, without systemd. Anyway, I've resolved the
issue: I had to set
fs.suid_dumpable BEFORE starting dovecot.

I've got a core dump, and here is the backtrace. Let me know if you want the
core file.

gdb) bt
#0  0xb76e4428 in __kernel_vsyscall ()
#1  0xb74636c1 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2  0xb7466af2 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3  0xb76485ae in default_fatal_finish (type=<optimized out>,
status=status at entry=0) at
failures.c:201
#4  0xb7648641 in i_internal_fatal_handler (ctx=0xbf839cc0, format=0x805c274
"file %s:
line %d (%s): assertion failed: (%s)", args=0xbf839ce4
"4\370\005\bI\004") at failures.c:670
#5  0xb75ce35e in i_panic (format=format at entry=0x805c274 "file %s: line
%d (%s):
assertion failed: (%s)") at failures.c:275
#6  0x080574f7 in doveadm_connection_deinit (_conn=_conn at entry=0xbf839d60) at
doveadm-connection.c:1097
#7  0x08057f03 in doveadm_connection_input (conn=0x0) at
doveadm-connection.c:1051
#8  0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600
#9  0xb7662e1e in io_loop_handler_run_internal (ioloop=ioloop at
entry=0x82bd648) at
ioloop-epoll.c:223
#10 0xb7661496 in io_loop_handler_run (ioloop=ioloop at entry=0x82bd648) at
ioloop.c:649
#11 0xb7661658 in io_loop_run (ioloop=0x82bd648) at ioloop.c:624
#12 0xb75da45e in master_service_run (service=0x82bd578, 
callback=callback at entry=0x804d360 <client_connected>) at
master-service.c:719
#13 0x0804cf5e in main (argc=1, argv=0x82bd300) at main.c:366



*Simone Lazzaris*
*Qcom S.p.A.*
simone.lazzaris at qcom.it[1] | www.qcom.it[2]
* LinkedIn[3]* | *Facebook*[4]
[5] 






--------
[1] mailto:simone.lazzaris at qcom.it
[2] https://www.qcom.it
[3] https://www.linkedin.com/company/qcom-spa
[4] http://www.facebook.com/qcomspa
[5] https://www.qcom.it/includes/email-banner.gif
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180918/ddd391d1/attachment.html>

Aki Tuomi

2018-Sep-18 12:25 UTC

head link

Auth process sometimes stop responding after upgrade

Can you provide 'bt full'

Aki


On 18.09.2018 15:15, Simone Lazzaris wrote:>
> In data marted? 18 settembre 2018 14:07:26 CEST, Aki Tuomi ha scritto:
>
> > If you are using systemd, create
>
> > /etc/systemd/system/dovecot.service.d/limits.conf and put
>
> > [Service]LimitCORE=infinity
>
> >
>
> > and run?
>
> > systemctl daemon-reloadsystemctl restart dovecot
>
> ?
>
> Nope, I'm on a debian 7, without systemd. Anyway, I've resolved the
> issue: I had to set fs.suid_dumpable BEFORE starting dovecot.
>
> ?
>
> I've got a core dump, and here is the backtrace. Let me know if you
> want the core file.
>
> ?
>
> gdb) bt
>
> #0 0xb76e4428 in __kernel_vsyscall ()
>
> #1 0xb74636c1 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
>
> #2 0xb7466af2 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
>
> #3 0xb76485ae in default_fatal_finish (type=<optimized out>,
> status=status at entry=0) at failures.c:201
>
> #4 0xb7648641 in i_internal_fatal_handler (ctx=0xbf839cc0,
> format=0x805c274 "file %s: line %d (%s): assertion failed: (%s)",
> args=0xbf839ce4 "4\370\005\bI\004") at failures.c:670
>
> #5 0xb75ce35e in i_panic (format=format at entry=0x805c274 "file %s:
line
> %d (%s): assertion failed: (%s)") at failures.c:275
>
> #6 0x080574f7 in doveadm_connection_deinit
> (_conn=_conn at entry=0xbf839d60) at doveadm-connection.c:1097
>
> #7 0x08057f03 in doveadm_connection_input (conn=0x0) at
> doveadm-connection.c:1051
>
> #8 0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600
>
> #9 0xb7662e1e in io_loop_handler_run_internal
> (ioloop=ioloop at entry=0x82bd648) at ioloop-epoll.c:223
>
> #10 0xb7661496 in io_loop_handler_run (ioloop=ioloop at entry=0x82bd648)
> at ioloop.c:649
>
> #11 0xb7661658 in io_loop_run (ioloop=0x82bd648) at ioloop.c:624
>
> #12 0xb75da45e in master_service_run (service=0x82bd578,
> callback=callback at entry=0x804d360 <client_connected>) at
> master-service.c:719
>
> #13 0x0804cf5e in main (argc=1, argv=0x82bd300) at main.c:366
>
> ?
>
> -- 
>
> Simone LazzarisResponsabile datacenter Qcom S.p.A.Via Roggia Vignola,
> 9 | 24047 Treviglio (BG) T +39036347905 | D +3903631970352| M
> +393938111237 simone.lazzaris at qcom.it
> <mailto:simone.lazzaris at qcom.it>| www.qcom.it
> <https://www.qcom.it>Qcom Official PagesLinkedIn
> <https://www.linkedin.com/company/qcom-spa>| Facebook
> <http://www.facebook.com/qcomspa>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180918/91f19b1c/attachment-0001.html>

Simone Lazzaris

2018-Sep-18 12:31 UTC

head link

Auth process sometimes stop responding after upgrade

In data marted? 18 settembre 2018 14:25:25 CEST, Aki Tuomi ha
scritto:> Can you provide 'bt full'
> 
Sure:

(gdb) bt full
#0  0xb76e4428 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb74636c1 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#2  0xb7466af2 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#3  0xb76485ae in default_fatal_finish (type=<optimized out>,
status=status at entry=0) at
failures.c:201
        backtrace = 0x82b5168
"/usr/local/lib/dovecot/libdovecot.so.0(+0xa15be)
[0xb76485be] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xa1641) [0xb7648641]
-> /usr/
local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb75ce35e] -> dove"...
#4  0xb7648641 in i_internal_fatal_handler (ctx=0xbf839cc0, format=0x805c274
"file %s:
line %d (%s): assertion failed: (%s)", args=0xbf839ce4
"4\370\005\bI\004") at failures.c:670
        status = 0
#5  0xb75ce35e in i_panic (format=format at entry=0x805c274 "file %s: line
%d (%s):
assertion failed: (%s)") at failures.c:275
        ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0,
timestamp_usecs = 0}
        args = 0xbf839ce4 "4\370\005\bI\004"
#6  0x080574f7 in doveadm_connection_deinit (_conn=_conn at entry=0xbf839d60) at
doveadm-connection.c:1097
        conn = 0x82fb580
        __FUNCTION__ = "doveadm_connection_deinit"
#7  0x08057f03 in doveadm_connection_input (conn=0x0) at
doveadm-connection.c:1051
        line = <optimized out>
        ret = <optimized out>
#8  0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600
        ioloop = 0x82bd648
        t_id = 2
        __FUNCTION__ = "io_loop_call_io"
#9  0xb7662e1e in io_loop_handler_run_internal (ioloop=ioloop at
entry=0x82bd648) at
ioloop-epoll.c:223
        ctx = 0x82c9a40
        events = 0x0
        event = 0x82c9a80
        list = 0x82e1830
        io = <optimized out>
        tv = {tv_sec = 0, tv_usec = 236182}
        events_count = 0
        msecs = <optimized out>
        ret = 1
        i = <optimized out>
        j = <optimized out>
        call = <optimized out>
        __FUNCTION__ = "io_loop_handler_run_internal"
#10 0xb7661496 in io_loop_handler_run (ioloop=ioloop at entry=0x82bd648) at
ioloop.c:649
No locals.
#11 0xb7661658 in io_loop_run (ioloop=0x82bd648) at ioloop.c:624
        __FUNCTION__ = "io_loop_run"
#12 0xb75da45e in master_service_run (service=0x82bd578, 
callback=callback at entry=0x804d360 <client_connected>) at
master-service.c:719
No locals.
#13 0x0804cf5e in main (argc=1, argv=0x82bd300) at main.c:366
        set_roots = {0x805f6c0, 0x0}
        test_port = 0
        error = <optimized out>
        debug = false
        c = <optimized out>
(gdb) 


I have realized that on these machine used to run dovecot 2.1.x and were
upgraded with
"make install" on 2.2.36. I've found a library that is not
upgraded (maybe is missing or not
compiled on the new version).


-rw-r--r-- 1 root staff 1963428 Jun 17  2016
/usr/local/lib/dovecot/libdovecot-ssl.a
-rwxr-xr-x 1 root staff    1014 Jun 17  2016
/usr/local/lib/dovecot/libdovecot-ssl.la
lrwxrwxrwx 1 root staff      23 Jun 17  2016
/usr/local/lib/dovecot/libdovecot-ssl.so ->
libdovecot-ssl.so.0.0.0
lrwxrwxrwx 1 root staff      23 Jun 17  2016
/usr/local/lib/dovecot/libdovecot-ssl.so.0 ->
libdovecot-ssl.so.0.0.0
-rwxr-xr-x 1 root staff 1284527 Jun 17  2016
/usr/local/lib/dovecot/libdovecot-ssl.so.0.0.0

Can that be the ultimate cause?


*Simone Lazzaris*
*Qcom S.p.A.*
simone.lazzaris at qcom.it[1] | www.qcom.it[2]
* LinkedIn[3]* | *Facebook[4]*
[5] 






--------
[1] mailto:simone.lazzaris at qcom.it
[2] https://www.qcom.it
[3] https://www.linkedin.com/company/qcom-spa
[4] http://www.facebook.com/qcomspa
[5] https://www.qcom.it/includes/email-banner.gif
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180918/0185a01f/attachment.html>

Timo Sirainen

2018-Sep-19 07:30 UTC

head link

Auth process sometimes stop responding after upgrade

On 18 Sep 2018, at 15.15, Simone Lazzaris <s.lazzaris at interactive.eu>
wrote:> 
> I've got a core dump, and here is the backtrace. Let me know if you
want the core file.
It would be useful if we're able to use it. Could you use
https://dovecot.org/tools/core-tar.sh
<https://dovecot.org/tools/core-tar.sh> to send the core and the related
binaries (e.g. just email to me)? The usage is explained at the beginning of the
script. At least in theory we could then debug with the core file, although
I've had some trouble even then.

But just in case the core doesn't work, could you also do:

bt full
fr 8
p *((struct doveadm_connection *)io->context)
p *((struct doveadm_connection *)io->context)->input

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180919/c452a2e6/attachment.html>

Simone Lazzaris

2018-Sep-19 08:11 UTC

head link

Auth process sometimes stop responding after upgrade

In data mercoled? 19 settembre 2018 09:30:47 CEST, Timo Sirainen ha
scritto:> On 18 Sep 2018, at 15.15, Simone Lazzaris <s.lazzaris at
interactive.eu> wrote:
> > I've got a core dump, and here is the backtrace. Let me know if
you want
> > the core file.
> It would be useful if we're able to use it. Could you use
> https://dovecot.org/tools/core-tar.sh
> <https://dovecot.org/tools/core-tar.sh> to send the core and the
related
> binaries (e.g. just email to me)? The usage is explained at the beginning
> of the script. At least in theory we could then debug with the core file,
> although I've had some trouble even then.
> 
> But just in case the core doesn't work, could you also do:
> 
> bt full
> fr 8
> p *((struct doveadm_connection *)io->context)
> p *((struct doveadm_connection *)io->context)->input
I'm sending you the tarball created with core-tar; and just in case:

root at imap-front4:/usr/local/src/dovecot-2.2.36# gdb
./src/director/.libs/director /var/tmp/
core.10733 
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/usr/local/src/dovecot-2.2.36/src/director/.libs/director...done.
[New LWP 10733]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `dovecot/director'.
Program terminated with signal 6, Aborted.
#0  0xb76e4428 in __kernel_vsyscall ()
(gdb) bt full
#0  0xb76e4428 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb74636c1 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#2  0xb7466af2 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#3  0xb76485ae in default_fatal_finish (type=<optimized out>,
status=status at entry=0) at
failures.c:201
        backtrace = 0x82b5168
"/usr/local/lib/dovecot/libdovecot.so.0(+0xa15be)
[0xb76485be] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xa1641) [0xb7648641]
-> /usr/
local/lib/dovecot/libdovecot.so.0(i_fatal+0) [0xb75ce35e] -> dove"...
#4  0xb7648641 in i_internal_fatal_handler (ctx=0xbf839cc0, format=0x805c274
"file %s:
line %d (%s): assertion failed: (%s)", args=0xbf839ce4
"4\370\005\bI\004") at failures.c:670
        status = 0
#5  0xb75ce35e in i_panic (format=format at entry=0x805c274 "file %s: line
%d (%s):
assertion failed: (%s)") at failures.c:275
        ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0,
timestamp_usecs = 0}
        args = 0xbf839ce4 "4\370\005\bI\004"
#6  0x080574f7 in doveadm_connection_deinit (_conn=_conn at entry=0xbf839d60) at
doveadm-connection.c:1097
        conn = 0x82fb580
        __FUNCTION__ = "doveadm_connection_deinit"
#7  0x08057f03 in doveadm_connection_input (conn=0x0) at
doveadm-connection.c:1051
        line = <optimized out>
        ret = <optimized out>
#8  0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600
        ioloop = 0x82bd648
        t_id = 2
        __FUNCTION__ = "io_loop_call_io"
#9  0xb7662e1e in io_loop_handler_run_internal (ioloop=ioloop at
entry=0x82bd648) at
ioloop-epoll.c:223
        ctx = 0x82c9a40
        events = 0x0
        event = 0x82c9a80
        list = 0x82e1830
        io = <optimized out>
        tv = {tv_sec = 0, tv_usec = 236182}
        events_count = 0
        msecs = <optimized out>
        ret = 1
        i = <optimized out>
        j = <optimized out>
        call = <optimized out>
        __FUNCTION__ = "io_loop_handler_run_internal"
#10 0xb7661496 in io_loop_handler_run (ioloop=ioloop at entry=0x82bd648) at
ioloop.c:649
No locals.
#11 0xb7661658 in io_loop_run (ioloop=0x82bd648) at ioloop.c:624
        __FUNCTION__ = "io_loop_run"
#12 0xb75da45e in master_service_run (service=0x82bd578, 
callback=callback at entry=0x804d360 <client_connected>) at
master-service.c:719
No locals.
#13 0x0804cf5e in main (argc=1, argv=0x82bd300) at main.c:366
        set_roots = {0x805f6c0, 0x0}
        test_port = 0
        error = <optimized out>
        debug = false
        c = <optimized out>
(gdb) fr 8
#8  0xb76613db in io_loop_call_io (io=0x82fb780) at ioloop.c:600
600             io->callback(io->context);
(gdb) p *((struct doveadm_connection *)io->context)
$1 = {prev = 0x0, next = 0x0, fd = 21, io = 0x0, input = 0x82fc478, output =
0x82fc598, dir =
0x82c20a8, to_ring_sync_abort = 0x82e8de0, reset_cmd = 0x0, kick_cmd = 0x0, 
  ring_sync_callback = 0x80570d0 <doveadm_connection_ret_ok>,
cmd_pending_args =
0x0, cmd_pending_idx = 0, handshaked = 1}
(gdb) p *((struct doveadm_connection *)io->context)->input
$2 = {v_offset = 51, stream_errno = 0, mmaped = 0, blocking = 0, closed = 0,
readable_fd =
1, seekable = 0, eof = 1, real_stream = 0x82fc440}
(gdb) quit

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://dovecot.org/pipermail/dovecot/attachments/20180919/cec2e15b/attachment-0001.html>

Possibly Parallel Threads

Search for more possibly parallel threads

dovecot - Sep 2018 - Auth process sometimes stop responding after upgrade

Auth process sometimes stop responding after upgrade

Auth process sometimes stop responding after upgrade

Auth process sometimes stop responding after upgrade

Auth process sometimes stop responding after upgrade

Auth process sometimes stop responding after upgrade

Possibly Parallel Threads