netfilter buglog - Nov 2019 - [Bug 1381] New: Conntrackd segfaults when committing external caches

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

            Bug ID: 1381
           Summary: Conntrackd segfaults when committing external caches
           Product: conntrack-tools
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: conntrack-daemon
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: ralf at os-s.de

Hi,
I have a setup based on Ubuntu 18.10 using conntrackd and keepalived for a
ha-firewall. When stress-testing the system by stopping and starting keepalived
eventually the conntrackd segfaults.

Nov 4 15:29:46 host conntrackd[1122]: [Mon Nov 4 15:29:46 2019] (pid=1122)
[notice] committing all external caches
Nov 4 15:29:46 host Keepalived_logscript: user.info GROUP VG11 MASTER 0
Nov 4 15:29:46 host kernel: [20484.383544] show_signal_msg: 7 callbacks
suppressed
Nov 4 15:29:46 host kernel: [20484.383546] conntrackd[1122]: segfault at 10 ip
00005649a5693ab4 sp 00007ffeb5aafec0 error 4 in conntrackd[5649a5688000+34000]

It does not segfault every time. Sometimes it takes 5-10 iterations starting
and stopping the keepalived daemon which will invoke conntrackd to flush the
caches and subsequently segfault.

Kernel: 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64
x86_64 x86_64 GNU/Linux

Conntrackd: 1.4.4+snapshot20161117

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191111/6471ebbc/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

Ralf Spenneberg <ralf at os-s.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P5                          |P3

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191111/0612cc8a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
                 CC|                            |pablo at netfilter.org

--- Comment #1 from Pablo Neira Ayuso <pablo at netfilter.org> ---
could you run conntrack under valgrind and post the stacktrace?

Please, send me your conntrackd.conf configuration file.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191111/48eefa3a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #2 from Ralf Spenneberg <ralf at os-s.de> ---
Created attachment 575
  --> https://bugzilla.netfilter.org/attachment.cgi?id=575&action=edit
conntrackd.conf

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191113/b9cdca45/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #3 from Ralf Spenneberg <ralf at os-s.de> ---
I added the conntrackd.conf. I could provide the coredump but unfortunately the
file is too large to upload. Would that already help?
I will try to retrigger the problem with valgrind running.

Kind regards,
Ralf

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191113/77f6dd6b/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #4 from Ralf Spenneberg <ralf at os-s.de> ---
Created attachment 576
  --> https://bugzilla.netfilter.org/attachment.cgi?id=576&action=edit
valgrind.log

On Line 36030 is the Segfault.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191118/7f4b8f07/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

csmith2 at gmx.de changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |csmith2 at gmx.de

--- Comment #5 from csmith2 at gmx.de ---
Hello,
I also have this problem with conntrackd in ubuntu 18.04.3.
Is there a known workaround while this is getting fixed?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191130/0b9cfde7/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #6 from Pablo Neira Ayuso <pablo at netfilter.org> ---
You are using 1.4.4.

Could you try latest git version?

It already contain a fix that might be the reason for this:

commit 6abd0b7efdcd808167b95242d2f8cfdbd51f7e8e
Author: Simon Kirby <sim at hostway.ca>
Date:   Thu May 9 14:41:46 2019 -0700

Can you also post what conntrackd commands does keepalived invoke?

Thanks.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191130/d8976f98/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #7 from Ralf Spenneberg <ralf at os-s.de> ---
Hi,
we first tried backporting the fix to our version. Unfortunately that did not
do the trick. We will try now the current version from git.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191209/2952e3e7/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #8 from Ralf Spenneberg <ralf at os-s.de> ---
The current git version still crashes. Unfortunately we are currently not able
to reproduce the crash when running under control of valgrind. We will continue
trying to trigger the crash. 

We might be able to provide a coredump without valgrind.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191217/2ba55d40/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #9 from Ralf Spenneberg <ralf at os-s.de> ---
Created attachment 579
  --> https://bugzilla.netfilter.org/attachment.cgi?id=579&action=edit
valgrind.log of current git version, SEGV at line 6038

Attached is the valgrind log of a crash of the current git version.
Crash occured on Dcc 17 09:57:41

Dec 17 09:57:36 p15x41 Keepalived[25061]: Starting Keepalived v1.3.9
(10/21,2017)
Dec 17 09:57:36 p15x41 Keepalived[25061]: Opening file
'/etc/keepalived/keepalived.conf'.
Dec 17 09:57:36 p15x41 Keepalived[25062]: Starting Healthcheck child process,
pid=25063
Dec 17 09:57:36 p15x41 Keepalived[25062]: Starting VRRP child process,
pid=25064
Dec 17 09:57:36 p15x41 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 17 09:57:36 p15x41 valgrind[1202]: [Tue Dec 17 09:57:36 2019] (pid=1202)
[notice] flushing conntrack table 
in 5 secs
Dec 17 09:57:41 p15x41 valgrind[1202]: [Tue Dec 17 09:57:41 2019] (pid=1202)
[notice] flushing kernel conntrack
 table (scheduled)
Dec 17 09:57:41 p15x41 systemd[1]: Created slice
system-systemd\x2dcoredump.slice.
Dec 17 09:57:41 p15x41 systemd[1]: Started Process Core Dump (PID 25096/UID 0).
Dec 17 09:57:41 p15x41 systemd[1]: conntrackd.service: Main process exited,
code=dumped, status=11/SEGV
Dec 17 09:57:41 p15x41 systemd[1]: conntrackd.service: Failed with result
'core-dump'.
Dec 17 09:57:42 p15x41 systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Dec 17 09:57:42 p15x41 Keepalived[25062]: Stopping

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191217/83595bfa/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

Ralf Spenneberg <ralf at os-s.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P3                          |P4
           Severity|enhancement                 |critical

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20191219/7a6b2446/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #10 from Ralf Spenneberg <ralf at os-s.de> ---
Hi Pablo,

did you have a chance to take a look at the valgrind log?

Ralf

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200120/873b940d/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1381

--- Comment #11 from Ralf Spenneberg <ralf at os-s.de> ---
We are testing currently the FTFW mode. Apparently in this mode the bug does
not trigger. While we can trigger the segfault using Alarm mode easily be
switching master/backup several times using FTFW this does not happen. Maybe
this helps in debugging the code.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200207/aa0a04d0/attachment.html>