Hi,
On 9/28/16 1:59 PM, Slawa Olhovchenkov wrote:> On Wed, Sep 28, 2016 at 12:06:47PM +0200, Julien Charbon wrote:
>> <SNIP>
>> I am still trying to reproduce your issue, without success so far.
Thanks for Slawa effort and multiple debug report we start seeing the
bottom of this issue and it seems to be a generic one. The most useful
report being:
panic: tcp_detach: INP_TIMEWAIT && INP_DROPPED && tp != NULL
cpuid = 4
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8032467b db_trace_self_wrapper+0x2b/frame
0xfffffe1f9e1f8730
vpanic() at 0xffffffff804b5672 = vpanic+0x182/frame 0xfffffe1f9e1f87b0
kassert_panic() at 0xffffffff804b54e6 = kassert_panic+0x126/frame
0xfffffe1f9e1f8820
tcp_usr_detach() at 0xffffffff806564dc = tcp_usr_detach+0x1bc/frame
0xfffffe1f9e1f8850
sofree() at 0xffffffff8053de66 = sofree+0x1a6/frame 0xfffffe1f9e1f8880
tcp_close() at 0xffffffff8064dd8e = tcp_close+0x11e/frame 0xfffffe1f9e1f88b0
tcp_timer_2msl() at 0xffffffff80653c28 = tcp_timer_2msl+0x278/frame
0xfffffe1f9e1f88e0
softclock_call_cc() at 0xffffffff804cbacc softclock_call_cc+0x19c/frame
0xfffffe1f9e1f89c0
softclock() at 0xffffffff804cbec7 = softclock+0x47/frame 0xfffffe1f9e1f89e0
intr_event_execute_handlers() at 0xffffffff8047aa86
intr_event_execute_handlers+0x96/frame 0xfffffe1f9e1f8a20
ithread_loop() at 0xffffffff8047b106 = ithread_loop+0xa6/frame
0xfffffe1f9e1f8a70
fork_exit() at 0xffffffff804781b4 = fork_exit+0x84/frame 0xfffffe1f9e1f8ab0
fork_trampoline() at 0xffffffff80713fce = fork_trampoline+0xe/frame
0xfffffe1f9e1f8ab0
The scenario:
1. thread1: tcp_timer_2msl() expires and tcp_close() is called to clean
this TCP connection.
2. thread1: In tcp_close() the inp is marked with INP_DROPPED flag, the
process continues and calls INP_WUNLOCK() here:
https://github.com/freebsd/freebsd/blob/releng/11.0/sys/netinet/tcp_subr.c#L1568
3. thread2: Now because INP_WLOCK is released, the inp can transition
to INP_TIMEWAIT state and nothing is preventing it.
4. thread2: During the INP_TIMEWAIT state transition, the inp is marked
with INP_TIMEWAIT flag.
5. thread1: Back in business and tcp_close() call continues with
sofree() -> tcp_usr_detach() -> tcp_detach(). Then as inp is marked
with INP_DROPPED|INP_TIMEWAIT flags, in_pcbfree() is called. w/
INVARIANTS you have an assertion here, w/o INVARIANTS process continues.
6. Later: tcp_twclose() cleans up this INP_TIMEWAIT inp and calls
in_pcbfree() again to achieve a fancy inp double-free.
This issue is a tricky one and seems here since quite a while. It has
been witness at least once in 10.1 and by two different people in 11.0.
Astute questions:
o Why INP_DROPPED flag is not tested in tcp_input() in the first place?
When you are marked as INP_DROPPED, you are almost dead, you should not
be allowed to transition to a different state!
Good point, and tcp_input() relies on the fact that INP_DROPPED inps
are no more in TCP hash table. But tcp_input() in some cases do relock
INP (see relocked: label) and if it does check a lot of things after
having relocked the inp it does not check for a recently added
INP_DROPPED flag.
o Why tcp_detach() does an unconditional in_pcbfree() for inps in
TIMEWAIT state? This because inps in TIMEWAIT state have only one exit:
Being freed. And it is the duty of tcp_detach() to free all inps with
INP_DROPPED|INP_TIMEWAIT.
o Why this issue is so rare?
Good question, I can see how to have a specific TCP traffic to make it
more frequent but no definitive answer yet.
Fix proposal:
This issue description is still a bit fresh but I would enforce that an
inp with INP_DROPPED flag should not be allowed to change state.
Thing learned:
When re-locking an inp, it might have changed a lot, and you might not
like what it became.
Thanks again to Slawa, for his numerous debug reports and always
questioning my explanations. His last question directly led to this
finding. He is testing a quick workaround patch to check if there is more.
I will create a review with a fix proposal, and don't hesitate if you
have other comments on this issue.
--
Julien
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20161006/742521da/attachment.sig>