netfilter buglog - Oct 2016 - [Bug 1087] New: Window Tracking not disabled

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

            Bug ID: 1087
           Summary: Window Tracking not disabled
           Product: conntrack-tools
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: conntrack-daemon
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: mat999 at gmail.com

Hi,

I have been conducting testing and I noticed that Window Tracking does not
appear to be disabled on a per connection level upon receipt.

Kernel: 4.1.17
Version: v1.4.3

To Replicate:

First ensure Configuration contains "TCPWindowTracking Off"
Ensure that /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal is set to
"0"

Sync'ed connections out of window will fail. I tested this by triggering a
BGP
PoP switch.

Then Set /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal to
"1".

It now works.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161004/502ee3bc/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #1 from Mathew <mat999 at gmail.com> ---
Please note I will be attempting to resolve this issue myself and am exploring
state with GDB currently.

I will try and isolate it as either a bug with conntrack-tools, or a bug with
nf_conntrack. I suspect conntrack-tools.

Any information you can provide would be appreciated.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161004/aeeeb224/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #2 from Mathew <mat999 at gmail.com> ---
To try and identify the cause of the issue I wrote a peice of software to
extract CTA_IP_V4_DST, CTA_PROTOINFO_TCP_FLAGS_ORIGINAL &
CTA_PROTOINFO_TCP_FLAGS_REPLY.

These are the values (in that order for a synced connection):
"**.**.56.135: 10
3"

given that IP_CT_TCP_FLAG_BE_LIBERAL is 0x08, it looks only to have been set on
the ORIG side of the connection track. I am unsure if this is the root cause of
the issue, but its my best bet so far.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161004/d74c1f07/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #3 from Mathew <mat999 at gmail.com> ---
Perhaps the flow of events relates to this:
http://lxr.free-electrons.com/source/net/netfilter/nf_conntrack_proto_tcp.c?v=4.1#L930

By my understanding this should also have IP_CT_TCP_FLAG_BE_LIBERAL.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161005/50c2d03e/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

Arturo Borrero Gonzalez <arturo.borrero.glez at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |arturo.borrero.glez at gmail.c
                   |                            |om

--- Comment #4 from Arturo Borrero Gonzalez <arturo.borrero.glez at
gmail.com> ---
Mathew, I would need additional details to understand what's happening here.

You need "TCPWindowTracking On" in conntrackd.conf to ensure the
window
tracking information is exchanged between the nodes of the cluster. I.E, from
your first comment I understand that using "TCPWindowTracking Off" and
pushing
out-of-window traffic leads to dropped packages. If my understanding is right,
then this is the *expected*  behaviour.

As I said, to properly deal with this situation, you need
"TCPWindowTracking
On".

This is independent of the behaviour of tcp_be_liberal.
Each node of the cluster should have it's own tcp_be_liberal configuration
using the sysctl variable.

Could you please share additional details?

thanks

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/857d69be/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #5 from Mathew <mat999 at gmail.com> ---
Hi,
>> As I said, to properly deal with this situation, you need
"TCPWindowTracking On".
You are confused, "TCPWindowTracking Off" (default) sets the
IP_CT_TCP_FLAG_BE_LIBERAL on the resulting installed connection which disabled
TCP window tracking

/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal does the same thing
globally.

--

I can confirm my diagnosis was correct. We patched the kernel to correctly
propagate the IP_CT_TCP_FLAG_BE_LIBERAL flag (which *IS* the flag set by
"TCPWindowTracking") to both sides of the connection. This showed the
correct
behaviour.

I have not at this time done extensive testing of the patch or taken it any
further as of yet (plans are to formalise the patch for staging testing within
the fortnight).

The patch is just adding: 
if (seen.flags & IP_CT_TCP_FLAG_BE_LIBERAL ) {
ct->proto.tcp.last_flags |= IP_CT_TCP_FLAG_BE_LIBERAL;
}

to the line linked. This correctly propagates the flag to the last_flags
variable which goes on to form the reply side of the connection.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/c2e5e5e4/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #6 from Arturo Borrero Gonzalez <arturo at debian.org> ---
Ok I will try to generate a patch for you to test.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/dbc0dcda/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #7 from Arturo Borrero Gonzalez <arturo at debian.org> ---
BTW, could you please test a newer version from both conntrack-tools and the
kernel?

The conntrackd from conntrack-tools has now 1.4.4 and kernel is around 4.9.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/653d184a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #8 from Mathew <mat999 at gmail.com> ---
At this time I am restricted to these software versions as these are the
versions deployed for this system that I need to test. There are other
dependencies that make an upgrade unfeasible at this time.

I may test a newer kernel in 2017, but currently I extremely busy. I did
however check there is no patches in this area that I was able to find in newer
kernel versions.

Initial testing in a development VM showed that the bug was fixed with the
patch. I will try and find time to more extensively test this patch within the
fortnight by patching the production kernel and putting it through the systems
QA tests in staging (should highlight any regressions or issues).

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/0d85b549/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #9 from Arturo Borrero Gonzalez <arturo at debian.org> ---
One more question.

In this line
http://lxr.free-electrons.com/source/net/netfilter/nf_conntrack_proto_tcp.c?v=4.1#L935

the code calls tcp_options() which parses the TCP header to fill our conntrack
entry (the 'seen' one).

In that function we don't put IP_CT_TCP_FLAG_BE_LIBERAL into flags, so a
later
check if (seen.flags & IP_CT_TCP_FLAG_BE_LIBERAL) should always fail.

It seems I'm missing something.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/b0701d55/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #10 from Mathew <mat999 at gmail.com> ---
(In reply to Arturo Borrero Gonzalez from comment #9)
> One more question.
> 
> In this line
> http://lxr.free-electrons.com/source/net/netfilter/nf_conntrack_proto_tcp.
> c?v=4.1#L935
> 
> the code calls tcp_options() which parses the TCP header to fill our
> conntrack entry (the 'seen' one).
> 
> In that function we don't put IP_CT_TCP_FLAG_BE_LIBERAL into flags, so
a
> later check if (seen.flags & IP_CT_TCP_FLAG_BE_LIBERAL) should always
fail.
> 
> It seems I'm missing something.
I am fairly certain (from memory) that's all I did to fix the issue. I
don't
have the patched kernel in front of me however. I'll double check when I am
on-site on Thursday.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161018/ad0cb96c/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #11 from Mathew <mat999 at gmail.com> ---
ah, not seen it was ct->proto.tcp.seen[IP_CT_DIR_ORIGINAL].flags

But I discovered that this path doesnt get called for outgoing connections.

So below "tuple = &ct->tuplehash[dir].tuple;" I added

        if(ct->proto.tcp.seen[IP_CT_DIR_ORIGINAL].flags &
IP_CT_TCP_FLAG_BE_LIBERAL) {
                ct->proto.tcp.seen[IP_CT_DIR_REPLY].flags
|IP_CT_TCP_FLAG_BE_LIBERAL;
        }


This worked well, even if it is hacky.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161020/70f2f69a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #12 from Arturo Borrero Gonzalez <arturo at debian.org> ---
Created attachment 483
  --> https://bugzilla.netfilter.org/attachment.cgi?id=483&action=edit
conntrack patch

RFC patch for discussion

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161020/6c1b174a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1087

--- Comment #13 from Mathew <mat999 at gmail.com> ---
Please note, I fully expect there are better places to set this / fix this.

Despite a few years experience building kernel modules in the xtables/conntrack
area I don't know all the in's and out's yet.

I am happy to test anyone elses patches.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20161020/24f64013/attachment.html>