netfilter buglog - Jan 2020 - [Bug 1402] New: Race errors with nft

https://bugzilla.netfilter.org/show_bug.cgi?id=1402

            Bug ID: 1402
           Summary: Race errors with nft
           Product: nftables
           Version: unspecified
          Hardware: All
                OS: Debian GNU/Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: netfilter at morp.org

Summary
-------
To evaluate the atomicity/stability of nftables, I set up the following
experiment:
 - Created two distinct sets of rules, R1 and R2 oaded via `nft -f`
 - In two different screens, started two processes to constantly re-load the
rulesets with no pause between reloads
 - Created a small script that would dump the currently active ruleset and save
it to a file, named after the hashed ruleset contents

The expectation is that the `nft list ruleset` output would always display
either R1 or R2 - no empty ruleset, or a cross between R1 or R2, or any other
output. Those expectations were true the vast majority of the time, but in some
rare cases I get an unexpeced output - hence the report.

Bug Summary
-------
When competing nft ruleset imports are running, and the ruleset is dumped (with
`nft list ruleset`), a couple of rare errors can be seen:
 1. An error message returned by `nft list ruleset`: "free(): double free
detected in tcache 2"
 2. Output is composed of duplicate (or even trippled) dumps of the same
tables/rules 


Test files to reproduce
-----------------------
R1.tf
```
#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    set DROP-NETS-V4 {
        type ipv4_addr
        flags interval
        elements = { 1.2.3.0/27 }
    }

    chain predefrag {
        type filter hook prerouting priority -450; policy accept;
        ip frag-off & 49151 != 0 counter packets 0 bytes 0 drop
        ip6 nexthdr ipv6-frag counter packets 0 bytes 0 drop
    }

    chain INPUT {
        type filter hook input priority 0; policy drop;
        udp sport domain udp dport 1024-65535 accept
        tcp sport domain tcp dport 1024-65535 accept
        udp dport domain ip daddr 127.0.0.1 accept
        udp sport 1024-65535 udp dport 1024-65535 ip ttl 1 counter reject
        udp sport 1024-65535 udp dport 1024-65535 ip6 hoplimit 1 counter reject
        ip saddr @DROP-NETS-V4 counter drop
        ct state invalid drop
        ct state established,related accept
        iifname "lo" counter accept
        meta l4proto icmp counter accept
        meta l4proto ipv6-icmp counter accept
        tcp dport 22 accept
    }

}
```

R2.tf
```
#!/usr/sbin/nft -f

flush ruleset
add table inet my-filter

add set inet my-filter ALLOW_SSH_NET {
  type ipv4_addr;
  flags interval;
  elements={
    0.0.0.0/0
  }
}

add chain inet my-filter FORWARD {type filter hook forward priority 0; policy
drop;}

add chain inet my-filter INPUT {type filter hook input priority 0; policy
drop;}
add rule inet my-filter INPUT ip protocol icmp accept
add rule inet my-filter INPUT ip saddr @ALLOW_SSH_NET accept
add rule inet my-filter INPUT ct state related,established accept
```

nftables_hash.sh:
```
#!/usr/bin/env bash

output=$(nft list ruleset  2>&1 | grep -v counter)
md5=$(md5sum <<< $output | cut -f 1 -d ' ')
$(cat <<< "$output" > "hash_${md5}.out")
echo $(date) "$md5"
```

Test commands
-------------
screen -S R1 bash -c 'while true; do ./R1.nft && echo $(date)
"Reloaded R1
ruleset"; done'
screen -S R2 bash -c 'while true; do ./R2.nft && echo $(date)
"Reloaded R2
ruleset"; done'
screen -S ruleset_hash bash -c 'while true; do ./nftables_hash.sh | tee
hashes_nft.txt; done'

Results
-------
After running for a few minutes, you would see more than 2 modes of
'nftables
list ruleset' outputs:
# ls -la hash_*
-rw-r--r-- 1 root root   41 Jan 30 14:30
hash_45b12278a9565f2b5d7107d1be71c93a.out
-rw-r--r-- 1 root root  966 Jan 30 14:30
hash_911e7f4ec5194058a36774026e22001b.out
-rw-r--r-- 1 root root 1035 Jan 30 14:30
hash_c4d068fa75ff27bccc495a3e3a550018.out
-rw-r--r-- 1 root root  363 Jan 30 14:30
hash_ce58eea5fcc69506faccd30dc0b68868.out
-rw-r--r-- 1 root root  483 Jan 30 14:30
hash_d33d213b4cfcc7bc056cab22b8d23196.out

The most frequently found error mode is where nft ruleset dumping:
```
# cat hash_45b12278a9565f2b5d7107d1be71c93a.out
free(): double free detected in tcache 2
```

Another error mode is where the dump contains duplicated version of the
ruleset, e.g.
```
# cat hash_911e7f4ec5194058a36774026e22001b.out
table inet filter {
    set DROP-NETS-V4 {
        type ipv4_addr
        flags interval
        elements = { 1.2.3.0/27 }
    }

    chain predefrag {
        type filter hook prerouting priority -450; policy accept;
    }

    chain INPUT {
        type filter hook input priority 0; policy drop;
        udp sport domain udp dport 1024-65535 accept
        tcp sport domain tcp dport 1024-65535 accept
        udp dport domain ip daddr 127.0.0.1 accept
        ct state invalid drop
        ct state established,related accept
        tcp dport ssh accept
    }
}
table inet filter {
    set DROP-NETS-V4 {
        type ipv4_addr
        flags interval
        elements = { 1.2.3.0/27 }
    }

    chain predefrag {
        type filter hook prerouting priority -450; policy accept;
    }

    chain INPUT {
        type filter hook input priority 0; policy drop;
        udp sport domain udp dport 1024-65535 accept
        tcp sport domain tcp dport 1024-65535 accept
        udp dport domain ip daddr 127.0.0.1 accept
        ct state invalid drop
        ct state established,related accept
        tcp dport ssh accept
    }
}
```

In (more) rare cases, the same table can even be displayed 3 times, rather than
just doubled.

System details:
---
OS: Debian 10.2
Kernel: 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
GNU/Linux
nftables version: nftables v0.9.0 (Fearless Fosdick)
Related LKMs:
  # lsmod | grep -i tables
  nf_tables_set          32768  18
  nf_tables             143360  248
nft_ct,nft_reject_inet,nft_counter,nf_tables_set,nft_reject
  nfnetlink              16384  1 nf_tables

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200130/b6f4f0f3/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1402

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kfm at plushkava.net

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200201/90e4d9e8/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1402

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #1 from Pablo Neira Ayuso <pablo at netfilter.org> ---
nftables 0.9.0 is almost two years old.

Would you please test this with lastest release and confirm you are still
experiencing problems.

Thanks.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200415/074d0511/attachment.html>