netfilter buglog - Apr 2023 - [Bug 1674] New: ebtables causing packet loss

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

            Bug ID: 1674
           Summary: ebtables causing packet loss
           Product: ebtables
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: critical
          Priority: P5
         Component: ebtables-nft
          Assignee: pablo at netfilter.org
          Reporter: ryder1ross at gmail.com

Hi,

Package version : ebtables 1.8.4 (nf_tables)

Kernel version : 4.18.0-425.13.1.el8_7.x86_64

OS version : AlmaLinux release 8.7 (Stone Smilodon)



I am using AlmaLinux 8 on a dedicated server and hosting VPSes using KVM.
Applying ebtables rules on those VPSes is causing packet loss issue.

Bridge chain: v1001, entries: 1, policy: DROP -p IPv4 -s 00:xx:xx:52:69:ac
--ip-src 192.168.122.204 -j ACCEPT

Bridge chain: v1001IPV6, entries: 0, policy: DROP

Bridge chain: v1001ARPIN, entries: 2, policy: ACCEPT -p ARP --arp-ip-src
192.168.122.204 --arp-mac-src 00:xx:xx:52:69:ac -j ACCEPT -p ARP -j DROP

There are around 100 VPSes hosted. Applying/loading rules for all those VPSes
all at once would cause issues ?

Where else can I check for logs ?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230424/281e07d9/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

Simon G. Trajkovski <neur0armitage at proton.me> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |neur0armitage at proton.me

--- Comment #1 from Simon G. Trajkovski <neur0armitage at proton.me> ---
hello,

what is your ruleset in ebtables-restore format, are you restoring it on the
host or the guest? It is not clear.

This is an upstream project and you are reporting a problem from a
distribution. At least if you give me to reproduce, I can confirm that
everything is ok, but I need an easy way to reproduce.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230424/8ff6fc25/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #2 from ryder1ross at gmail.com ---
Where can I report the issue if this is not the place ?


ebtables-restore is not in use.
I am applying these rules on host node to avoid IP spoofing on the VPS :

------------------------------------------------------------------------------
ebtables -L 'v1001' 

ebtables -L 'v1001IPV6' 

ebtables -L 'v1001ARPIN' 

ebtables -L 'v1001' 

ebtables -L 'v1001IPV6' 

ebtables -L 'v1001ARPIN' 

ebtables -N 'v1001' 

ebtables -P 'v1001' DROP 

ebtables -N 'v1001IPV6' 

ebtables -P 'v1001IPV6' DROP 

 ebtables -N 'v1001'ARPIN 

 ebtables -A FORWARD -p IPv4 -i 'vifv1001' -j 'v1001' 

ebtables -A INPUT -p ARP -i 'vifv1001' -j 'v1001'ARPIN 

 ebtables -A FORWARD -p ARP -i 'vifv1001' -j 'v1001'ARPIN 

 ebtables -A FORWARD -p IPv4 -i 'vinfv1001' -j 'v1001' 

ebtables -A INPUT -p ARP -i 'vinfv1001' -j 'v1001'ARPIN 

ebtables -A FORWARD -p ARP -i 'vinfv1001' -j 'v1001'ARPIN 

 ebtables -A FORWARD -p IPv6 -i 'vifv1001' -j 'v1001IPV6' 

ebtables -A 'v1001' -p IPv4 --ip-src '192.168.122.204' -s
'00:xx:xx:52:69:ac'
-j ACCEPT 

 ebtables -A 'v1001'ARPIN -p ARP --arp-ip-src '192.168.122.204'
--arp-mac-src
'00:xx:xx:52:69:ac' -j ACCEPT 

 ebtables -A 'v1001'ARPIN -p ARP -j DROP 
------------------------------------------------------------------------------
and then save after adding these rules for every VPS.

ebtables-save

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230426/984a0117/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

Phil Sutter <phil at nwl.cc> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |phil at nwl.cc

--- Comment #3 from Phil Sutter <phil at nwl.cc> ---
(In reply to ryder1ross from comment #0)
> I am using AlmaLinux 8 on a dedicated server and hosting VPSes using KVM.
> Applying ebtables rules on those VPSes is causing packet loss issue.
What are the precise symptoms? Just some traffic lost, no connectivity at all?
> Bridge chain: v1001, entries: 1, policy: DROP -p IPv4 -s 00:xx:xx:52:69:ac
> --ip-src 192.168.122.204 -j ACCEPT
> 
> Bridge chain: v1001IPV6, entries: 0, policy: DROP
> 
> Bridge chain: v1001ARPIN, entries: 2, policy: ACCEPT -p ARP --arp-ip-src
> 192.168.122.204 --arp-mac-src 00:xx:xx:52:69:ac -j ACCEPT -p ARP -j DROP
Are you aware that not all ARP traffic may have a source IP address set?
> There are around 100 VPSes hosted. Applying/loading rules for all those
> VPSes all at once would cause issues ?
Cause issues other than packet loss? If not, does packet loss occur only if you
apply the rules to all VPS or for a single one also?
> Where else can I check for logs ?
You could add '--log' to the rules dropping packets and check dmesg.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230426/768d6bbe/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #4 from ryder1ross at gmail.com ---
It starts with packet loss and once the rule count increases, it starts to drop
connections for SSH as well.

ebtables -L --Lx |wc -l shows around 3000+ rules 

when it starts to happen.

It is back to normal once I stop ebtables service on Host node.
------------------------------------------

 ebtables -A 'v1001'ARPIN -p ARP -j DROP --log

Only for this rules do I try with logging ?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230426/af57a11b/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #5 from Phil Sutter <phil at nwl.cc> ---
(In reply to ryder1ross from comment #4)
> It starts with packet loss and once the rule count increases, it starts to
> drop connections for SSH as well.
> 
> ebtables -L --Lx |wc -l shows around 3000+ rules 
> 
> when it starts to happen.
> 
> It is back to normal once I stop ebtables service on Host node.
> ------------------------------------------
> 
>  ebtables -A 'v1001'ARPIN -p ARP -j DROP --log
> 
> Only for this rules do I try with logging ?
That and append rules to the drop-policy chains which only log ('-A
<chain>
--log').

Or you disable the drop policies and remove the DROP rules for testing to see
if the packet loss vanishes.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230426/390b3065/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #6 from ryder1ross at gmail.com ---
I have around 150 VPSes on the server so I tried to apply the rules at around
10 minutes of interval without drop rules

Applied for 1 VPS

Applied for more 5 VPS at once : ebtables -L --Lx |wc -l : 107 (count of
output)

Applied for more10 VPS at once : ebtables -L --Lx |wc -l : 269 (count of
output)

Applied for more 20 VPS at once : ebtables -L --Lx |wc -l : 595 (count of
output)

---  ping statistics ---
239 packets transmitted, 234 received, 2.09205% packet loss, time 238439ms
rtt min/avg/max/mdev = 5.434/6.621/87.034/8.033 ms


--- ping statistics ---
228 packets transmitted, 228 received, 0% packet loss, time 227303ms
rtt min/avg/max/mdev = 5.436/7.445/318.067/21.496 ms


------------------------------------------------------------------------------------------

Applied for more 50 VPS at once : ebtables -L --Lx |wc -l : 1400 (count of
output)

--- ping statistics ---
212 packets transmitted, 197 received, 7.07547% packet loss, time 211530ms
rtt min/avg/max/mdev = 5.458/25.077/348.502/52.653 ms


--- ping statistics ---
204 packets transmitted, 197 received, 3.43137% packet loss, time 203383ms
rtt min/avg/max/mdev = 5.455/10.922/214.666/23.377 ms

------------------------------------------------------------------------------------------
after 20 mins


--- ping statistics ---
202 packets transmitted, 190 received, 5.94059% packet loss, time 201482ms
rtt min/avg/max/mdev = 5.447/24.766/275.033/46.939 ms


------------------------------------------------------------------------------------------

Applied for 20 VPS at once:  ebtables -L --Lx |wc -l : 1753

--- ping statistics ---
205 packets transmitted, 167 received, 18.5366% packet loss, time 204985ms
rtt min/avg/max/mdev = 5.458/31.679/310.276/60.866 ms

------------------------------------------------------------------------------------------

100% ksoftirqd/57 starts to show up after around 1500 lines of output and
packet loss increases too.

RAM/CPU was free 

%Cpu(s): 46.9 us, 10.6 sy,  0.0 ni, 37.8 id,  0.0 wa,  1.4 hi,  3.3 si,  0.0 st
MiB Mem : 515835.8 total,  90607.5 free, 283876.2 used, 141352.1 buff/cache
MiB Swap:   8198.0 total,   8198.0 free,      0.0 used. 223415.2 avail Mem

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230504/04118c1d/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #7 from Phil Sutter <phil at nwl.cc> ---
OK, so packet loss doesn't occur due to the drop rules but from system
overload
(it seems). At least when you see 100% ksoftirqd, you've maxed out your
server.

What hypervisor are you using? Is this maybe related to retbleed mitigation?
(It hits performance pretty badly.)

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230504/6145185c/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #8 from Simon G. Trajkovski <neur0armitage at proton.me> ---
It is 2023, why use ebtables?

### step no.1; add skeleton ruleset w/maps

table bridge filter {
   map proto_ifname_map {
            typeof meta protocol . meta iifname : verdict
   }

   chain input {
            type filter hook input priority filter; policy accept;
            meta protocol . meta iifname vmap @proto_ifname_map
            counter drop
   }

   chain forward {
            type filter hook forward priority filter; policy accept;
            meta protocol . meta iifname vmap @proto_ifname_map
            counter drop
   }
}

### step no.2; add rules for every virtual machine

add chain bridge filter v1001
add chain bridge filter v1001IPV6
add chain bridge filter v1001ARPIN

add map bridge filter v1001_ip_map { typeof ether saddr . ip saddr : verdict; }
add element bridge filter v1001_ip_map { 00:ff:ff:52:69:ac . 192.168.122.204 :
accept }
add rule bridge filter v1001 ether saddr . ip saddr vmap @v1001_ip_map

add map bridge filter v1001_arp_map { typeof arp saddr ether . arp saddr ip :
verdict; }
add element bridge filter v1001_arp_map { 00:ff:ff:52:69:ac . 192.168.122.204 :
accept }
add rule bridge filter v1001ARPIN arp saddr ether . arp saddr ip vmap
@v1001_arp_map

add rule bridge filter v1001IPV6 drop

add element bridge filter proto_ifname_map { arp . vifv1001 : jump v1001ARPIN }
add element bridge filter proto_ifname_map { ip . vifv1001 : jump v1001 }
add element bridge filter proto_ifname_map { ip6 . vifv1001 : jump v1001IPV6 }

--
try banana pi - opensource router with hardware acceleration!

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230504/199ce453/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #9 from ryder1ross at gmail.com ---
its KVM on AMD EPYC 7502P 32-Core Processor

It does not looks affected.

Also I am not sure what rules were sent in previous response by Simon.
I want to be able to avoid IP/MAC spoofing in VPS.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230505/59228fbf/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #10 from ryder1ross at gmail.com ---
Is it NFT rules ?
Would it clash with iptables rules on server ?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230505/ef088add/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #11 from Phil Sutter <phil at nwl.cc> ---
Yes, these are nftables rules. They may clash or not, depends on the remaining
ruleset. In general, excessive use of sets and maps (either with ipset or
nftables) is a much better approach than long lists of rules for packets to
traverse.

What kernel version are you using? Did the packet loss occur after an upgrade
or did this never perform? In the latter case, maybe your hardware is just not
up to the task?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230505/6a778819/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #12 from ryder1ross at gmail.com ---
Kernel version : 4.18.0-425.13.1.el8_7.x86_64

Its been happening since 8-10 months.
It is un-clear if it started happening after specific kernel update.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230505/55b79672/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1674

--- Comment #13 from ryder1ross at gmail.com ---
Would using nftables lower the occurrence of the issue by large ?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20230508/fe4270d9/attachment.html>