netfilter buglog - Sep 2017 - [Bug 1185] New: counter flag proposal for sets and maps

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

            Bug ID: 1185
           Summary: counter flag proposal for sets and maps
           Product: nftables
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: karel at unitednetworks.cz

Now when we have stateful objects, one can use map to emulate counting of hits
in set elements, but counters have to be created first.

It would be nice to have "counter" flag for sets and maps with similar
function
as counters in rules, just to count packets and bytes hitting elements of set
or map. No need to list them as stateful object, just show them in set or map
listing.

I am aware that this behaviour can be emulated by combining set with flow table
with same keys in one rule, but having "counter" flags is more
straightforward
and cleaner solution. Not to mention that counting hits in verdict maps cant be
easily emulated like that.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20170922/ce2ced2c/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

Pablo Neira Ayuso <pablo at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #1 from Pablo Neira Ayuso <pablo at netfilter.org> ---
Hi Karel,

We can add something like this:

table x {
    chain y {
         ...
         map m {
                type ipv4_addr . type ipv4_addr : counter;
         }

         map add @m { ip saddr . ip daddr : counter }
}

This would populate the map 'm' with entries whose keys are the 'ip
saddr . ip
daddr' tuple, where each entry gets a counter.

The idea is that you can use the usual command:

    nft list map x y

as well as any other existing map operations.

The double 'counter' above is a bit redundant, but we could use any
stateful
object instead with its parameters, eg.

table x {
    chain y {
         ...
         map m {
                type ipv4_addr : quota;
         }

         map add @m { ip saddr : quota 10 mbytes } accept
}

In this case, we can populate 'm' via different rules, that add
different quota
policies.

Is this close to you would like to see in place?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171002/38c44219/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

alzeih at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alzeih at gmail.com

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171006/746a5918/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #2 from Karel Rericha <karel at unitednetworks.cz> ---
Looks good and actually cleaner solution than flag. Thanks Pablo

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171009/2fc3d15e/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #3 from Karel Rericha <karel at unitednetworks.cz> ---
Actually on second thought I just want to know, how many times set or map
element was hit. Your proposal (which I would call "anonymous stateful
objects") is nice too, but for other purpose (e.g. when you want to use
counter
value right in rule). But I want to be able to use sets and maps unchanged as
they are, just let me see counter value of each item when listing sets or maps.

Sry for previous post, first day in work after vacation :)

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171009/31618626/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #4 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to Karel Rericha from comment #3)
> Actually on second thought I just want to know, how many times set or map
> element was hit.
I see. So it would be a global counter for this set/map that updated if an
element lookup succeeds.

Right?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171009/568f9f60/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #5 from Karel Rericha <karel at unitednetworks.cz> ---
(In reply to Pablo Neira Ayuso from comment #4)
> (In reply to Karel Rericha from comment #3)
> > Actually on second thought I just want to know, how many times set or
map
> > element was hit.
> 
> I see. So it would be a global counter for this set/map that updated if an
> element lookup succeeds.
> 
> Right?
No, separate counter for each element. Definition could look like:

table x {
    chain y {
        ...
        set s {
            type ipv4_addr . type ipv4_addr; flags counter;
        }

        map m {
            type inet_service : ipv4_addr; flags counter;
        }
    }
}

Listing could look like (similar to listing of timeout flag):

table x y {
        set s {
                type ipv4_addr
                elements = { 192.168.100.1 counter 5, 192.168.100.2 counter 0 }
        }
}

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171009/a2cf32c3/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #6 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to Karel Rericha from comment #5)
> (In reply to Pablo Neira Ayuso from comment #4)
> > (In reply to Karel Rericha from comment #3)
> > > Actually on second thought I just want to know, how many times
set or map
> > > element was hit.
> > 
> > I see. So it would be a global counter for this set/map that updated
if an
> > element lookup succeeds.
> > 
> > Right?
> 
> No, separate counter for each element. Definition could look like:
> 
> table x {
>     chain y {
>         ...
>         set s {
>             type ipv4_addr . type ipv4_addr; flags counter;
>         }
> 
>         map m {
>             type inet_service : ipv4_addr; flags counter;
>         }
>     }
> }
OK, we can do this via anonymous stateful objects, needs to be implemented.

We can also support it with maps like this, proposed syntax (please suggest any
better if you like):

         map m {
             type inet_service : ipv4_addr, counter;
         }

Where the counter is an anonymous stateful object.

Semantics would be:

1) Look up for inet_service key.
2) If found, bump counter and then fetch ipv4_addr in the mapping.

Let me know.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171017/59011c1a/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #7 from Karel Rericha <karel at unitednetworks.cz> ---
(In reply to Pablo Neira Ayuso from comment #6)
> 
> OK, we can do this via anonymous stateful objects, needs to be implemented.
> 
> We can also support it with maps like this, proposed syntax (please suggest
> any better if you like):
> 
>          map m {
>              type inet_service : ipv4_addr, counter;
>          }
>   
> Where the counter is an anonymous stateful object.
> 
> Semantics would be:
> 
> 1) Look up for inet_service key.
> 2) If found, bump counter and then fetch ipv4_addr in the mapping.
> 
> Let me know.
Easy way:
I dont think that extending map definition syntax is good idea. Adding counter
flag seems to be much cleaner and intuitive for this purpose. See my Comment 5.
Semantic would be exactly as you are proposing.

Hard (generic) way:
Let sets, maps, vmaps and flow tables can specify actions if element was hit.
And let this action return accept when we "accept" hit or drop when we
"drop"
hit and signal upper levels that element is not there. See syntax:

table x {
    chain y {
        ...
        set s {
            type ipv4_addr . type inet_service; flags action;
        }

        map m {
            type inet_service : ipv4_addr; flags action;
        }
    }
}

# this will count hits of set items (the original idea)
nft add element x s { 192.168.1.1 . 80 counter accept}

# this will hit packets from the start until their payload reaches 50MB
nft add element x s { 192.168.1.2 . 8080 quota 50 mbytes accept}

# this will count packets and hit only first 10 packets per second
nft add element x s { 192.168.1.3 . 22 counter limit rate 10/second accept}

# this will count packets and hit only packets over 10 packets per second
nft add element x s { 192.168.1.4 . 443 counter limit rate 10/second drop}

# this will hit only packets over 10 packets per second and count false hits
(drops)
nft add element x s { 192.168.1.4 . 443 limit rate 10/second count drop}

# same as previous, but for map
nft add element x m { 88: 192.168.1.5 counter limit rate 10/second drop}

Default verdict for action is accept so "accept" keyword is redundant
here.

Is is little bit similar like vmaps, but no verdict is used and original
functionality of sets, maps, flows and even vmaps is unchanged.

Seems to me as best idea, but I am not sure if it is worth the effort.


Regarding anonymous stateful objects:
Anonymous stateful objects, which you proposed, are not exactly what I wanted,
but they seems to me as nice thing to have too. And they would fit right into
syntax because we already have anonymous sets, maps, counters, limits etc. So
idea is if we can have anonymous stateful objects bonded to rule, why couldnt
we have anonymous stateful object bonded to map element.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20171017/a15aa7b2/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kfm at plushkava.net

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200128/63b5e0f8/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

Frank Myhr <fmyhr at fhmtech.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |fmyhr at fhmtech.com

--- Comment #8 from Frank Myhr <fmyhr at fhmtech.com> ---
I too would like set element counters in nftables. Having them (preferably via
set "counter" flag) would much simplify switching from ipset.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200130/8e0a62e9/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #9 from Frank Myhr <fmyhr at fhmtech.com> ---
It seems that per-element counters work since nftables 0.9.1:
https://marc.info/?l=netfilter-devel&m=156139496310278&w=2

Syntax shown at above link under bullet point "Improve support for dynamic
set
updates" works for me in nftables 0.9.3:

table ip test_tab {
  set test_set {
    type ipv4_addr
    size 65535
    flags dynamic,timeout
    timeout 1h
  }
  chain INPUT {
    type filter hook input priority 0; policy accept;
    update @test_set { ip saddr counter }
  }
}

Can this bug be closed?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200303/763077da/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

--- Comment #10 from Karel Rericha <karel at unitednetworks.cz> ---
Frank what you are pointing out are actually "add/update" counters in
dynamic
sets/maps. Not lookup "hit" counters I am proposing.

See my new bug:

https://bugzilla.netfilter.org/show_bug.cgi?id=1411

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200303/ba7a6300/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1185

Karel Rericha <karel at unitednetworks.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #11 from Karel Rericha <karel at unitednetworks.cz> ---
Looks like Pablo recently commited nftables and kernel patches that allows
adding stateful expressions to set items so this proposal can be closed.

See

http://git.netfilter.org/nftables/commit/?id=6d80e0f154920b5d26aa764459ec0450a8a12b58

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200324/ae749415/attachment.html>