netfilter buglog - Nov 2018 - [Bug 1300] New: nft(8) - man page - SETS - missing descriptions and explanations - flags, auto-merge

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

            Bug ID: 1300
           Summary: nft(8) - man page - SETS - missing descriptions and
                    explanations - flags, auto-merge
           Product: nftables
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: james at nurealm.net

Arch Linux
nftables 1:0.9.0-1

nft(8) man page

SETS

add set [<address_family>] <table_name> <set_name> { type
<type_spec> ; [flags
<flag_spec> ;] [timeout <timeout_spec> ;] [gc-interval
<gc-interval_spec> ;]
[elements = { <element>[,...] } ;] [size <size_spec> ;] [policy
<policy_spec>
;] [auto-merge <auto-merge> ;] }

The man page has just:

flags        │ set flags        │ string: constant, interval, timeout
...
auto-merge   │ automatic  merge of adjacent/overlapping set elements (only for
interval sets)        │ <no type specification>

There is not enough information to make use of these these flags or make use of
"auto-merge".  "constant", "interval", and
"timeout" have no explanation in the
man page.

The nftables wiki page has:

flags, the available flags are:
constant - set content may not change while bound
interval - set contains intervals
timeout - elements can be added with a timeout

The explanation "interval - set contains intervals" provides no
information.
You cannot "explain" something by simply repeating the name of a
thing.  That
represents the logical fallacy of "affirming the consequent" or
"presuming the
conclusion", the idea that someone already knows the meaning of the word
being
repeated.

What does it mean to "contain intervals"?  Especially when the members
of the
element list do not look anything like "intervals", except that they
contain
CIDR notation?  Or, is CIDR notation itself considered an "interval"?

In a thread dated 27 Oct 2016, Pablo Neira Ayuso explains "With named sets,
you
have to specify this flag since the kernel uses [it] to select what is the best
data structure [to use] to represent what you need."

But, if "flags interval", *always* has to be specified with named
sets, then,
when creating a named set, why does this flag have to be specified at all? 
I'd
call that a bug.  A named set should instead, then, just automatically include
"interval", whatever that is.

A named set without the redundant "flag interval" configuration causes
"add
element" to throw an error, "Error: Set member cannot be prefix,
missing
interval flag on declaration".  The error message makes no sense, simply
highlighting a member of the set, and given that "being a prefix" is
not
defined and has no explanation itself.

With the configuration "auto-merge", no "Type" description
is given, though the
"add set" synopsis shows "auto-merge" requiring a value or
type specification.
Is this value simply "yes" or "on"?  And, if so, why does it
require a value at
all?  Simply providing the configuration item "auto-merge" should
automatically
turn-on the feature.  Or, does "auto-merge" require some distinct type
of
"auto-merge" technique?  Or, is the man page in error?

"auto-merge" is not referenced at all in the nftables wiki.

A mailing list archive entry "[ANNOUNCE] nftables 0.8.2 release",
dated Feb 2,
2018, shows an example of "auto-merge" with *no* configuration value
or type
specification.  The explanation only says "a new explicit option for
interval
sets, that enables auto-merge of adjacent/overlapping elements when adding them
to the set".  The notion of "interval sets" is still undefined.

When would someone ever *not* want an "interval set" to automatically
merge an
adjacent or overlapping set?  The idea of *not* merging seems to suggest
redundant CPU cycles in the kernel, when processing network packets.

And, again, if a "named set" must always be an "interval
set", then why would a
"flag interval" configuration be required when it should be automatic?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20181117/c5961271/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kfm at plushkava.net

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20200128/a5d23446/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

--- Comment #1 from kfm at plushkava.net ---
(In reply to James Feeney from comment #0)
> The explanation "interval - set contains intervals" provides no
information.
> You cannot "explain" something by simply repeating the name of a
thing.
> That represents the logical fallacy of "affirming the consequent"
or
> "presuming the conclusion", the idea that someone already knows
the meaning
> of the word being repeated.
It's probably because the underlying data structure is formally implemented
as
an interval tree. That aside, your criticism is valid. There is no reason for
the layman to understand this, and nor should they have to. I wholeheartedly
agree that the documentation could stand to be markedly improved.
> 
> What does it mean to "contain intervals"?  Especially when the
members of
> the element list do not look anything like "intervals", except
that they
> contain CIDR notation?  Or, is CIDR notation itself considered an
"interval"?
The short answers are see above, and yes, respectively. Likewise, in the case
of a hyphenated range.
> 
> In a thread dated 27 Oct 2016, Pablo Neira Ayuso explains "With named
sets,
> you have to specify this flag since the kernel uses [it] to select what is
> the best data structure [to use] to represent what you need."
> 
> But, if "flags interval", *always* has to be specified with named
sets,
> then, when creating a named set, why does this flag have to be specified at
> all?  I'd call that a bug.  A named set should instead, then, just
> automatically include "interval", whatever that is.
This is a misunderstanding. A named set does not require the interval flag to
be defined. However, without it, you will only be able to add elements of type
ipv4_addr or ipv6_addr that refer to a single such address.

# nft create table ip t
# nft create set ip t s { type ipv4_addr\; }
# nft add element ip t s { 10.0.0.0/8 }
Error: You must add 'flags interval' to your set declaration if you want
to add
prefix elements
add element ip t s { 10.0.0.0/8 }
                     ^^^^^^^^^^
# # nft add element ip t s { 192.168.1.1-192.168.1.2 }
Error: You must add 'flags interval' to your set declaration if you want
to add
range elements
add element ip t s { 192.168.1.1-192.168.1.2 }
                     ^^^^^^^^^^^^^^^^^^^^^^^
Note that "prefix" refers to the network prefix that an address in
CIDR
notation contains, /8 in this case.

Keep in mind that users may not require every set to store ranges/prefixes. In
that case, eschewing the interval flag allows for a more optimal data structure
to be employed for the task at hand. Just as ipset has an "ip" type
and a "net"
type, for that matter.
> 
> A named set without the redundant "flag interval" configuration
causes "add
> element" to throw an error, "Error: Set member cannot be prefix,
missing
> interval flag on declaration".  The error message makes no sense,
simply
> highlighting a member of the set, and given that "being a prefix"
is not
> defined and has no explanation itself.
The error messages have been improved, as shown above. This came about as a
result of a discussion in bug 1380. It might still be a little confusing if you
don't know what a prefix is, but the error messages of nft might not be the
best place to expound upon networking jargon.
> 
> With the configuration "auto-merge", no "Type"
description is given, though
> the "add set" synopsis shows "auto-merge" requiring a
value or type
> specification.  Is this value simply "yes" or "on"? 
And, if so, why does it
> require a value at all?  Simply providing the configuration item
> "auto-merge" should automatically turn-on the feature.  Or, does
As it does.
> "auto-merge" require some distinct type of "auto-merge"
technique?  Or, is
> the man page in error?
> 
> "auto-merge" is not referenced at all in the nftables wiki.
I'll take a look at that, as I'm able to edit the wiki.
> 
> A mailing list archive entry "[ANNOUNCE] nftables 0.8.2 release",
dated Feb
> 2, 2018, shows an example of "auto-merge" with *no* configuration
value or
> type specification.  The explanation only says "a new explicit option
for
> interval sets, that enables auto-merge of adjacent/overlapping elements
when
> adding them to the set".  The notion of "interval sets" is
still undefined.
One way to look at is that auto-merge is analagous to an option flag that
stands alone, as in "ls -l". Compare and contrast to an option flag
that has to
work in tandem with an additional argument, such as the --quoting-style option.
You can't just write "ls --quoting-style", that would be an error.
But you can,
for instance, write "ls --quoting-style=c". As weak as the man page
may
generally be, I don't see how one can get the impression that auto-merge
takes
any other parameters than itself from the synopsis of add set.
> 
> When would someone ever *not* want an "interval set" to
automatically merge
> an adjacent or overlapping set?  The idea of *not* merging seems to suggest
> redundant CPU cycles in the kernel, when processing network packets.
We'd have to hear from a developer to know for sure but I can say that there
is
a computational cost in updating and balancing the tree, with potential issues
of resource consumption. Imagine if it were possible for these costs to be
imposed from the packet path i.e. by way of a rule that automatically interacts
with a set. It could open up the possibility of an attacker mounting a DoS
attack revolving around set maintenance. Indeed, I'm guessing that's why
nftables doesn't currently allow for the "dynamic" and
"interval" flags to be
combined.

As an aside, and while it doesn't directly address the question, there have
been some unpleasant bugs around the auto-merge functionality - mainly on the
kernel side - that are only just recently being addressed.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20201203/1949dff1/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

kfm at plushkava.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |https://bugzilla.netfilter.
                   |                            |org/show_bug.cgi?id=1380

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20201203/002c7dbe/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

--- Comment #2 from kfm at plushkava.net ---
I just realised that the synopsis was, indeed, broken at the time this bug was
opened. It's now expressed as "[auto-merge ;]" in the man page.
One down, at
least.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20201203/4b265cd6/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1300

--- Comment #3 from James Feeney <james at nurealm.net> ---
Many thanks for your follow-up and explanation.

I still wonder, though, that nft is not able to automatically recognize an
"interval expression", and then, automatically set the interval flag. 
I
suppose that the CIDR "/" character and the "-" character
between individual
addresses are rather obvious "give-aways" that an interval expression
has been
encountered.  "Automation" is sort of what might be expected from a
computer
program.

But then, perhaps it is too difficult to go back and have the kernel change the
type declaration for the named set?  Then again, maybe nft should just first
delete an existing named set, and then re-declare the named set, including all
old and new members, should any subsequent addition to the set be seen
expressed as a range?

A Feature Request then, if someone is looking for something to do...

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20201203/12772e19/attachment.html>