netfilter buglog - Mar 2019 - [Bug 1328] New: Please allow ipset add and del via the /proc/net/xt_ipset mechanism

https://bugzilla.netfilter.org/show_bug.cgi?id=1328

            Bug ID: 1328
           Summary: Please allow ipset add and del via the
                    /proc/net/xt_ipset mechanism
           Product: ipset
           Version: unspecified
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: default
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: murf at parsetree.com

I measure the xt_recent hash table's speed to enter an ip into the hash, and
compared with ipset add speed. recent, via this method:

echo +<addr> >/proc/net/xt_recent/DEFAULT

is 13 times faster than ipset add <addr>

Now, ipset has these advantages against recent:

recent has size limits you must define when you load the module. Ugh.
ipset, on the other hand, at least in the newer versions, (the ones NOT on
centos 6, btw), has no size limit.

So, a list of IP's with CIDRs, like voipbl, takes around 90 seconds to load
the
63k ip's.  recent will do it in about 7 seconds. (well, estimated, as recent
can handle only 8k entries (or at least so it did over a year ago).)

Can you guys, someday, sometime, enable this other, faster way to load up the
ipset with multiple thousands of addresses?

You know, it might even be faster if you allow a whole batch of + and -addrs in
a file, to be dumped at a single shot:

cat ipfile > /proc/net/xt_ipset/someipset

Many thanks!

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20190325/8cb79af1/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1328

Jozsef Kadlecsik <kadlec at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kadlec at netfilter.org

--- Comment #1 from Jozsef Kadlecsik <kadlec at netfilter.org> ---
It is not surprising that the proc interface is faster than the netlink one.
But you can already use the restore mode of ipset to add multiple number of
elements to a set. It's much faster than adding the elements individually.

Sorry, but I do not plan to add a proc file interface to ipset. The main reason
is that ipset elements may have options (timeout, comment, counters, etc.)
which would need to be parsed in the proc interface too.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20190326/ca996fbb/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1328

--- Comment #2 from Steve Murphy <murf at parsetree.com> ---
So many thanks for your response. I had no idea that the save format was so
easy to generate. I wrote a go program to translate the 70k+ ip ranges from
voipbl to the save format, and fed it to restore. I've timed it from several
runs now. The runtime of my converter + the run time of the ipset restore was
as low as 1 second wall time. That places it as about 90 times faster than
individual ipset adds being used. That easily beats the xt_recent /proc
interface.

I was delighted it ran so fast. It was SO FAST, that I had to double and triple
and quadruple check that it actually worked!

So, I respectfully withdraw this request. I will close the bug.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20190329/79edc951/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1328

Steve Murphy <murf at parsetree.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #3 from Steve Murphy <murf at parsetree.com> ---
See the comments, there is no need for this request; it might not even be as
fast as the ipset restore command, even if the request were coded.

You guys are GOOOOOOD!!! Wow! 90 sec runtime cut to 1 sec. A huge time savings,
and not one line of code needed to be changed.

Again, many thanks for the pointer!

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20190329/c7028b04/attachment.html>