netfilter buglog - Dec 2023 - [Bug 1726] New: invalid json generated by ipset list -output json

https://bugzilla.netfilter.org/show_bug.cgi?id=1726

            Bug ID: 1726
           Summary: invalid json generated by ipset list -output json
           Product: ipset
           Version: unspecified
          Hardware: x86_64
                OS: Debian GNU/Linux
            Status: NEW
          Severity: trivial
          Priority: P5
         Component: default
          Assignee: netfilter-buglog at lists.netfilter.org
          Reporter: mark at glines.org

I tried to use the -output json parameter with ipset list, and found that it
does not emit valid json.

This is true in both the Debian Trixie package, version 7.19-1, as well as a
fresh ipset build from today's git.

I can see two problems with the output:

1. the "initval" field is emitted as hex, which is not a part of the
json
syntax [1].
2. in combination with '-name', it seems to be printing key:value pairs
in an
array, not a map.

I set up a test ipset as follows:

---
root at gir:~# ipset list
root at gir:~# ipset new test hash:ip
root at gir:~# ipset add test 1.2.3.4
root at gir:~# ipset list --output json
[
  {
    "name" : "test",
    "type" : "hash:ip",
    "revision" : 6,
    "header" : {
      "family" : "inet",
      "hashsize" : 1024,
      "maxelem" : 65536,
      "bucketsize" : 12,
      "initval" : 0xdcadf93b,
---

This is the point where parsing fails.  The "0x" prefix is not part of
the JSON
grammar [1].  Popular JSON parsers fail on this:

---
root at gir:~# ipset list -output json | jq .
jq: parse error: Invalid numeric literal at line 11, column 29
---

It is strange that the JSON spec allows hex literals for unicode characters
within strings, but not 0x-prefixes in numbers.  But it is what it is.
I think that this could be resolved by emitting the value as a decimal (%d), or
as a quoted string.


When combined with the -name parameter, I see:

---
root at gir:~# ipset list -output json -name
[
"name" : "test"

]
---

The problem with this is that key:value pairs are valid in a map/object (curly
braces), but not an array (square braces).  Popular JSON parsers fail on this:

---
root at gir:~# ipset list -output json -name | jq .
jq: parse error: ':' not as part of an object at line 2, column 8
---

This output looks even more odd when multiple ipsets are present.

---
root at gir:~# ipset new test2 hash:ip
root at gir:~# ipset list -output json -name
[
"name" : "test"
"name" : "test2"

]
---

If the outer braces were curly, then this would have multiple instances of the
same key at the same level.  It's also missing a comma separator.

I think the best way to solve this would be to put curly braces around each
name, like this:

---
[
  { "name" : "test" },
  { "name" : "test2" }
]
---


[1]:
https://ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf
Figure 4 and surrounding text

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20231205/969443c6/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1726

--- Comment #1 from mark at glines.org ---
Created attachment 732
  --> https://bugzilla.netfilter.org/attachment.cgi?id=732&action=edit
Fix ipset list -output json -name

This fixes -output json -name.

Adds curly braces.  Adds a static flag to keep track of whether a comma is
necessary.  This is very similar to how list_create() handles it, in the
non-"-name" case.

Output before:

---
% sudo src/ipset list -output json -name
[
"name" : "blocklist.de.ipv4"
"name" : "blocklist.de.ipv6"

]
---

Output after:

---
 sudo src/ipset list -output json -name
[
  { "name" : "blocklist.de.ipv4" },
  { "name" : "blocklist.de.ipv6" }
]
---

While testing this, I added an "else" to solve a logic problem.  It
wants to do
one thing for xml, another thing for json, and a third thing for plain text. 
But because of the missing "else", xml output contains both xml and
plaintext.
Like this:

---
% sudo src/ipset list -output xml -name
<ipsets>
<ipset name="blocklist.de.ipv4"/>
blocklist.de.ipv4
<ipset name="blocklist.de.ipv6"/>
blocklist.de.ipv6
</ipsets>
---

The added "else" resolves that.

---
% sudo src/ipset list -output xml -name
<ipsets>
<ipset name="blocklist.de.ipv4"/>
<ipset name="blocklist.de.ipv6"/>
</ipsets>
---

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20231205/bb2d32d6/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1726

--- Comment #2 from mark at glines.org ---
I don't have a patch for the first problem (initval printed as hex).  It
seems
to be caused by the way ipset_args[] works, calling ipset_print_hexnumber()
unconditionally.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20231205/35584828/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1726

Jozsef Kadlecsik <kadlec at netfilter.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |kadlec at netfilter.org
         Resolution|---                         |FIXED

--- Comment #3 from Jozsef Kadlecsik <kadlec at netfilter.org> ---
I applied your patch to fix the json listing of the -name output. Also, fixed
the printing of hex numbers as quoted strings. Both patches are available from
the ipset git tree at https://git.netfilter.org/ipset/ and there'll be a new
ipset release in a few days.

Thanks for the reporting and the patch,
Jozsef

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20231212/7870c20c/attachment.html>