netfilter buglog - Feb 2024 - [Bug 1736] New: nftables - dynamic update for verdict map from the packet path

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

            Bug ID: 1736
           Summary: nftables - dynamic update for verdict map from the
                    packet path
           Product: nftables
           Version: 1.0.x
          Hardware: x86_64
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: nft
          Assignee: pablo at netfilter.org
          Reporter: dinhtrason at gmail.com

I'm not sure if this is a bug or a feature, not yet implemented.

I am trying to use a verdict map to associate a client to a chain to implement
the session affinity function for my load balancer.

The map is defined with the dynamic and timeout flag. I plan to add source
address of new client retrieved from the packet path to a map with the `update
@` action like below


add table ip loadbalancer
add map ip loadbalancer epToChain { type ipv4_addr : verdict ; flags
dynamic,timeout ; timeout 4m ;}
add chain ip loadbalancer service-ABC
add rule ip loadbalancer service-ABC ip saddr vmap @epToChain
add chain ip loadbalancer endpoint-1
add rule ip loadbalancer endpoint-1 update @epToChain { ip saddr : goto
endpoint-1 }

add chain ip loadbalancer endpoint-2
add rule ip loadbalancer endpoint-2 update @epToChain { ip saddr : goto
endpoint-2 }


But I got the error below with nft 1.0.8

---
vm-001 ~ # nft --file /tmp/test.txt
/tmp/test.txt:6:68-71: Error: syntax error, unexpected goto
add rule ip loadbalancer endpoint-1 update @epToChain { ip saddr : goto
endpoint-1 }
                                                                   ^^^^
/tmp/test.txt:9:68-71: Error: syntax error, unexpected goto
add rule ip loadbalancer endpoint-2 update @epToChain { ip saddr : goto
endpoint-2 }
                                                                   ^^^^
vm-001 ~ # nft -v
nftables v1.0.8 (Old Doc Yak #2)

vm-001 ~ # uname -a
Linux vm-001 5.9.1 #32 SMP Thu Jan 14 09:40:07 CET 2021 x86_64 GNU/Linux
---

As a verdict map looks similar to a map or set from user configuration
perspective, it would be nice to have the same support of dynamic update from
the packet path for verdict map as set and map.

I also tried to use another map instead of the verdict map as a workaround, but
got another error (see below).

---
add table ip loadbalancer
add map ip loadbalancer affinity-mappings { type ipv4_addr : ipv4_addr ; flags
dynamic,timeout ; timeout 4m ; }
add map ip loadbalancer epToChain { type ipv4_addr : verdict ; }

add chain ip loadbalancer endpoint-1
add chain ip loadbalancer endpoint-2

add rule ip loadbalancer endpoint-1 update @affinity-mappings { ip saddr :
11.0.2.1 }
add rule ip loadbalancer endpoint-2 update @affinity-mappings { ip saddr :
11.0.2.2 }

add element ip loadbalancer epToChain { 11.0.2.1 : goto endpoint-1, 11.0.2.2 :
goto endpoint-2 }

add element ip loadbalancer affinity-mappings { 192.168.0.1 : 11.0.2.1 }

add chain ip loadbalancer service-ABC
add rule ip loadbalancer service-ABC ip saddr map @affinity-mappings vmap
@epToChain


vm-001 ~ # nft --file /tmp/test.txt
/tmp/test.txt:17:70-73: Error: syntax error, unexpected vmap
add rule ip loadbalancer service-ABC ip saddr map @affinity-mappings vmap
@epToChain
                                                                    ^^^^
---

Is it considered a bug or a new feature that will be fixed in the next nftables
version?

Is there any other alternatives for this issue with the latest nft version?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240213/cc726786/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #1 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to dinhtrason from comment #0)
> Is there any other alternatives for this issue with the latest nft version?
I have seen rulesets which rely in meta mark to achieve this, thus, you use the
'update' statement to add mappings using any key : meta mark. Then, use
the
meta mark for the verdict map lookup to know what chain to visit in the
ruleset.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240319/cf292713/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #2 from dinhtrason at gmail.com ---
> I have seen rulesets which rely in meta mark to achieve this, thus, you use
> the 'update' statement to add mappings using any key : meta mark.
Then, use
> the meta mark for the verdict map lookup to know what chain to visit in the
> ruleset.
Thanks for the suggestion.

There is a restriction on using meta mark for this purpose because the packet
mark has been used for an existing function (namely for masquerading) in my
project. 

It is also hard to have a unique meta mark for each target chain because its
value will be generated by a hash of the target chain's ids (e.g.
endpoint's
destIP and destport) and collision is not avoidable with hashing function even
the target chain's ids are different.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/d3c0f84d/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #3 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to dinhtrason from comment #2)
> > I have seen rulesets which rely in meta mark to achieve this, thus,
you use
> > the 'update' statement to add mappings using any key : meta
mark. Then, use
> > the meta mark for the verdict map lookup to know what chain to visit
in the
> > ruleset.
> 
> Thanks for the suggestion.
> 
> There is a restriction on using meta mark for this purpose because the
> packet mark has been used for an existing function (namely for
masquerading)
> in my project.
Are you fully using the 32 bits in the mark _only_ for masquerading?

If you use conntrack, then can you use connlabel?
> It is also hard to have a unique meta mark for each target chain because
its
> value will be generated by a hash of the target chain's ids (e.g.
endpoint's
> destIP and destport) and collision is not avoidable with hashing function
> even the target chain's ids are different.
I don't have access to your ruleset, I would need a sketch ruleset of you to
understand better what you are trying to do and make better suggestions.

Thanks.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/4a778b5d/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #4 from dinhtrason at gmail.com ---
> Are you fully using the 32 bits in the mark _only_ for masquerading?
No, masquerading takes one bit of the packet mark. The location of the bit
however is not fixed (i.e. it is a configuration option), making the usage of
meta mark is even more difficult. 

You can refer to masqueradeBit in the link for more details.
https://kubernetes.io/docs/reference/config-api/kube-proxy-config.v1alpha1/#kubeproxy-config-k8s-io-v1alpha1-KubeProxyNFTablesConfiguration
> 
> If you use conntrack, then can you use connlabel?
> 
No, conntrack is not used in the context of this chain.
> 
> I don't have access to your ruleset, I would need a sketch ruleset of
you to
> understand better what you are trying to do and make better suggestions.
> 
> Thanks.
You can refer to the snippet of ruleset highlighted in k8s's pull request
for
more details.

https://github.com/kubernetes/kubernetes/pull/123168#issuecomment-1931674294

Note that: I use the trick "ip daddr set ip saddr map
@affinityMapToEP-DBUHUTQG-default/alpine-service/tcp/iperf" instead of meta
mark in this example. That works fine for this use-case, but it is not a
recommended solution from the community.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/ddcceb5e/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #5 from Pablo Neira Ayuso <pablo at netfilter.org> ---
Created attachment 737
  --> https://bugzilla.netfilter.org/attachment.cgi?id=737&action=edit
k8s ct mark ruleset

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/20d6baed/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #6 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to dinhtrason from comment #4)
> > Are you fully using the 32 bits in the mark _only_ for masquerading?
> 
> No, masquerading takes one bit of the packet mark. The location of the bit
> however is not fixed (i.e. it is a configuration option), making the usage
> of meta mark is even more difficult. 
Can you use the conntrack mark (instead of the packet mark)?

Looking at your ruleset, that makes sense to me, because this also allows to
debug via `conntrack -L' what endpoint has being selected for a given flow,
also for netfilter logging as well as `conntrack -E' for event reporting.
> You can refer to masqueradeBit in the link for more details.
> https://kubernetes.io/docs/reference/config-api/kube-proxy-config.v1alpha1/
> #kubeproxy-config-k8s-io-v1alpha1-KubeProxyNFTablesConfiguration
> 
> > 
> > If you use conntrack, then can you use connlabel?
> > 
> 
> No, conntrack is not used in the context of this chain.
You do use conntrack, because I can see 'dnat to' is used in your
ruleset after
the endpoint is selected based on the affinity, note that the stateful NAT
engine requires conntrack.
> > I don't have access to your ruleset, I would need a sketch ruleset
of you to
> > understand better what you are trying to do and make better
suggestions.
> > 
> > Thanks.
> 
> You can refer to the snippet of ruleset highlighted in k8s's pull
request
> for more details.
> 
>
https://github.com/kubernetes/kubernetes/pull/123168#issuecomment-1931674294
> 
> Note that: I use the trick "ip daddr set ip saddr map
> @affinityMapToEP-DBUHUTQG-default/alpine-service/tcp/iperf" instead of
meta
> mark in this example. That works fine for this use-case, but it is not a
> recommended solution from the community.
I have attached a sketch ruleset I build from your link, I mangled it to use ct
mark.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/a94330df/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #7 from dinhtrason at gmail.com ---
> Can you use the conntrack mark (instead of the packet mark)?
> 
> Looking at your ruleset, that makes sense to me, because this also allows
to
> debug via `conntrack -L' what endpoint has being selected for a given
flow,
> also for netfilter logging as well as `conntrack -E' for event
reporting.
> 
> You do use conntrack, because I can see 'dnat to' is used in your
ruleset
> after the endpoint is selected based on the affinity, note that the
stateful
> NAT engine requires conntrack.
> 
That makes sense. 
> I have attached a sketch ruleset I build from your link, I mangled it to
use
> ct mark.
Thanks for your quick reply. I'll give it a try.

> vm-001 ~ # nft --file /tmp/test.txt
> /tmp/test.txt:17:70-73: Error: syntax error, unexpected vmap
> add rule ip loadbalancer service-ABC ip saddr map @affinity-mappings vmap
@epToChain
BTW, I had a commit to support the case. Could you please let me know how I can
send the patch? I refer to the guide
https://wiki.nftables.org/wiki-nftables/index.php/Portal:DeveloperDocs/Patch_submission_guidelines,
but could not see the email address that I can send the patch to.

Thanks!

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/2935a166/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #8 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to dinhtrason from comment #7)
> > Can you use the conntrack mark (instead of the packet mark)?
> > 
> > Looking at your ruleset, that makes sense to me, because this also
allows to
> > debug via `conntrack -L' what endpoint has being selected for a
given flow,
> > also for netfilter logging as well as `conntrack -E' for event
reporting.
> > 
> > You do use conntrack, because I can see 'dnat to' is used in
your ruleset
> > after the endpoint is selected based on the affinity, note that the
stateful
> > NAT engine requires conntrack.
> > 
> 
> That makes sense. 
BTW, you probably want to reserve ct mark == 0 for flow which is not yet pinned
to an endpoint, which is the initial value for the conntrack mark anyway when
the flow is created. Then, you have to adjust maps.

Note that numgen random allows for offset, you could even try to make the ct
mark unique, that is, ensure that each service has its own ct mark space. This
goes with the idea of allowing you to use the existing conntrack userspace tool
and as well netfilter logging to debug issues (eg. a flow going somewhere it
should not).
> > I have attached a sketch ruleset I build from your link, I mangled it
to use
> > ct mark.
> 
> Thanks for your quick reply. I'll give it a try.
> 
> 
> > vm-001 ~ # nft --file /tmp/test.txt
> > /tmp/test.txt:17:70-73: Error: syntax error, unexpected vmap
> > add rule ip loadbalancer service-ABC ip saddr map @affinity-mappings
vmap @epToChain
> 
> BTW, I had a commit to support the case. Could you please let me know how I
> can send the patch? I refer to the guide
> https://wiki.nftables.org/wiki-nftables/index.php/Portal:DeveloperDocs/
> Patch_submission_guidelines, but could not see the email address that I can
> send the patch to.
You mean you would like to support for map lookup then use the result as input
for another verdict map lookup?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/10ae42cd/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #9 from dinhtrason at gmail.com ---
> BTW, you probably want to reserve ct mark == 0 for flow which is not yet
> pinned to an endpoint, which is the initial value for the conntrack mark
> anyway when the flow is created. Then, you have to adjust maps.
> 
> Note that numgen random allows for offset, you could even try to make the
ct
> mark unique, that is, ensure that each service has its own ct mark space.
> This goes with the idea of allowing you to use the existing conntrack
> userspace tool and as well netfilter logging to debug issues (eg. a flow
> going somewhere it should not).
Got the idea. Thanks for your suggestion about using offset.
> 
> You mean you would like to support for map lookup then use the result as
> input for another verdict map lookup?
Yes, Do you think it is reasonable to support such nested lookup? 

Here is one test example of my commit.

# nft list ruleset
table ip loadbalancer {
        map affinity-mappings {
                type ipv4_addr : ipv4_addr
                size 65535
                flags dynamic,timeout
                timeout 4m
                elements = { 192.168.0.156 timeout 4m expires 3m59s996ms :
11.0.2.2, 192.168.0.211 timeout 4m expires 3m56s64ms : 11.0.2.1,
                             192.168.10.254 timeout 4m expires 3m53s974ms :
11.0.2.2 }
        }

        map epToChain {
                type ipv4_addr : verdict
                elements = { 11.0.2.1 : goto endpoint-1, 11.0.2.2 : goto
endpoint-2 }
        }

        chain endpoint-1 {
                counter packets 2 bytes 224
                update @affinity-mappings { ip saddr : 11.0.2.1 }
        }

        chain endpoint-2 {
                counter packets 203 bytes 11940
                update @affinity-mappings { ip saddr : 11.0.2.2 }
        }

        chain service-ABC {
                ip saddr map @affinity-mappings vmap @epToChain
                numgen random mod 2 vmap { 0 : goto endpoint-1, 1 : goto
endpoint-2 }
        }

        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
                goto service-ABC
        }
}

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240320/37894050/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

Simon G. Trajkovski <neuroarmitage at proton.me> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |neuroarmitage at proton.me

--- Comment #10 from Simon G. Trajkovski <neuroarmitage at proton.me> ---
Thanks for simplifying the ruleset.

Looking at your previous ruleset snippet, it looks like the goal is to find the
endpoint for DNAT?

If so, then something like this should be fine?

table ip loadbalancer {
        map affinity-mappings {
                type ipv4_addr : ipv4_addr
                size 65535
                flags dynamic,timeout
                timeout 4m
                elements = { 192.168.0.156 timeout 4m expires 3m59s996ms :
11.0.2.2, 192.168.0.211 timeout 4m expires 3m56s64ms : 11.0.2.1,
                             192.168.10.254 timeout 4m expires 3m53s974ms :
11.0.2.2 }
        }

        map epToChain {
                type ipv4_addr : verdict
                elements = { 11.0.2.1 : goto endpoint-1, 11.0.2.2 : goto
endpoint-2 }
        }

        chain endpoint-1 {
                counter packets 2 bytes 224
                update @affinity-mappings { ip saddr : 11.0.2.1 }
                meta l4proto tcp dnat to 11.0.2.1 : 5001
        }

        chain endpoint-2 {
                counter packets 203 bytes 11940
                update @affinity-mappings { ip saddr : 11.0.2.2 }
                meta l4proto tcp dnat to 11.0.2.2 : 5001
        }

        chain service-ABC {
                meta l4proto tcp dnat to ip saddr map @affinity-mappings : 5001
                numgen random mod 2 vmap { 0 : goto endpoint-1, 1 : goto
endpoint-2 }
        }

        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                goto service-ABC
        }
}

note that the relevant part is:

                meta l4proto tcp dnat to ip saddr map @affinity-mappings : 5001
                numgen random mod 2 vmap { 0 : goto endpoint-1, 1 : goto
endpoint-2 }

look up for an existing affinity mapping, it exists, use it for dnat.

Otherwise, update the affinity map and set dnat.

You can combine dnat with map lookups, it is also possible to use
concatenations, such as 11.0.2.1 . 5001 in the affinity map.

(I am all the time assuming port 5001 is the destination port, as in your
previous snippet).

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240321/2591f104/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #11 from Pablo Neira Ayuso <pablo at netfilter.org> ---
This affinity map also needs to deal with two packets racing to set up the
affinity?

         chain endpoint-1 {
                 counter packets 2 bytes 224
                 update @affinity-mappings { ip saddr : 11.0.2.2 } meta l4proto
tcp dnat to 11.0.2.2 : 5001
                 # lost race, re-lookup to use existing mapping
                 meta l4proto tcp dnat to ip saddr map @affinity-mappings :
5001
                 # not found, should not ever happen
                 counter drop
         }

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240321/bbfefec3/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #12 from dinhtrason at gmail.com ---
> Looking at your previous ruleset snippet, it looks like the goal is to find
> the endpoint for DNAT?
> 
> If so, then something like this should be fine?
> 
Many thanks for your suggestion and the clear example!

Yes, the ultimate goal of these chains are handling session affinity and doing
DNAT to the correct target's endpoint eventually based on the cached source
IP.

> note that the relevant part is:
> 
>                 meta l4proto tcp dnat to ip saddr map @affinity-mappings :
> 5001
> look up for an existing affinity mapping, it exists, use it for dnat.
> 
> Otherwise, update the affinity map and set dnat.

I was not aware that nft supports the syntax that directly DNAT to a port after
a map lookup. It works for my test at least. 

The rule however does not refresh timeout of the cached source ip stored in the
map affinity-mappings for subsequence packets from the same session like what
the rule "update @affinity-mappings ..." in the endpoint-1/2 does.

Do you have any idea how to do that with the new ruleset?

Thanks!

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240321/42912b91/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #13 from dinhtrason at gmail.com ---
(In reply to Pablo Neira Ayuso from comment #11)
> This affinity map also needs to deal with two packets racing to set up the
> affinity?
> 
>          chain endpoint-1 {
>                  counter packets 2 bytes 224
>                  update @affinity-mappings { ip saddr : 11.0.2.2 } meta
> l4proto tcp dnat to 11.0.2.2 : 5001
>                  # lost race, re-lookup to use existing mapping
>                  meta l4proto tcp dnat to ip saddr map @affinity-mappings :
> 5001
>                  # not found, should not ever happen
>                  counter drop
>          }
I did not even think about the race condition. Thanks for highlighting this
special case.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240321/c79a35d9/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #14 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to dinhtrason from comment #13)
> I did not even think about the race condition. Thanks for highlighting this
> special case.
table ip loadbalancer {
        map affinity-mappings {
                type ipv4_addr : ipv4_addr
                size 65535
                flags dynamic,timeout
                timeout 4m
        }

        chain candidate-endpoint-1 {
                update @affinity-mappings { ip saddr counter : 11.0.2.1 }
        }

        chain candidate-endpoint-2 {                                            
                update @affinity-mappings { ip saddr counter : 11.0.2.2 }
        }

        chain service-ABC {                                                     
                numgen random mod 2 vmap { 0 : goto candidate-endpoint-1, 1 :
goto candidate-endpoint-2 }
                meta l4proto tcp dnat to ip saddr map @affinity-mappings : 5001 
        }                                                                       
}

The idea is:

1) Update/refresh the mapping first. If the mapping already exists, refresh the
timeout for such existing mapping, note that the existing mapping is left as is
if it exists (update does not override an existing mapping). If the mapping
does not exist, then the new mapping entry gets added.
2) Then, look up for the DNAT mapping.

BTW; I have placed counters after the key in the update statement, then update
bumps the corresponding counters for the existing/new mapping in your
@affinity-mapping.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240325/e13ecb58/attachment-0001.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #15 from Pablo Neira Ayuso <pablo at netfilter.org> ---
(In reply to Pablo Neira Ayuso from comment #14)
> table ip loadbalancer {
>         map affinity-mappings {
>                 type ipv4_addr : ipv4_addr
>                 size 65535
>                 flags dynamic,timeout
>                 timeout 4m
>         }
> 
>         chain candidate-endpoint-1 {
>                 update @affinity-mappings { ip saddr counter : 11.0.2.1 }
>         }
> 
>         chain candidate-endpoint-2 {
> 
>                 update @affinity-mappings { ip saddr counter : 11.0.2.2 }
>         }
> 
>         chain service-ABC {
> 
>                 numgen random mod 2 vmap { 0 : goto candidate-endpoint-1, 1
> : goto candidate-endpoint-2 }
>                 meta l4proto tcp dnat to ip saddr map @affinity-mappings :
> 5001  
>         }
> 
> }
This should be 'jump' not 'goto' BTW, so the dnat lookup happens
after
refreshing @affinity-mappings.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240325/de96bff0/attachment.html>

https://bugzilla.netfilter.org/show_bug.cgi?id=1736

--- Comment #16 from dinhtrason at gmail.com ---
Jumping always to chain 'candidate-endpoint-x' prior to looking up the
affinity
map seems to be a trick that helps update the timeout of the cached source IPs.
It looks a bit unusual, but seems to work fine.

There are two options suggested so far:

1/ Use two maps with 'ct mark' to lookup the target endpoint 

table ip loadbalancer {
        map affinityMapToEP {
                type ipv4_addr : mark
                size 65535
                flags dynamic,timeout
                timeout 4m
                elements = { 192.168.0.156 timeout 4m expires 3m58s644ms :
0x00000002 }
        }

        map affinityGotoChain {
                type mark : verdict
                flags dynamic
                elements = { 0x00000001 : goto endpoint-1, 0x00000002 : goto
endpoint-2 }
        }

        chain endpoint-1 {
                update @affinityMapToEP { ip saddr : 0x00000001 }
                meta l4proto tcp dnat to 11.0.2.1:5001
        }

        chain endpoint-2 {
                update @affinityMapToEP { ip saddr : 0x00000002 }
                meta l4proto tcp dnat to 11.0.2.2:5001
        }

        chain service-ABC {
                ct mark set ip saddr map @affinityMapToEP
                ct mark vmap @affinityGotoChain
                numgen random mod 2 vmap { 0 : goto endpoint-1, 1 : goto
endpoint-2 }
        }

        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                tcp dport 32001 goto service-ABC
        }
}

2/ Use one map only and DNAT directly to endpoint address

table ip loadbalancer {
        map affinity-mappings {
                type ipv4_addr : ipv4_addr
                size 65535
                flags dynamic,timeout
                timeout 4m
                elements = { 192.168.0.156 timeout 4m expires 3m58s702ms :
11.0.2.1 }
        }

        chain endpoint-1 {
                update @affinity-mappings { ip saddr : 11.0.2.1 }
        }

        chain endpoint-2 {
                update @affinity-mappings { ip saddr : 11.0.2.2 }
        }

        chain service-ABC {
                numgen random mod 2 vmap { 0 : jump endpoint-1, 1 : jump
endpoint-2 }
                meta l4proto tcp dnat to ip saddr map @affinity-mappings:5001
        }

        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                tcp dport 32001 goto service-ABC
        }
}

I verified both options. They work for my case with some PROS and CONS. I will
need to tailor and promote one of them to k8s community.

Thanks again for your support with the detailed examples.

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.netfilter.org/pipermail/netfilter-buglog/attachments/20240326/e445f723/attachment.html>