Linux Ethernet Bridging - May 2023 - [Bridge] [PATCH net-next 1/2] bridge: Add a limit on FDB entries

Hi,

On Tue, May 16, 2023 at 11:56:41AM +0300, Nikolay Aleksandrov
wrote:
> Hmm.. perhaps we can add a flag mask of entries to count. Initially it can
be
> only dynamic entries. We should include more people in this discussion (+CC
Ido and Vladimir).
> Switchdev folks might have more specific requirements and restrictions, so
it'd be nice to get
> their input as well.
I have some other things to do until I can take a closer look at this
discussion, but in principle, switchdev drivers will likely want to
impose their own limit on FDB entries because the hardware itself is
inherently limited in size, so I'm thinking there should be another way
for the software bridge to be informed about this limit other than UAPI.
Which ports that limit should affect (think bridging between ports of
different switches with different FDB sizes) I don't know. If we only
consider switchdev, FDB limits should probably be per hwdom.

Also, in terms of static vs dynamic limits, I've seen hardware
implementations where static FDB entries go to a different FDB table
compared to dynamic ones (Microchip KSZ DSA switches), implementations
where static partitioning between static and dynamic FDB entries is
possible but configurable, and implementations where they all consume
from the shared space and you'd have to evict a dynamic entry to install
a static one. So it's hard to really say what's the size. That, plus not
to mention, many hardware FDBs are not fully associative, and due to
hash collisions, you may be unable to install an entry in the 4-way
associative bin where its {MAC,VID} hash says it should go, even though
the FDB at large is not full.

It sounds sexy to take switchdev into consideration, but I'm not really
sure what we want. Something flexible to cater for the above, probably.
This discussion should probably be merged with:
https://lore.kernel.org/netdev/20230324144917.32lnpgtw5auuyovy at
skbuf/T/#ma600839815582ca61886e83ba533b1dfbe447557
so I'm CCing Oleksij too, since he probably knows better than me what he
wants.

In the thread with DSA trace events, there also was a short talk about
user space theoretically being able to infer FDB sizes and utilization
degree based on instrumenting with ftrace, which is something we wouldn't
like to have to maintain. So I'm adding the DSA maintainers too, since
there is interest for agreeing on a different API.
https://lore.kernel.org/netdev/2f150ad4-34f4-4af9-b3ce-c1aff208ec7e at
lunn.ch/T/#mfa895245fd012e8f66db784fa568109dba396aa7

On 16/05/2023 13:21, Vladimir Oltean wrote:
> Hi,
> 
> On Tue, May 16, 2023 at 11:56:41AM +0300, Nikolay Aleksandrov wrote:
>> Hmm.. perhaps we can add a flag mask of entries to count. Initially it
can be
>> only dynamic entries. We should include more people in this discussion
(+CC Ido and Vladimir).
>> Switchdev folks might have more specific requirements and restrictions,
so it'd be nice to get
>> their input as well.
> 
> I have some other things to do until I can take a closer look at this
> discussion, but in principle, switchdev drivers will likely want to
> impose their own limit on FDB entries because the hardware itself is
> inherently limited in size, so I'm thinking there should be another way
> for the software bridge to be informed about this limit other than UAPI.
Yep, that's ok but it can be added later. This is pretty much internal.
> Which ports that limit should affect (think bridging between ports of
> different switches with different FDB sizes) I don't know. If we only
> consider switchdev, FDB limits should probably be per hwdom.
> 
Now, that's a whole different issue (per-port limits). I've prototype
patches
for that too, but it's a much harder problem to solve and scale in software.
Let's please focus on the single global limit for the moment.
> Also, in terms of static vs dynamic limits, I've seen hardware
> implementations where static FDB entries go to a different FDB table
> compared to dynamic ones (Microchip KSZ DSA switches), implementations
> where static partitioning between static and dynamic FDB entries is
> possible but configurable, and implementations where they all consume
> from the shared space and you'd have to evict a dynamic entry to
install
> a static one. So it's hard to really say what's the size. That,
plus not
> to mention, many hardware FDBs are not fully associative, and due to
> hash collisions, you may be unable to install an entry in the 4-way
> associative bin where its {MAC,VID} hash says it should go, even though
> the FDB at large is not full.
> 
> It sounds sexy to take switchdev into consideration, but I'm not really
> sure what we want. Something flexible to cater for the above, probably.
> This discussion should probably be merged with:
> https://lore.kernel.org/netdev/20230324144917.32lnpgtw5auuyovy at
skbuf/T/#ma600839815582ca61886e83ba533b1dfbe447557
> so I'm CCing Oleksij too, since he probably knows better than me what
he
> wants.
> 
Let's take a step back, I wasn't suggesting we start with a full-fledged
switchdev
implementation. :) I meant only to see if the minimum global limit
implementation
suggested would suffice and would be able to later extend so switchdev can use
and
potentially modify (e.g. drivers setting limits etc). We can start with a simple
support for limits and then extend accordingly. The important part here is to
not add any uAPI that can't be changed later which would impact future
changes.
> In the thread with DSA trace events, there also was a short talk about
> user space theoretically being able to infer FDB sizes and utilization
> degree based on instrumenting with ftrace, which is something we
wouldn't
> like to have to maintain. So I'm adding the DSA maintainers too, since
> there is interest for agreeing on a different API.
> https://lore.kernel.org/netdev/2f150ad4-34f4-4af9-b3ce-c1aff208ec7e at
lunn.ch/T/#mfa895245fd012e8f66db784fa568109dba396aa7