Linux Ethernet Bridging - Jan 2020 - [Bridge] [RFC net-next Patch 0/3] net: bridge: mrp: Add support for Media Redundancy Protocol(MRP)

Hi Horatiu,

On Fri, 10 Jan 2020 at 18:04, Horatiu Vultur
<horatiu.vultur at microchip.com> wrote:
>
> Hi Nik,
>
> > I agree with Stephen here, IMO you have to take note of how STP has
progressed
> > and that bringing it in the kernel was a mistake, these days mstpd has
an active
> > community and much better support which is being extended. This looks
best implemented
> > in user-space in my opinion with minimal kernel changes to support it.
You could simply
> > open a packet socket with a filter and work through that, you
don't need new netlink
> > sockets. I'm not familiar with the protocol so can't really be
the judge of that, if
> > you present a good argument for needing a new netlink socket for these
packets - then
> > sure, ok.
>
> We are aware of the STP story, and in case of STP I do agree, it is much
> better to have this in user-space. But while MRP has much in common with
> STP, it also differs in some important areas.
>
> Most importantly, MRP requires sending and receiving thousands of frames
> per second. To achieve the 10ms recovery time, the tx period per
> interface is 500us, on two interfaces, adding up to 4000 frames per
> second to RX and 4000 to TX(if the ring is closed). And this is per
> ring...
>
> The CPU systems in the kind of switches we are working on can not handle
> this load, and it was not meant to handle this. Instead the switch core
> can do the periodic injection of frames and automatic terminate them.
>
> In patch posted, we have not added this HW offload (we have this in our
> internal repos, where we also have implemented the remaining part of the
> protocol). The reason for this is that we wanted to do a proper SW
> implementation and then HW offload it.
>
> Looking back, I can see that what we have presented here could be done
> equally good in user-space (roughly), but that is because the HW offload
> is not part of this patch.
>
> The problem in putting it in user-space is that we do not have a nice a
> clean API where it is just putting a port in forwarding/blocking state
> (like we have with STP). To do an abstraction that actually allow us to
> utilize the HW to offload a protocol like MRP will very easy become too
> specific for our SoC and rejected with that argument.
>
> >
> > If you do decide to continue with the kernel version (which I would
again discourage)
> > a few general points (from a quick scan):
> >  - the single 1.6+k line patch is just hard to review, please break it
into more digestable
> >    and logical pieces
> We will work in this.
>
> >  - the locking is wrong, also there're a few use-after-free bugs
> Oops, that is not good - happy that you caught it. A hint on where,
> would be great.
>
> >  - please re-work the bridge integration code, it can be simplified
and tests can be eliminated
> We will have a second look at that.
>
> >  - your netlink helpers usage is generally wrong and needs more work
> Ok - some hints on what we did wrong would be great.
>
> >  - use the already existing port states instead of adding new ones and
you can avoid some tests in fast-path
> I assume you want us to re-use the STP concept of forwarding/blocking
> and relay on the checks it already has.
>
> >  - perhaps look into using br_afspec() for configuration/retrieval
initially ? I don't think you need the new rtm messages yet.
> Is that a good example on how to do the netlink interface, and you want
> us to use that as a reference?
>
> >  - I'm sure I can go on, but I really think all of this should be
put in user-space -
> >    in-kernel STP is a great example of how _not_ to do it. :) As a
bonus you'll avoid 90% of the
> >    problems above just by making your own abstractions and using them
for it.
> Please continue.
>
> We do not see any good paths for getting user-space based solutions
> which actually does use the HW offloading accepted upstream. If this
> path exists then we would like to understand it and evaluate it
> properly.
>
> --
> /Horatiu
I think it would help your case if you explained a bit more about the
hw offload primitives you have implemented internally. I believe you
are talking about the frame generation engine in the Ocelot switch
which has 1024 frame slots that are periodically sent based on one of
8 timers. For receive, I believe that the functionality is to offload
the consumption of these periodic frames, and just raise an interrupt
if frames were expected but not received.
For your use case of MRP, it makes perfect sense to have this. I am
just not sure (and not knowledgeable enough in Linux) what this engine
is offloading from the operating system's perspective.

Your justification for implementing MRP in the kernel seems to be that
it's better to make MRP part of the kernel uapi than a configuration
interface for your periodic engine, which in principle I agree with.
I'm just not sure if the offload that you propose will have a trivial
path into the kernel either, so it would make sense for reviewers to
see everything put together first.

Thanks,
-Vladimir

> I think it would help your case if you explained a bit more about
> the hw offload primitives you have implemented internally.
Agreed.

Horatiu, could you also give some references to the frames that need
to be sent. I've no idea what information they need to contain, if the
contents is dynamic, or static, etc.

	 Andrew

Hi Valdimir and Andrew

The 01/10/2020 18:21, Vladimir Oltean wrote:
> I think it would help your case if you explained a bit more about the
> hw offload primitives you have implemented internally. I believe you
> are talking about the frame generation engine in the Ocelot switch
> which has 1024 frame slots that are periodically sent based on one of
> 8 timers. For receive, I believe that the functionality is to offload
> the consumption of these periodic frames, and just raise an interrupt
> if frames were expected but not received.
Yes something like this. But it is worth mention that it is not just about
injecting frames, sequence number needs to be incremented (by HW) etc.
> For your use case of MRP, it makes perfect sense to have this. I am
> just not sure (and not knowledgeable enough in Linux) what this engine
> is offloading from the operating system's perspective.
We will try to make that more clear.
> Your justification for implementing MRP in the kernel seems to be that
> it's better to make MRP part of the kernel uapi than a configuration
> interface for your periodic engine, which in principle I agree with.
> I'm just not sure if the offload that you propose will have a trivial
> path into the kernel either, so it would make sense for reviewers to
> see everything put together first.
You are right. The decision of start by publishing a pure SW implementation with
no HW offload was not the best.

I can do a new RFC that does including the HW offload hooks, and
describe what configurations we do when these hooks are called. The
actual HW which implements these hooks is still not released (and the
SwitchDev driver for this device is still not submitted).
> Horatiu, could you also give some references to the frames that need
> to be sent. I've no idea what information they need to contain, if the
> contents is dynamic, or static, etc.
It is dynamic - but trivial... Here is a dump from WireShark with
annotation on what our HW can update:

Ethernet II, Src: 7a:8b:b1:35:96:e1 (7a:8b:b1:35:96:e1), Dst: Iec_00:00:01
(01:15:4e:00:00:01)
    Destination: Iec_00:00:01 (01:15:4e:00:00:01)
    Source: 7a:8b:b1:35:96:e1 (7a:8b:b1:35:96:e1)
    Type: MRP (0x88e3)
PROFINET MRP MRP_Test, MRP_Common, MRP_End
    MRP_Version: 1
    MRP_TLVHeader.Type: MRP_Test (0x02)
        MRP_TLVHeader.Type: MRP_Test (0x02)
        MRP_TLVHeader.Length: 18
        MRP_Prio: 0x1f40 High priorities
        MRP_SA: 7a:8b:b1:35:96:e1 (7a:8b:b1:35:96:e1)
        MRP_PortRole: Primary ring port (0x0000)
        MRP_RingState: Ring closed (0x0001)
        MRP_Transition: 0x0001
        MRP_TimeStamp [ms]: 0x000cf574             <---------- Updated
automatic
    MRP_TLVHeader.Type: MRP_Common (0x01)
        MRP_TLVHeader.Type: MRP_Common (0x01)
        MRP_TLVHeader.Length: 18
        MRP_SequenceID: 0x00e9                     <---------- Updated
automatic
        MRP_DomainUUID: ffffffff-ffff-ffff-ffff-ffffffffffff
    MRP_TLVHeader.Type: MRP_End (0x00)
        MRP_TLVHeader.Type: MRP_End (0x00)
        MRP_TLVHeader.Length: 0

But all the fields can change, but to change the other fields we need to
interact with the HW. Other SoC may have other capabilities in their
offload. As an example, if the ring becomes open then the fields
MRP_RingState and MRP_Transition need to change and in our case this
requires SW interference.

Would you like a PCAP file as an example? Or do you want a better
description of the frame format.

/Horatiu