freebsd stable - Feb 2016 - 82576 + NETMAP + VLAN

On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote:
> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto:
> > Can you look also on second issue?
> >
> > PS: What need from me? May be open PR?
> 
> May you provide some example code that triggers the issue?
This is about 700 lines of code (not very clear), may be I can describe it?
First: this is thread code. I think it impotant point.
All sync with netmap go throw kevent() interface.

Main thread got information about two NIC (determine number of rings)
and spawn 1 balancer thread and NCPU worker thread.
All thread pinned to cores.

Balancer thread open all rings on both NIC separate (i.e. open 6+6
NETMAP fds for hardware rings) and allocate on first NIC and open
NCPU*2 master NETMAP pipes.
All fds register in kevent for read event as

EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx);

and for future write event as

EV_SET(kp, fd, EVFILT_WRITE, EV_ADD  | EV_DISABLE, 0, 0, 0);

after write occur next event will be added:

EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0);

Each worker thread open two slave NETMAP pipes in same maner as above.

Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside
master pipe.
Worker got packet from inside slave pipe and zero-copy to outside slave pipe.
Balancer got  packet from outside master pipe and dispatch (zero-copy) to
outside hardware ring.
And vise versa.

After program start I have deadlock instantly (with WITNESS enabled kernel).
W/o WITNESS deadloc occur only under load.

W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets.
NETMAP don't forward incoming packet to program when open and attach to
NETMAP do in main thread.

Il 10/02/2016 14:53, Slawa Olhovchenkov ha scritto:
> On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote:
>
>> Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto:
>>> Can you look also on second issue?
>>>
>>> PS: What need from me? May be open PR?
>>
>> May you provide some example code that triggers the issue?
>
> This is about 700 lines of code (not very clear), may be I can describe it?
I just need some code to trigger the problem locally. Don't worry about 
the clarity and the line count, unless you cannot share the code for 
other reasons.

Cheers,
Giuseppe
> First: this is thread code. I think it impotant point.
> All sync with netmap go throw kevent() interface.
>
> Main thread got information about two NIC (determine number of rings)
> and spawn 1 balancer thread and NCPU worker thread.
> All thread pinned to cores.
>
> Balancer thread open all rings on both NIC separate (i.e. open 6+6
> NETMAP fds for hardware rings) and allocate on first NIC and open
> NCPU*2 master NETMAP pipes.
> All fds register in kevent for read event as
>
> EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx);
>
> and for future write event as
>
> EV_SET(kp, fd, EVFILT_WRITE, EV_ADD  | EV_DISABLE, 0, 0, 0);
>
> after write occur next event will be added:
>
> EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0);
>
> Each worker thread open two slave NETMAP pipes in same maner as above.
>
> Balancer got packet from inside hardware ring, dispatch (zero-copy) to
inside master pipe.
> Worker got packet from inside slave pipe and zero-copy to outside slave
pipe.
> Balancer got  packet from outside master pipe and dispatch (zero-copy) to
outside hardware ring.
> And vise versa.
>
> After program start I have deadlock instantly (with WITNESS enabled
kernel).
> W/o WITNESS deadloc occur only under load.
>
> W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets.
> NETMAP don't forward incoming packet to program when open and attach to
NETMAP do in main thread.
>

-- 
Dr. Ing. Giuseppe Lettieri
Dipartimento di Ingegneria della Informazione
Universita' di Pisa
Largo Lucio Lazzarino 1, 56122 Pisa - Italy
Ph. : (+39) 050-2217.649 (direct) .599 (switch)
Fax : (+39) 050-2217.600
e-mail: g.lettieri at iet.unipi.it