Linux Virtualization - Jan 2014 - [PATCH net-next 2/3] virtio-net: use per-receive queue page frag alloc for mergeable bufs

On Thu, Dec 26, 2013 at 02:00:58PM -0800, Eric Dumazet
wrote:
> On Thu, 2013-12-26 at 23:37 +0200, Michael S. Tsirkin wrote:
> 
> > Interesting. But if we can't allocate a buffer how can we
> > do network processing?
> 
> How typical NIC drivers handle this case ?
> 
> Answer : nothing special should happen, we drop incoming traffic,
> and make sure the driver recovers properly. (like not NULL deref or
> crazy things like that)
> 
> Why virtio_net should be different ?
Basically yes, we could start dropping packets immediately
once GFP_ATOMIC allocations fail and repost the buffer to host,
and hope memory is available by the time we get the next interrupt.
But we wanted host to have visibility into the fact that
we are out of memory and packets are dropped, so we did not want to
repost.
If we don't repost how do we know memory is finally available?
We went for a timer based workqueue thing.
What do you suggest?


> > 
> > If we can reproduce the problem, we can maybe move
> > allocation out of napi disabled section, but then
> > we'll need to add more locking.
> 
> More exactly, use appropriate locking ;)
>

On Wed, 2014-01-08 at 19:21 +0200, Michael S. Tsirkin wrote:
> Basically yes, we could start dropping packets immediately
> once GFP_ATOMIC allocations fail and repost the buffer to host,
> and hope memory is available by the time we get the next interrupt.
> But we wanted host to have visibility into the fact that
> we are out of memory and packets are dropped, so we did not want to
> repost.
bufferbloat alert :)
> If we don't repost how do we know memory is finally available?
> We went for a timer based workqueue thing.
> What do you suggest?

In normal networking land, when a host A sends frames to host B,
nothing prevents A to pause the traffic to B if B is dropping packets
under stress.

A physical NIC do not use a workqueue to refill its RX queue but uses
the following strategy :

0) Pre filling of RX ring buffer with N frames. This can use GFP_KERNEL
   allocations with all needed (sleep/retry/shout) logic...
1) IRQ is handled.
2) Can we allocate a new buffer (GFP_ATOMIC) ?
   If yes, we accept the frame,
      and post the new buffer for the 'next frame'
   If no, we drop the frame and recycle the memory for next round.

On Wed, Jan 08, 2014 at 10:09:47AM -0800, Eric Dumazet
wrote:
> On Wed, 2014-01-08 at 19:21 +0200, Michael S. Tsirkin wrote:
> 
> > Basically yes, we could start dropping packets immediately
> > once GFP_ATOMIC allocations fail and repost the buffer to host,
> > and hope memory is available by the time we get the next interrupt.
> 
> > But we wanted host to have visibility into the fact that
> > we are out of memory and packets are dropped, so we did not want to
> > repost.
> 
> bufferbloat alert :)
>
I guess you are saying we never need to signal host/device
that we are out of memory, it's enough that packets are dropped?
It seemed like a useful thing for hypervisor to know about on general
principles, even though I don't think kvm uses this info at this point.
> > If we don't repost how do we know memory is finally available?
> > We went for a timer based workqueue thing.
> > What do you suggest?
> 
> 
> In normal networking land, when a host A sends frames to host B,
> nothing prevents A to pause the traffic to B if B is dropping packets
> under stress.
> 
> A physical NIC do not use a workqueue to refill its RX queue but uses
> the following strategy :
> 
> 0) Pre filling of RX ring buffer with N frames. This can use GFP_KERNEL
>    allocations with all needed (sleep/retry/shout) logic...
> 1) IRQ is handled.
> 2) Can we allocate a new buffer (GFP_ATOMIC) ?
>    If yes, we accept the frame,
>       and post the new buffer for the 'next frame'
>    If no, we drop the frame and recycle the memory for next round.
> 
Exactly, this is what I tried to describe in the part that
you have snipped out - but this means queue is always full.

Also, I wonder whether allocating before passing
frame to the stack might slow us down a tiny bit e.g. if an application
is polling this socket on another CPU.

Maybe a slightly better strategy is to do the above when queue depth
is running low. E.g. when queue is 3/4 empty, try
allocating before giving frames to net core,
and recycle buffers on error.

Not sure how much of a win this is.

-- 
MST

From: Eric Dumazet <eric.dumazet at gmail.com>
Date: Wed, 08 Jan 2014 10:09:47 -0800
> A physical NIC do not use a workqueue to refill its RX queue but uses
> the following strategy :
> 
> 0) Pre filling of RX ring buffer with N frames. This can use GFP_KERNEL
>    allocations with all needed (sleep/retry/shout) logic...
> 1) IRQ is handled.
> 2) Can we allocate a new buffer (GFP_ATOMIC) ?
>    If yes, we accept the frame,
>       and post the new buffer for the 'next frame'
>    If no, we drop the frame and recycle the memory for next round.
+1

On 01/08/2014 10:09 AM, Eric Dumazet wrote:
> In normal networking land, when a host A sends frames to host B,
> nothing prevents A to pause the traffic to B if B is dropping packets
> under stress.
>
> A physical NIC do not use a workqueue to refill its RX queue but uses
> the following strategy :
>
> 0) Pre filling of RX ring buffer with N frames. This can use GFP_KERNEL
>     allocations with all needed (sleep/retry/shout) logic...
> 1) IRQ is handled.
> 2) Can we allocate a new buffer (GFP_ATOMIC) ?
>     If yes, we accept the frame,
>        and post the new buffer for the 'next frame'
>     If no, we drop the frame and recycle the memory for next round.
          and increment a suitably specific statistic so someone trying 
to diagnose performance/other problems can know we dropped the frame.

rick jones