Virtualization - Jan 2017 - GFP_REPEAT usage in vhost_net_open resp. vhost_vsock_dev_open

Hi Michael,
I am currently cleaning up opencoded kmalloc with vmalloc fallback users
[1] and my current kvmalloc_node helper doesn't support GFP_REPEAT
because there are no users which would need it. At least that's what I
thought until I've encountered vhost_vsock_dev_open resp.
vhost_vsock_dev_open which are trying to use GFP_REPEAT for kmalloc.
23cc5a991c7a ("vhost-net: extend device allocation to vmalloc")
explains
the motivation as follows:
"
As vmalloc() adds overhead on a critical network path, add __GFP_REPEAT
to kzalloc() flags to do this fallback only when really needed.
"

I am wondering whether vmalloc adds more overhead than GFP_REPEAT which
can get pretty costly for order-4 allocation which will be used here as
struct vhost_net seems to be 36104 (at least in with my config). Have
you ever measured the difference?

So I am just trying to understand whether we should teach kvmalloc_node
to understand GFP_REPEAT or there is no strong reason to keep the repeat
flag.

[1] http://lkml.kernel.org/r/20170102133700.1734-1-mhocko at kernel.org
-- 
Michal Hocko
SUSE Labs

On Wed, Jan 04, 2017 at 04:08:00PM +0100, Michal Hocko
wrote:
> Hi Michael,
> I am currently cleaning up opencoded kmalloc with vmalloc fallback users
> [1] and my current kvmalloc_node helper doesn't support GFP_REPEAT
> because there are no users which would need it. At least that's what I
> thought until I've encountered vhost_vsock_dev_open resp.
> vhost_vsock_dev_open which are trying to use GFP_REPEAT for kmalloc.
> 23cc5a991c7a ("vhost-net: extend device allocation to vmalloc")
explains
> the motivation as follows:
> "
> As vmalloc() adds overhead on a critical network path, add __GFP_REPEAT
> to kzalloc() flags to do this fallback only when really needed.
> "
> 
> I am wondering whether vmalloc adds more overhead than GFP_REPEAT
Yes but the GFP_REPEAT overhead is during allocation time.
Using vmalloc means all accesses are slowed down.
Allocation is not on data path, accesses are.
> which
> can get pretty costly for order-4 allocation which will be used here as
> struct vhost_net seems to be 36104 (at least in with my config). Have
> you ever measured the difference?
I think it was measureable.
> So I am just trying to understand whether we should teach kvmalloc_node
> to understand GFP_REPEAT or there is no strong reason to keep the repeat
> flag.
> 
> [1] http://lkml.kernel.org/r/20170102133700.1734-1-mhocko at kernel.org
> -- 
> Michal Hocko
> SUSE Labs

On Wed 04-01-17 19:56:42, Michael S. Tsirkin wrote:
> On Wed, Jan 04, 2017 at 04:08:00PM +0100, Michal Hocko wrote:
> > Hi Michael,
> > I am currently cleaning up opencoded kmalloc with vmalloc fallback
users
> > [1] and my current kvmalloc_node helper doesn't support GFP_REPEAT
> > because there are no users which would need it. At least that's
what I
> > thought until I've encountered vhost_vsock_dev_open resp.
> > vhost_vsock_dev_open which are trying to use GFP_REPEAT for kmalloc.
> > 23cc5a991c7a ("vhost-net: extend device allocation to
vmalloc") explains
> > the motivation as follows:
> > "
> > As vmalloc() adds overhead on a critical network path, add
__GFP_REPEAT
> > to kzalloc() flags to do this fallback only when really needed.
> > "
> > 
> > I am wondering whether vmalloc adds more overhead than GFP_REPEAT
> 
> Yes but the GFP_REPEAT overhead is during allocation time.
> Using vmalloc means all accesses are slowed down.
> Allocation is not on data path, accesses are.
OK, that wasn't clear to me. Thanks for the clarification. If the access
path can compensate the allocation cost then I agree that GFP_REPEAT
makes a lot of sense. I will cook up a patch to allow GFP_REPEAT in the
current kvmalloc_node and convert vhost users to it.

Thanks!

-- 
Michal Hocko
SUSE Labs