Linux Virtualization - Sep 2015 - DEFINE_IDA causing memory leaks? (was Re: [PATCH 1/2] virtio: fix memory leak of virtio ida cache layers)

Hi,

The following patches fix couple of memory leaks in the virtio and
remoteproc cores when using these as modules, and going through a
cycle of insmod and rmmod with at least a device registered with
the corresponding cores in between. I ran into this on our downstream
product kernels on both 3.14 and 4.1 based kernels, and should apply
to the latest kernel as well.

Patches can be picked up independently through the respective trees.

regards
Suman

Suman Anna (2):
  virtio: fix memory leak of virtio ida cache layers
  remoteproc: fix memory leak of remoteproc ida cache layers

 drivers/remoteproc/remoteproc_core.c | 2 ++
 drivers/virtio/virtio.c              | 1 +
 2 files changed, 3 insertions(+)

-- 
2.5.0

The virtio core uses a static ida named virtio_index_ida for
assigning index numbers to virtio devices during registration.
The ida core may allocate some internal idr cache layers and
an ida bitmap upon any ida allocation, and all these layers are
truely freed only upon the ida destruction. The virtio_index_ida
is not destroyed at present, leading to a memory leak when using
the virtio core as a module and atleast one virtio device is
registered and unregistered.

Fix this by invoking ida_destroy() in the virtio core module
exit.

Cc: "Michael S. Tsirkin" <mst at redhat.com>
Signed-off-by: Suman Anna <s-anna at ti.com>
---
 drivers/virtio/virtio.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index b1877d73fa56..7062bb0975a5 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -412,6 +412,7 @@ static int virtio_init(void)
 static void __exit virtio_exit(void)
 {
 	bus_unregister(&virtio_bus);
+	ida_destroy(&virtio_index_ida);
 }
 core_initcall(virtio_init);
 module_exit(virtio_exit);
-- 
2.5.0

The remoteproc core uses a static ida named rproc_dev_index for
assigning an automatic index number to a registered remoteproc.
The ida core may allocate some internal idr cache layers and ida
bitmap upon any ida allocation, and all these layers are truely
freed only upon the ida destruction. The rproc_dev_index ida is
not destroyed at present, leading to a memory leak when using the
remoteproc core as a module and atleast one rproc device is
registered and unregistered.

Fix this by invoking ida_destroy() in the remoteproc core module
exit.

Cc: Ohad Ben-Cohen <ohad at wizery.com>
Signed-off-by: Suman Anna <s-anna at ti.com>
---
 drivers/remoteproc/remoteproc_core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/remoteproc/remoteproc_core.c
b/drivers/remoteproc/remoteproc_core.c
index 8b3130f22b42..9e03d158f411 100644
--- a/drivers/remoteproc/remoteproc_core.c
+++ b/drivers/remoteproc/remoteproc_core.c
@@ -1478,6 +1478,8 @@ module_init(remoteproc_init);
 
 static void __exit remoteproc_exit(void)
 {
+	ida_destroy(&rproc_dev_index);
+
 	rproc_exit_debugfs();
 }
 module_exit(remoteproc_exit);
-- 
2.5.0

On Wed, Sep 16, 2015 at 07:29:17PM -0500, Suman Anna
wrote:
> The virtio core uses a static ida named virtio_index_ida for
> assigning index numbers to virtio devices during registration.
> The ida core may allocate some internal idr cache layers and
> an ida bitmap upon any ida allocation, and all these layers are
> truely freed only upon the ida destruction. The virtio_index_ida
> is not destroyed at present, leading to a memory leak when using
> the virtio core as a module and atleast one virtio device is
> registered and unregistered.
> 
> Fix this by invoking ida_destroy() in the virtio core module
> exit.
> 
> Cc: "Michael S. Tsirkin" <mst at redhat.com>
> Signed-off-by: Suman Anna <s-anna at ti.com>
Interesting.
Will the same apply to e.g. sd_index_ida in drivers/scsi/sd.c
or iscsi_sess_ida in drivers/scsi/scsi_transport_iscsi.c?

If no, why not?

One doesn't generally expect to have to free global variables.
Maybe we should forbid DEFINE_IDA in modules?

James, could you comment on this please?
> ---
>  drivers/virtio/virtio.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> index b1877d73fa56..7062bb0975a5 100644
> --- a/drivers/virtio/virtio.c
> +++ b/drivers/virtio/virtio.c
> @@ -412,6 +412,7 @@ static int virtio_init(void)
>  static void __exit virtio_exit(void)
>  {
>  	bus_unregister(&virtio_bus);
> +	ida_destroy(&virtio_index_ida);
>  }
>  core_initcall(virtio_init);
>  module_exit(virtio_exit);
> -- 
> 2.5.0

Hello,

On Thu, Sep 17, 2015 at 07:15:44AM -0700, James Bottomley
wrote:
> I don't understand why you'd want to forbid DEFINE_IDA ... all it
does
I guess to require the use of explicit init / creation so that it's
clear the data structure needs to be destroyed?
> is pre-initialise a usually static ida structure.  The initialised
> structure will have a NULL bitmap cache that's allocated in the first
> ida_pre_get() ... that all seems to work as expected and no different
> from a dynamically allocated struct ida.  Or are you thinking because
> ida_destory() doesn't set bitmap to NULL, it damages the reuse?  In
> which case I'm not sure there's much benefit to making it reusable,
but
> I suppose we could by adding a memset into ida_destroy().
I don't know.  Data structures which do lazy anything would likely
need explicit destruction and I'm not sure we'd wanna ban static
initialization for all such cases.  Seems like an unnecessary
restriction.

Thanks.

-- 
tejun

On Thu, Sep 17, 2015 at 07:15:44AM -0700, James Bottomley
wrote:
> On Thu, 2015-09-17 at 08:33 +0300, Michael S. Tsirkin wrote:
> > On Wed, Sep 16, 2015 at 07:29:17PM -0500, Suman Anna wrote:
> > > The virtio core uses a static ida named virtio_index_ida for
> > > assigning index numbers to virtio devices during registration.
> > > The ida core may allocate some internal idr cache layers and
> > > an ida bitmap upon any ida allocation, and all these layers are
> > > truely freed only upon the ida destruction. The virtio_index_ida
> > > is not destroyed at present, leading to a memory leak when using
> > > the virtio core as a module and atleast one virtio device is
> > > registered and unregistered.
> > > 
> > > Fix this by invoking ida_destroy() in the virtio core module
> > > exit.
> > > 
> > > Cc: "Michael S. Tsirkin" <mst at redhat.com>
> > > Signed-off-by: Suman Anna <s-anna at ti.com>
> > 
> > Interesting.
> > Will the same apply to e.g. sd_index_ida in drivers/scsi/sd.c
> > or iscsi_sess_ida in drivers/scsi/scsi_transport_iscsi.c?
> > 
> > If no, why not?
> > 
> > One doesn't generally expect to have to free global variables.
> > Maybe we should forbid DEFINE_IDA in modules?
> > 
> > James, could you comment on this please?
> 
> ida is Tejun's baby (cc'd).  However, it does look like without
> ida_destroy() you will leave a cached ida->bitmap dangling because
we're
> trying to be a bit clever in ida_remove() so we cache the bitmap to
> relieve ida_pre_get() of the burden if we would otherwise free it.
> 
> I don't understand why you'd want to forbid DEFINE_IDA ... all it
does
> is pre-initialise a usually static ida structure.  The initialised
> structure will have a NULL bitmap cache that's allocated in the first
> ida_pre_get() ... that all seems to work as expected and no different
> from a dynamically allocated struct ida.  Or are you thinking because
> ida_destory() doesn't set bitmap to NULL, it damages the reuse?  In
> which case I'm not sure there's much benefit to making it reusable,
but
> I suppose we could by adding a memset into ida_destroy().
> 
> James
It's just unusual to have  a descructor without a constructor.
I bet more drivers misuse this AI because of this.
> > > ---
> > >  drivers/virtio/virtio.c | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
> > > index b1877d73fa56..7062bb0975a5 100644
> > > --- a/drivers/virtio/virtio.c
> > > +++ b/drivers/virtio/virtio.c
> > > @@ -412,6 +412,7 @@ static int virtio_init(void)
> > >  static void __exit virtio_exit(void)
> > >  {
> > >  	bus_unregister(&virtio_bus);
> > > +	ida_destroy(&virtio_index_ida);
> > >  }
> > >  core_initcall(virtio_init);
> > >  module_exit(virtio_exit);
> > > -- 
> > > 2.5.0
> > --
> > To unsubscribe from this list: send the line "unsubscribe
linux-scsi" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> 
>

Hello,

On Thu, Sep 17, 2015 at 09:48:37AM -0700, James Bottomley
wrote:
> Well, there's an easy fix for that.  We could have ida_remove()
actually
> free the bitmap and not cache it if it's the last layer.  That way ida
> would naturally empty and we wouldn't need a destructor.   Tejun, would
> that work?
Yeah, that definitely is one way to go about it.  It kinda muddles the
purpose of ida_destroy() tho.  I suppose we can rename it to
idr_remove_all() and then do the same to idr.  I'm not particularly
objecting to all that but what's wrong with just calling idr_destroy()
on exit paths?  If missing the call in modules is an issue, maybe we
can just annotate idr/ida with debugobj?

Thanks.

-- 
tejun

Hello, James.

On Thu, Sep 17, 2015 at 10:58:29AM -0700, James Bottomley
wrote:
> The argument is that we shouldn't have to explicitly destroy a
> statically initialized object, so 
> 
> DEFINE_IDA(someida);
> 
> Should just work without having to explicitly do
> 
> ida_destory(someida);
> 
> somewhere in the exit code.  It's about usage patterns.  Michael's
> argument is that if we can't follow the no destructor pattern for
> DEFINE_IDA() then we shouldn't have it at all, because it's
confusing
> kernel design patterns.  The pattern we would have would be
> 
> struct ida someida:
> 
> ida_init(&someida);
> 
> ...
> 
> ida_destroy(&someida);
> 
> so the object explicitly has a constructor matched to a destructor.
Yeah, I get that.  I'm just not convinced that this matters enough
especially if we can get debugobj/ksan/whatever trip on it.

Thanks.

-- 
tejun