Linux Virtualization - Jan 2015 - [PATCH 0/3] virtio_pci: fix DEBUG_KOBJECT_RELEASE

This is based on Sasha's patch, with some tweaks.


Michael S. Tsirkin (2):
  virtio_pci: device-specific release callback
  virtio_pci: document why we defer kfree

Sasha Levin (1):
  virtio_pci: defer kfree until release callback

 drivers/virtio/virtio_pci_common.h |  1 -
 drivers/virtio/virtio_pci_common.c |  9 ---------
 drivers/virtio/virtio_pci_legacy.c | 12 +++++++++++-
 3 files changed, 11 insertions(+), 11 deletions(-)

-- 
MST

It turns out we need to add device-specific code
in release callback.
In particular, we need to free memory allocated in probe.

Move it to virtio_pci_legacy.c.

Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
---
 drivers/virtio/virtio_pci_common.h | 1 -
 drivers/virtio/virtio_pci_common.c | 9 ---------
 drivers/virtio/virtio_pci_legacy.c | 9 +++++++++
 3 files changed, 9 insertions(+), 10 deletions(-)

diff --git a/drivers/virtio/virtio_pci_common.h
b/drivers/virtio/virtio_pci_common.h
index 1584833..24956c5 100644
--- a/drivers/virtio/virtio_pci_common.h
+++ b/drivers/virtio/virtio_pci_common.h
@@ -146,7 +146,6 @@ const char *vp_bus_name(struct virtio_device *vdev);
  * - ignore the affinity request if we're using INTX
  */
 int vp_set_vq_affinity(struct virtqueue *vq, int cpu);
-void virtio_pci_release_dev(struct device *);
 
 int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
 			    const struct pci_device_id *id);
diff --git a/drivers/virtio/virtio_pci_common.c
b/drivers/virtio/virtio_pci_common.c
index cb08a68..557cbcb 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -422,15 +422,6 @@ int vp_set_vq_affinity(struct virtqueue *vq, int cpu)
 	return 0;
 }
 
-void virtio_pci_release_dev(struct device *_d)
-{
-	/*
-	 * No need for a release method as we allocate/free
-	 * all devices together with the pci devices.
-	 * Provide an empty one to avoid getting a warning from core.
-	 */
-}
-
 #ifdef CONFIG_PM_SLEEP
 static int virtio_pci_freeze(struct device *dev)
 {
diff --git a/drivers/virtio/virtio_pci_legacy.c
b/drivers/virtio/virtio_pci_legacy.c
index 6c76f0f..08d1915 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -211,6 +211,15 @@ static const struct virtio_config_ops virtio_pci_config_ops
= {
 	.set_vq_affinity = vp_set_vq_affinity,
 };
 
+static void virtio_pci_release_dev(struct device *_d)
+{
+	/*
+	 * No need for a release method as we allocate/free
+	 * all devices together with the pci devices.
+	 * Provide an empty one to avoid getting a warning from core.
+	 */
+}
+
 /* the PCI probing function */
 int virtio_pci_legacy_probe(struct pci_dev *pci_dev,
 			    const struct pci_device_id *id)
-- 
MST

From: Sasha Levin <sasha.levin at oracle.com>

A struct device which has just been unregistered can live on past the
point at which a driver decides to drop it's initial reference to the
kobject gained on allocation.

This implies that when releasing a virtio device, we can't free a struct
virtio_device until the underlying struct device has been released,
which might not happen immediately on device_unregister().

Unfortunately, this is exactly what virtio pci does:
it has an empty release callback, and frees memory immediately
after unregistering the device.

This causes an easy to reproduce crash if CONFIG_DEBUG_KOBJECT_RELEASE
it enabled.

To fix, free the memory only once we know the device is gone in the release
callback.

Cc: stable at vger.kernel.org
Signed-off-by: Sasha Levin <sasha.levin at oracle.com>
Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
---
 drivers/virtio/virtio_pci_legacy.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/drivers/virtio/virtio_pci_legacy.c
b/drivers/virtio/virtio_pci_legacy.c
index 08d1915..4beaee3 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -213,11 +213,10 @@ static const struct virtio_config_ops
virtio_pci_config_ops = {
 
 static void virtio_pci_release_dev(struct device *_d)
 {
-	/*
-	 * No need for a release method as we allocate/free
-	 * all devices together with the pci devices.
-	 * Provide an empty one to avoid getting a warning from core.
-	 */
+	struct virtio_device *vdev = dev_to_virtio(_d);
+	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+
+	kfree(vp_dev);
 }
 
 /* the PCI probing function */
@@ -311,5 +310,4 @@ void virtio_pci_legacy_remove(struct pci_dev *pci_dev)
 	pci_iounmap(pci_dev, vp_dev->ioaddr);
 	pci_release_regions(pci_dev);
 	pci_disable_device(pci_dev);
-	kfree(vp_dev);
 }
-- 
MST

The reason we defer kfree until release function is because it's a
general rule for kobjects: kfree of the reference counter itself is only
legal in the release function.

Previous patch didn't make this clear, document this in code.

Cc: stable at vger.kernel.org
Signed-off-by: Michael S. Tsirkin <mst at redhat.com>
---
 drivers/virtio/virtio_pci_legacy.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/virtio/virtio_pci_legacy.c
b/drivers/virtio/virtio_pci_legacy.c
index 4beaee3..a5486e6 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -216,6 +216,9 @@ static void virtio_pci_release_dev(struct device *_d)
 	struct virtio_device *vdev = dev_to_virtio(_d);
 	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
 
+	/* As struct device is a kobject, it's not safe to
+	 * free the memory (including the reference counter itself)
+	 * until it's release callback. */
 	kfree(vp_dev);
 }
 
-- 
MST

On Sun, Jan 04, 2015 at 06:02:13PM +0200, Michael S. Tsirkin
wrote:
> This is based on Sasha's patch, with some tweaks.
> 

Sasha, I would appreciate it if you can send
a Tested-by tag to confirm this + the del_vqs
idempotent patches are sufficient to address
all issues you have with 3.19.


> Michael S. Tsirkin (2):
>   virtio_pci: device-specific release callback
>   virtio_pci: document why we defer kfree
> 
> Sasha Levin (1):
>   virtio_pci: defer kfree until release callback
> 
>  drivers/virtio/virtio_pci_common.h |  1 -
>  drivers/virtio/virtio_pci_common.c |  9 ---------
>  drivers/virtio/virtio_pci_legacy.c | 12 +++++++++++-
>  3 files changed, 11 insertions(+), 11 deletions(-)
> 
> -- 
> MST
> 
> _______________________________________________
> Virtualization mailing list
> Virtualization at lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

On 01/04/2015 11:21 AM, Michael S. Tsirkin wrote:
> On Sun, Jan 04, 2015 at 06:02:13PM +0200, Michael S. Tsirkin wrote:
>> > This is based on Sasha's patch, with some tweaks.
>> > 
> 
> Sasha, I would appreciate it if you can send
> a Tested-by tag to confirm this + the del_vqs
> idempotent patches are sufficient to address
> all issues you have with 3.19.
I've tested the series, which fixed the issues I've reported:

	Tested-by: Sasha Levin <sasha.levin at oracle.com>


Thanks,
Sasha