Cornelia Huck
2022-Apr-25 08:54 UTC
[PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()
On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst at redhat.com> wrote:> On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote: >> This patch tries to implement the synchronize_cbs() for ccw. For the >> vring_interrupt() that is called via virtio_airq_handler(), the >> synchronization is simply done via the airq_info's lock. For the >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per >> device spinlock for irq is introduced ans used in the synchronization >> method. >> >> Cc: Thomas Gleixner <tglx at linutronix.de> >> Cc: Peter Zijlstra <peterz at infradead.org> >> Cc: "Paul E. McKenney" <paulmck at kernel.org> >> Cc: Marc Zyngier <maz at kernel.org> >> Cc: Halil Pasic <pasic at linux.ibm.com> >> Cc: Cornelia Huck <cohuck at redhat.com> >> Signed-off-by: Jason Wang <jasowang at redhat.com> > > > This is the only one that is giving me pause. Halil, Cornelia, > should we be concerned about the performance impact here? > Any chance it can be tested?We can have a bunch of devices using the same airq structure, and the sync cb creates a choke point, same as registering/unregistering. If invoking the sync cb is a rare operation (same as (un)registering), it should not affect interrupt processing for other devices too much, but it really should be rare. For testing, you would probably want to use a setup with many devices that share the same airq area (you can fit a lot of devices if they have few queues), generate traffic on the queues, and then do something that triggers the callback (adding/removing a new device in a loop?) I currently don't have such a setup handy; Halil, would you be able to test that?> >> --- >> drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++ >> 1 file changed, 27 insertions(+) >> >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c >> index d35e7a3f7067..c19f07a82d62 100644 >> --- a/drivers/s390/virtio/virtio_ccw.c >> +++ b/drivers/s390/virtio/virtio_ccw.c >> @@ -62,6 +62,7 @@ struct virtio_ccw_device { >> unsigned int revision; /* Transport revision */ >> wait_queue_head_t wait_q; >> spinlock_t lock; >> + spinlock_t irq_lock; >> struct mutex io_lock; /* Serializes I/O requests */ >> struct list_head virtqueues; >> bool is_thinint; >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev) >> return dev_name(&vcdev->cdev->dev); >> } >> >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev) >> +{ >> + struct virtio_ccw_device *vcdev = to_vc_device(vdev); >> + struct airq_info *info = vcdev->airq_info; >> + >> + /* >> + * Synchronize with the vring_interrupt() called by >> + * virtio_ccw_int_handler(). >> + */ >> + spin_lock(&vcdev->irq_lock); >> + spin_unlock(&vcdev->irq_lock); >> + >> + if (info) { >> + /* >> + * Synchronize with the vring_interrupt() with airq indicator >> + */ >> + write_lock(&info->lock); >> + write_unlock(&info->lock); >> + }I think we can make this an either/or operation (devices will either use classic interrupts or adapter interrupts)?>> +} >> + >> static const struct virtio_config_ops virtio_ccw_config_ops = { >> .get_features = virtio_ccw_get_features, >> .finalize_features = virtio_ccw_finalize_features,
Michael S. Tsirkin
2022-Apr-25 13:59 UTC
[PATCH V3 6/9] virtio-ccw: implement synchronize_cbs()
On Mon, Apr 25, 2022 at 10:54:24AM +0200, Cornelia Huck wrote:> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote: > >> This patch tries to implement the synchronize_cbs() for ccw. For the > >> vring_interrupt() that is called via virtio_airq_handler(), the > >> synchronization is simply done via the airq_info's lock. For the > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per > >> device spinlock for irq is introduced ans used in the synchronization > >> method. > >> > >> Cc: Thomas Gleixner <tglx at linutronix.de> > >> Cc: Peter Zijlstra <peterz at infradead.org> > >> Cc: "Paul E. McKenney" <paulmck at kernel.org> > >> Cc: Marc Zyngier <maz at kernel.org> > >> Cc: Halil Pasic <pasic at linux.ibm.com> > >> Cc: Cornelia Huck <cohuck at redhat.com> > >> Signed-off-by: Jason Wang <jasowang at redhat.com> > > > > > > This is the only one that is giving me pause. Halil, Cornelia, > > should we be concerned about the performance impact here? > > Any chance it can be tested? > > We can have a bunch of devices using the same airq structure, and the > sync cb creates a choke point, same as registering/unregistering.BTW can callbacks for multiple VQs run on multiple CPUs at the moment? this patch serializes them on a spinlock.> If > invoking the sync cb is a rare operation (same as (un)registering), it > should not affect interrupt processing for other devices too much, but > it really should be rare. > > For testing, you would probably want to use a setup with many devices > that share the same airq area (you can fit a lot of devices if they have > few queues), generate traffic on the queues, and then do something that > triggers the callback (adding/removing a new device in a loop?) > > I currently don't have such a setup handy; Halil, would you be able to > test that? > > > > >> --- > >> drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++ > >> 1 file changed, 27 insertions(+) > >> > >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > >> index d35e7a3f7067..c19f07a82d62 100644 > >> --- a/drivers/s390/virtio/virtio_ccw.c > >> +++ b/drivers/s390/virtio/virtio_ccw.c > >> @@ -62,6 +62,7 @@ struct virtio_ccw_device { > >> unsigned int revision; /* Transport revision */ > >> wait_queue_head_t wait_q; > >> spinlock_t lock; > >> + spinlock_t irq_lock; > >> struct mutex io_lock; /* Serializes I/O requests */ > >> struct list_head virtqueues; > >> bool is_thinint; > >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev) > >> return dev_name(&vcdev->cdev->dev); > >> } > >> > >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev) > >> +{ > >> + struct virtio_ccw_device *vcdev = to_vc_device(vdev); > >> + struct airq_info *info = vcdev->airq_info; > >> + > >> + /* > >> + * Synchronize with the vring_interrupt() called by > >> + * virtio_ccw_int_handler(). > >> + */ > >> + spin_lock(&vcdev->irq_lock); > >> + spin_unlock(&vcdev->irq_lock); > >> + > >> + if (info) { > >> + /* > >> + * Synchronize with the vring_interrupt() with airq indicator > >> + */ > >> + write_lock(&info->lock); > >> + write_unlock(&info->lock); > >> + } > > I think we can make this an either/or operation (devices will either use > classic interrupts or adapter interrupts)? > > >> +} > >> + > >> static const struct virtio_config_ops virtio_ccw_config_ops = { > >> .get_features = virtio_ccw_get_features, > >> .finalize_features = virtio_ccw_finalize_features,
On Mon, 25 Apr 2022 10:54:24 +0200 Cornelia Huck <cohuck at redhat.com> wrote:> On Mon, Apr 25 2022, "Michael S. Tsirkin" <mst at redhat.com> wrote: > > > On Mon, Apr 25, 2022 at 10:44:15AM +0800, Jason Wang wrote: > >> This patch tries to implement the synchronize_cbs() for ccw. For the > >> vring_interrupt() that is called via virtio_airq_handler(), the > >> synchronization is simply done via the airq_info's lock. For the > >> vring_interrupt() that is called via virtio_ccw_int_handler(), a per > >> device spinlock for irq is introduced ans used in the synchronization > >> method. > >> > >> Cc: Thomas Gleixner <tglx at linutronix.de> > >> Cc: Peter Zijlstra <peterz at infradead.org> > >> Cc: "Paul E. McKenney" <paulmck at kernel.org> > >> Cc: Marc Zyngier <maz at kernel.org> > >> Cc: Halil Pasic <pasic at linux.ibm.com> > >> Cc: Cornelia Huck <cohuck at redhat.com> > >> Signed-off-by: Jason Wang <jasowang at redhat.com> > > > > > > This is the only one that is giving me pause. Halil, Cornelia, > > should we be concerned about the performance impact here? > > Any chance it can be tested? > > We can have a bunch of devices using the same airq structure, and the > sync cb creates a choke point, same as registering/unregistering. If > invoking the sync cb is a rare operation (same as (un)registering), it > should not affect interrupt processing for other devices too much, but > it really should be rare.With the notable difference that the critical section in sync_cb is basically empty, so it should be less intrusive that register/unregister. I would also argue, that since after the reset we (re-)discover our virtqueues and (re-)register adapter interrupts, and thus before or as a part of the reset we probably do an unregister to clean up the adapter interrupts and de-allocate the bits in the info, this should not incur any mayor overhead for the airq case, which is the preferred one. Or am I missing something?> > For testing, you would probably want to use a setup with many devices > that share the same airq area (you can fit a lot of devices if they have > few queues), generate traffic on the queues, and then do something that > triggers the callback (adding/removing a new device in a loop?) > > I currently don't have such a setup handy; Halil, would you be able to > test that?Neither do I. I would also have to start from scratch. I guess it would be also sufficient to do a setup with two devices: a nic with many busy queues, and another device that is responsible for generating the resets. Regards, Halil> > > > >> --- > >> drivers/s390/virtio/virtio_ccw.c | 27 +++++++++++++++++++++++++++ > >> 1 file changed, 27 insertions(+) > >> > >> diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c > >> index d35e7a3f7067..c19f07a82d62 100644 > >> --- a/drivers/s390/virtio/virtio_ccw.c > >> +++ b/drivers/s390/virtio/virtio_ccw.c > >> @@ -62,6 +62,7 @@ struct virtio_ccw_device { > >> unsigned int revision; /* Transport revision */ > >> wait_queue_head_t wait_q; > >> spinlock_t lock; > >> + spinlock_t irq_lock; > >> struct mutex io_lock; /* Serializes I/O requests */ > >> struct list_head virtqueues; > >> bool is_thinint; > >> @@ -984,6 +985,27 @@ static const char *virtio_ccw_bus_name(struct virtio_device *vdev) > >> return dev_name(&vcdev->cdev->dev); > >> } > >> > >> +static void virtio_ccw_synchronize_cbs(struct virtio_device *vdev) > >> +{ > >> + struct virtio_ccw_device *vcdev = to_vc_device(vdev); > >> + struct airq_info *info = vcdev->airq_info; > >> + > >> + /* > >> + * Synchronize with the vring_interrupt() called by > >> + * virtio_ccw_int_handler(). > >> + */ > >> + spin_lock(&vcdev->irq_lock); > >> + spin_unlock(&vcdev->irq_lock); > >> + > >> + if (info) { > >> + /* > >> + * Synchronize with the vring_interrupt() with airq indicator > >> + */ > >> + write_lock(&info->lock); > >> + write_unlock(&info->lock); > >> + } > > I think we can make this an either/or operation (devices will either use > classic interrupts or adapter interrupts)?Right, for virtqueue notifications. I second Connie's motion!> > >> +} > >> + > >> static const struct virtio_config_ops virtio_ccw_config_ops = { > >> .get_features = virtio_ccw_get_features, > >> .finalize_features = virtio_ccw_finalize_features, >