similar to: [PATCH v6 00/11] vhost: multiple worker support

For vhost-scsi with 3 vqs and a workload like that tries to use those vqs like: fio --filename=/dev/sdb --direct=1 --rw=randrw --bs=4k \ --ioengine=libaio --iodepth=128 --numjobs=3 the single vhost worker thread will become a bottlneck and we are stuck at around 500K IOPs no matter how many jobs, virtqueues, and CPUs are used. To better utilize virtqueues and available CPUs, this patch allows

This patch has the core work flush function take a worker. When we support multiple workers we can then flush each worker during device removal, stoppage, etc. Signed-off-by: Mike Christie <michael.christie at oracle.com> --- drivers/vhost/vhost.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c

On Wed, Apr 5, 2023 at 7:08?AM Mike Christie <michael.christie at oracle.com> wrote: > > On 4/4/23 3:00 AM, Jason Wang wrote: > >> > >> -static void vhost_worker_free(struct vhost_dev *dev) > >> +static void vhost_worker_put(struct vhost_dev *dev, struct vhost_worker *worker) > >> { > >> - struct vhost_worker *worker = dev->worker;

If userspace does VHOST_VSOCK_SET_GUEST_CID before VHOST_SET_OWNER we can race where: 1. thread0 calls vhost_transport_send_pkt -> vhost_work_queue 2. thread1 does VHOST_SET_OWNER which calls vhost_worker_create. 3. vhost_worker_create will set the dev->worker pointer before setting the worker->vtsk pointer. 4. thread0's vhost_work_queue will see the dev->worker pointer is set and

If userspace does VHOST_VSOCK_SET_GUEST_CID before VHOST_SET_OWNER we can race where: 1. thread0 calls vhost_transport_send_pkt -> vhost_work_queue 2. thread1 does VHOST_SET_OWNER which calls vhost_worker_create. 3. vhost_worker_create will set the dev->worker pointer before setting the worker->vtsk pointer. 4. thread0's vhost_work_queue will see the dev->worker pointer is set and

On 5/31/23 10:15 AM, Mike Christie wrote: >>> rcu would work for your case and for what Jason had requested. >> Yeah, so you already have some patches? >> >> Do you want to send it to solve this problem? >> > Yeah, I'll break them out and send them later today when I can retest > rebased patches. > Just one question. Do you core vhost developers

On Tue, Jun 06, 2023 at 12:19:10PM -0500, Mike Christie wrote: > On 6/6/23 4:49 AM, Stefano Garzarella wrote: > > On Mon, Jun 05, 2023 at 01:57:30PM -0500, Mike Christie wrote: > >> If userspace does VHOST_VSOCK_SET_GUEST_CID before VHOST_SET_OWNER we > >> can race where: > >> 1. thread0 calls vhost_transport_send_pkt -> vhost_work_queue > >> 2.

On 6/6/23 4:49 AM, Stefano Garzarella wrote: > On Mon, Jun 05, 2023 at 01:57:30PM -0500, Mike Christie wrote: >> If userspace does VHOST_VSOCK_SET_GUEST_CID before VHOST_SET_OWNER we >> can race where: >> 1. thread0 calls vhost_transport_send_pkt -> vhost_work_queue >> 2. thread1 does VHOST_SET_OWNER which calls vhost_worker_create. >> 3. vhost_worker_create

When switching from kthreads to vhost_tasks two bugs were added: 1. The vhost worker tasks's now show up as processes so scripts doing ps or ps a would not incorrectly detect the vhost task as another process. 2. kthreads disabled freeze by setting PF_NOFREEZE, but vhost tasks's didn't disable or add support for them. To fix both bugs, this switches the vhost task to be thread in the

In the next patches each vq might have different workers so one could have work but others do not. For net, we only want to check specific vqs, so this adds a helper to check if a vq has work pending and converts vhost-net to use it. Signed-off-by: Mike Christie <michael.christie at oracle.com> --- drivers/vhost/net.c | 2 +- drivers/vhost/vhost.c | 6 +++--- drivers/vhost/vhost.h | 2 +-

On Mon, Jun 05, 2023 at 01:57:30PM -0500, Mike Christie wrote: >If userspace does VHOST_VSOCK_SET_GUEST_CID before VHOST_SET_OWNER we >can race where: >1. thread0 calls vhost_transport_send_pkt -> vhost_work_queue >2. thread1 does VHOST_SET_OWNER which calls vhost_worker_create. >3. vhost_worker_create will set the dev->worker pointer before setting >the worker->vtsk

Hi Mike, sorry, but somehow I can't understand this patch... I'll try to read it with a fresh head on Weekend, but for example, On 06/01, Mike Christie wrote: > > static int vhost_task_fn(void *data) > { > struct vhost_task *vtsk = data; > - int ret; > + bool dead = false; > + > + for (;;) { > + bool did_work; > + > + /* mb paired w/

On 6/6/23 7:16 AM, Oleg Nesterov wrote: > On 06/05, Mike Christie wrote: >> >> On 6/5/23 10:10 AM, Oleg Nesterov wrote: >>> On 06/03, michael.christie at oracle.com wrote: >>>> >>>> On 6/2/23 11:15 PM, Eric W. Biederman wrote: >>>> The problem is that as part of the flush the drivers/vhost/scsi.c code >>>> will wait for

The following patches made over Linus's tree fix the 2 bugs: 1. vhost worker task shows up as a process forked from the parent that did VHOST_SET_OWNER ioctl instead of a process under root/kthreadd. This was causing breaking scripts. 2. vhost_tasks didn't disable or add support for freeze requests. The following patches fix these issues by making the vhost_task task a thread under the

On Wed, May 31, 2023 at 11:27:12AM -0500, Mike Christie wrote: >On 5/31/23 10:15 AM, Mike Christie wrote: >>>> rcu would work for your case and for what Jason had requested. >>> Yeah, so you already have some patches? >>> >>> Do you want to send it to solve this problem? >>> >> Yeah, I'll break them out and send them later today when I

On 06/06, Mike Christie wrote: > > On 6/6/23 7:16 AM, Oleg Nesterov wrote: > > On 06/05, Mike Christie wrote: > > > >> So it works like if we were using a kthread still: > >> > >> 1. Userapce thread0 opens /dev/vhost-$something. > >> 2. thread0 does VHOST_SET_OWNER ioctl. This calls vhost_task_create() to > >> create the task_struct

The following patches were made over Linus tree. The patches fix 3 issues: 1. If a user performs LIO LUN unmapping before the endpoint has been cleared then we can end up trying to free a bogus tmf struct if the TMF is still exucuting when we do the unmap. 2. If vhost_scsi_setup_vq_cmds fails we can leave the tpg->vhost_scsi pointer set and we can end up trying to access a freed struct. 3.

vhost_task_create is supposed to return the vhost_task or NULL on failure. This fixes it to return the correct value when the allocation of the struct fails. Fixes: 77feab3c4156 ("vhost_task: Allow vhost layer to use copy_process") # mainline only Reported-by: syzbot+6b27b2d2aba1c80cc13b at syzkaller.appspotmail.com Signed-off-by: Mike Christie <michael.christie at oracle.com> ---

On Fri, Mar 10, 2023 at 2:04?PM Mike Christie <michael.christie at oracle.com> wrote: > > The following patches were made over Linus's tree and apply over next. They > allow the vhost layer to use copy_process instead of using > workqueue_structs to create worker threads for VM's devices. Ok, all these patches looked fine to me from a quick scan - nothing that I reacted

On 3/21/23 12:03 PM, syzbot wrote: > RIP: 0010:vhost_task_start+0x22/0x40 kernel/vhost_task.c:115 > Code: 00 00 00 00 00 0f 1f 00 f3 0f 1e fa 53 48 89 fb e8 c3 67 2c 00 48 8d 7b 70 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 <80> 3c 02 00 75 0a 48 8b 7b 70 5b e9 fe bd 02 00 e8 79 ec 7e 00 eb > RSP: 0018:ffffc90003a9fc38 EFLAGS: 00010207 > RAX: dffffc0000000000 RBX: