Libguestfs - Oct 2017 - Re: a question about multithreading with libguestfs

On Mon, Jun 26, 2017 at 10:27:54AM +0300, Maxim Kozover
wrote:
> I'd like to ask you, please, about multithreading with libguestfs.
> I'm using libguestfs 1.36.4 with Perl front-end.
>
> Should it be possible to have the same libguestfs appliance provide both
> fuse interface and periodically perform some longer APIs like find0?
>
> I mean make one Perl thread do mount_local_run and another thread
> periodically run find0 that executes simultaneously with fuse (hopefully
> not blocking fuse functionality internally until find0 exits)?
Unfortunately not, and as you discovered it will (currently) crash.
> When I'm trying this, the whole thing crashes, so the question is if it
> should be possible, please, and if yes, how could the problem be fixed?
This is really a bug, and something we've been meaning to fix.  What
should happen is there is a lock in the handle which every API takes.

It requires some changes to the generator to implement but is not
necessary hard to do.  In fact I posted patches for it a while back,
but they did not get upstream:

  https://www.redhat.com/archives/libguestfs/2015-June/msg00048.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

FYI I rebased those patches on top of master here:

https://github.com/rwmjones/libguestfs/commits/threads

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Many many thanks, Richard, for your prompt help!

Thanks,

Maxim.

On Jun 27, 2017 1:01 PM, "Richard W.M. Jones"
<rjones@redhat.com> wrote:
>
> FYI I rebased those patches on top of master here:
>
> https://github.com/rwmjones/libguestfs/commits/threads
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~
> rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> Fedora Windows cross-compiler. Compile Windows programs, test, and
> build Windows installers. Over 100 libraries supported.
> http://fedoraproject.org/wiki/MinGW
>

On Mon, Oct 16, 2017 at 10:16:27PM +0300, Maxim Kozover
wrote:
> I mean if after applying multithreading patches I can change fuse_loop to
> fuse_loop_mt...
OK, I've not actually tested how well that works myself, so make sure
you test it properly before doing anything important with it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

[Please keep replies on the mailing list so that others can benefit]

On Tue, Oct 17, 2017 at 10:07:31PM +0300, Maxim Kozover
wrote:
> Hi Richard,
> Unfortunately I achieved a negative result, maybe you could help, please?
.
> I'm using libguestfs-1.36.4 as a base since I changed for myself a bit
some
> detection stuff that you recently moved from C to OCaml and I can't
rewrite
> it immediately.
>
> WIth vanilla 1.36.4 just changing fuse_loop to fuse_loop_mt gives almost
> immediate crash when the filesystem is accessed.
That's expected because plain 1.36 doesn't support multithreading, but
...
> I've put the most recent gnulib, changed a bit bootstrap etc, put the
> patches from https://www.redhat.com/archives/libguestfs/2017-June/
> msg00287.html and rebuilt.
... OK.
> The system seems to work (with single-threaded fuse).
> Changed fuse_loop to fuse_loop_mt, rebuilt and fuse is stuck on the first
> call to appropriate fuse "system call" like mount_local_getattr
in case of
> ls.
> It is stuck at guestfs_lstatns possibly at ACQUIRE_LOCK_FOR_CURRENT_SCOPE,
> while being called from fuse.
Can you get a stack trace from gdb.  Use the command ‘t a a bt’ to
show stacks from all threads at the same time.
> What do you think could be a problem? Some lock already taken?
> Also, maybe less concern for me as I use fuse read-only, is the directory
> cache guarded?
Possibly not, I've not really looked at the fuse code w.r.t
multithreading at all.  Patches welcome as usual.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Hi Richard!
Maybe the function guestfs_mount_local_run shouldn't
ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and sits
in the loop? What do you think?
If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't know
how to properly remove it from ml generator), fuse_loop_mt works, but I
still don't understand how it worked with fuse_loop (single threaded) when
guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE.
Unfortunately multiple ls -lR to the same directory tree lead to crash
while it seems multiple ls -lR to different directory trees (if not
repeated for a while) don't lead to crash, so I suspect directory cache in
fuse without guard.
Is there a quick way to disable the directory cache in fuse so I can see if
it is directory cache that leads to crash or we have to look at other parts?

Thanks much,

Maxim.

On Tue, Oct 17, 2017 at 11:25 PM, Richard W.M. Jones <rjones@redhat.com>
wrote:
> [Please keep replies on the mailing list so that others can benefit]
>
> On Tue, Oct 17, 2017 at 10:07:31PM +0300, Maxim Kozover wrote:
> > Hi Richard,
> > Unfortunately I achieved a negative result, maybe you could help,
please?
> .
> > I'm using libguestfs-1.36.4 as a base since I changed for myself a
bit
> some
> > detection stuff that you recently moved from C to OCaml and I
can't
> rewrite
> > it immediately.
> >
> > WIth vanilla 1.36.4 just changing fuse_loop to fuse_loop_mt gives
almost
> > immediate crash when the filesystem is accessed.
>
> That's expected because plain 1.36 doesn't support multithreading,
but ...
>
> > I've put the most recent gnulib, changed a bit bootstrap etc, put
the
> > patches from https://www.redhat.com/archives/libguestfs/2017-June/
> > msg00287.html and rebuilt.
>
> ... OK.
>
> > The system seems to work (with single-threaded fuse).
> > Changed fuse_loop to fuse_loop_mt, rebuilt and fuse is stuck on the
first
> > call to appropriate fuse "system call" like
mount_local_getattr in case
> of
> > ls.
> > It is stuck at guestfs_lstatns possibly at ACQUIRE_LOCK_FOR_CURRENT_
> SCOPE,
> > while being called from fuse.
>
> Can you get a stack trace from gdb.  Use the command ‘t a a bt’ to
> show stacks from all threads at the same time.
>
> > What do you think could be a problem? Some lock already taken?
> > Also, maybe less concern for me as I use fuse read-only, is the
directory
> > cache guarded?
>
> Possibly not, I've not really looked at the fuse code w.r.t
> multithreading at all.  Patches welcome as usual.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~
> rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
>