Libguestfs - Oct 2017 - Re: a question about multithreading with libguestfs

Hi Richard!
Maybe the function guestfs_mount_local_run shouldn't
ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and sits
in the loop? What do you think?
If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't know
how to properly remove it from ml generator), fuse_loop_mt works, but I
still don't understand how it worked with fuse_loop (single threaded) when
guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE.
Unfortunately multiple ls -lR to the same directory tree lead to crash
while it seems multiple ls -lR to different directory trees (if not
repeated for a while) don't lead to crash, so I suspect directory cache in
fuse without guard.
Is there a quick way to disable the directory cache in fuse so I can see if
it is directory cache that leads to crash or we have to look at other parts?

Thanks much,

Maxim.

On Tue, Oct 17, 2017 at 11:25 PM, Richard W.M. Jones <rjones@redhat.com>
wrote:
> [Please keep replies on the mailing list so that others can benefit]
>
> On Tue, Oct 17, 2017 at 10:07:31PM +0300, Maxim Kozover wrote:
> > Hi Richard,
> > Unfortunately I achieved a negative result, maybe you could help,
please?
> .
> > I'm using libguestfs-1.36.4 as a base since I changed for myself a
bit
> some
> > detection stuff that you recently moved from C to OCaml and I
can't
> rewrite
> > it immediately.
> >
> > WIth vanilla 1.36.4 just changing fuse_loop to fuse_loop_mt gives
almost
> > immediate crash when the filesystem is accessed.
>
> That's expected because plain 1.36 doesn't support multithreading,
but ...
>
> > I've put the most recent gnulib, changed a bit bootstrap etc, put
the
> > patches from https://www.redhat.com/archives/libguestfs/2017-June/
> > msg00287.html and rebuilt.
>
> ... OK.
>
> > The system seems to work (with single-threaded fuse).
> > Changed fuse_loop to fuse_loop_mt, rebuilt and fuse is stuck on the
first
> > call to appropriate fuse "system call" like
mount_local_getattr in case
> of
> > ls.
> > It is stuck at guestfs_lstatns possibly at ACQUIRE_LOCK_FOR_CURRENT_
> SCOPE,
> > while being called from fuse.
>
> Can you get a stack trace from gdb.  Use the command ‘t a a bt’ to
> show stacks from all threads at the same time.
>
> > What do you think could be a problem? Some lock already taken?
> > Also, maybe less concern for me as I use fuse read-only, is the
directory
> > cache guarded?
>
> Possibly not, I've not really looked at the fuse code w.r.t
> multithreading at all.  Patches welcome as usual.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~
> rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
>

On Wed, Oct 18, 2017 at 01:33:49AM +0300, Maxim Kozover
wrote:
> Hi Richard!
> Maybe the function guestfs_mount_local_run shouldn't
> ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon and
sits
> in the loop? What do you think?
The lock is needed for any access to the guestfs_h structure, so I'm
pretty sure that is not safe.
> If I remove it from guestfs_mount_local_run (in lib/action-1.c, don't
know
> how to properly remove it from ml generator), fuse_loop_mt works, but I
> still don't understand how it worked with fuse_loop (single threaded)
when
> guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE.
> Unfortunately multiple ls -lR to the same directory tree lead to crash
> while it seems multiple ls -lR to different directory trees (if not
> repeated for a while) don't lead to crash, so I suspect directory cache
in
> fuse without guard.
> Is there a quick way to disable the directory cache in fuse so I can see if
> it is directory cache that leads to crash or we have to look at other
parts?
You could try removing the directory cache altogether from the FUSE
code (it may run much more slowly).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Hi Richard!
Finally it seems it works after removing ACQUIRE_LOCK_FOR_CURRENT_SCOPE in
guestfs_mount_local_run and throwing away all directory cache code.
Although I didn't try to mix fuse libguestfs client with other clients, I
think guestfs_mount_local_run shouldn't take that specific lock as the
function won't exit until umount and will sit in the loop. I think it does
nothing libguestfs-related, only fuse loop activation.
What I don't understand is how that lock in guestfs_mount_local_run
didn't
make problems with single threaded fuse_loop.
I'll try to run it a bit more to see if any problems are seen.
If everything is OK (hope not a false success as directory lists are slower
and race condition maybe less likely), then it should be worth to see why
directory cache made problems, maybe guard it with the same lock or
different one?
What do you think, Richard?

Many thanks for your prompt help,

Maxim.

On Wed, Oct 18, 2017 at 1:39 AM, Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Wed, Oct 18, 2017 at 01:33:49AM +0300, Maxim Kozover wrote:
> > Hi Richard!
> > Maybe the function guestfs_mount_local_run shouldn't
> > ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the daemon
and
> sits
> > in the loop? What do you think?
>
> The lock is needed for any access to the guestfs_h structure, so I'm
> pretty sure that is not safe.
>
> > If I remove it from guestfs_mount_local_run (in lib/action-1.c,
don't
> know
> > how to properly remove it from ml generator), fuse_loop_mt works, but
I
> > still don't understand how it worked with fuse_loop (single
threaded)
> when
> > guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE.
> > Unfortunately multiple ls -lR to the same directory tree lead to crash
> > while it seems multiple ls -lR to different directory trees (if not
> > repeated for a while) don't lead to crash, so I suspect directory
cache
> in
> > fuse without guard.
> > Is there a quick way to disable the directory cache in fuse so I can
see
> if
> > it is directory cache that leads to crash or we have to look at other
> parts?
>
> You could try removing the directory cache altogether from the FUSE
> code (it may run much more slowly).
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~
> rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>

Hi Richard,
Can you tell, please, if the daemon at the appliance side is multithreaded
as well and is able to execute several simultaneous reads, for example? If
yes, how many threads it uses? Or every request starts its own thread?

Thanks much,

Maxim.

On Oct 18, 2017 2:17 AM, "Maxim Kozover" <maximkoz@gmail.com>
wrote:
> Hi Richard!
> Finally it seems it works after removing ACQUIRE_LOCK_FOR_CURRENT_SCOPE
> in guestfs_mount_local_run and throwing away all directory cache code.
> Although I didn't try to mix fuse libguestfs client with other clients,
I
> think guestfs_mount_local_run shouldn't take that specific lock as the
> function won't exit until umount and will sit in the loop. I think it
does
> nothing libguestfs-related, only fuse loop activation.
> What I don't understand is how that lock in guestfs_mount_local_run
didn't
> make problems with single threaded fuse_loop.
> I'll try to run it a bit more to see if any problems are seen.
> If everything is OK (hope not a false success as directory lists are
> slower and race condition maybe less likely), then it should be worth to
> see why directory cache made problems, maybe guard it with the same lock or
> different one?
> What do you think, Richard?
>
> Many thanks for your prompt help,
>
> Maxim.
>
> On Wed, Oct 18, 2017 at 1:39 AM, Richard W.M. Jones
<rjones@redhat.com>
> wrote:
>
>> On Wed, Oct 18, 2017 at 01:33:49AM +0300, Maxim Kozover wrote:
>> > Hi Richard!
>> > Maybe the function guestfs_mount_local_run shouldn't
>> > ACQUIRE_LOCK_FOR_CURRENT_SCOPE as it doesn't talk with the
daemon and
>> sits
>> > in the loop? What do you think?
>>
>> The lock is needed for any access to the guestfs_h structure, so
I'm
>> pretty sure that is not safe.
>>
>> > If I remove it from guestfs_mount_local_run (in lib/action-1.c,
don't
>> know
>> > how to properly remove it from ml generator), fuse_loop_mt works,
but I
>> > still don't understand how it worked with fuse_loop (single
threaded)
>> when
>> > guestfs_mount_local_run did ACQUIRE_LOCK_FOR_CURRENT_SCOPE.
>> > Unfortunately multiple ls -lR to the same directory tree lead to
crash
>> > while it seems multiple ls -lR to different directory trees (if
not
>> > repeated for a while) don't lead to crash, so I suspect
directory cache
>> in
>> > fuse without guard.
>> > Is there a quick way to disable the directory cache in fuse so I
can
>> see if
>> > it is directory cache that leads to crash or we have to look at
other
>> parts?
>>
>> You could try removing the directory cache altogether from the FUSE
>> code (it may run much more slowly).
>>
>> Rich.
>>
>> --
>> Richard Jones, Virtualization Group, Red Hat
>> http://people.redhat.com/~rjones
>> Read my programming and virtualization blog: http://rwmj.wordpress.com
>> virt-builder quickly builds VMs from scratch
>> http://libguestfs.org/virt-builder.1.html
>>
>
>

On Wed, Oct 18, 2017 at 09:16:38AM +0300, Maxim Kozover
wrote:
> Hi Richard,
> Can you tell, please, if the daemon at the appliance side is multithreaded
> as well and is able to execute several simultaneous reads, for example? If
> yes, how many threads it uses? Or every request starts its own thread?
It's single threaded.  That's why we have a lock on the library side.

We did have a look at making the daemon multithreaded, but it requires
us to stop using virtio-serial and use something else instead (there's
some alternate thing which I've forgotten the name of, but it doesn't
quite work for our use case because it uses global addresses for
ports).  IOW, it's complicated.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org