Libguestfs - Oct 2020 - Re: virt-sparsify failed (was: [oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 479 - Failure!)

On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > I think this is the right solution - when virt-something tool fails,
> > it should log the reason for the failure - the error that caused the
> > tool to fail. I'm not sure this is easy to do as the failing code
> > run inside a special VM. Maybe the code running in the VM should log
> > the output in a machine readable way, so once an error is detected
> > virt-something can report the error as the reason, without running
> > in debug mode.
>
> All the virt-* tools that I've written have a non-zero exit code and
> print an error message on stderr when they fail.  Errors from inside
> the appliance are propagated to the library and thence to the tool
> correctly.
>
> I think the best thing to do is:
>
>  - spool up stdout + stderr from the tool
>
>  - if the exit code != 0, save the spooled output for analysis
>
>  - if the exit code == 0, discard it (or keep it if you like)
This is what we already do, and the result is not helpful. If you look
at the log message in the previous message, basically the only
info about the error is:

    libguestfs error: guestfs_launch failed

I don't see what we can do with this error message.

If this is an error running qemu, the error message from qemu can help
to understand why qemu failed to start.
> Without LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE stdout + stderr should be
> quite minimal for all tools.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>

On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer
wrote:
> On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones@redhat.com> wrote:
> >
> > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > I think this is the right solution - when virt-something tool
fails,
> > > it should log the reason for the failure - the error that caused
the
> > > tool to fail. I'm not sure this is easy to do as the failing
code
> > > run inside a special VM. Maybe the code running in the VM should
log
> > > the output in a machine readable way, so once an error is
detected
> > > virt-something can report the error as the reason, without
running
> > > in debug mode.
> >
> > All the virt-* tools that I've written have a non-zero exit code
and
> > print an error message on stderr when they fail.  Errors from inside
> > the appliance are propagated to the library and thence to the tool
> > correctly.
> >
> > I think the best thing to do is:
> >
> >  - spool up stdout + stderr from the tool
> >
> >  - if the exit code != 0, save the spooled output for analysis
> >
> >  - if the exit code == 0, discard it (or keep it if you like)
> 
> This is what we already do, and the result is not helpful. If you look
> at the log message in the previous message, basically the only
> info about the error is:
> 
>     libguestfs error: guestfs_launch failed
> 
> I don't see what we can do with this error message.
Right, so in this particular instance the error message would tell us
that you should run libguestfs-test-tool because your qemu/kernel/etc
is broken in some way :-/

There's not a particularly good answer here if you don't want to ever
use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
libguestfs-test-tool if you see any error which matches the substring
/guestfs_launch/ ?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Tue, Oct 13, 2020 at 8:40 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Tue, Oct 13, 2020 at 07:56:29PM +0300, Nir Soffer wrote:
> > On Tue, Oct 13, 2020 at 7:15 PM Richard W.M. Jones
<rjones@redhat.com> wrote:
> > >
> > > On Tue, Oct 13, 2020 at 06:45:42PM +0300, Nir Soffer wrote:
> > > > I think this is the right solution - when virt-something
tool fails,
> > > > it should log the reason for the failure - the error that
caused the
> > > > tool to fail. I'm not sure this is easy to do as the
failing code
> > > > run inside a special VM. Maybe the code running in the VM
should log
> > > > the output in a machine readable way, so once an error is
detected
> > > > virt-something can report the error as the reason, without
running
> > > > in debug mode.
> > >
> > > All the virt-* tools that I've written have a non-zero exit
code and
> > > print an error message on stderr when they fail.  Errors from
inside
> > > the appliance are propagated to the library and thence to the
tool
> > > correctly.
> > >
> > > I think the best thing to do is:
> > >
> > >  - spool up stdout + stderr from the tool
> > >
> > >  - if the exit code != 0, save the spooled output for analysis
> > >
> > >  - if the exit code == 0, discard it (or keep it if you like)
> >
> > This is what we already do, and the result is not helpful. If you look
> > at the log message in the previous message, basically the only
> > info about the error is:
> >
> >     libguestfs error: guestfs_launch failed
> >
> > I don't see what we can do with this error message.
>
> Right, so in this particular instance the error message would tell us
> that you should run libguestfs-test-tool because your qemu/kernel/etc
> is broken in some way :-/
>
> There's not a particularly good answer here if you don't want to
ever
> use LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE, but perhaps you could run
> libguestfs-test-tool if you see any error which matches the substring
> /guestfs_launch/ ?
Another (orthogonal?) option:

Make LIBGUESTFS_DEBUG/LIBGUESTFS_TRACE log elsewhere, not to stdout/err
(e.g. some other file descriptor, or to a file passed via env or whatever).
This way, it might make sense for vdsm to always pass these vars, continue
logging all stdout/err, and log/keep debug/trace logs only on errors.

Best regards,
-- 
Didi