Libguestfs - Mar 2020 - Re: Having integration tests "test the future" with QEMU

I understand libguestfs comes with integration tests.  You might be
interested in something I'm developing for QEMU 5.0 to permit "testing
the future".  I'd like to ensure it is actually useful before I
continue.  Let me know what you think.

= Motivation 
When layers above QEMU somehow miss feature deprecation, things continue
to work until the deprecated interface goes away, at which point we have
an entirely avoidable regression.

We've been trying to get better at communicating feature deprecation to
the layers above QEMU.  An obvious first step was systematic
documentation.

However, documentation is all too easy to miss.  Even if you don't miss
it, you may need to continue using the deprecated feature with old
versions of QEMU, and that logic needs to be tested.

I'm working on a way to run QEMU with the deprecated features disabled.
This permits "testing the future".
>From my cover letter:
    This series extends QMP introspection to cover deprecation.
    Additionally, new option -compat lets you configure what to do when
    deprecated interfaces get used.  This is intended for testing users of
    the management interfaces.  It is experimental.

    -compat deprecated-input=<in-policy> configures what to do when
    deprecated input is received.  Available policies:

    * accept: Accept deprecated commands and arguments (default)
    * reject: Reject them
    * crash: Crash

    -compat deprecated-output=<out-policy> configures what to do when
    deprecated output is sent.  Available output policies:

    * accept: Emit deprecated command results and events (default)
    * hide: Suppress them

    For now, -compat covers only deprecated syntactic aspects of QMP.  We
    may want to extend it to cover semantic aspects, CLI, and experimental
    features.

If you want to learn more, check out the last three commit messages in
the series.

= Patches 
    [PATCH v2 00/30] Configurable policy for handling deprecated interfaces
    Message-Id: <20200303163505.32041-1-armbru@redhat.com>
    https://lists.nongnu.org/archive/html/qemu-devel/2020-03/msg00645.html

= Git 
    Tag patchew/20200303163505.32041-1-armbru@redhat.com in repository
    https://github.com/patchew-project/qemu.git

On Wed, Mar 04, 2020 at 03:48:34PM +0100, Markus Armbruster
wrote:
> I understand libguestfs comes with integration tests.  You might be
> interested in something I'm developing for QEMU 5.0 to permit
"testing
> the future".  I'd like to ensure it is actually useful before I
> continue.  Let me know what you think.
> 
> = Motivation > 
> When layers above QEMU somehow miss feature deprecation, things continue
> to work until the deprecated interface goes away, at which point we have
> an entirely avoidable regression.
> 
> We've been trying to get better at communicating feature deprecation to
> the layers above QEMU.  An obvious first step was systematic
> documentation.
> 
> However, documentation is all too easy to miss.  Even if you don't miss
> it, you may need to continue using the deprecated feature with old
> versions of QEMU, and that logic needs to be tested.
> 
> I'm working on a way to run QEMU with the deprecated features disabled.
> This permits "testing the future".
> 
> >From my cover letter:
> 
>     This series extends QMP introspection to cover deprecation.
>     Additionally, new option -compat lets you configure what to do when
>     deprecated interfaces get used.  This is intended for testing users of
>     the management interfaces.  It is experimental.
> 
>     -compat deprecated-input=<in-policy> configures what to do when
>     deprecated input is received.  Available policies:
> 
>     * accept: Accept deprecated commands and arguments (default)
>     * reject: Reject them
>     * crash: Crash
> 
>     -compat deprecated-output=<out-policy> configures what to do when
>     deprecated output is sent.  Available output policies:
> 
>     * accept: Emit deprecated command results and events (default)
>     * hide: Suppress them
> 
>     For now, -compat covers only deprecated syntactic aspects of QMP.  We
>     may want to extend it to cover semantic aspects, CLI, and experimental
>     features.
Yes we would add a -compat option when running qemu (only when running
non-production builds of libguestfs) so that it would fail loudly when
we use a deprecated feature.  This is the best way I've heard of so
far to get advanced warning of deprecated features.

Rich.
> If you want to learn more, check out the last three commit messages in
> the series.
> 
> = Patches > 
>     [PATCH v2 00/30] Configurable policy for handling deprecated interfaces
>     Message-Id: <20200303163505.32041-1-armbru@redhat.com>
>     https://lists.nongnu.org/archive/html/qemu-devel/2020-03/msg00645.html
> 
> = Git > 
>     Tag patchew/20200303163505.32041-1-armbru@redhat.com in repository
>     https://github.com/patchew-project/qemu.git
> 
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

"Richard W.M. Jones" <rjones@redhat.com> writes:
> On Wed, Mar 04, 2020 at 03:48:34PM +0100, Markus Armbruster wrote:
>> I understand libguestfs comes with integration tests.  You might be
>> interested in something I'm developing for QEMU 5.0 to permit
"testing
>> the future".  I'd like to ensure it is actually useful before
I
>> continue.  Let me know what you think.
>> 
>> = Motivation >> 
>> When layers above QEMU somehow miss feature deprecation, things
continue
>> to work until the deprecated interface goes away, at which point we
have
>> an entirely avoidable regression.
>> 
>> We've been trying to get better at communicating feature
deprecation to
>> the layers above QEMU.  An obvious first step was systematic
>> documentation.
>> 
>> However, documentation is all too easy to miss.  Even if you don't
miss
>> it, you may need to continue using the deprecated feature with old
>> versions of QEMU, and that logic needs to be tested.
>> 
>> I'm working on a way to run QEMU with the deprecated features
disabled.
>> This permits "testing the future".
>> 
>> From my cover letter:
>> 
>>     This series extends QMP introspection to cover deprecation.
>>     Additionally, new option -compat lets you configure what to do when
>>     deprecated interfaces get used.  This is intended for testing users
of
>>     the management interfaces.  It is experimental.
>> 
>>     -compat deprecated-input=<in-policy> configures what to do
when
>>     deprecated input is received.  Available policies:
>> 
>>     * accept: Accept deprecated commands and arguments (default)
>>     * reject: Reject them
>>     * crash: Crash
>> 
>>     -compat deprecated-output=<out-policy> configures what to do
when
>>     deprecated output is sent.  Available output policies:
>> 
>>     * accept: Emit deprecated command results and events (default)
>>     * hide: Suppress them
>> 
>>     For now, -compat covers only deprecated syntactic aspects of QMP. 
We
>>     may want to extend it to cover semantic aspects, CLI, and
experimental
>>     features.
>
> Yes we would add a -compat option when running qemu (only when running
> non-production builds of libguestfs) so that it would fail loudly when
> we use a deprecated feature.  This is the best way I've heard of so
> far to get advanced warning of deprecated features.
Excellent.  I'll let you know when my work is upstream.