Markus Armbruster
2020-Mar-17 11:54 UTC
[Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
This series extends QMP introspection to cover deprecation.
Additionally, new option -compat lets you configure what to do when
deprecated interfaces get used. This is intended for testing users of
the management interfaces. It is experimental.
-compat deprecated-input=<in-policy> configures what to do when
deprecated input is received. Available policies:
* accept: Accept deprecated commands and arguments (default)
* reject: Reject them
* crash: Crash
-compat deprecated-output=<out-policy> configures what to do when
deprecated output is sent. Available output policies:
* accept: Emit deprecated command results and events (default)
* hide: Suppress them
For now, -compat covers only deprecated syntactic aspects of QMP. We
may want to extend it to cover semantic aspects, CLI, and experimental
features.
PATCH 01-04: Documentation fixes
PATCH 05-10: Test improvements
PATCH 11-24: Add feature flags to remaining user-defined types and to
struct members
PATCH 25-26: New special feature 'deprecated', visible in
introspection
PATCH 27-34: New -compat to set policy for handling stuff marked with
feature 'deprecated'
v4:
PATCH 05+07: Temporary memory leak plugged [Marc-André]
PATCH 23: Rewritten [Marc-André]
PATCH 24: Comment typo [Marc-André]
PATCH 30: Memory leaks plugged
v3:
* Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST
conversion and code motion
* PATCH 28-29: Old PATCH 28 split up to ease review
* PATCH 30-31: New
* PATCH 32-33: Old PATCH 29 split up to ease review
Comparison to RFC (24 Oct 2019):
* Cover arguments and results in addition to commands and events
* Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the
response to a deprecated command" dropped
See also last item of
Subject: Minutes of KVM Forum BoF on deprecating stuff
Date: Fri, 26 Oct 2018 16:03:51 +0200
Message-ID: <87mur0ls8o.fsf@dusky.pond.sub.org>
https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html
Cc: Lukáš Doktor <ldoktor@redhat.com>
Cc: libguestfs@redhat.com
Cc: libvir-list@redhat.com
Cc: Daniel P. Berrange <berrange@redhat.com>
Cc: Peter Krempa <pkrempa@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Markus Armbruster (34):
qemu-doc: Belatedly document QMP command arg & result deprecation
qapi: Belatedly update doc comment for @wait deprecation
docs/devel/qapi-code-gen: Clarify allow-oob introspection
docs/devel/qapi-code-gen: Document 'features' introspection
tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers
tests/test-qmp-cmds: Check responses more thoroughly
tests/test-qmp-cmds: Simplify test data setup
tests/test-qmp-event: Simplify test data setup
tests/test-qmp-event: Use qobject_is_equal()
tests/test-qmp-event: Check event is actually emitted
qapi/schema: Clean up around QAPISchemaEntity.connect_doc()
qapi: Add feature flags to remaining definitions
qapi: Consistently put @features parameter right after @ifcond
qapi/introspect: Rename *qlit* to reduce confusion
qapi/introspect: Factor out _make_tree()
qapi/schema: Change _make_features() to a take feature list
qapi/schema: Reorder classes so related ones are together
qapi/schema: Rename QAPISchemaObjectType{Variant,Variants}
qapi/schema: Call QAPIDoc.connect_member() in just one place
qapi: Add feature flags to struct members
qapi: Inline do_qmp_dispatch() into qmp_dispatch()
qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP
qapi: Simplify how qmp_dispatch() gets the request ID
qapi: Replace qmp_dispatch()'s TODO comment by an explanation
qapi: New special feature flag "deprecated"
qapi: Mark deprecated QMP parts with feature 'deprecated'
qemu-options: New -compat to set policy for deprecated interfaces
qapi: Implement deprecated-output=hide for QMP command results
qapi: Implement deprecated-output=hide for QMP events
qapi: Implement deprecated-output=hide for QMP event data
qapi: Implement deprecated-output=hide for QMP introspection
qapi: Implement deprecated-input=reject for QMP commands
qapi: Implement deprecated-input=reject for QMP command arguments
qapi: New -compat deprecated-input=crash
docs/devel/qapi-code-gen.txt | 79 ++-
docs/system/deprecated.rst | 48 +-
tests/qapi-schema/doc-good.texi | 32 ++
qapi/block-core.json | 48 +-
qapi/block.json | 30 +-
qapi/char.json | 1 +
qapi/compat.json | 52 ++
qapi/control.json | 11 +-
qapi/introspect.json | 28 +-
qapi/machine.json | 34 +-
qapi/migration.json | 36 +-
qapi/misc.json | 13 +-
qapi/qapi-schema.json | 1 +
include/qapi/compat-policy.h | 20 +
include/qapi/qmp/dispatch.h | 1 +
include/qapi/qobject-input-visitor.h | 9 +
include/qapi/qobject-output-visitor.h | 9 +
include/qapi/visitor-impl.h | 3 +
include/qapi/visitor.h | 9 +
monitor/monitor-internal.h | 3 -
monitor/misc.c | 2 -
monitor/qmp-cmds-control.c | 102 +++-
qapi/qapi-visit-core.c | 9 +
qapi/qmp-dispatch.c | 149 +++---
qapi/qobject-input-visitor.c | 29 ++
qapi/qobject-output-visitor.c | 20 +
qemu-storage-daemon.c | 2 -
softmmu/vl.c | 17 +
tests/test-qmp-cmds.c | 249 +++++----
tests/test-qmp-event.c | 203 +++-----
qapi/Makefile.objs | 8 +-
qapi/trace-events | 1 +
qemu-options.hx | 22 +
scripts/qapi/commands.py | 20 +-
scripts/qapi/doc.py | 16 +-
scripts/qapi/events.py | 24 +-
scripts/qapi/expr.py | 14 +-
scripts/qapi/introspect.py | 104 ++--
scripts/qapi/schema.py | 488 ++++++++++--------
scripts/qapi/types.py | 8 +-
scripts/qapi/visit.py | 28 +-
tests/Makefile.include | 1 +
tests/qapi-schema/alternate-base.err | 2 +-
tests/qapi-schema/doc-good.json | 22 +-
tests/qapi-schema/doc-good.out | 18 +
.../qapi-schema/features-deprecated-type.err | 2 +
.../qapi-schema/features-deprecated-type.json | 3 +
.../qapi-schema/features-deprecated-type.out | 0
tests/qapi-schema/qapi-schema-test.json | 51 +-
tests/qapi-schema/qapi-schema-test.out | 48 +-
tests/qapi-schema/test-qapi.py | 26 +-
51 files changed, 1393 insertions(+), 762 deletions(-)
create mode 100644 qapi/compat.json
create mode 100644 include/qapi/compat-policy.h
create mode 100644 tests/qapi-schema/features-deprecated-type.err
create mode 100644 tests/qapi-schema/features-deprecated-type.json
create mode 100644 tests/qapi-schema/features-deprecated-type.out
--
2.21.1
Marc-André Lureau
2020-Mar-17 13:10 UTC
Re: [Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
Hi On Tue, Mar 17, 2020 at 12:55 PM Markus Armbruster <armbru@redhat.com> wrote:> > This series extends QMP introspection to cover deprecation. > Additionally, new option -compat lets you configure what to do when > deprecated interfaces get used. This is intended for testing users of > the management interfaces. It is experimental. > > -compat deprecated-input=<in-policy> configures what to do when > deprecated input is received. Available policies: > > * accept: Accept deprecated commands and arguments (default) > * reject: Reject them > * crash: Crash > > -compat deprecated-output=<out-policy> configures what to do when > deprecated output is sent. Available output policies: > > * accept: Emit deprecated command results and events (default) > * hide: Suppress them > > For now, -compat covers only deprecated syntactic aspects of QMP. We > may want to extend it to cover semantic aspects, CLI, and experimental > features.I suggest to use a qmp- prefix for qmp-related policies.> PATCH 01-04: Documentation fixes > PATCH 05-10: Test improvements > PATCH 11-24: Add feature flags to remaining user-defined types and to > struct members > PATCH 25-26: New special feature 'deprecated', visible in > introspection > PATCH 27-34: New -compat to set policy for handling stuff marked with > feature 'deprecated' > > v4: > PATCH 05+07: Temporary memory leak plugged [Marc-André] > PATCH 23: Rewritten [Marc-André] > PATCH 24: Comment typo [Marc-André] > PATCH 30: Memory leaks plugged > > v3: > * Rebased, non-trivial conflicts in PATCH 01+26+27+34 due to RST > conversion and code motion > * PATCH 28-29: Old PATCH 28 split up to ease review > * PATCH 30-31: New > * PATCH 32-33: Old PATCH 29 split up to ease review > > Comparison to RFC (24 Oct 2019): > * Cover arguments and results in addition to commands and events > * Half-baked "[RFC PATCH 18/19] qapi: Include a warning in the > response to a deprecated command" dropped > > See also last item of > Subject: Minutes of KVM Forum BoF on deprecating stuff > Date: Fri, 26 Oct 2018 16:03:51 +0200 > Message-ID: <87mur0ls8o.fsf@dusky.pond.sub.org> > https://lists.nongnu.org/archive/html/qemu-devel/2018-10/msg05828.html > > Cc: Lukáš Doktor <ldoktor@redhat.com> > Cc: libguestfs@redhat.com > Cc: libvir-list@redhat.com > Cc: Daniel P. Berrange <berrange@redhat.com> > Cc: Peter Krempa <pkrempa@redhat.com> > Cc: Kevin Wolf <kwolf@redhat.com> > > Markus Armbruster (34): > qemu-doc: Belatedly document QMP command arg & result deprecation > qapi: Belatedly update doc comment for @wait deprecation > docs/devel/qapi-code-gen: Clarify allow-oob introspection > docs/devel/qapi-code-gen: Document 'features' introspection > tests/test-qmp-cmds: Factor out qmp_dispatch() test helpers > tests/test-qmp-cmds: Check responses more thoroughly > tests/test-qmp-cmds: Simplify test data setup > tests/test-qmp-event: Simplify test data setup > tests/test-qmp-event: Use qobject_is_equal() > tests/test-qmp-event: Check event is actually emitted > qapi/schema: Clean up around QAPISchemaEntity.connect_doc() > qapi: Add feature flags to remaining definitions > qapi: Consistently put @features parameter right after @ifcond > qapi/introspect: Rename *qlit* to reduce confusion > qapi/introspect: Factor out _make_tree() > qapi/schema: Change _make_features() to a take feature list > qapi/schema: Reorder classes so related ones are together > qapi/schema: Rename QAPISchemaObjectType{Variant,Variants} > qapi/schema: Call QAPIDoc.connect_member() in just one place > qapi: Add feature flags to struct members > qapi: Inline do_qmp_dispatch() into qmp_dispatch() > qapi: Simplify how qmp_dispatch() deals with QCO_NO_SUCCESS_RESP > qapi: Simplify how qmp_dispatch() gets the request ID > qapi: Replace qmp_dispatch()'s TODO comment by an explanation > qapi: New special feature flag "deprecated" > qapi: Mark deprecated QMP parts with feature 'deprecated' > qemu-options: New -compat to set policy for deprecated interfaces > qapi: Implement deprecated-output=hide for QMP command results > qapi: Implement deprecated-output=hide for QMP events > qapi: Implement deprecated-output=hide for QMP event data > qapi: Implement deprecated-output=hide for QMP introspection > qapi: Implement deprecated-input=reject for QMP commands > qapi: Implement deprecated-input=reject for QMP command arguments > qapi: New -compat deprecated-input=crash > > docs/devel/qapi-code-gen.txt | 79 ++- > docs/system/deprecated.rst | 48 +- > tests/qapi-schema/doc-good.texi | 32 ++ > qapi/block-core.json | 48 +- > qapi/block.json | 30 +- > qapi/char.json | 1 + > qapi/compat.json | 52 ++ > qapi/control.json | 11 +- > qapi/introspect.json | 28 +- > qapi/machine.json | 34 +- > qapi/migration.json | 36 +- > qapi/misc.json | 13 +- > qapi/qapi-schema.json | 1 + > include/qapi/compat-policy.h | 20 + > include/qapi/qmp/dispatch.h | 1 + > include/qapi/qobject-input-visitor.h | 9 + > include/qapi/qobject-output-visitor.h | 9 + > include/qapi/visitor-impl.h | 3 + > include/qapi/visitor.h | 9 + > monitor/monitor-internal.h | 3 - > monitor/misc.c | 2 - > monitor/qmp-cmds-control.c | 102 +++- > qapi/qapi-visit-core.c | 9 + > qapi/qmp-dispatch.c | 149 +++--- > qapi/qobject-input-visitor.c | 29 ++ > qapi/qobject-output-visitor.c | 20 + > qemu-storage-daemon.c | 2 - > softmmu/vl.c | 17 + > tests/test-qmp-cmds.c | 249 +++++---- > tests/test-qmp-event.c | 203 +++----- > qapi/Makefile.objs | 8 +- > qapi/trace-events | 1 + > qemu-options.hx | 22 + > scripts/qapi/commands.py | 20 +- > scripts/qapi/doc.py | 16 +- > scripts/qapi/events.py | 24 +- > scripts/qapi/expr.py | 14 +- > scripts/qapi/introspect.py | 104 ++-- > scripts/qapi/schema.py | 488 ++++++++++-------- > scripts/qapi/types.py | 8 +- > scripts/qapi/visit.py | 28 +- > tests/Makefile.include | 1 + > tests/qapi-schema/alternate-base.err | 2 +- > tests/qapi-schema/doc-good.json | 22 +- > tests/qapi-schema/doc-good.out | 18 + > .../qapi-schema/features-deprecated-type.err | 2 + > .../qapi-schema/features-deprecated-type.json | 3 + > .../qapi-schema/features-deprecated-type.out | 0 > tests/qapi-schema/qapi-schema-test.json | 51 +- > tests/qapi-schema/qapi-schema-test.out | 48 +- > tests/qapi-schema/test-qapi.py | 26 +- > 51 files changed, 1393 insertions(+), 762 deletions(-) > create mode 100644 qapi/compat.json > create mode 100644 include/qapi/compat-policy.h > create mode 100644 tests/qapi-schema/features-deprecated-type.err > create mode 100644 tests/qapi-schema/features-deprecated-type.json > create mode 100644 tests/qapi-schema/features-deprecated-type.out > > -- > 2.21.1 >-- Marc-André Lureau
Markus Armbruster
2020-Mar-17 15:32 UTC
Re: [Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
Marc-André Lureau <marcandre.lureau@gmail.com> writes:> Hi > > On Tue, Mar 17, 2020 at 12:55 PM Markus Armbruster <armbru@redhat.com> wrote: >> >> This series extends QMP introspection to cover deprecation. >> Additionally, new option -compat lets you configure what to do when >> deprecated interfaces get used. This is intended for testing users of >> the management interfaces. It is experimental. >> >> -compat deprecated-input=<in-policy> configures what to do when >> deprecated input is received. Available policies: >> >> * accept: Accept deprecated commands and arguments (default) >> * reject: Reject them >> * crash: Crash >> >> -compat deprecated-output=<out-policy> configures what to do when >> deprecated output is sent. Available output policies: >> >> * accept: Emit deprecated command results and events (default) >> * hide: Suppress them >> >> For now, -compat covers only deprecated syntactic aspects of QMP. We >> may want to extend it to cover semantic aspects, CLI, and experimental >> features. > > I suggest to use a qmp- prefix for qmp-related policies.The interface is general. The implemented infrastructure is QAPI-only. Its application is QMP-only. If our CLI was QAPIfied, I'd certainly apply it there, too. I intend to resume exploring CLI QAPIfication "real soon now". Not covering CLI now is a bit like not covering semantic aspects of QMP. Imagine the thing covered CLI. Would we want to have separate -compat deprecated-qmp-input, deprecated-cli-input? I doubt it. I think we want a single policy for all host interfaces. Imagine it covered deprecated semantic aspects of QMP. Would we want to have a separate flag for that? Again, I doubt it. For what it's worth, the interface is documented as experimental.
Markus Armbruster
2020-Mar-17 20:58 UTC
Re: [Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
Markus Armbruster <armbru@redhat.com> writes:> This series extends QMP introspection to cover deprecation. > Additionally, new option -compat lets you configure what to do when > deprecated interfaces get used. This is intended for testing users of > the management interfaces. It is experimental. > > -compat deprecated-input=<in-policy> configures what to do when > deprecated input is received. Available policies: > > * accept: Accept deprecated commands and arguments (default) > * reject: Reject them > * crash: Crash > > -compat deprecated-output=<out-policy> configures what to do when > deprecated output is sent. Available output policies: > > * accept: Emit deprecated command results and events (default) > * hide: Suppress them > > For now, -compat covers only deprecated syntactic aspects of QMP. We > may want to extend it to cover semantic aspects, CLI, and experimental > features. > > PATCH 01-04: Documentation fixes > PATCH 05-10: Test improvements > PATCH 11-24: Add feature flags to remaining user-defined types and to > struct members > PATCH 25-26: New special feature 'deprecated', visible in > introspection > PATCH 27-34: New -compat to set policy for handling stuff marked with > feature 'deprecated'Queued PATCH 01-26. Thanks for the reviews that made this possible! PATCH 27-34 will have to wait for 5.1.
Peter Krempa
2020-May-04 08:24 UTC
Re: [Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
On Tue, Mar 17, 2020 at 12:54:25 +0100, Markus Armbruster wrote:> This series extends QMP introspection to cover deprecation. > Additionally, new option -compat lets you configure what to do when > deprecated interfaces get used. This is intended for testing users of > the management interfaces. It is experimental. > > -compat deprecated-input=<in-policy> configures what to do when > deprecated input is received. Available policies: > > * accept: Accept deprecated commands and arguments (default) > * reject: Reject them > * crash: CrashI've noticed that the 'crash' option doesn't manage to log the reason to stderr. Relevant section of libvirt's log file which agregates the stderr/out 2020-04-30 13:24:22.006+0000: 2072883: debug : virCommandHandshakeChild:418 : Handshake with parent is done char device redirected to /dev/pts/0 (label charserial0) 2020-04-30 13:24:31.879+0000: Domain id=4 is tainted: custom-monitor 2020-04-30 13:24:32.330+0000: shutting down, reason=crashed 'handshake' line is last of libvirt's messages pre-start of the qemu process. 'char device redirected' is reported by qemu. 'domain is tainted' is added by libvirt when I've issued the deprecated API via virsh qemu-monitor-command. 'reason=crashed' is added by libvirts VM shutdown hanlder.> > -compat deprecated-output=<out-policy> configures what to do when > deprecated output is sent. Available output policies: > > * accept: Emit deprecated command results and events (default) > * hide: Suppress them > > For now, -compat covers only deprecated syntactic aspects of QMP. We > may want to extend it to cover semantic aspects, CLI, and experimental > features. > > PATCH 01-04: Documentation fixes > PATCH 05-10: Test improvements > PATCH 11-24: Add feature flags to remaining user-defined types and to > struct members > PATCH 25-26: New special feature 'deprecated', visible in > introspectionThese are cool. I've added support for verifying that any command excercised by the libvirt unit test suite is not deprecated, or we at least know that it is and have a replacement. https://www.redhat.com/archives/libvir-list/2020-April/msg01444.html> PATCH 27-34: New -compat to set policy for handling stuff marked with > feature 'deprecated'While implementing support for this feature I noticed that it's impossible for libvirt to detect that it's available. The idea is to make a developer-centred setting in our config which will enable the compat setting if available and ignore it if not available to prevent us having to fiddle with the settings when testing various qemu versions. Thanks! Peter
Markus Armbruster
2020-May-08 13:01 UTC
Re: [Libguestfs] [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
Peter Krempa <pkrempa@redhat.com> writes:> On Tue, Mar 17, 2020 at 12:54:25 +0100, Markus Armbruster wrote: >> This series extends QMP introspection to cover deprecation. >> Additionally, new option -compat lets you configure what to do when >> deprecated interfaces get used. This is intended for testing users of >> the management interfaces. It is experimental. >> >> -compat deprecated-input=<in-policy> configures what to do when >> deprecated input is received. Available policies: >> >> * accept: Accept deprecated commands and arguments (default) >> * reject: Reject them >> * crash: Crash > > I've noticed that the 'crash' option doesn't manage to log the reason to > stderr. Relevant section of libvirt's log file which agregates the > stderr/out > > 2020-04-30 13:24:22.006+0000: 2072883: debug : virCommandHandshakeChild:418 : Handshake with parent is done > char device redirected to /dev/pts/0 (label charserial0) > 2020-04-30 13:24:31.879+0000: Domain id=4 is tainted: custom-monitor > 2020-04-30 13:24:32.330+0000: shutting down, reason=crashed > > 'handshake' line is last of libvirt's messages pre-start of the qemu > process. 'char device redirected' is reported by qemu. 'domain is > tainted' is added by libvirt when I've issued the deprecated API via > virsh qemu-monitor-command. 'reason=crashed' is added by libvirts VM > shutdown hanlder.I'm up to my ears in QOM right now. I'd like to follow up when I pivot back to QAPI/QMP. Please pester me if that takes too long.>> >> -compat deprecated-output=<out-policy> configures what to do when >> deprecated output is sent. Available output policies: >> >> * accept: Emit deprecated command results and events (default) >> * hide: Suppress them >> >> For now, -compat covers only deprecated syntactic aspects of QMP. We >> may want to extend it to cover semantic aspects, CLI, and experimental >> features. >> >> PATCH 01-04: Documentation fixes >> PATCH 05-10: Test improvements >> PATCH 11-24: Add feature flags to remaining user-defined types and to >> struct members >> PATCH 25-26: New special feature 'deprecated', visible in >> introspection > > These are cool. I've added support for verifying that any command > excercised by the libvirt unit test suite is not deprecated, or we at > least know that it is and have a replacement. > > https://www.redhat.com/archives/libvir-list/2020-April/msg01444.htmlAwesome!>> PATCH 27-34: New -compat to set policy for handling stuff marked with >> feature 'deprecated' > > While implementing support for this feature I noticed that it's > impossible for libvirt to detect that it's available. The idea is to > make a developer-centred setting in our config which will enable the > compat setting if available and ignore it if not available to prevent us > having to fiddle with the settings when testing various qemu versions.Again, I'd like to follow up when I pivot back to QAPI/QMP.
Possibly Parallel Threads
- Re: [PATCH v4 00/34] Configurable policy for handling deprecated interfaces
- [PATCH v3 00/34] Configurable policy for handling deprecated interfaces
- [PATCH v5 0/8] Configurable policy for handling deprecated interfaces
- Having integration tests "test the future" with QEMU
- Re: Does Libvirt's json parser support single quoted string in qmp json string?