similar to: [PATCH nbdkit 0/8] Implement nbdkit API v2 for Python plugins.

v1: https://www.redhat.com/archives/libguestfs/2019-November/msg00153.html v2: - Fix implementation of can_cache. - Add implementation of can_fua. - Add a very thorough test suite which tests every command + flag combination.

v2 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00163.html I pushed patch 1 (with spelling fix), patch 4 and patch 5 since those were previously ACKed on the list. Differences in v3: - Add error checking to PyModule_AddIntConstant. - Use API_VERSION constant instead of function. - Add max API version supported to --dump-plugin output. - Print API_VERSION selected by

v3 was here: https://www.redhat.com/archives/libguestfs/2019-November/msg00209.html In v4: - Rebase on top of current master. Includes various fixes and updates required because of Nir's patches that went into master. - Fix api_version() -> API_VERSION in patch 2 noted previously on the mailing list. Rich.

When I added .can_FOO caching in 1.16, I missed the case that the sh plugin itself was calling .can_flush twice in some situations (in order to default .can_fua). Then right after, I regressed it to call .can_zero twice (in order to default .can_fast_zero). I also missed that .thread_model could use better caching, because at the time, I did not add testsuite coverage. Fix that now. Eric Blake

As mentioned in my reviews, I wonder if we should make our python callbacks look a bit more Pythonic by having kwargs added for each new flag that we want to expose. The idea was first floated here: https://www.redhat.com/archives/libguestfs/2018-April/msg00108.html Note that with my proposal, there is no need for a python script to expose a global API_VERSION variable; new flags are added

I noticed these while working on adding fua support into python, these are independent enough to push now (and I'll have to rebase my 'optional may_trim' patch on top of this). Eric Blake (2): python: Use Py_XDEFREF() python: Simplify calling into plugin plugins/python/python.c | 106 ++++++++---------------------------------------- 1 file changed, 18 insertions(+), 88

First out of our four language bindings to add FUA support (for reference, I added 'zero' support for python, perl, and ruby back in 1.1.13, then Rich had to add it for ocaml in 1.1.20). I tested this heavily under python 2, but for now only compile tested under python 3; I plan to do further testing there and make any tweaks if necessary. I wrote patch 5 early on, but then realized I

--- plugins/python/nbdkit-python-plugin.pod | 19 ++++++- plugins/python/python.c | 75 +++++++++++++++++++++++++ tests/test-python-plugin.py | 8 ++- tests/test_python.py | 73 +++++++++++++++++++++++- 4 files changed, 169 insertions(+), 6 deletions(-) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod

This is working for me now, although possibly only on Python 3.9. Dan suggested PyEval_InitThreads but that was deprecated in Python 3.7. Rich.

This picks up python and ocaml. Some of our languages are lacking a number of bindings (for example, lua and perl lack .extents, so I didn't have anything to copy from), and I felt less comfortable with golang and rust. But for python and ocaml, I was able to test a working implementation. Eric Blake (2): python: Implement .list_exports and friends ocaml: Implement .list_exports and

Fix some things I noticed while reviewing v1, and follow Rich's idea to add a new nbdkit_set_error() utility function with a binding for Python users to request a particular error (rather than being forced to live with whatever stale value is in errno after all the intermediate binding glue code). I could not easily find out how to register a C function callable from perl bindings, and have

Begin the language binding followups to my new .zero callback, since Rich was indeed correct that we want them. I'm more familiar with python and perl (at least to the point that I was able to modify the appropriate example files and prove to myself that the bindings worked), so I've started with those. I'm less familiar with ruby and ocaml, so I've left those for tomorrow (it

On 11/22/19 1:53 PM, Richard W.M. Jones wrote: > To avoid breaking existing plugins, Python plugins wishing to use > version 2 of the API must opt in by declaring: > > def api_version(): > return 2 > > (Plugins which do not do this are assumed to want API version 1). Could we also permit the python code to declare a global variable instead of a function? But a

Passing a memoryview we avoid unneeded copy of the original buffer. On the python side memoryview object can be used for slicing, writing to file, or sending to socket. This may break plugins assuming that the they get a bytearray, but good python code should not care about the type of the buffer, only about the behaviour. Testing with a plugin writing to /dev/null shows 2.7x speedup. Real

To avoid breaking existing plugins, Python plugins wishing to use version 2 of the API must opt in by declaring: def api_version(): return 2 (Plugins which do not do this are assumed to want API version 1). --- plugins/python/example.py | 15 +++- plugins/python/nbdkit-python-plugin.pod | 61 ++++++++++----- plugins/python/python.c | 100

To avoid breaking existing plugins, Python plugins wishing to use version 2 of the API must opt in by declaring: API_VERSION = 2 (Plugins which do not do this are assumed to want API version 1). --- plugins/python/example.py | 14 ++- plugins/python/nbdkit-python-plugin.pod | 56 +++++++----- plugins/python/python.c | 114 ++++++++++++++++++++---- tests/test.py

Thanks: Nir Soffer. Enhances: commit c12e3cb150259f0058727a50341a2d14bb0015a3 --- plugins/python/nbdkit-python-plugin.pod | 3 +- plugins/python/python.c | 39 +++++++++++++++---------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod index d7b6033f..ddae677e 100644 ---

This patch proposes to drop support for Python 2 in nbdkit. Rather than abruptly drop it everywhere, my proposal is that we point people to nbdkit 1.14 (the current stable version) if they want to continue with Python 2 plugins, while gently reminding them of the upcoming Python 2.7 end of life announcement. Libnbd never supported Python 2. Libguestfs in theory supports Python 2 but I dropped

PyObject_CallObject is powerful, but awkward - we have to wrap all arguments into a temporary tuple before using it. Let python do more of the work by using PyObject_CallFunction anywhere that all arguments can be described by a Py_BuildValue() format string, instead of creating one-shot arguments ourselves. In fact, for our py_config(), this makes it easier to not worry about python 2 String vs.

--- plugins/python/nbdkit-python-plugin.pod | 22 +++++++++++-- plugins/python/python.c | 41 +++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 2 deletions(-) diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod index 0fd4dcb..2bc4722 100644 --- a/plugins/python/nbdkit-python-plugin.pod +++