similar to: python: Avoid leaking py_array and py_args in event callbacks

Version 1 was here: https://listman.redhat.com/archives/libguestfs/2023-February/030732.html (Ignore version 2 which had a mistake, this is version 3) Following Eric's suggestion here: https://listman.redhat.com/archives/libguestfs/2023-February/030746.html let's decrement the reference of py_array right after adding it to the args. (This works even if args fails to be built).

On Thu, Feb 16, 2023 at 03:23:51PM +0000, Richard W.M. Jones wrote: > See-also: https://listman.redhat.com/archives/libguestfs/2023-February/030730.html > Fixes: commit 6ef5837e2d8c5d4d83eff51c0201eb2e08f719de > Thanks: Laszlo Ersek > --- > python/handle.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/python/handle.c b/python/handle.c > index

The event callback gets a buffer parameter which is usually something like a log message. However as it comes from C it is not necessarily well-formed (eg) UTF-8 but could contain any old byte sequence. In the test case provided by the reported, we failed to encode the buffer as str with this error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 137: unexpected

In the case that building the parameters to the Python event callback fails, args was returned as NULL. We immediately tried to call Py_INCREF on this which crashed. Returning NULL means the Python function threw an exception, so print the exception and return (there is no way to return an error here - the event is lost). Reported-by: Yonatan Shtarkman See:

Tested by reverting bbf396fc5562b4f so that the error path is used, and re-running the reproducer supplied by Google. Rich.

On Thu, Feb 16, 2023 at 03:09:02PM +0100, Laszlo Ersek wrote: > On 2/14/23 19:51, Richard W.M. Jones wrote: > > In the case that building the parameters to the Python event callback > > fails, args was returned as NULL. We immediately tried to call > > Py_INCREF on this which crashed. Returning NULL means the Python > > function threw an exception, so print the

A couple of fixes for Python 3 to the Python binding. Unfortunately a behaviour change is needed, although it fixes broken types used. Changes from v1: - handle also FBuffer in structs Pino Toscano (2): python: fix call of Python handlers of events python: change types for RBufferOut/FBuffer with Python 3 (RHBZ#1661871) generator/python.ml | 9 +++++++++ python/handle.c | 3 ++- 2

On 2/17/23 17:52, Eric Blake wrote: > On Thu, Feb 16, 2023 at 03:09:02PM +0100, Laszlo Ersek wrote: >> - Py_BuildValue with the "O" format specifier transfers the new list's >> *sole* reference (= ownership) to the just-built higher-level object "args" > > Reference transfer is done with "N", not "O". That would be an >

A couple of fixes for Python 3 to the Python binding. Unfortunately a behaviour change is needed, although it fixes broken types used. Pino Toscano (2): python: fix call of Python handlers of events python: change return type for RBufferOut with Python 3 (RHBZ#1661871) generator/python.ml | 4 ++++ python/handle.c | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) -- 2.20.1

On Mon, Feb 20, 2023 at 10:45 AM Laszlo Ersek <lersek at redhat.com> wrote: > > On 2/17/23 17:52, Eric Blake wrote: > > On Thu, Feb 16, 2023 at 03:09:02PM +0100, Laszlo Ersek wrote: > > >> - Py_BuildValue with the "O" format specifier transfers the new list's > >> *sole* reference (= ownership) to the just-built higher-level object

All Py_* functions should be protected by the GIL. Otherwise internal python data structures can get corrupted. Move PyGILState_Ensure to the beginning of the block and PyGILState_Release to the bottom. Signed-off-by: Sam Eiderman <sameid@google.com> --- python/handle.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/python/handle.c b/python/handle.c

We were previously generating one instance of the Python closure wrapper per (function * Closure arg). However these wrappers didn't actually differ across functions. We can therefore save a lot of code by only generating one wrapper per closure globally. This reduces the amount of generated code by nearly 25%. Before and after: $ wc -l python/methods.c 3275 python/methods.c $ wc -l

[RFC because generated OCaml code needs the same treatment, but I ran out of time to play with that. At least 'make -C python check' passes, although coverage is not complete yet, as there is no python/t/5XX-pread-structured-callback.py...] [Also RFC because I'm not sure if the use of a record type for 'callback' is the best approach or most idiomatic OCaml - but hey, it

The freeing feature can now be done by associating a free callback with a closure's user_data, so having the valid_flag is no longer necessary and it can be completely removed. This mostly reverts commit 2d9b98e96772e282d51dafac07f16387dadc8afa. --- TODO | 2 - docs/libnbd.pod | 64 ++-------------- examples/glib-main-loop.c |

Posting now that I got something to compile (at the expense of breaking OCaml bindings), but I'm open to ideas on how to improve it. Eric Blake (2): generator: Tweak print_c_arg_list to take alternate first arg RFC: generator: Handle shared callbacks in Python generator/generator | 556 ++++++++++++++++++++++---------------------- 1 file changed, 280 insertions(+), 276 deletions(-) --

Change the way that we deal with freeing closures in language bindings: * The valid_flag is removed (mostly reverting commit 2d9b98e96772e282d51dafac07f16387dadc8afa). * An extra ‘free’ parameter is added to all callback structures. This is called by the library whenever the closure won't be called again (so the user_data can be freed). This is analogous to valid_flag ==

In preparation for closure lifetimes, split up the Closure so it no longer describes a list of closures, but a single callback. This changes the API because functions which take 2 or more closures now pass a separate user_data for each one. --- docs/libnbd.pod | 3 +- examples/strict-structured-reads.c | 2 +- generator/generator | 760

On 8/10/19 12:02 PM, Richard W.M. Jones wrote: > See equivalent change for OCaml in > commit d881d160e1cd9c9964782300a7652ffb4e506c27. > --- > generator/generator | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > The idea makes sense, but I'm not sure if the code is correct: > diff --git a/generator/generator b/generator/generator > index 0107724..0523f0a

Patches 1-4 are basically uncontroversial, straightforward refactoring and IMHO we should just push them. Possibly 1-3 should be squashed together, but I posted them separately so they are easier to review. Patches 5 and 6 together implement OClosure. Patch 5 adds the feature and is simple to understand. Patch 6 changes the Closure completion callbacks into OClosure, but because it doesn't

v1 was here: https://www.redhat.com/archives/libguestfs/2019-July/thread.html#00231 The changes address everything that Eric picked up in his review of the first two patches. I have also added two more patches (4 and 5) which respectively fix docs and change int status -> unsigned status, as discussed. Passes make, check, check-valgrind. Rich.