similar to: [Bug #1406906] [PATCH] python: fix segmentation fault when setting non UTF-8 strings

Addressing issue #1406906. When constructing the returned objects, check the return value of Python APIs. This is especially relevant when dealing with non UTF-8 strings. A RuntimeError will be raised on failure pointing to the problematic entry and the field name. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 143

The current need for #ifdef's based on the presence of PyString_FromString makes both the OCaml code of the generator, and the generated C code a mess to read. Hence, add three simple wrappers to make both the OCaml, and C code more readable, and easier to tweak in the future. --- generator/python.ml | 72 ++++++++++++----------------------------------------- python/handle.c | 52

The current need for #ifdef's based on the presence of PyString_FromString makes both the OCaml code of the generator, and the generated C code a mess to read. Hence, add three simple wrappers to make both the OCaml, and C code more readable, and easier to tweak in the future. --- generator/python.ml | 72 ++++++++++++----------------------------------------- python/handle.c | 65

Verify the returned values of Python Object constructor functions are not NULL before adding them to a collection. This is particularly relevant when constructing Unicode strings in Python 3 as they will return NULL if non UTF-8 characters are present. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 102 ++++++++++++++++++++++++++---------------

Verify the returned values of Python Object constructor functions are not NULL before adding them to a collection. This is particularly relevant when constructing Unicode strings in Python 3 as they will return NULL if non UTF-8 characters are present. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 102 ++++++++++++++++++++++++++---------------

This series addresses the issue where non UTF8 file names in a guest image lead to libguestfs segfault with Python 3 APIs. The core issue is the APIs are not checking the return value when constructing a new PyObject. Therefore NULL pointers are added to Python collections (lists and dictionaries) crashing the application. Few notes regarding the comments on the previous patch. - Added a

--- generator/generator.ml | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/generator/generator.ml b/generator/generator.ml index 1f2690d..908c5f3 100755 --- a/generator/generator.ml +++ b/generator/generator.ml @@ -2925,11 +2925,7 @@ put_string_list (char * const * const argv) list = PyList_New (argc); for (i = 0; i < argc; ++i) { -#ifdef

When a struct containing FChar fields is defined in generator/structs.ml, the Python code generator will generate code referencing a dirent struct instead of the correct one thus breaking at compile time. This bug was left unnoticed because the dirent struct is the only one using FChar fields. Yet it prevents any other struct from using such fields. Signed-off-by: Matteo Cafasso <noxdafox at

In Python 2 code branches use PyString_FromStringAndSize directly instead of the guestfs_int_py_fromstringsize wrapper (which calls PyString_FromStringAndSize anyway on Python < 3). --- generator/python.ml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/generator/python.ml b/generator/python.ml index 2a1a48fc0..0e1ed20d8 100644 --- a/generator/python.ml +++

EDIT: forgot to add the list to the subject, sorry for the spam. When a struct containing FChar fields is defined in generator/structs.ml, the Python code generator will generate code referencing a dirent struct instead of the correct one thus breaking at compile time. This bug was left unnoticed because the dirent struct is the only one using FChar fields. Yet it prevents any other struct from

So far RBufferOut return values, and FBuffer struct fields are 'str' on all the versions of Python. Python 3 distinguishes between 'str' (unicode strings), and 'bytes', with 'str' no more able to hold arbitrary data. For this reason, switch the return value of RBufferOut functions, and FBuffer struct fields to bytes on Python 3: while this is a potentially

Before this patch, Python would segfault once you pass a non-string key or value to node_set_value. It was also not possible to set bytes on Python 3 as Unicode was assumed (Python 2 was not affected by this). This patch fixes recognition of bytes for Python 3, but in addition it recognizes ints (includes 'long' in Python 2) for DWORD (LE + BE) and QWORDs. For this purpose, a new field

The interesting line is this one ... checking for PyString_AsString in -lc... no and if you compare that to the code, eg in python/guestfs-py.c: #ifdef HAVE_PYSTRING_ASSTRING PyDict_SetItemString (dict, "ftyp", PyString_FromStringAndSize (&dirent->ftyp, 1)); #else PyDict_SetItemString (dict, "ftyp",

We also need to be more careful about PyString_FromString and similar functions returning NULL on failure. Currently we don't check this every time. This commit adds more checks, but is still not complete. --- TODO | 6 ------ generator/python.ml | 19 +++++++++++++++---- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/TODO b/TODO index 075bdc6..666400e

Per discussion on IRC, here are two changes to convert all TABs-as-indentation to spaces. The first one is the fully-automated conversion. However, note that the command mentioned uses a file (the .x-sc* one) that is added only in the following patch. The second patch adds rules to help keep things that way: Document and enforce the new spaces-only indentation policy. * cfg.mk

Make use of functions and types that fit more, and that do the same job: - use PyErr_NoMemory() on malloc failure - use PyErr_SetFromErrno when setting an exception from an errno - throw TypeError if not getting a list when required --- python/handle.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/python/handle.c b/python/handle.c index 88024e1..9746dca 100644 ---

The set_decode_error_handler function allows the User to set the decoding error scheme to be used when non UTF8 characters are encountered in Python 3. The function has no effect in Python 2. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 16 ++++++++++++++++ python/handle.c | 18 ++++++++++++++++-- python/t/test830RHBZ1406906.py |

For a very long time we have maintained two sets of utility functions, in mllib/common_utils.ml and generator/utils.ml. This changes things so that the same set of utility functions can be shared with both directories. It's not possible to use common_utils.ml directly in the generator because it provides several functions that use modules outside the OCaml stdlib. Therefore we add some

The Python 3 bindings currently are unable to deal with non UTF8 characters. This series continues what proposed in RHBZ#1406906. A new function 'set_decode_error_handler' allows the User to specify how to deal with decoding errors. The default behaviour will be raising a UnicodeDecodeError. If the handler is changed to 'surrogateescape', non UTF8 characters will be escaped in a

So far RBufferOut return values are 'str' on all the versions of Python. Python 3 distinguishes between 'str' (unicode strings), and 'bytes', with 'str' no more able to hold arbitrary data. For this reason, switch the return value of RBufferOut functions to bytes on Python 3: while this is a potentially incompatibile change, this is the only way to handle safely