similar to: [PATCH] python: add simple wrappers for PyObject<->string functions

The current need for #ifdef's based on the presence of PyString_FromString makes both the OCaml code of the generator, and the generated C code a mess to read. Hence, add three simple wrappers to make both the OCaml, and C code more readable, and easier to tweak in the future. --- generator/python.ml | 72 ++++++++++++----------------------------------------- python/handle.c | 65

Addressing issue #1406906. When constructing the returned objects, check the return value of Python APIs. This is especially relevant when dealing with non UTF-8 strings. A RuntimeError will be raised on failure pointing to the problematic entry and the field name. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 143

When constructing the returned objects, check the return value of Python APIs. A RuntimeError will be raised on failure pointing to the problematic entry and the field name. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 143 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 46 deletions(-) diff --git

Verify the returned values of Python Object constructor functions are not NULL before adding them to a collection. This is particularly relevant when constructing Unicode strings in Python 3 as they will return NULL if non UTF-8 characters are present. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 102 ++++++++++++++++++++++++++---------------

Verify the returned values of Python Object constructor functions are not NULL before adding them to a collection. This is particularly relevant when constructing Unicode strings in Python 3 as they will return NULL if non UTF-8 characters are present. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 102 ++++++++++++++++++++++++++---------------

--- generator/generator.ml | 23 ++++------------------- 1 file changed, 4 insertions(+), 19 deletions(-) diff --git a/generator/generator.ml b/generator/generator.ml index 1f2690d..908c5f3 100755 --- a/generator/generator.ml +++ b/generator/generator.ml @@ -2925,11 +2925,7 @@ put_string_list (char * const * const argv) list = PyList_New (argc); for (i = 0; i < argc; ++i) { -#ifdef

The set_decode_error_handler function allows the User to set the decoding error scheme to be used when non UTF8 characters are encountered in Python 3. The function has no effect in Python 2. Signed-off-by: Matteo Cafasso <noxdafox@gmail.com> --- generator/python.ml | 16 ++++++++++++++++ python/handle.c | 18 ++++++++++++++++-- python/t/test830RHBZ1406906.py |

On Sun, Apr 26, 2020 at 09:14:03PM +0300, Sam Eiderman wrote: > The python3 bindings create PyUnicode objects from application strings > on the guest (i.e. installed rpm, deb packages). > It is documented that rpm package fields such as description should be > utf8 encoded - however in some cases they are not a valid unicode > string, on SLES11 SP4 the encoding of the description of

gentle ping On Wed, Jun 3, 2020 at 2:52 PM Sam Eiderman <sameid@google.com> wrote: > On Wed, May 13, 2020 at 10:06 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > > On Sun, Apr 26, 2020 at 09:14:03PM +0300, Sam Eiderman wrote: > > > The python3 bindings create PyUnicode objects from application strings > > > on the guest (i.e. installed

On Wed, May 13, 2020 at 10:06 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Sun, Apr 26, 2020 at 09:14:03PM +0300, Sam Eiderman wrote: > > The python3 bindings create PyUnicode objects from application strings > > on the guest (i.e. installed rpm, deb packages). > > It is documented that rpm package fields such as description should be > > utf8 encoded

On Mon, Apr 20, 2020 at 03:37:16PM +0300, Sam Eiderman wrote: > The python3 bindings create unicode objects from application strings > on the guest (i.e. installed rpm, deb packages). > It is documented that rpm package fields such as description should be > utf8 encoded - however in some cases they are not a valid unicode > string, on SLES11 SP4 the following packages fail to be

On Mon, Apr 20, 2020 at 3:38 PM Sam Eiderman <sameid@google.com> wrote: > > The python3 bindings create unicode objects from application strings > on the guest (i.e. installed rpm, deb packages). > It is documented that rpm package fields such as description should be > utf8 encoded - however in some cases they are not a valid unicode > string, So what are they? latin1

The python3 bindings create PyUnicode objects from application strings on the guest (i.e. installed rpm, deb packages). It is documented that rpm package fields such as description should be utf8 encoded - however in some cases they are not a valid unicode string, on SLES11 SP4 the encoding of the description of the following packages is latin1 and they fail to be converted to unicode using

On Sunday, 26 April 2020 20:14:03 CEST Sam Eiderman wrote: > The python3 bindings create PyUnicode objects from application strings > on the guest (i.e. installed rpm, deb packages). > It is documented that rpm package fields such as description should be > utf8 encoded - however in some cases they are not a valid unicode > string, on SLES11 SP4 the encoding of the description of

Hey Pino, Can you search for the previous patches I submitted? I had some discussions regarding this with Daniel and Nir. Thanks! On Tue, Jun 30, 2020 at 11:43 AM Pino Toscano <ptoscano@redhat.com> wrote: > On Sunday, 26 April 2020 20:14:03 CEST Sam Eiderman wrote: > > The python3 bindings create PyUnicode objects from application strings > > on the guest (i.e. installed

We also need to be more careful about PyString_FromString and similar functions returning NULL on failure. Currently we don't check this every time. This commit adds more checks, but is still not complete. --- TODO | 6 ------ generator/python.ml | 19 +++++++++++++++---- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/TODO b/TODO index 075bdc6..666400e

The python3 bindings create unicode objects from application strings on the guest (i.e. installed rpm, deb packages). It is documented that rpm package fields such as description should be utf8 encoded - however in some cases they are not a valid unicode string, on SLES11 SP4 the following packages fail to be converted to unicode using guestfs_int_py_fromstring() (which invokes

Hi Nir, I think latin1, How do you think we should handle latin1 errors then? Replace on latin1 or replace on utf-8? for codec in ["utf8", "latin1"]: try: return decode(b, codec) except: pass return decode(b, "utf8", errors="replace") (Pseudocode, will be implemented in c) On Thu, Apr 23, 2020, 21:34 Nir Soffer <nsoffer@redhat.com>

In Python 2 code branches use PyString_FromStringAndSize directly instead of the guestfs_int_py_fromstringsize wrapper (which calls PyString_FromStringAndSize anyway on Python < 3). --- generator/python.ml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/generator/python.ml b/generator/python.ml index 2a1a48fc0..0e1ed20d8 100644 --- a/generator/python.ml +++

So far RBufferOut return values, and FBuffer struct fields are 'str' on all the versions of Python. Python 3 distinguishes between 'str' (unicode strings), and 'bytes', with 'str' no more able to hold arbitrary data. For this reason, switch the return value of RBufferOut functions, and FBuffer struct fields to bytes on Python 3: while this is a potentially