Yonatan Shtarkman
2023-Feb-14 18:02 UTC
[Libguestfs] Issue with downloading files whose path contains multi-byte utf-8 characters
The attached code snippet reproduces the issue when running: python3 libugestfs_segfault_repro.py With: guestfs version: 1.48.6 guestfs-python version: 1.48.6 Stacktrace: #0 0x00007f6dd17b4701 in _Py_INCREF (op=<optimized out>) at /usr/include/python3.9/object.h:408 408 op->ob_refcnt++; (gdb) bt #0 0x00007f6dd17b4701 in _Py_INCREF (op=<optimized out>) at /usr/include/python3.9/object.h:408 #1 guestfs_int_py_event_callback_wrapper (g=<optimized out>, flags=<optimized out>, array_len=0, array=0x0, buf_len=43, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\347\232\204\303 available\r\ned7 #2 guestfs_int_py_event_callback_wrapper (g=<optimized out>, callback=0x7f6dd19ec040, event=16, event_handle=0, flags=<optimized out>, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\347\232\204\303 4 #3 0x00007f6dd166f9aa in guestfs_int_call_callbacks_message (g=0xae06f0, event=16, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\347\232\204\303 available\r\ned to rvf:e run in, ignor data av", buf_len=43) at events.c:117 #4 0x00007f6dd168560e in guestfs_int_log_message_callback (g=g at entry=0xae06f0, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\347\232\204\303 available\r\ned to rvf:e run in, ignor data av", len=len at entry=43) at proto.c:145 #5 0x00007f6dd166ac29 in handle_log_message (g=g at entry=0xae06f0, conn=conn at entry=0xd19570) at conn-socket.c:395 #6 0x00007f6dd166b233 in read_data (len=4, bufv=<optimized out>, connv=<optimized out>, g=<optimized out>) at conn-socket.c:179 #7 read_data (g=0xae06f0, connv=0xd19570, bufv=<optimized out>, len=4) at conn-socket.c:142 #8 0x00007f6dd1685c2a in recv_from_daemon (buf_rtn=0x7fffa95cc548, size_rtn=0x7fffa95cc544, g=0xae06f0) at proto.c:545 #9 guestfs_int_recv_from_daemon (g=g at entry=0xae06f0, size_rtn=size_rtn at entry=0x7fffa95cc544, buf_rtn=buf_rtn at entry=0x7fffa95cc548) at proto.c:623 #10 0x00007f6dd168603a in guestfs_int_recv (g=g at entry=0xae06f0, fn=fn at entry=0x7f6dd16aa308 "download", hdr=hdr at entry=0x7fffa95cc610, err=err at entry=0x7fffa95cc5e0, xdrp=xdrp at entry=0x0, ret=ret at entry=0x0) at proto.c:668 #11 0x00007f6dd163825e in guestfs_download (g=g at entry=0xae06f0, remotefilename=<optimized out>, filename=0x7f6dd17edea0 "/tmp/1") at actions-5.c:2984 #12 0x00007f6dd17ab01a in guestfs_int_py_download (self=<optimized out>, args=<optimized out>) at actions-5.c:794 On Mon, Feb 13, 2023 at 10:16 PM Richard W.M. Jones <rjones at redhat.com> wrote:> On Mon, Feb 13, 2023 at 09:38:50PM +0200, Yonatan Shtarkman wrote: > > Also, as a workaround, I avoided calling the event callback if null is > returned > > by Py_BuildValue (still print the error and release the thread). > > This seems to work for our usage (we only use the event callbacks for > logging), > > any potential issues I'm missing? > > Oh I see so the issue is actually when logging and not in the download > call? That's strange that it should be happening intermittently. I'd > expect a UTF-8 encoding error would happen every time not sometimes. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat > http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-top is 'top' for virtual machines. Tiny program with many > powerful monitoring features, net stats, disk stats, logging, etc. > http://people.redhat.com/~rjones/virt-top > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://listman.redhat.com/archives/libguestfs/attachments/20230214/4e9b3d19/attachment.htm> -------------- next part -------------- A non-text attachment was scrubbed... Name: libguestfs_segfault_repro.py Type: text/x-python Size: 469 bytes Desc: not available URL: <http://listman.redhat.com/archives/libguestfs/attachments/20230214/4e9b3d19/attachment.py>
Richard W.M. Jones
2023-Feb-14 18:10 UTC
[Libguestfs] Issue with downloading files whose path contains multi-byte utf-8 characters
On Tue, Feb 14, 2023 at 08:02:59PM +0200, Yonatan Shtarkman wrote:> The attached code snippet?reproduces the issue when running: > python3 libugestfs_segfault_repro.py > > With: > guestfs version: 1.48.6 > guestfs-python version: 1.48.6Can confirm with {python3-,}libguestfs-1.51.1-1.fc39.x86_64 The stack trace is roughly the same as yours. Rich.> Stacktrace: > #0 ?0x00007f6dd17b4701 in _Py_INCREF (op=<optimized out>) at /usr/include/ > python3.9/object.h:408 > 408 ? ?op->ob_refcnt++; > (gdb) bt > #0 ?0x00007f6dd17b4701 in _Py_INCREF (op=<optimized out>) at /usr/include/ > python3.9/object.h:408 > #1 ?guestfs_int_py_event_callback_wrapper > ? ? (g=<optimized out>, flags=<optimized out>, array_len=0, array=0x0, buf_len> 43, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\347\232\204\303 > available\r\ned7 > #2 ?guestfs_int_py_event_callback_wrapper > ? ? (g=<optimized out>, callback=0x7f6dd19ec040, event=16, event_handle=0, > flags=<optimized out>, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\ > 347\232\204\303 4 > #3 ?0x00007f6dd166f9aa in guestfs_int_call_callbacks_message > ? ? (g=0xae06f0, event=16, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E > /\347\232\204\303 available\r\ned to rvf:e run ?in, ignor data av", buf_len=43) > ? ? at events.c:117 > #4 ?0x00007f6dd168560e in guestfs_int_log_message_callback > ? ? (g=g at entry=0xae06f0, buf=0xd1dc90 "commandrvf: udevadm --debug settle -E /\ > 347\232\204\303 available\r\ned to rvf:e run ?in, ignor data av", len=len at entry > =43) > ? ? at proto.c:145 > #5 ?0x00007f6dd166ac29 in handle_log_message (g=g at entry=0xae06f0, conn> conn at entry=0xd19570) at conn-socket.c:395 > #6 ?0x00007f6dd166b233 in read_data (len=4, bufv=<optimized out>, connv> <optimized out>, g=<optimized out>) at conn-socket.c:179 > #7 ?read_data (g=0xae06f0, connv=0xd19570, bufv=<optimized out>, len=4) at > conn-socket.c:142 > #8 ?0x00007f6dd1685c2a in recv_from_daemon (buf_rtn=0x7fffa95cc548, size_rtn> 0x7fffa95cc544, g=0xae06f0) at proto.c:545 > #9 ?guestfs_int_recv_from_daemon (g=g at entry=0xae06f0, size_rtn=size_rtn at entry> 0x7fffa95cc544, buf_rtn=buf_rtn at entry=0x7fffa95cc548) at proto.c:623 > #10 0x00007f6dd168603a in guestfs_int_recv > ? ? (g=g at entry=0xae06f0, fn=fn at entry=0x7f6dd16aa308 "download", hdr=hdr at entry> 0x7fffa95cc610, err=err at entry=0x7fffa95cc5e0, xdrp=xdrp at entry=0x0, ret> ret at entry=0x0) > ? ? at proto.c:668 > #11 0x00007f6dd163825e in guestfs_download (g=g at entry=0xae06f0, remotefilename> <optimized out>, filename=0x7f6dd17edea0 "/tmp/1") at actions-5.c:2984 > #12 0x00007f6dd17ab01a in guestfs_int_py_download (self=<optimized out>, args> <optimized out>) at actions-5.c:794 > ? > > > On Mon, Feb 13, 2023 at 10:16 PM Richard W.M. Jones <rjones at redhat.com> wrote: > > On Mon, Feb 13, 2023 at 09:38:50PM +0200, Yonatan Shtarkman wrote: > > Also, as a workaround, I avoided calling the event callback if null is > returned > > by?Py_BuildValue (still print the error and release the thread). > > This seems to work for our usage (we only use the event callbacks for > logging), > > any potential?issues?I'm missing? > > Oh I see so the issue is actually when logging and not in the download > call?? That's strange that it should be happening intermittently.? I'd > expect a UTF-8 encoding error would happen every time not sometimes. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/ > ~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-top is 'top' for virtual machines.? Tiny program with many > powerful monitoring features, net stats, disk stats, logging, etc. > http://people.redhat.com/~rjones/virt-top > >> import guestfs > from guestfs import GuestFS > > g = GuestFS(python_return_dict=True) > def callback(event, _event_handle, buf: str, array): > pass > > g.set_event_callback(callback, guestfs.EVENT_ALL) > g.set_verbose(True) > path = '/tmp/tmp_disk' > g.disk_create(path, format='raw', size=32*1024*1024) > g.add_drive(path) > g.launch() > g.mkfs('ext4', '/dev/sda') > g.mount('/dev/sda', '/') > for _ in range(20000): > try: > g.download ('/??', '/tmp/1') > except: > pass >-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Maybe Matching Threads
- Issue with downloading files whose path contains multi-byte utf-8 characters
- Issue with downloading files whose path contains multi-byte utf-8 characters
- [PATCH 0/3] protocol: Abstract out socket operations.
- [PATCH 1/2] python: Avoid crash if callback parameters cannot be built
- [PATCH 0/2] Add mkfs-opts API with optional arguments