Christian Schwamborn
2018-Jun-04 07:05 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
Package: xen-utils-4.8 Version: 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u7 Severity: normal after updating to the latest version from security, the xen xl (binary date: May 22 19:41) toolstack causes a "Segmentation fault" during the creation of a domu and exits with 139 thought the domu if created anyways. An other machine, still running ...+deb9u6 doesn't show this issue.
Christian Schwamborn
2018-Jun-06 06:21 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
Some additional infos: I found the following entry in my syslog syslog: Jun 6 07:11:13 zero2 kernel: [212430.899942] xl[3941]: segfault at 7fc9239c2ff8 ip 00007fc9237be0b5 sp 00007fc9239c3000 error 6 in ld-2.24.so[7fc9237b4000+23000] ... and the error also occurs during a 'xl block-attach ...'
Damian Pietras
2018-Jun-28 07:58 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
I've also hit it on one of my boxes with 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8 This is related to too small stack size set for threads in XEN utils which explicitly set it to use 16KB. Similar issue is reported here for NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527 I've recompilled the package with the attached patch to increase the stack size from 16KB to 32KB and it works. Technical details: The issue appears with modern CPU that support AVX-512 instruction set, in my case it's Intel(R) Xeon(R) Gold 6148. More details are in this bug report against glibc: https://bugzilla.redhat.com/show_bug.cgi?id=1 527887#c18 There was a post on xen-users acknowledging the bug that says it's fixed in XEN 4.11: https://lists.xenproject.org/archives/html/xen-users /2018-05/msg00034.html -- Damian Pietras -------------- next part -------------- A non-text attachment was scrubbed... Name: Fix-for-thread-stack-size-on-CPU-with-AVX-512.patch Type: text/x-patch Size: 934 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20180628/5f249b47/attachment.bin>
Ander Punnar
2018-Jun-28 08:30 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
confirming segfault on two indentical amd64 machines. in dmesg:> segfault at 7ffff7fefff8 ip 00007ffff7de30b5 sp 00007ffff7ff0000 error 6 in ld-2.24.so[7ffff7dd9000+23000]output of gdb backtrace:> Starting program: /usr/lib/xen-4.8/bin/xl create /etc/xen/test.cfg > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > [New Thread 0x7ffff7ff2700 (LWP 4193)] > > Thread 2 "xl" received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7ffff7ff2700 (LWP 4193)] > 0x00007ffff7de30b5 in _dl_lookup_symbol_x (undef_name=0x7ffff5478c63 "_Unwind_Find_FDE", undef_map=0x555555788310, ref=ref at entry=0x7ffff7ff0128, > symbol_scope=0x555555788668, version=0x555555788960, type_class=type_class at entry=1, flags=5, skip_map=0x0) at dl-lookup.c:833 > 833 dl-lookup.c: No such file or directory. > #0 0x00007ffff7de30b5 in _dl_lookup_symbol_x (undef_name=0x7ffff5478c63 "_Unwind_Find_FDE", undef_map=0x555555788310, ref=ref at entry=0x7ffff7ff0128, > symbol_scope=0x555555788668, version=0x555555788960, type_class=type_class at entry=1, flags=5, skip_map=0x0) at dl-lookup.c:833 > #1 0x00007ffff7de7c54 in _dl_fixup (l=<optimized out>, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:111 > #2 0x00007ffff7def35a in _dl_runtime_resolve_xsavec () at ../sysdeps/x86_64/dl-trampoline.h:125 > #3 0x00007ffff5485b13 in uw_frame_state_for (context=context at entry=0x7ffff7ff0da0, fs=fs at entry=0x7ffff7ff0bf0) at ../../../src/libgcc/unwind-dw2.c:1241 > #4 0x00007ffff5486d30 in uw_init_context_1 (context=context at entry=0x7ffff7ff0da0, outer_cfa=outer_cfa at entry=0x7ffff7ff0fd0, > outer_ra=0x7ffff72fad60 <__GI___pthread_unwind+64>) at ../../../src/libgcc/unwind-dw2.c:1562 > #5 0x00007ffff5487336 in _Unwind_ForcedUnwind (exc=0x7ffff7ff2d70, stop=stop at entry=0x7ffff72fabd0 <unwind_stop>, stop_argument=0x7ffff7ff1d50) > at ../../../src/libgcc/unwind.inc:201 > #6 0x00007ffff72fad60 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:121 > #7 0x00007ffff72f0c5a in __do_cancel () at ./pthreadP.h:283 > #8 sigcancel_handler (sig=<optimized out>, si=0x7ffff7ff1130, ctx=<optimized out>) at nptl-init.c:220 > #9 <signal handler called> > #10 0x00007ffff72fb20d in read () at ../sysdeps/unix/syscall-template.S:84 > #11 0x00007ffff66e6847 in ?? () from /usr/lib/x86_64-linux-gnu/libxenstore.so.3.0 > #12 0x00007ffff66e692f in ?? () from /usr/lib/x86_64-linux-gnu/libxenstore.so.3.0 > #13 0x00007ffff66e6c61 in ?? () from /usr/lib/x86_64-linux-gnu/libxenstore.so.3.0 > #14 0x00007ffff72f2494 in start_thread (arg=0x7ffff7ff2700) at pthread_create.c:333 > #15 0x00007ffff7034acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97versions:> libc6:amd64 2.24-11+deb9u3 > libxen-4.8:amd64 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 > libxenstore3.0:amd64 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9
Hans van Kranenburg
2018-Jun-28 16:58 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
Hi, On 06/28/2018 09:58 AM, Damian Pietras wrote:> I've also hit it on one of my boxes with > 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8 > > This is related to too small stack size set for threads in XEN utils > which explicitly set it to use 16KB. Similar issue is reported here for > NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527 > > I've recompilled the package with the attached patch to increase the > stack size from 16KB to 32KB and it works.Thanks for doing "bug triaging"!> Technical details: > > The issue appears with modern CPU that support AVX-512 instruction set, > in my case it's Intel(R) Xeon(R) Gold 6148. More details are in this > bug report against glibc: https://bugzilla.redhat.com/show_bug.cgi?id=1 > 527887#c18 > > There was a post on xen-users acknowledging the bug that says it's > fixed in XEN 4.11: https://lists.xenproject.org/archives/html/xen-users > /2018-05/msg00034.htmlFollowing that post and what happened afterwards leads me to upstream commit 448c03b3cb "tools/xenstore: try to get minimum thread stack size for watch thread", which seems to solve this problem without hardcoding some size. Maybe this should be a nice candidate for upstream backport to stable branches, since users are buying newer hardware and otherwise cannot use Debian Stable without recompiling their Xen packages? Ian? Thanks, Hans
Stephen Gelman
2019-Jan-24 21:15 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
On Thu, 28 Jun 2018 18:58:24 +0200 Hans van Kranenburg <hans at knorrie.org> wrote:> Hi, > > On 06/28/2018 09:58 AM, Damian Pietras wrote: > > I've also hit it on one of my boxes with > > 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8 > > > > This is related to too small stack size set for threads in XEN utils > > which explicitly set it to use 16KB. Similar issue is reported here for > > NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527 > > > > I've recompilled the package with the attached patch to increase the > > stack size from 16KB to 32KB and it works. > > Thanks for doing "bug triaging"! > > > Technical details: > > > > The issue appears with modern CPU that support AVX-512 instruction set, > > in my case it's Intel(R) Xeon(R) Gold 6148. More details are in this > > bug report against glibc: https://bugzilla.redhat.com/show_bug.cgi?id=1 > > 527887#c18 > > > > There was a post on xen-users acknowledging the bug that says it's > > fixed in XEN 4.11: https://lists.xenproject.org/archives/html/xen-users > > /2018-05/msg00034.html > > Following that post and what happened afterwards leads me to upstream > commit 448c03b3cb "tools/xenstore: try to get minimum thread stack size > for watch thread", which seems to solve this problem without hardcoding > some size. > > Maybe this should be a nice candidate for upstream backport to stable > branches, since users are buying newer hardware and otherwise cannot use > Debian Stable without recompiling their Xen packages? > > Ian? > > Thanks, > HansThis patch fixed segfault problems for us on servers with Intel Xeon Gold 6148 CPUs. What would it take to get this patch included in the next stretch update? It's a pretty big deal for us right now and while we'll be ok running this patched version for now it'd be great to not have to maintain our own patch. Thanks, Stephen
Stephen Gelman
2019-Jan-31 20:20 UTC
[Pkg-xen-devel] Bug#900746: xen toolstack xl causes a Segmentation fault on create domu
Any chance this can make it into the next stretch point release? We’re currently maintaining our own package and it’d be really nice to not have to. Stephen
Debian Bug Tracking System
2020-May-10 21:45 UTC
[Pkg-xen-devel] Bug#900746: marked as done (xen toolstack xl causes a Segmentation fault on create domu)
Your message dated Sun, 10 May 2020 23:32:33 +0200 with message-id <4428184.u1DFsJF7ir at bagend> and subject line Fixed in 4.8.5.final+shim4.10.4-1+deb9u12 has caused the Debian Bug report #900746, regarding xen toolstack xl causes a Segmentation fault on create domu to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner at bugs.debian.org immediately.) -- 900746: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900746 Debian Bug Tracking System Contact owner at bugs.debian.org with problems -------------- next part -------------- An embedded message was scrubbed... From: Christian Schwamborn <christian.schwamborn at nswit.de> Subject: xen toolstack xl causes a Segmentation fault on create domu Date: Mon, 4 Jun 2018 09:05:44 +0200 Size: 5206 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200510/adb135cb/attachment.mht> -------------- next part -------------- An embedded message was scrubbed... From: Diederik de Haas <didi.debian at cknow.org> Subject: Fixed in 4.8.5.final+shim4.10.4-1+deb9u12 Date: Sun, 10 May 2020 23:32:33 +0200 Size: 5788 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200510/adb135cb/attachment-0001.mht>