Pkg xen devel - Jun 2018 - Bug#900746: xen toolstack xl causes a Segmentation fault on create domu

Package: xen-utils-4.8
Version: 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u7
Severity: normal

after updating to the latest version from security, the xen xl (binary
date: May 22 19:41) toolstack causes a "Segmentation fault" during the
creation of a domu and exits with 139 thought the domu if created anyways.
An other machine, still running ...+deb9u6 doesn't show this issue.

Some additional infos:

I found the following entry in my syslog syslog:
Jun  6 07:11:13 zero2 kernel: [212430.899942] xl[3941]: segfault at
7fc9239c2ff8 ip 00007fc9237be0b5 sp 00007fc9239c3000 error 6 in
ld-2.24.so[7fc9237b4000+23000]

... and the error also occurs during a 'xl block-attach ...'

I've also hit it on one of my boxes with
4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8

This is related to too small stack size set for threads in XEN utils
which explicitly set it to use 16KB. Similar issue is reported here for
NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527

I've recompilled the package with the attached patch to increase the
stack size from 16KB to 32KB and it works.

Technical details:

The issue appears with modern CPU that support AVX-512 instruction set,
in my case it's Intel(R) Xeon(R) Gold 6148. More details are in this
bug report against glibc: https://bugzilla.redhat.com/show_bug.cgi?id=1
527887#c18

There was a post on xen-users acknowledging the bug that says it's
fixed in XEN 4.11: https://lists.xenproject.org/archives/html/xen-users
/2018-05/msg00034.html





-- 
Damian Pietras
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Fix-for-thread-stack-size-on-CPU-with-AVX-512.patch
Type: text/x-patch
Size: 934 bytes
Desc: not available
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20180628/5f249b47/attachment.bin>

confirming segfault on two indentical amd64 machines.

in dmesg:
> segfault at 7ffff7fefff8 ip 00007ffff7de30b5 sp 00007ffff7ff0000 error 6 in
ld-2.24.so[7ffff7dd9000+23000]
output of gdb backtrace:
> Starting program: /usr/lib/xen-4.8/bin/xl create /etc/xen/test.cfg
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
> [New Thread 0x7ffff7ff2700 (LWP 4193)]
> 
> Thread 2 "xl" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff7ff2700 (LWP 4193)]
> 0x00007ffff7de30b5 in _dl_lookup_symbol_x (undef_name=0x7ffff5478c63
"_Unwind_Find_FDE", undef_map=0x555555788310, ref=ref at
entry=0x7ffff7ff0128,
>     symbol_scope=0x555555788668, version=0x555555788960,
type_class=type_class at entry=1, flags=5, skip_map=0x0) at dl-lookup.c:833
> 833     dl-lookup.c: No such file or directory.
> #0  0x00007ffff7de30b5 in _dl_lookup_symbol_x (undef_name=0x7ffff5478c63
"_Unwind_Find_FDE", undef_map=0x555555788310, ref=ref at
entry=0x7ffff7ff0128,
>     symbol_scope=0x555555788668, version=0x555555788960,
type_class=type_class at entry=1, flags=5, skip_map=0x0) at dl-lookup.c:833
> #1  0x00007ffff7de7c54 in _dl_fixup (l=<optimized out>,
reloc_arg=<optimized out>) at ../elf/dl-runtime.c:111
> #2  0x00007ffff7def35a in _dl_runtime_resolve_xsavec () at
../sysdeps/x86_64/dl-trampoline.h:125
> #3  0x00007ffff5485b13 in uw_frame_state_for (context=context at
entry=0x7ffff7ff0da0, fs=fs at entry=0x7ffff7ff0bf0) at
../../../src/libgcc/unwind-dw2.c:1241
> #4  0x00007ffff5486d30 in uw_init_context_1 (context=context at
entry=0x7ffff7ff0da0, outer_cfa=outer_cfa at entry=0x7ffff7ff0fd0,
>     outer_ra=0x7ffff72fad60 <__GI___pthread_unwind+64>) at
../../../src/libgcc/unwind-dw2.c:1562
> #5  0x00007ffff5487336 in _Unwind_ForcedUnwind (exc=0x7ffff7ff2d70,
stop=stop at entry=0x7ffff72fabd0 <unwind_stop>,
stop_argument=0x7ffff7ff1d50)
>     at ../../../src/libgcc/unwind.inc:201
> #6  0x00007ffff72fad60 in __GI___pthread_unwind (buf=<optimized out>)
at unwind.c:121
> #7  0x00007ffff72f0c5a in __do_cancel () at ./pthreadP.h:283
> #8  sigcancel_handler (sig=<optimized out>, si=0x7ffff7ff1130,
ctx=<optimized out>) at nptl-init.c:220
> #9  <signal handler called>
> #10 0x00007ffff72fb20d in read () at ../sysdeps/unix/syscall-template.S:84
> #11 0x00007ffff66e6847 in ?? () from
/usr/lib/x86_64-linux-gnu/libxenstore.so.3.0
> #12 0x00007ffff66e692f in ?? () from
/usr/lib/x86_64-linux-gnu/libxenstore.so.3.0
> #13 0x00007ffff66e6c61 in ?? () from
/usr/lib/x86_64-linux-gnu/libxenstore.so.3.0
> #14 0x00007ffff72f2494 in start_thread (arg=0x7ffff7ff2700) at
pthread_create.c:333
> #15 0x00007ffff7034acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
versions:
> libc6:amd64          2.24-11+deb9u3
> libxen-4.8:amd64     4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9
> libxenstore3.0:amd64 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9

Hi,

On 06/28/2018 09:58 AM, Damian Pietras wrote:
> I've also hit it on one of my boxes with
> 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8
> 
> This is related to too small stack size set for threads in XEN utils
> which explicitly set it to use 16KB. Similar issue is reported here for
> NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527
> 
> I've recompilled the package with the attached patch to increase the
> stack size from 16KB to 32KB and it works.
Thanks for doing "bug triaging"!
> Technical details:
> 
> The issue appears with modern CPU that support AVX-512 instruction set,
> in my case it's Intel(R) Xeon(R) Gold 6148. More details are in this
> bug report against glibc: https://bugzilla.redhat.com/show_bug.cgi?id=1
> 527887#c18
> 
> There was a post on xen-users acknowledging the bug that says it's
> fixed in XEN 4.11: https://lists.xenproject.org/archives/html/xen-users
> /2018-05/msg00034.html
Following that post and what happened afterwards leads me to upstream
commit 448c03b3cb "tools/xenstore: try to get minimum thread stack size
for watch thread", which seems to solve this problem without hardcoding
some size.

Maybe this should be a nice candidate for upstream backport to stable
branches, since users are buying newer hardware and otherwise cannot use
Debian Stable without recompiling their Xen packages?

Ian?

Thanks,
Hans

On Thu, 28 Jun 2018 18:58:24 +0200 Hans van Kranenburg <hans at
knorrie.org> wrote:
> Hi,
>
> On 06/28/2018 09:58 AM, Damian Pietras wrote:
> > I've also hit it on one of my boxes with
> > 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u8
> >
> > This is related to too small stack size set for threads in XEN utils
> > which explicitly set it to use 16KB. Similar issue is reported here
for
> > NTP: https://bugzilla.redhat.com/show_bug.cgi?id=1564527
> >
> > I've recompilled the package with the attached patch to increase
the
> > stack size from 16KB to 32KB and it works.
>
> Thanks for doing "bug triaging"!
>
> > Technical details:
> >
> > The issue appears with modern CPU that support AVX-512 instruction
set,
> > in my case it's Intel(R) Xeon(R) Gold 6148. More details are in
this
> > bug report against glibc:
https://bugzilla.redhat.com/show_bug.cgi?id=1
> > 527887#c18
> >
> > There was a post on xen-users acknowledging the bug that says it's
> > fixed in XEN 4.11:
https://lists.xenproject.org/archives/html/xen-users
> > /2018-05/msg00034.html
>
> Following that post and what happened afterwards leads me to upstream
> commit 448c03b3cb "tools/xenstore: try to get minimum thread stack
size
> for watch thread", which seems to solve this problem without
hardcoding
> some size.
>
> Maybe this should be a nice candidate for upstream backport to stable
> branches, since users are buying newer hardware and otherwise cannot use
> Debian Stable without recompiling their Xen packages?
>
> Ian?
>
> Thanks,
> Hans
This patch fixed segfault problems for us on servers with Intel Xeon
Gold 6148 CPUs.  What would it take to get this patch included in the
next stretch update?  It's a pretty big deal for us right now and
while we'll be ok running this patched version for now it'd be great
to not have to maintain our own patch.

Thanks,

Stephen

Any chance this can make it into the next stretch point release?  We’re
currently maintaining our own package and it’d be really nice to not have to.

Stephen

Your message dated Sun, 10 May 2020 23:32:33 +0200
with message-id <4428184.u1DFsJF7ir at bagend>
and subject line Fixed in 4.8.5.final+shim4.10.4-1+deb9u12
has caused the Debian Bug report #900746,
regarding xen toolstack xl causes a Segmentation fault on create domu
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
900746: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900746
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Christian Schwamborn <christian.schwamborn at nswit.de>
Subject: xen toolstack xl causes a Segmentation fault on create domu
Date: Mon, 4 Jun 2018 09:05:44 +0200
Size: 5206
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200510/adb135cb/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Diederik de Haas <didi.debian at cknow.org>
Subject: Fixed in 4.8.5.final+shim4.10.4-1+deb9u12
Date: Sun, 10 May 2020 23:32:33 +0200
Size: 5788
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200510/adb135cb/attachment-0001.mht>