Pkg xen devel - Jun 2015 - Bug#787229: Bug#787229: xen-hypervisor-4.4-amd64: xl command hangs on Dell R720

Package: xen-hypervisor-4.4-amd64
Version: 4.4.1-9
Severity: grave
Justification: renders package unusable

Hi,

I was running Debian Wheezy (amd64) and xen-hypervisor 4.1 on a Dell
PowerEdge
R720.
Xen was working fine.
Xen was configured with LVM

Now, I'm trying Debian Jessie (amd64) and I have a strange problem.

After booting, Xen seems to be working : /etc/init.d/xen status

? xen.service - LSB: Xen daemons
   Loaded: loaded (/etc/init.d/xen)
   Active: activating (start) since sam. 2015-05-29 23:20:50 CEST; 33s ago
  Control: 1150 (xen)
   CGroup: /system.slice/xen.service
           ??1150 /bin/sh /etc/init.d/xen start
           ??1229 xenstore-write /local/domain/0/name Domain-0

mai 29 23:20:50 jango xen[1150]: Starting Xen daemons: xenfs xenstored
mai 29 22:20:50 jango xen[1150]: Warning: Fake start-stop-daemon called,
doing
nothing.
mai 29 23:20:50 jango xen[1150]: Warning: Fake start-stop-daemon called,
doing
nothing.


When I try to run a simple 'xl list' : I got this in console :

root at jango:~# xl list
Name                                        ID   Mem VCPUs      State
Time(s)

The cursor is blinking, nothing happens.
I have not touched any config files yet.
(/etc/xen contains default config files).

/var/log/xen is empty.

In /var/log/syslog, I got this :

May 29 23:28:01 jango kernel: [  245.721670] INFO: task xl:1440 blocked for
more than 120 seconds.
May 29 23:28:01 jango kernel: [  245.721780]       Not tainted
3.16.0-4-amd64
#1
May 29 23:28:01 jango kernel: [  245.721872] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 29 23:28:01 jango kernel: [  245.721973] xl              D
ffff8807ee86a528
0  1440   1425 0x00000000
May 29 23:28:01 jango kernel: [  245.721981]  ffff8807ee86a0d0
0000000000000286
0000000000012f00 ffff8800bc41bfd8
May 29 23:28:01 jango kernel: [  245.721985]  0000000000012f00
ffff8807ee86a0d0
ffffffff81adaf70 ffff8800bc41be48
May 29 23:28:01 jango kernel: [  245.721989]  ffffffff81adaf74
ffff8807ee86a0d0
00000000ffffffff ffffffff81adaf78
May 29 23:28:01 jango kernel: [  245.721994] Call Trace:
May 29 23:28:01 jango kernel: [  245.722007]  [<ffffffff8150da55>] ?
schedule_preempt_disabled+0x25/0x70
May 29 23:28:01 jango kernel: [  245.722012]  [<ffffffff8150f503>] ?
__mutex_lock_slowpath+0xd3/0x1c0
May 29 23:28:01 jango kernel: [  245.722019]  [<ffffffff810a74a2>] ?
remove_wait_queue+0x12/0x50
May 29 23:28:01 jango kernel: [  245.722023]  [<ffffffff8150f60b>] ?
mutex_lock+0x1b/0x2a
May 29 23:28:01 jango kernel: [  245.722029]  [<ffffffff8135bc75>] ?
xenbus_dev_request_and_reply+0x25/0xb0
May 29 23:28:01 jango kernel: [  245.722034]  [<ffffffff8135e2de>] ?
xenbus_file_write+0x27e/0x540
May 29 23:28:01 jango kernel: [  245.722041]  [<ffffffff811aa442>] ?
__sb_start_write+0x42/0xd0
May 29 23:28:01 jango kernel: [  245.722047]  [<ffffffff811a8082>] ?
vfs_write+0xb2/0x1f0
May 29 23:28:01 jango kernel: [  245.722053]  [<ffffffff811a8bc2>] ?
SyS_write+0x42/0xa0
May 29 23:28:01 jango kernel: [  245.722058]  [<ffffffff81510e4d>] ?
system_call_fast_compare_end+0x10/0x15


I tried with xen-hypervisor 4.5 with no luck.
I tried with linux-image-4.xxxx from unstable with no luck.

xl info command output :

host                   : jango
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24)
machine                : x86_64
nr_cpus                : 32
max_cpu_id             : 47
nr_nodes               : 2
cores_per_socket       : 8
threads_per_core       : 2
cpu_mhz                : 2400
hw_caps                :
bfebfbff:2c100800:00000000:00003f00:17bee3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 32722
free_memory            : 127
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 4
xen_extra              : .1
xen_version            : 4.4.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder
cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by          : waldi
cc_compile_domain      : debian.org
cc_compile_date        : Mon Apr  6 18:24:28 UTC 2015
xend_config_format     : 4


Unfortunatly, this server is not a test server, so I'll have to reinstall
wheezy.

Have you any clue about this problem ?

Is there anything I can test to debug this ?

Thanks a lot.

Benoit



-- System Information:
Debian Release: 8.0
  APT prefers stable-updates
  APT policy: (900, 'stable-updates'), (900, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20150530/c697ab8c/attachment.html>

On Sat, 2015-05-30 at 05:23 +0200, Beno?t Tonnerre
wrote:
> mai 29 23:20:50 jango xen[1150]: Starting Xen daemons: xenfs xenstored
> mai 29 22:20:50 jango xen[1150]: Warning: Fake start-stop-daemon called,
doing
> nothing.
> mai 29 23:20:50 jango xen[1150]: Warning: Fake start-stop-daemon called,
doing
> nothing.
This suggests that the xen daemons and in particular xenstored are not
actually running, the result of which will be any command which tries to
talk to xenstored (like xl list, most xl commands and the xenstore-write
seen in the status log) will never get an answer and will appear to
hang.

I don't know what "Fake start-stop-daemon" is, but it sounds like
the
root of your problems.

Ian.

Hi,

Thanks for your answer.

During the Debian jessie installation, I had a problem at grub step.
The os-prober part was stuck : grub was at 66 % (os-prober tried to browse
all the VM on LVM partitions)
After more than 15 minutes, os-prober seems to be stuck, so I rebooted the
server.
I had to boot in rescue mode, then, reinstalled grub and modified it to add
"GRUB_DISABLE_OS_PROBER=true".
It seems it solved the problem, I was able to boot the server.

Maybe there are some missing things in my Debian installation ?

Do you want me to fill an other bug report (concerning the installation
problem ?)

I have not this problem with Debian Wheezy installation.

Thanks for your time and for your help.

Benoit






2015-05-30 10:31 GMT+02:00 Ian Campbell <ijc at debian.org>:
> On Sat, 2015-05-30 at 05:23 +0200, Beno?t Tonnerre wrote:
> > mai 29 23:20:50 jango xen[1150]: Starting Xen daemons: xenfs xenstored
> > mai 29 22:20:50 jango xen[1150]: Warning: Fake start-stop-daemon
called,
> doing
> > nothing.
> > mai 29 23:20:50 jango xen[1150]: Warning: Fake start-stop-daemon
called,
> doing
> > nothing.
>
> This suggests that the xen daemons and in particular xenstored are not
> actually running, the result of which will be any command which tries to
> talk to xenstored (like xl list, most xl commands and the xenstore-write
> seen in the status log) will never get an answer and will appear to
> hang.
>
> I don't know what "Fake start-stop-daemon" is, but it sounds
like the
> root of your problems.
>
> Ian.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20150602/22d9f16a/attachment.html>

Hi Ian,

You can close this bug report.
I finally succeed in installing Debian Jessie on the Dell PowerEdge R720.

The problem was indeed, a faulty installation due to os-prober, unable to
complete the scan of all the Xen LVM.

I have installed Debian Jessie without grub, than reboot in rescue mode.
I have created manually /etc/default/grub and add
"GRUB_DISABLE_OS_PROBER=true".
I have installed grub, reboot and everything worked with xl commands.

Thank you again for your help.

Have a nice day.

Benoit

2015-05-30 10:31 GMT+02:00 Ian Campbell <ijc at debian.org>:
> On Sat, 2015-05-30 at 05:23 +0200, Beno?t Tonnerre wrote:
> > mai 29 23:20:50 jango xen[1150]: Starting Xen daemons: xenfs xenstored
> > mai 29 22:20:50 jango xen[1150]: Warning: Fake start-stop-daemon
called,
> doing
> > nothing.
> > mai 29 23:20:50 jango xen[1150]: Warning: Fake start-stop-daemon
called,
> doing
> > nothing.
>
> This suggests that the xen daemons and in particular xenstored are not
> actually running, the result of which will be any command which tries to
> talk to xenstored (like xl list, most xl commands and the xenstore-write
> seen in the status log) will never get an answer and will appear to
> hang.
>
> I don't know what "Fake start-stop-daemon" is, but it sounds
like the
> root of your problems.
>
> Ian.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20150810/29ac90eb/attachment.html>

Your message dated Mon, 10 Aug 2015 19:56:42 +0100
with message-id <1439233002.18644.64.camel at debian.org>
and subject line Re: [Pkg-xen-devel] Bug#787229: xen-hypervisor-4.4-amd64: xl
command hangs on Dell R720
has caused the Debian Bug report #787229,
regarding xen-hypervisor-4.4-amd64: xl command hangs on Dell R720
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
787229: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787229
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: =?UTF-8?Q?Beno=C3=AEt_Tonnerre?= <bevito at gmail.com>
Subject: xen-hypervisor-4.4-amd64: xl command hangs on Dell R720
Date: Sat, 30 May 2015 05:23:58 +0200
Size: 15930
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20150810/8232d8b1/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Campbell <ijc at debian.org>
Subject: Re: [Pkg-xen-devel] Bug#787229: xen-hypervisor-4.4-amd64: xl command
hangs on Dell R720
Date: Mon, 10 Aug 2015 19:56:42 +0100
Size: 3594
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20150810/8232d8b1/attachment-0001.mht>