Pkg xen devel - Mar 2012 - Bug#666135: Multiple "Domain-0", slow libvirt

Package: xen-hypervisor-4.0-amd64
Version: 4.0.1-4
Severity: normal

Hi all,

I have a problem with libvirt, it works slow. For example, "virsh
list" takes 6..7 seconds.

I ask xen-users list with no luck. According to google, this problem
is not wide spreaded, so maybe it's debian specific. If not, please
hint where I should go next.

I found this in libvirt log:

...
11:38:12.765: debug : virEventDispatchHandles:477 : Dispatch n=8 f=56
w=9 e=1 0x25ce830
11:38:12.765: debug : virEventUpdateHandleImpl:147 : Update handle w=8 e=3
11:38:12.765: debug : virEventInterruptLocked:668 : Interrupting
11:38:12.766: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.867: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:12.968: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
(about 20 such entries)
11:38:14.681: debug : xenStoreDomainIntroduced:1421 : Some domains
were missing, trying again
11:38:14.782: debug : virEventDispatchHandles:464 : i=9 w=10
11:38:14.782: debug : virEventCleanupTimeouts:495 : Cleanup 3
11:38:14.782: debug : virEventCleanupHandles:536 : Cleanupo 10
...

At least 2 seconds spent here. And there is a missing domains
somewhere in configs.

Here they are (xenstored config via script from
http://wiki.xensource.com/xenwiki/XenStore):

/vm/00000000-0000-0000-0000-000000000000/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000/image/kernel/vm/00000000-0000-0000-0000-000000000000/image/cmdline/vm/00000000-0000-0000-0000-000000000000/image/ramdisk/vm/00000000-0000-0000-0000-000000000000/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000/bootloader_args/vm/00000000-0000-0000-0000-000000000000/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000/vcpus=4
/vm/00000000-0000-0000-0000-000000000000/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000/bootloader/vm/00000000-0000-0000-0000-000000000000/name=Domain-0
/vm/00000000-0000-0000-0000-000000000000-1/on_xend_stop=ignore
/vm/00000000-0000-0000-0000-000000000000-1/shadow_memory=0
/vm/00000000-0000-0000-0000-000000000000-1/uuid=00000000-0000-0000-0000-000000000000
/vm/00000000-0000-0000-0000-000000000000-1/on_reboot=restart
/vm/00000000-0000-0000-0000-000000000000-1/image/ostype=linux
/vm/00000000-0000-0000-0000-000000000000-1/image/kernel/vm/00000000-0000-0000-0000-000000000000-1/image/cmdline/vm/00000000-0000-0000-0000-000000000000-1/image/ramdisk/vm/00000000-0000-0000-0000-000000000000-1/on_poweroff=destroy
/vm/00000000-0000-0000-0000-000000000000-1/bootloader_args/vm/00000000-0000-0000-0000-000000000000-1/on_xend_start=ignore
/vm/00000000-0000-0000-0000-000000000000-1/on_crash=restart
/vm/00000000-0000-0000-0000-000000000000-1/xend/restart_count=0
/vm/00000000-0000-0000-0000-000000000000-1/vcpus=4
/vm/00000000-0000-0000-0000-000000000000-1/vcpu_avail=15
/vm/00000000-0000-0000-0000-000000000000-1/bootloader/vm/00000000-0000-0000-0000-000000000000-1/name=Domain-0
/local/domain/0/vm=/vm/00000000-0000-0000-0000-000000000000-1
/local/domain/0/device...

I have two (maybe more) "Domain-0" in "vm" section, and
every reboot
adds a new one.

I run "xenstore-rm /vm/00000000-0000-0000-0000-000000000000",
"xenstore-rm /vm/00000000-0000-0000-0000-000000000000-1" and so on for
every entry. After reboot I have one "Domain-0" and "virsh
list" takes
0,4 second until next reboot.

Part of xend.log (reboot):

[2012-03-21 12:17:51 1768] INFO (SrvDaemon:220) Xend exited with status 0.
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:332) Xend Daemon started
[2012-03-21 12:20:53 1791] INFO (SrvDaemon:336) Xend changeset: unavailable.
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:149)
XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time':
10554784758L,
'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255,
'dying': 0,
'online_vcpus': 4, 'domid': 0, 'paused': 0,
'crashed': 0, 'running':
1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb':
3400960L,
'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'blocked':
0, 'name': 'Domain-0'})
[2012-03-21 12:20:55 1791] INFO (XendDomainInfo:167) Recreating domain
0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:3400) Storing VM
details: {'on_xend_stop': 'ignore', 'shadow_memory':
'0', 'uuid':
'00000000-0000-0000-0000-000000000000', 'on_reboot':
'restart',
'image': '(linux (kernel ) (superpages 0) (tsc_mode 0) (nomigrate
0))', 'on_poweroff': 'destroy', 'bootloader_args':
'',
'on_xend_start': 'ignore', 'on_crash':
'restart',
'xend/restart_count': '0', 'vcpus': '4',
'vcpu_avail': '15',
'bootloader': '', 'name': 'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1804) Storing domain
details: {'cpu/3/availability': 'online', 'description':
'',
'console/limit': '1048576', 'memory/target':
'3400960',
'cpu/2/availability': 'online', 'vm':
'/vm/00000000-0000-0000-0000-000000000000-1', 'domid':
'0',
'cpu/0/availability': 'online', 'cpu/1/availability':
'online',
'control/platform-feature-multiprocessor-suspend': '1',
'console/type': 'xenconsoled', 'name':
'Domain-0'}
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:464) Adding Domain: 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomain:398) number of vcpus to use is 0
[2012-03-21 12:20:55 1791] DEBUG (XendDomainInfo:1891)
XendDomainInfo.handleShutdownWatch
[2012-03-21 12:20:55 1791] INFO (SrvServer:177) unix
path=/var/lib/xend/xend-socket
...

Xen creates a new vm "Domain-0" instead of using an existing one. I
have no idea how I can tune it, so I suppose this is a bug.

My software:
root at sl03500008180:~# dpkg -l | grep xen
ii  libxenstore3.0                      4.0.1-4
Xenstore communications library for Xen
ii  linux-image-2.6.32-5-xen-amd64      2.6.32-41
Linux 2.6.32 for 64-bit PCs, Xen dom0 support
ii  xen-hypervisor-4.0-amd64            4.0.1-4
The Xen Hypervisor on AMD64
ii  xen-linux-system-2.6-xen-amd64      2.6.32+29
Xen system with Linux 2.6 for 64-bit PCs (meta-package)
ii  xen-linux-system-2.6.32-5-xen-amd64 2.6.32-41
Xen system with Linux 2.6.32 on 64-bit PCs (meta-package)
ii  xen-qemu-dm-4.0                     4.0.1-2
Xen Qemu Device Model virtual machine hardware emulator
ii  xen-utils-4.0                       4.0.1-4
XEN administrative tools
ii  xen-utils-common                    4.0.0-1
XEN administrative tools - common files
ii  xenstore-utils                      4.0.1-4
Xenstore utilities for Xen

Regards,
Dmitry

-- System Information:
Debian Release: 6.0.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-xen-amd64 (SMP w/4 CPU cores)
Locale: LANG=ru_RU.UTF-8, LC_CTYPE=ru_RU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

xen-hypervisor-4.0-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.0-amd64 recommends:
ii  xen-utils-4.0                 4.0.1-4    XEN administrative tools

Versions of packages xen-hypervisor-4.0-amd64 suggests:
pn  xen-docs-4.0                  <none>     (no description available)

-- no debconf information

On 03/29/2012 07:39 AM, Dmitry Ogorodnikov wrote:
> Package: xen-hypervisor-4.0-amd64
> Version: 4.0.1-4
> Severity: normal
> 
> Hi all,
> 
> I have a problem with libvirt, it works slow. For example, "virsh
> list" takes 6..7 seconds.
> 
> I ask xen-users list with no luck. According to google, this problem
> is not wide spreaded, so maybe it's debian specific. If not, please
> hint where I should go next.
> 
> I found this in libvirt log:
Hi Dmitry,

If you are having an issue with libvirt, why do you send a bug report
against the Xen hypervisor? It doesn't make sense.
> I have no idea how I can tune it, so I suppose this is a bug.
If you have no idea, IMO you should ask in the relevant user lists,
rather than just sending a bug report against any random package. Also,
you can't expect the maintainer of the Xen hypervisor to know specific
libvirt issues, I believe (I am a heavy user of Xen myself, but never
used Xen with libvirt).

Cheers,

Thomas

Your message dated Fri, 24 Aug 2012 18:25:21 +0200
with message-id <20120824162521.GA6536 at wavehammer.waldi.eu.org>
and subject line Re: [Pkg-xen-devel] Bug#666135: Multiple "Domain-0",
slow libvirt
has caused the Debian Bug report #666135,
regarding Multiple "Domain-0", slow libvirt
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
666135: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666135
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Dmitry Ogorodnikov <dmitry.b.ogorodnikov at gmail.com>
Subject: Multiple "Domain-0", slow libvirt
Date: Thu, 29 Mar 2012 10:39:12 +1100
Size: 9746
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20120824/d6e541f2/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Bastian Blank <waldi at debian.org>
Subject: Re: [Pkg-xen-devel] Bug#666135: Multiple "Domain-0", slow
libvirt
Date: Fri, 24 Aug 2012 18:25:21 +0200
Size: 2026
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20120824/d6e541f2/attachment-0001.mht>