thr3ads.net - Xen devel - [Xen-devel] VM start time slows with large number of VMs [Sep 2007]

If this information is useful, please help other people find it:
Share via:

Carb, Brian A

2007-Sep-04 16:51 UTC

[Xen-devel] VM start time slows with large number of VMs

Hello,
 
We''re running Xen Unstable changeset 15468 with SLES10 on a Unisys
ES7000/one 56x cpu and 160GB memory.  Xen is built with NR_DYNIRQS=1024
and max_phys_cpus=64, and booted with dom0_mem=1024 and
xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on
external storage.
 
Our script starts a VM, waits for the start to complete, then starts the
next: "for i in *.cfg; do xm create $i; done". As more VMs start, the
time to start each one increases. I''ve attached a config file for one
of
the VMs, as well as a portion of the xend.log which shows timestamps of
the events for 2 VMS, vm002 and vm263.
 
The total elapsed time of the start from "create" to
"unpaused" for
vm002 is 13 seconds, but by the time 262 VMs are started, the start of
vm263 has slowed to 238 seconds. The host itself is not sluggish, as we
can open putty sessions to and interact with dom0.  However, if we run
"xm top", its interface is very unresponsive.  Could writing to the
xenstore be taking longer and longer? Could there be possible contention
for vmlinuz and initrd since they are being used by every VM to boot?
Would it be better to use bootloader/bootentry and store the vmlinuz and
initrd within each VM?
 
Any insight would be appreciated.
 
brian carb
unisys corporation, malvern pa
 
----- domU config file 002.cfg -----
kernel = "/boot/vmlinuz-xen"
ramdisk = "/boot/initrd-xen"
root = "/dev/sda2 ro"
disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
memory = 512
vcpus = 4
name = ''xen002''
vif = [ ''mac=00:16:3e:0d:a0:01'' ]
ip = ''10.0.27.2''
netmask = ''255.254.0.0''
extra = '' xencons=tty1 3''

----- xend.log -----
[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create([''vm'', [''name'',
''xen002''], [''memory'', 512],
[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
''ignore''],
[''image'', [''linux'',
[''kernel'', ''/boot/vmlinuz-xen''],
[''ramdisk'',
''/boot/initrd-xen''], [''ip'',
''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
[''args'', '' xencons=tty1 3'']]],
[''device'',
[''vbd'', [''uname'',
''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'',
''w'']]], [''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 KiB
free; need 2048; done.
[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 16 256
[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 KiB
free; need 524288; done.
[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux dom=16
vcpus=4
[2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        = 512
[2007-09-02 05:26:43 8597] DEBUG (image:199) image          /boot/vmlinuz-xen
[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
[2007-09-02 05:26:43 8597] DEBUG (image:205) features       = 
[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: vbd
: {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'': 1,
''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
''phy:/tmp/xendevs/xen002'', ''mode'':
''w''}
[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController:
writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
''protocol'':
''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
''/local/domain/0/backend/vbd/16/2048''} to
/local/domain/16/device/vbd/2048.
[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen002'',
''frontend'':
''/local/domain/16/device/vbd/2048'', ''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'',
''params'': ''/tmp/xendevs/xen002'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''16'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/16/2048.
[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: vif
: {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController:
writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
''/local/domain/16/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'', ''online'':
''1'', ''frontend-id'': ''16'',
''type'': ''netfront''} to
/local/domain/0/backend/vif/16/0.
[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM
details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
''start_time'': ''1188725206.15'',
''on_poweroff'': ''destroy'',
''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
''xend/restart_count'':
''0'', ''vcpus'': ''4'',
''vcpu_avail'': ''15'',
''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args ''root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') (notes
(FEATURES
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", ''name'':
''xen002''}
[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {''console/ring-ref'': ''34194913'',
''image/entry'': ''-2145386496'',
''console/port'': ''2'',
''cpu/3/availability'': ''online'',
''store/ring-ref'':
''34194914'', ''image/loader'':
''generic'', ''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController:
writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
''backend'':
''/local/domain/0/backend/console/16/0''} to
/local/domain/16/device/console/0.
[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'',
''online'': ''1'',
''frontend-id'': ''16''} to
/local/domain/0/backend/console/16/0.
[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices
vif.
[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {''console/ring-ref'': ''34194913'',
''image/entry'': ''-2145386496'',
''console/port'': ''2'',
''cpu/3/availability'': ''online'',
''store/ring-ref'':
''34194914'', ''image/loader'':
''generic'', ''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
[2007-09-02 05:26:53 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
[2007-09-02 05:26:53 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
usb.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vbd.
[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
[2007-09-02 05:26:54 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/16/2048/hotplug-status.
[2007-09-02 05:26:54 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
irq.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vfb.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
console.
[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
pci.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
ioports.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
tap.
[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
unpaused.
...
... [vms xen003 to xen262 deleted]
...
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create([''vm'', [''name'',
''xen263''], [''memory'', 512],
[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
''ignore''],
[''image'', [''linux'',
[''kernel'', ''/boot/vmlinuz-xen''],
[''ramdisk'',
''/boot/initrd-xen''], [''ip'',
''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
[''args'', '' xencons=tty1 3'']]],
[''device'', [''vbd'',
[''uname'',
''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
[''device'',
[''vif'', [''mac'',
''00:16:3e:0d:a0:01'']]]])
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB
free; need 2048; done.
[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 277 256
[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB
free; need 524288; done.
[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux dom=277
vcpus=4
[2007-09-02 17:44:29 8597] DEBUG (image:197) domid          = 277
[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        = 512
[2007-09-02 17:44:29 8597] DEBUG (image:199) image          /boot/vmlinuz-xen
[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
[2007-09-02 17:44:29 8597] DEBUG (image:205) features       = 
[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vbd
: {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'': 1,
''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
''phy:/tmp/xendevs/xen263'', ''mode'':
''w''}
[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController:
writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
''protocol'':
''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
''/local/domain/0/backend/vbd/277/2048''} to
/local/domain/277/device/vbd/2048.
[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen263'',
''frontend'':
''/local/domain/277/device/vbd/2048'', ''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'',
''params'': ''/tmp/xendevs/xen263'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''277'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/277/2048.
[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vif
: {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
''a126d815-fd4a-332b-c881-235a7e003cfb''}
[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController:
writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
''/local/domain/277/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'', ''online'':
''1'', ''frontend-id'': ''277'',
''type'': ''netfront''} to
/local/domain/0/backend/vif/277/0.
[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM
details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
''start_time'': ''1188769556.11'',
''on_poweroff'': ''destroy'',
''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
''xend/restart_count'':
''0'', ''vcpus'': ''4'',
''vcpu_avail'': ''15'',
''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args ''root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') (notes
(FEATURES
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", ''name'':
''xen263''}
[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {''console/ring-ref'': ''5227479'',
''image/entry'': ''-2145386496'',
''console/port'': ''2'',
''cpu/3/availability'': ''online'',
''store/ring-ref'':
''5227480'', ''image/loader'':
''generic'', ''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController:
writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
''backend'':
''/local/domain/0/backend/console/277/0''} to
/local/domain/277/device/console/0.
[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController:
writing {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'',
''online'': ''1'',
''frontend-id'': ''277''} to
/local/domain/0/backend/console/277/0.
[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain
details: {''console/ring-ref'': ''5227479'',
''image/entry'': ''-2145386496'',
''console/port'': ''2'',
''cpu/3/availability'': ''online'',
''store/ring-ref'':
''5227480'', ''image/loader'':
''generic'', ''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices
vif.
[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 17:48:23 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
[2007-09-02 17:48:23 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
usb.
[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
vbd.
[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
[2007-09-02 17:48:25 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/277/2048/hotplug-status.
[2007-09-02 17:48:25 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
irq.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vfb.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
console.
[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
pci.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
ioports.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
tap.
[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 (277)
unpaused.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Sep-05 17:11 UTC

head link

Re: [Xen-devel] VM start time slows with large number of VMs

xenstore slowness seems a reasonable hypothesis. How does performance of
time xenstore-ls >/dev/null¹ change as you increase the number of guests?

 -- Keir

On 4/9/07 17:51, "Carb, Brian A" <Brian.Carb@unisys.com> wrote:
> Hello,
>  
> We''re running Xen Unstable changeset 15468 with SLES10 on a Unisys
ES7000/one
> 56x cpu and 160GB memory.  Xen is built with NR_DYNIRQS=1024 and
> max_phys_cpus=64, and booted with dom0_mem=1024 and xenheap_megabytes=64.
We
> have 320 para-virtualized SLES10 VMs on external storage.
>  
> Our script starts a VM, waits for the start to complete, then starts the
next:
> "for i in *.cfg; do xm create $i; done". As more VMs start, the
time to start
> each one increases. I''ve attached a config file for one of the
VMs, as well as
> a portion of the xend.log which shows timestamps of the events for 2 VMS,
> vm002 and vm263.
>  
> The total elapsed time of the start from "create" to
"unpaused" for vm002 is
> 13 seconds, but by the time 262 VMs are started, the start of vm263 has
slowed
> to 238 seconds. The host itself is not sluggish, as we can open putty
sessions
> to and interact with dom0.  However, if we run "xm top", its
interface is very
> unresponsive.  Could writing to the xenstore be taking longer and longer?
> Could there be possible contention for vmlinuz and initrd since they are
being
> used by every VM to boot? Would it be better to use bootloader/bootentry
and
> store the vmlinuz and initrd within each VM?
>  
> Any insight would be appreciated.
>  
> brian carb
> unisys corporation, malvern pa
>  
> ----- domU config file 002.cfg -----
> kernel = "/boot/vmlinuz-xen"
> ramdisk = "/boot/initrd-xen"
> root = "/dev/sda2 ro"
> disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
> memory = 512
> vcpus = 4
> name = ''xen002''
> vif = [ ''mac=00:16:3e:0d:a0:01'' ]
> ip = ''10.0.27.2''
> netmask = ''255.254.0.0''
> extra = '' xencons=tty1 3''
> ----- xend.log -----
> [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78)
> XendDomainInfo.create([''vm'', [''name'',
''xen002''], [''memory'', 512],
[''vcpus'',
> 4], [''on_xend_start'', ''ignore''],
[''on_xend_stop'', ''ignore''],
[''image'',
> [''linux'', [''kernel'',
''/boot/vmlinuz-xen''], [''ramdisk'',
''/boot/initrd-xen''],
> [''ip'',
''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
> [''args'', '' xencons=tty1 3'']]],
[''device'', [''vbd'',
[''uname'',
> ''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'', ''w'']]],
[''device'',
> [''vif'', [''mac'',
''00:16:3e:0d:a0:01'']]]])
> [2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418)
> XendDomainInfo.constructDomain
> [2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 KiB free;
> need 2048; done.
> [2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
> [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472)
> XendDomainInfo.initDomain: 16 256
> [2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503)
> _initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
> memory_static_min=0x0.
> [2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 KiB free;
> need 524288; done.
> [2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux dom=16
> vcpus=4
> [2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
> [2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        = 512
> [2007-09-02 05:26:43 8597] DEBUG (image:199) image          >
/boot/vmlinuz-xen
> [2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
> [2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
> [2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        =
root=/dev/sda2
> ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3
> [2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk        =
/boot/initrd-xen
> [2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
> [2007-09-02 05:26:43 8597] DEBUG (image:205) features       >
[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice: vbd :
> {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'': 1, ''driver'':
> ''paravirtualised'', ''dev'':
''sda'', ''uname'':
''phy:/tmp/xendevs/xen002'', ''mode'':
> ''w''}
> [2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: writing
> {''virtual-device'': ''2048'',
''device-type'': ''disk'',
''protocol'': ''x86_64-abi'',
> ''backend-id'': ''0'',
''state'': ''1'', ''backend'':
> ''/local/domain/0/backend/vbd/16/2048''} to
/local/domain/16/device/vbd/2048.
> [2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen002'',
''frontend'':
''/local/domain/16/device/vbd/2048'', ''uuid'':
> ''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'', ''params'':
> ''/tmp/xendevs/xen002'', ''mode'':
''w'', ''online'': ''1'',
''frontend-id'': ''16'',
> ''type'': ''phy''} to
/local/domain/0/backend/vbd/16/2048.
> [2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: vif :
> {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
> [2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: writing
> {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
''x86_64-abi'',
> ''backend-id'': ''0'',
''state'': ''1'', ''backend'':
> ''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
> [2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
> ''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
> ''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
> ''/local/domain/16/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'', ''online'':
''1'',
> ''frontend-id'': ''16'',
''type'': ''netfront''} to
/local/domain/0/backend/vif/16/0.
> [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM details:
> {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
> ''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
''start_time'':
> ''1188725206.15'', ''on_poweroff'':
''destroy'', ''on_xend_start'':
''ignore'',
> ''on_crash'': ''restart'',
''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'':
> ''15'', ''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen)
> (args ''root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
> xencons=tty1 3'') (notes (FEATURES
>
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_p
> gdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION
> 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE
-2145361920)
> (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION
> xen-3.0)))", ''name'': ''xen002''}
> [2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain
details:
> {''console/ring-ref'': ''34194913'',
''image/entry'': ''-2145386496'',
''console/port'':
> ''2'', ''cpu/3/availability'':
''online'', ''store/ring-ref'':
''34194914'',
> ''image/loader'': ''generic'',
''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
> ''control/platform-feature-multiprocessor-suspend'':
''1'', ''cpu/2/availability'':
> ''online'', ''cpu/1/availability'':
''online'',
> ''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> ''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
> ''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
> ''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
> ''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
> ''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> ''image/features/writable-page-tables'':
''1'',
> ''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'':
> ''16'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
> [2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: writing
> {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
''backend'':
> ''/local/domain/0/backend/console/16/0''} to
/local/domain/16/device/console/0.
> [2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
> ''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
> ''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'', ''online'':
> ''1'', ''frontend-id'':
''16''} to /local/domain/0/backend/console/16/0.
> [2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices
vif.
> [2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
> [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain
details:
> {''console/ring-ref'': ''34194913'',
''image/entry'': ''-2145386496'',
''console/port'':
> ''2'', ''cpu/3/availability'':
''online'', ''store/ring-ref'':
''34194914'',
> ''image/loader'': ''generic'',
''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
> ''control/platform-feature-multiprocessor-suspend'':
''1'', ''cpu/2/availability'':
> ''online'', ''cpu/1/availability'':
''online'',
> ''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> ''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
> ''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
> ''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
> ''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
> ''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> ''image/features/writable-page-tables'':
''1'',
> ''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'':
> ''16'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
> [2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907)
> XendDomainInfo.handleShutdownWatch
> [2007-09-02 05:26:53 8597] DEBUG (DevController:541) hotplugStatusCallback
> /local/domain/0/backend/vif/16/0/hotplug-status.
> [2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
usb.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vbd.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:541) hotplugStatusCallback
> /local/domain/0/backend/vbd/16/2048/hotplug-status.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
irq.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
> [2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vfb.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
> console.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
pci.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
> ioports.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
tap.
> [2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
> [2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
unpaused.
> ...
> ... [vms xen003 to xen262 deleted]
> ...
> [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78)
> XendDomainInfo.create([''vm'', [''name'',
''xen263''], [''memory'', 512],
[''vcpus'',
> 4], [''on_xend_start'', ''ignore''],
[''on_xend_stop'', ''ignore''],
[''image'',
> [''linux'', [''kernel'',
''/boot/vmlinuz-xen''], [''ramdisk'',
''/boot/initrd-xen''],
> [''ip'',
''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
> [''args'', '' xencons=tty1 3'']]],
[''device'', [''vbd'',
[''uname'',
> ''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
[''device'',
> [''vif'', [''mac'',
''00:16:3e:0d:a0:01'']]]])
> [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418)
> XendDomainInfo.constructDomain
> [2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB free;
> need 2048; done.
> [2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
> [2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472)
> XendDomainInfo.initDomain: 277 256
> [2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503)
> _initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
> memory_static_min=0x0.
> [2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB free;
> need 524288; done.
> [2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux dom=277
> vcpus=4
> [2007-09-02 17:44:29 8597] DEBUG (image:197) domid          = 277
> [2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        = 512
> [2007-09-02 17:44:29 8597] DEBUG (image:199) image          >
/boot/vmlinuz-xen
> [2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
> [2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
> [2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        =
root=/dev/sda2
> ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3
> [2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk        =
/boot/initrd-xen
> [2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
> [2007-09-02 17:44:29 8597] DEBUG (image:205) features       >
[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vbd :
> {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'': 1, ''driver'':
> ''paravirtualised'', ''dev'':
''sda'', ''uname'':
''phy:/tmp/xendevs/xen263'', ''mode'':
> ''w''}
> [2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: writing
> {''virtual-device'': ''2048'',
''device-type'': ''disk'',
''protocol'': ''x86_64-abi'',
> ''backend-id'': ''0'',
''state'': ''1'', ''backend'':
> ''/local/domain/0/backend/vbd/277/2048''} to
/local/domain/277/device/vbd/2048.
> [2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen263'',
''frontend'':
''/local/domain/277/device/vbd/2048'', ''uuid'':
> ''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'', ''params'':
> ''/tmp/xendevs/xen263'', ''mode'':
''w'', ''online'': ''1'',
''frontend-id'': ''277'',
> ''type'': ''phy''} to
/local/domain/0/backend/vbd/277/2048.
> [2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: vif :
> {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
''a126d815-fd4a-332b-c881-235a7e003cfb''}
> [2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: writing
> {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
''x86_64-abi'',
> ''backend-id'': ''0'',
''state'': ''1'', ''backend'':
> ''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
> [2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
> ''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
> ''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
> ''/local/domain/277/device/vif/0'',
''mac'': ''00:16:3e:0d:a0:01'',
''online'': ''1'',
> ''frontend-id'': ''277'',
''type'': ''netfront''} to
> /local/domain/0/backend/vif/277/0.
> [2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM details:
> {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
> ''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
''start_time'':
> ''1188769556.11'', ''on_poweroff'':
''destroy'', ''on_xend_start'':
''ignore'',
> ''on_crash'': ''restart'',
''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'':
> ''15'', ''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen)
> (args ''root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
> xencons=tty1 3'') (notes (FEATURES
>
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_p
> gdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION
> 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux) (HYPERCALL_PAGE
-2145361920)
> (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION
> xen-3.0)))", ''name'': ''xen263''}
> [2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain
details:
> {''console/ring-ref'': ''5227479'',
''image/entry'': ''-2145386496'',
''console/port'':
> ''2'', ''cpu/3/availability'':
''online'', ''store/ring-ref'':
''5227480'',
> ''image/loader'': ''generic'',
''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
> ''control/platform-feature-multiprocessor-suspend'':
''1'', ''cpu/2/availability'':
> ''online'', ''cpu/1/availability'':
''online'',
> ''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> ''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
> ''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
> ''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
> ''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
> ''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> ''image/features/writable-page-tables'':
''1'',
> ''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'':
> ''277'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
> [2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: writing
> {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
''backend'':
> ''/local/domain/0/backend/console/277/0''} to
> /local/domain/277/device/console/0.
> [2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: writing
> {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
> ''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
> ''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'', ''online'':
> ''1'', ''frontend-id'':
''277''} to /local/domain/0/backend/console/277/0.
> [2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain
details:
> {''console/ring-ref'': ''5227479'',
''image/entry'': ''-2145386496'',
''console/port'':
> ''2'', ''cpu/3/availability'':
''online'', ''store/ring-ref'':
''5227480'',
> ''image/loader'': ''generic'',
''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
> ''control/platform-feature-multiprocessor-suspend'':
''1'', ''cpu/2/availability'':
> ''online'', ''cpu/1/availability'':
''online'',
> ''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> ''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
> ''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
> ''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
> ''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
> ''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> ''image/features/writable-page-tables'':
''1'',
> ''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'':
> ''277'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
> [2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907)
> XendDomainInfo.handleShutdownWatch
> [2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices
vif.
> [2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
> [2007-09-02 17:48:23 8597] DEBUG (DevController:541) hotplugStatusCallback
> /local/domain/0/backend/vif/277/0/hotplug-status.
> [2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
> [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
usb.
> [2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
vbd.
> [2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
> [2007-09-02 17:48:25 8597] DEBUG (DevController:541) hotplugStatusCallback
> /local/domain/0/backend/vbd/277/2048/hotplug-status.
> [2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
> [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
irq.
> [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
> [2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vfb.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
> console.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
pci.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
> ioports.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
tap.
> [2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
> [2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 (277)
> unpaused.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel P. Berrange

2007-Sep-05 17:37 UTC

head link

Re: [Xen-devel] VM start time slows with large number of VMs

On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser
wrote:> xenstore slowness seems a reasonable hypothesis. How does performance of
> time xenstore-ls >/dev/null¹ change as you increase the number of
guests?
There''s another issue with the way hotplug scripts do locking which
exhbits
fairly pathelogical behaviour when a large number of guests are started 
concurrently. We''ve got a patch we''re evaluating which changes
the use of
lock files + looping + sleeps, to instead use the ''flock''
shell util.

I suspect the performance issues will be a combination of both the locking stuff
and the xenstore I/O. You might also want to try putting /var/lib/xenstored
onto tmpfs eg in  initscripts before xenstored starts ...

  mkdir /dev/shm/xenstored
  mount --bind /dev/shm/xenstored /var/lib/xenstored


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Carb, Brian A

2007-Sep-06 16:24 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

keir
 
I''ll try timing this as I load up... 
 
but when I restart my test system, with only dom0 running, xenstore-ls
is already slow before any VMs are started:
 
  time xenstore-ls >/dev/null
  real    0m21.725s
  user    0m0.010s
  sys     0m0.090s
 
If I save the output from the xenstore-ls, the file has 61,318 lines
with information from all the VMs run yesterday. I conclude that
xenstore does not get cleared on startup. Is there a way to clear it?
 
brian carb
unisys corporation - malvern, pa
 

________________________________

From: Keir Fraser  
Sent: Wednesday, September 05, 2007 1:11 PM
To: Carb, Brian A; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] VM start time slows with large number of VMs


xenstore slowness seems a reasonable hypothesis. How does performance of
''time xenstore-ls >/dev/null'' change as you increase the
number of
guests?

 -- Keir

On 4/9/07 17:51, "Carb, Brian A" wrote:



	Hello,
	
	We''re running Xen Unstable changeset 15468 with SLES10 on a
Unisys ES7000/one 56x cpu and 160GB memory.  Xen is built with
NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 and
xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on
external storage.
	
	Our script starts a VM, waits for the start to complete, then
starts the next: "for i in *.cfg; do xm create $i; done". As more VMs
start, the time to start each one increases. I''ve attached a config
file
for one of the VMs, as well as a portion of the xend.log which shows
timestamps of the events for 2 VMS, vm002 and vm263.
	
	The total elapsed time of the start from "create" to
"unpaused"
for vm002 is 13 seconds, but by the time 262 VMs are started, the start
of vm263 has slowed to 238 seconds. The host itself is not sluggish, as
we can open putty sessions to and interact with dom0.  However, if we
run "xm top", its interface is very unresponsive.  Could writing to
the
xenstore be taking longer and longer? Could there be possible contention
for vmlinuz and initrd since they are being used by every VM to boot?
Would it be better to use bootloader/bootentry and store the vmlinuz and
initrd within each VM?
	
	Any insight would be appreciated.
	
	brian carb
	unisys corporation, malvern pa
	
	----- domU config file 002.cfg -----
	kernel = "/boot/vmlinuz-xen"
	ramdisk = "/boot/initrd-xen"
	root = "/dev/sda2 ro"
	disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
	memory = 512
	vcpus = 4
	name = ''xen002''
	vif = [ ''mac=00:16:3e:0d:a0:01'' ]
	ip = ''10.0.27.2''
	netmask = ''255.254.0.0''
	extra = '' xencons=tty1 3''
	----- xend.log -----
	[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create([''vm'', [''name'',
''xen002''], [''memory'', 512],
[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
''ignore''],
[''image'', [''linux'',
[''kernel'', ''/boot/vmlinuz-xen''],
[''ramdisk'',
''/boot/initrd-xen''], [''ip'',
''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
[''args'', '' xencons=tty1 3'']]],
[''device'',
[''vbd'', [''uname'',
''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'',
''w'']]], [''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
	[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
	[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon:
169819872 KiB free; need 2048; done.
	[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain:
16
	[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 16 256
	[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
	[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon:
169819864 KiB free; need 524288; done.
	[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux
dom=16 vcpus=4
	[2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
	[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        512
	[2007-09-02 05:26:43 8597] DEBUG (image:199) image          /boot/vmlinuz-xen
	[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
	[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
	[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
	[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
	[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
	[2007-09-02 05:26:43 8597] DEBUG (image:205) features       = 
	[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321)
createDevice: vbd : {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'': 1, ''driver'':
''paravirtualised'', ''dev'':
''sda'', ''uname'':
''phy:/tmp/xendevs/xen002'', ''mode'':
''w''}
	[2007-09-02 05:26:43 8597] DEBUG (DevController:115)
DevController: writing {''virtual-device'':
''2048'', ''device-type'':
''disk'',
''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vbd/16/2048''} to
/local/domain/16/device/vbd/2048.
	[2007-09-02 05:26:43 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen002'',
''frontend'':
''/local/domain/16/device/vbd/2048'', ''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'',
''params'': ''/tmp/xendevs/xen002'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''16'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/16/2048.
	[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321)
createDevice: vif : {''mac'':
''00:16:3e:0d:a0:01'', ''uuid'':
''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
	[2007-09-02 05:26:45 8597] DEBUG (DevController:115)
DevController: writing {''mac'':
''00:16:3e:0d:a0:01'', ''handle'':
''0'',
''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
	[2007-09-02 05:26:45 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
''/local/domain/16/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'', ''online'':
''1'', ''frontend-id'': ''16'',
''type'': ''netfront''} to
/local/domain/0/backend/vif/16/0.
	[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing
VM details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
''start_time'': ''1188725206.15'',
''on_poweroff'': ''destroy'',
''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
''xend/restart_count'':
''0'', ''vcpus'': ''4'',
''vcpu_avail'': ''15'',
''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args ''root=/dev/sda2 ro
ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') (notes
(FEATURES
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", ''name'':
''xen002''}
	[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing
domain details: {''console/ring-ref'':
''34194913'', ''image/entry'':
''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
	[2007-09-02 05:26:50 8597] DEBUG (DevController:115)
DevController: writing {''protocol'':
''x86_64-abi'', ''state'':
''1'',
''backend-id'': ''0'',
''backend'':
''/local/domain/0/backend/console/16/0''} to
/local/domain/16/device/console/0.
	[2007-09-02 05:26:50 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'',
''online'': ''1'',
''frontend-id'': ''16''} to
/local/domain/0/backend/console/16/0.
	[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for
devices vif.
	[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for
0.
	[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing
domain details: {''console/ring-ref'':
''34194913'', ''image/entry'':
''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
	[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
	[2007-09-02 05:26:53 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
	[2007-09-02 05:26:53 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices usb.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vbd.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for
2048.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/16/2048/hotplug-status.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices irq.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vkbd.
	[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vfb.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices console.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for
0.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices pci.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices ioports.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices tap.
	[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices vtpm.
	[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002
(16) unpaused.
	...
	... [vms xen003 to xen262 deleted]
	...
	[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78)
XendDomainInfo.create([''vm'', [''name'',
''xen263''], [''memory'', 512],
[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
''ignore''],
[''image'', [''linux'',
[''kernel'', ''/boot/vmlinuz-xen''],
[''ramdisk'',
''/boot/initrd-xen''], [''ip'',
''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2 ro''],
[''args'', '' xencons=tty1 3'']]],
[''device'', [''vbd'',
[''uname'',
''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
[''device'',
[''vif'', [''mac'',
''00:16:3e:0d:a0:01'']]]])
	[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418)
XendDomainInfo.constructDomain
	[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616
KiB free; need 2048; done.
	[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain:
277
	[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472)
XendDomainInfo.initDomain: 277 256
	[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503)
_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
memory_static_min=0x0.
	[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608
KiB free; need 524288; done.
	[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux
dom=277 vcpus=4
	[2007-09-02 17:44:29 8597] DEBUG (image:197) domid          277
	[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        512
	[2007-09-02 17:44:29 8597] DEBUG (image:199) image          /boot/vmlinuz-xen
	[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
	[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
	[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3
	[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
	[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
	[2007-09-02 17:44:29 8597] DEBUG (image:205) features       = 
	[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321)
createDevice: vbd : {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'': 1, ''driver'':
''paravirtualised'', ''dev'':
''sda'', ''uname'':
''phy:/tmp/xendevs/xen263'', ''mode'':
''w''}
	[2007-09-02 17:44:30 8597] DEBUG (DevController:115)
DevController: writing {''virtual-device'':
''2048'', ''device-type'':
''disk'',
''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vbd/277/2048''} to
/local/domain/277/device/vbd/2048.
	[2007-09-02 17:44:30 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen263'',
''frontend'':
''/local/domain/277/device/vbd/2048'', ''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'',
''params'': ''/tmp/xendevs/xen263'',
''mode'': ''w'', ''online'':
''1'',
''frontend-id'': ''277'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/277/2048.
	[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321)
createDevice: vif : {''mac'':
''00:16:3e:0d:a0:01'', ''uuid'':
''a126d815-fd4a-332b-c881-235a7e003cfb''}
	[2007-09-02 17:45:02 8597] DEBUG (DevController:115)
DevController: writing {''mac'':
''00:16:3e:0d:a0:01'', ''handle'':
''0'',
''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'', ''backend'':
''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
	[2007-09-02 17:45:02 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
''/local/domain/277/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'', ''online'':
''1'', ''frontend-id'': ''277'',
''type'': ''netfront''} to
/local/domain/0/backend/vif/277/0.
	[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing
VM details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
''start_time'': ''1188769556.11'',
''on_poweroff'': ''destroy'',
''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
''xend/restart_count'':
''0'', ''vcpus'': ''4'',
''vcpu_avail'': ''15'',
''image'': "(linux (kernel
/boot/vmlinuz-xen) (ramdisk /boot/initrd-xen) (args ''root=/dev/sda2 ro
ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') (notes
(FEATURES
''writable_page_tables|writable_descriptor_tables|auto_translated_physmap
|pae_pgdir_above_4gb|supervisor_mode_kernel'') (VIRT_BASE -2147483648)
(GUEST_VERSION 2.6) (PADDR_OFFSET -2147483648) (GUEST_OS linux)
(HYPERCALL_PAGE -2145361920) (LOADER generic) (SUSPEND_CANCEL 1) (ENTRY
-2145386496) (XEN_VERSION xen-3.0)))", ''name'':
''xen263''}
	[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing
domain details: {''console/ring-ref'':
''5227479'', ''image/entry'':
''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
	[2007-09-02 17:47:25 8597] DEBUG (DevController:115)
DevController: writing {''protocol'':
''x86_64-abi'', ''state'':
''1'',
''backend-id'': ''0'',
''backend'':
''/local/domain/0/backend/console/277/0''}
to /local/domain/277/device/console/0.
	[2007-09-02 17:47:25 8597] DEBUG (DevController:117)
DevController: writing {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'',
''online'': ''1'',
''frontend-id'': ''277''} to
/local/domain/0/backend/console/277/0.
	[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing
domain details: {''console/ring-ref'':
''5227479'', ''image/entry'':
''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
''2.6'',
''image/features/supervisor-mode-kernel'':
''1'', ''image/guest-os'':
''linux'',
''console/limit'': ''1048576'',
''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'': ''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'': ''online'',
''image/features/pae-pgdir-above-4gb'':
''1'', ''image/features/writable-page-tables'':
''1'',
''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
	[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907)
XendDomainInfo.handleShutdownWatch
	[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for
devices vif.
	[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for
0.
	[2007-09-02 17:48:23 8597] DEBUG (DevController:541)
hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
	[2007-09-02 17:48:23 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
	[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
devices usb.
	[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
devices vbd.
	[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for
2048.
	[2007-09-02 17:48:25 8597] DEBUG (DevController:541)
hotplugStatusCallback
/local/domain/0/backend/vbd/277/2048/hotplug-status.
	[2007-09-02 17:48:25 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.
	[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices irq.
	[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices vkbd.
	[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices vfb.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices console.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for
0.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices pci.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices ioports.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices tap.
	[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices vtpm.
	[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263
(277) unpaused.
	
	
________________________________

	_______________________________________________
	Xen-devel mailing list
	Xen-devel@lists.xensource.com
	http://lists.xensource.com/xen-devel
	





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Mats Petersson

2007-Sep-06 17:29 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

At 17:24 06/09/2007, Carb, Brian A wrote:>Content-class: urn:content-classes:message
>Content-Type: multipart/alternative;
>         boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0"
>
>keir
>
>I''ll try timing this as I load up...
>
>but when I restart my test system, with only dom0 running, 
>xenstore-ls is already slow before any VMs are started:
>
>   time xenstore-ls >/dev/null
>   real    0m21.725s
>   user    0m0.010s
>   sys     0m0.090s
>
>If I save the output from the xenstore-ls, the file has 61,318 lines 
>with information from all the VMs run yesterday. I conclude that 
>xenstore does not get cleared on startup. Is there a way to clear it?

It is perfectly fine to remove the xenstore.tdb [filename from 
memory] in /var/xen/lib [again, from memory]. Unless of course you 
actually NEED some of your old Xenstrore entries for soemthing, but 
in general, they are fine to be removed completely.

--
Mats
>
>brian carb
>unisys corporation - malvern, pa
>
>
>
>----------
>From: Keir Fraser
>Sent: Wednesday, September 05, 2007 1:11 PM
>To: Carb, Brian A; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] VM start time slows with large number of VMs
>
>xenstore slowness seems a reasonable hypothesis. How does 
>performance of ''time xenstore-ls >/dev/null'' change as
you increase
>the number of guests?
>
>  -- Keir
>
>On 4/9/07 17:51, "Carb, Brian A" wrote:
>
>Hello,
>
>We''re running Xen Unstable changeset 15468 with SLES10 on a Unisys 
>ES7000/one 56x cpu and 160GB memory.  Xen is built with 
>NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 
>and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on 
>external storage.
>
>Our script starts a VM, waits for the start to complete, then starts 
>the next: "for i in *.cfg; do xm create $i; done". As more VMs 
>start, the time to start each one increases. I''ve attached a config
>file for one of the VMs, as well as a portion of the xend.log which 
>shows timestamps of the events for 2 VMS, vm002 and vm263.
>
>The total elapsed time of the start from "create" to
"unpaused" for
>vm002 is 13 seconds, but by the time 262 VMs are started, the start 
>of vm263 has slowed to 238 seconds. The host itself is not sluggish, 
>as we can open putty sessions to and interact with dom0.  However, 
>if we run "xm top", its interface is very unresponsive.  Could 
>writing to the xenstore be taking longer and longer? Could there be 
>possible contention for vmlinuz and initrd since they are being used 
>by every VM to boot? Would it be better to use bootloader/bootentry 
>and store the vmlinuz and initrd within each VM?
>
>Any insight would be appreciated.
>
>brian carb
>unisys corporation, malvern pa
>
>----- domU config file 002.cfg -----
>kernel = "/boot/vmlinuz-xen"
>ramdisk = "/boot/initrd-xen"
>root = "/dev/sda2 ro"
>disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
>memory = 512
>vcpus = 4
>name = ''xen002''
>vif = [ ''mac=00:16:3e:0d:a0:01'' ]
>ip = ''10.0.27.2''
>netmask = ''255.254.0.0''
>extra = '' xencons=tty1 3''
>----- xend.log -----
>[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) 
>XendDomainInfo.create([''vm'', [''name'',
''xen002''], [''memory'', 512],
>[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
>''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
>[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
>''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
>ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
>''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'', ''w'']]],
>[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
>[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) 
>XendDomainInfo.constructDomain
>[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 
>KiB free; need 2048; done.
>[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
>[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) 
>XendDomainInfo.initDomain: 16 256
>[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) 
>_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, 
>memory_static_min=0x0.
>[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 
>KiB free; need 524288; done.
>[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux 
>dom=16 vcpus=4
>[2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
>[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        = 512
>[2007-09-02 05:26:43 8597] DEBUG (image:199) image          = 
>/boot/vmlinuz-xen
>[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
>[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
>[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        = 
>root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1
3
>[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk        =
/boot/initrd-xen
>[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
>[2007-09-02 05:26:43 8597] DEBUG (image:205) features       >[2007-09-02
05:26:43 8597] INFO (XendDomainInfo:1321) createDevice:
>vbd : {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'':
>1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
>''phy:/tmp/xendevs/xen002'', ''mode'':
''w''}
>[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: 
>writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
>''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
>''backend'':
''/local/domain/0/backend/vbd/16/2048''} to
>/local/domain/16/device/vbd/2048.
>[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''frontend'':
>''/local/domain/16/device/vbd/2048'',
''uuid'':
>''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'',
>''params'': ''/tmp/xendevs/xen002'',
''mode'': ''w'', ''online'':
''1'',
>''frontend-id'': ''16'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/16/2048.
>[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: 
>vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
>''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
>[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: 
>writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
>''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
>''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
>[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
>''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
>''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
>''/local/domain/16/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'',
>''online'': ''1'',
''frontend-id'': ''16'',
''type'': ''netfront''} to
>/local/domain/0/backend/vif/16/0.
>[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM 
>details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
>''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
>''start_time'': ''1188725206.15'',
''on_poweroff'': ''destroy'',
>''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
>''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
>''image'': "(linux (kernel /boot/vmlinuz-xen) (ramdisk 
>/boot/initrd-xen) (args ''root=/dev/sda2 ro 
>ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'')
(notes
>(FEATURES 
>''writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel'')
>(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET 
>-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER 
>generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION 
>xen-3.0)))", ''name'': ''xen002''}
>[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'':
''34194913'', ''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
>''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
>''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: 
>writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
>''backend'':
''/local/domain/0/backend/console/16/0''} to
>/local/domain/16/device/console/0.
>[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
>''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
>''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'',
>''online'': ''1'',
''frontend-id'': ''16''} to
/local/domain/0/backend/console/16/0.
>[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for devices
vif.
>[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'':
''34194913'', ''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
>''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
>''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) 
>XendDomainInfo.handleShutdownWatch
>[2007-09-02 05:26:53 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
>[2007-09-02 05:26:53 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
usb.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vbd.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vbd/16/2048/hotplug-status.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
irq.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for devices
vfb.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for 
>devices console.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
pci.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for 
>devices ioports.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
tap.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
>[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
unpaused.
>...
>... [vms xen003 to xen262 deleted]
>...
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) 
>XendDomainInfo.create([''vm'', [''name'',
''xen263''], [''memory'', 512],
>[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
>''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
>[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
>''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
>ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
>''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
>[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) 
>XendDomainInfo.constructDomain
>[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB 
>free; need 2048; done.
>[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) 
>XendDomainInfo.initDomain: 277 256
>[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) 
>_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, 
>memory_static_min=0x0.
>[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB 
>free; need 524288; done.
>[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux 
>dom=277 vcpus=4
>[2007-09-02 17:44:29 8597] DEBUG (image:197) domid          = 277
>[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        = 512
>[2007-09-02 17:44:29 8597] DEBUG (image:199) image          = 
>/boot/vmlinuz-xen
>[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
>[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
>[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        = 
>root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1
3
>[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk        =
/boot/initrd-xen
>[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
>[2007-09-02 17:44:29 8597] DEBUG (image:205) features       >[2007-09-02
17:44:30 8597] INFO (XendDomainInfo:1321) createDevice:
>vbd : {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'':
>1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
>''phy:/tmp/xendevs/xen263'', ''mode'':
''w''}
>[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: 
>writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
>''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
>''backend'':
''/local/domain/0/backend/vbd/277/2048''} to
>/local/domain/277/device/vbd/2048.
>[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''frontend'':
>''/local/domain/277/device/vbd/2048'',
''uuid'':
>''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'',
>''params'': ''/tmp/xendevs/xen263'',
''mode'': ''w'', ''online'':
''1'',
>''frontend-id'': ''277'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/277/2048.
>[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: 
>vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
>''a126d815-fd4a-332b-c881-235a7e003cfb''}
>[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: 
>writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
>''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
>''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
>[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
>''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
>''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
>''/local/domain/277/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'',
>''online'': ''1'',
''frontend-id'': ''277'',
''type'': ''netfront''} to
>/local/domain/0/backend/vif/277/0.
>[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM 
>details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
>''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
>''start_time'': ''1188769556.11'',
''on_poweroff'': ''destroy'',
>''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
>''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
>''image'': "(linux (kernel /boot/vmlinuz-xen) (ramdisk 
>/boot/initrd-xen) (args ''root=/dev/sda2 ro 
>ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') 
>(notes (FEATURES 
>''writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel'')
>(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET 
>-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER 
>generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION 
>xen-3.0)))", ''name'': ''xen263''}
>[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'': ''5227479'',
''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
>''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
>''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: 
>writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
>''backend'':
''/local/domain/0/backend/console/277/0''} to
>/local/domain/277/device/console/0.
>[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
>''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
>''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'',
>''online'': ''1'',
''frontend-id'': ''277''} to
/local/domain/0/backend/console/277/0.
>[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'': ''5227479'',
''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
>''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
>''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) 
>XendDomainInfo.handleShutdownWatch
>[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for devices
vif.
>[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 17:48:23 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
>[2007-09-02 17:48:23 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
>[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
usb.
>[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for devices
vbd.
>[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vbd/277/2048/hotplug-status.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:555) hotplugStatusCallback
1.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
irq.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vkbd.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for devices
vfb.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for 
>devices console.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
pci.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for 
>devices ioports.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
tap.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for devices
vtpm.
>[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 
>(277) unpaused.
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-devel
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Sep-06 17:31 UTC

head link

Re: [Xen-devel] VM start time slows with large number of VMs

On 6/9/07 18:29, "Mats Petersson" <mats@planetcatfish.com>
wrote:
>> If I save the output from the xenstore-ls, the file has 61,318 lines
>> with information from all the VMs run yesterday. I conclude that
>> xenstore does not get cleared on startup. Is there a way to clear it?
> 
> 
> It is perfectly fine to remove the xenstore.tdb [filename from
> memory] in /var/xen/lib [again, from memory]. Unless of course you
> actually NEED some of your old Xenstrore entries for soemthing, but
> in general, they are fine to be removed completely.
Yes, actually this would be something perhaps we should do from the xend
startup script, if we can determine that xenstored is not running. It''d
be
nice to get a patch.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel P. Berrange

2007-Sep-06 17:44 UTC

head link

Re: [Xen-devel] VM start time slows with large number of VMs

On Thu, Sep 06, 2007 at 06:31:28PM +0100, Keir Fraser
wrote:> On 6/9/07 18:29, "Mats Petersson" <mats@planetcatfish.com>
wrote:
> 
> >> If I save the output from the xenstore-ls, the file has 61,318
lines
> >> with information from all the VMs run yesterday. I conclude that
> >> xenstore does not get cleared on startup. Is there a way to clear
it?
> > 
> > 
> > It is perfectly fine to remove the xenstore.tdb [filename from
> > memory] in /var/xen/lib [again, from memory]. Unless of course you
> > actually NEED some of your old Xenstrore entries for soemthing, but
> > in general, they are fine to be removed completely.
> 
> Yes, actually this would be something perhaps we should do from the xend
> startup script, if we can determine that xenstored is not running.
It''d be
> nice to get a patch.
Or just put the database on /dev/shm so its in RAM - that way it just goes
away whenever you reboot & you reduce the disk I/O overhead xenstored has.
IIRC last time I tested it doubled xenstored performance using /dev/shm

Dan
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Carb, Brian A

2007-Sep-06 19:25 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

I can''t seem to clear out this information. I tried

  /etc/init.d/xend stop
  killall xensored xenconsoled

  rm -rf /var/lib/xenstored   <- has the tdb
  rm -rf /var/lib/xend <- has domains, state and storage dirs


but when i startup xend again, xenstore-ls still shows all the old
domains. I must be missing something.

brian carb
unisys corporation - malvern, pa

-----Original Message-----
From: mats petersson Sent: Thursday, September 06, 2007 1:30 PM
To: Carb, Brian A; Keir Fraser; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] VM start time slows with large number of VMs

At 17:24 06/09/2007, Carb, Brian A wrote:>Content-class: urn:content-classes:message
>Content-Type: multipart/alternative;
>         boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0"
>
>keir
>
>I''ll try timing this as I load up...
>
>but when I restart my test system, with only dom0 running, xenstore-ls 
>is already slow before any VMs are started:
>
>   time xenstore-ls >/dev/null
>   real    0m21.725s
>   user    0m0.010s
>   sys     0m0.090s
>
>If I save the output from the xenstore-ls, the file has 61,318 lines 
>with information from all the VMs run yesterday. I conclude that 
>xenstore does not get cleared on startup. Is there a way to clear it?

It is perfectly fine to remove the xenstore.tdb [filename from memory]
in /var/xen/lib [again, from memory]. Unless of course you actually NEED
some of your old Xenstrore entries for soemthing, but in general, they
are fine to be removed completely.

--
Mats
>
>brian carb
>unisys corporation - malvern, pa
>
>
>
>----------
>From: Keir Fraser
>Sent: Wednesday, September 05, 2007 1:11 PM
>To: Carb, Brian A; xen-devel@lists.xensource.com
>Subject: Re: [Xen-devel] VM start time slows with large number of VMs
>
>xenstore slowness seems a reasonable hypothesis. How does 
>performance of ''time xenstore-ls >/dev/null'' change as
you increase
>the number of guests?
>
>  -- Keir
>
>On 4/9/07 17:51, "Carb, Brian A" wrote:
>
>Hello,
>
>We''re running Xen Unstable changeset 15468 with SLES10 on a Unisys 
>ES7000/one 56x cpu and 160GB memory.  Xen is built with 
>NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024 
>and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on 
>external storage.
>
>Our script starts a VM, waits for the start to complete, then starts 
>the next: "for i in *.cfg; do xm create $i; done". As more VMs 
>start, the time to start each one increases. I''ve attached a config
>file for one of the VMs, as well as a portion of the xend.log which 
>shows timestamps of the events for 2 VMS, vm002 and vm263.
>
>The total elapsed time of the start from "create" to
"unpaused" for
>vm002 is 13 seconds, but by the time 262 VMs are started, the start 
>of vm263 has slowed to 238 seconds. The host itself is not sluggish, 
>as we can open putty sessions to and interact with dom0.  However, 
>if we run "xm top", its interface is very unresponsive.  Could 
>writing to the xenstore be taking longer and longer? Could there be 
>possible contention for vmlinuz and initrd since they are being used 
>by every VM to boot? Would it be better to use bootloader/bootentry 
>and store the vmlinuz and initrd within each VM?
>
>Any insight would be appreciated.
>
>brian carb
>unisys corporation, malvern pa
>
>----- domU config file 002.cfg -----
>kernel = "/boot/vmlinuz-xen"
>ramdisk = "/boot/initrd-xen"
>root = "/dev/sda2 ro"
>disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
>memory = 512
>vcpus = 4
>name = ''xen002''
>vif = [ ''mac=00:16:3e:0d:a0:01'' ]
>ip = ''10.0.27.2''
>netmask = ''255.254.0.0''
>extra = '' xencons=tty1 3''
>----- xend.log -----
>[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78) 
>XendDomainInfo.create([''vm'', [''name'',
''xen002''], [''memory'', 512],
>[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
>''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
>[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
>''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
>ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
>''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'', ''w'']]],
>[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
>[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418) 
>XendDomainInfo.constructDomain
>[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872 
>KiB free; need 2048; done.
>[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
>[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472) 
>XendDomainInfo.initDomain: 16 256
>[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503) 
>_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, 
>memory_static_min=0x0.
>[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864 
>KiB free; need 524288; done.
>[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux 
>dom=16 vcpus=4
>[2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
>[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        = 512
>[2007-09-02 05:26:43 8597] DEBUG (image:199) image          = 
>/boot/vmlinuz-xen
>[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
>[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
>[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        = 
>root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3>[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
>[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
>[2007-09-02 05:26:43 8597] DEBUG (image:205) features       >[2007-09-02
05:26:43 8597] INFO (XendDomainInfo:1321) createDevice:
>vbd : {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'':
>1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
>''phy:/tmp/xendevs/xen002'', ''mode'':
''w''}
>[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController: 
>writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
>''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
>''backend'':
''/local/domain/0/backend/vbd/16/2048''} to
>/local/domain/16/device/vbd/2048.
>[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''frontend'':
>''/local/domain/16/device/vbd/2048'',
''uuid'':
>''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'',
>''params'': ''/tmp/xendevs/xen002'',
''mode'': ''w'', ''online'':
''1'',
>''frontend-id'': ''16'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/16/2048.>[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice: 
>vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
>''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
>[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController: 
>writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
>''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
>''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
>[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
>''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
>''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
>''/local/domain/16/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'',
>''online'': ''1'',
''frontend-id'': ''16'',
''type'': ''netfront''} to
>/local/domain/0/backend/vif/16/0.
>[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM 
>details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
>''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
>''start_time'': ''1188725206.15'',
''on_poweroff'': ''destroy'',
>''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
>''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
>''image'': "(linux (kernel /boot/vmlinuz-xen) (ramdisk 
>/boot/initrd-xen) (args ''root=/dev/sda2 ro 
>ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'')
(notes
>(FEATURES 
>''writable_page_tables|writable_descriptor_tables|auto_translated_physma
p|pae_pgdir_above_4gb|supervisor_mode_kernel'') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET 
>-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER 
>generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION 
>xen-3.0)))", ''name'': ''xen002''}
>[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'':
''34194913'', ''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
>''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
>''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController: 
>writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
>''backend'':
''/local/domain/0/backend/console/16/0''} to
>/local/domain/16/device/console/0.
>[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
>''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
>''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'',
>''online'': ''1'',
''frontend-id'': ''16''} to
/local/domain/0/backend/console/16/0.>[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for
devices vif.>[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'':
''34194913'', ''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
>''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
>''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907) 
>XendDomainInfo.handleShutdownWatch
>[2007-09-02 05:26:53 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
>[2007-09-02 05:26:53 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices usb.>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vbd.>[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
>[2007-09-02 05:26:54 8597] DEBUG (DevController:541) 
>hotplugStatusCallback
/local/domain/0/backend/vbd/16/2048/hotplug-status.>[2007-09-02 05:26:54 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices irq.>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vkbd.>[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
devices vfb.>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for 
>devices console.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices pci.>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for 
>devices ioports.
>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices tap.>[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
devices vtpm.>[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
unpaused.>...
>... [vms xen003 to xen262 deleted]
>...
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78) 
>XendDomainInfo.create([''vm'', [''name'',
''xen263''], [''memory'', 512],
>[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
>''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
>[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
>''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
>ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
>''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
>[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418) 
>XendDomainInfo.constructDomain
>[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB 
>free; need 2048; done.
>[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
>[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472) 
>XendDomainInfo.initDomain: 277 256
>[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503) 
>_initDomain:shadow_memory=0x0, memory_static_max=0x20000000, 
>memory_static_min=0x0.
>[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB 
>free; need 524288; done.
>[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux 
>dom=277 vcpus=4
>[2007-09-02 17:44:29 8597] DEBUG (image:197) domid          = 277
>[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        = 512
>[2007-09-02 17:44:29 8597] DEBUG (image:199) image          = 
>/boot/vmlinuz-xen
>[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
>[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
>[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        = 
>root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
xencons=tty1 3>[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk        /boot/initrd-xen
>[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
>[2007-09-02 17:44:29 8597] DEBUG (image:205) features       >[2007-09-02
17:44:30 8597] INFO (XendDomainInfo:1321) createDevice:
>vbd : {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'':
>1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
>''phy:/tmp/xendevs/xen263'', ''mode'':
''w''}
>[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController: 
>writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
>''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
>''backend'':
''/local/domain/0/backend/vbd/277/2048''} to
>/local/domain/277/device/vbd/2048.
>[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''frontend'':
>''/local/domain/277/device/vbd/2048'',
''uuid'':
>''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'',
>''params'': ''/tmp/xendevs/xen263'',
''mode'': ''w'', ''online'':
''1'',
>''frontend-id'': ''277'',
''type'': ''phy''} to
/local/domain/0/backend/vbd/277/2048.>[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice: 
>vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
>''a126d815-fd4a-332b-c881-235a7e003cfb''}
>[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController: 
>writing {''mac'': ''00:16:3e:0d:a0:01'',
''handle'': ''0'', ''protocol'':
>''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
>''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
>[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
>''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
>''/etc/xen/scripts/vif-bridge'', ''state'':
''1'', ''frontend'':
>''/local/domain/277/device/vif/0'', ''mac'':
''00:16:3e:0d:a0:01'',
>''online'': ''1'',
''frontend-id'': ''277'',
''type'': ''netfront''} to
>/local/domain/0/backend/vif/277/0.
>[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM 
>details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
>''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
>''start_time'': ''1188769556.11'',
''on_poweroff'': ''destroy'',
>''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
>''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
>''image'': "(linux (kernel /boot/vmlinuz-xen) (ramdisk 
>/boot/initrd-xen) (args ''root=/dev/sda2 ro 
>ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'') 
>(notes (FEATURES 
>''writable_page_tables|writable_descriptor_tables|auto_translated_physma
p|pae_pgdir_above_4gb|supervisor_mode_kernel'') >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET 
>-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER 
>generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION 
>xen-3.0)))", ''name'': ''xen263''}
>[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'': ''5227479'',
''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
>''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
>''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController: 
>writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
>''backend'':
''/local/domain/0/backend/console/277/0''} to
>/local/domain/277/device/console/0.
>[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController: 
>writing {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
>''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
>''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'',
>''online'': ''1'',
''frontend-id'': ''277''} to
/local/domain/0/backend/console/277/0.>[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain 
>details: {''console/ring-ref'': ''5227479'',
''image/entry'':
>''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
>''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
>''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'', 
>''control/platform-feature-multiprocessor-suspend'':
''1'',
>''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
>''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
>''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
>''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
>''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
>''image/paddr-offset'': ''-2147483648'',
''image/hypercall-page'':
>''-2145361920'', ''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
>''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
>''image/features/writable-page-tables'':
''1'',
>''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
>''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
>[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907) 
>XendDomainInfo.handleShutdownWatch
>[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for
devices vif.>[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 17:48:23 8597] DEBUG (DevController:541) 
>hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
>[2007-09-02 17:48:23 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.>[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
devices usb.>[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
devices vbd.>[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
>[2007-09-02 17:48:25 8597] DEBUG (DevController:541) 
>hotplugStatusCallback
/local/domain/0/backend/vbd/277/2048/hotplug-status.>[2007-09-02 17:48:25 8597] DEBUG (DevController:555)
hotplugStatusCallback 1.>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices irq.>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices vkbd.>[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
devices vfb.>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for 
>devices console.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices pci.>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for 
>devices ioports.
>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices tap.>[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
devices vtpm.>[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263 
>(277) unpaused.
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-de
vel>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser

2007-Sep-06 21:25 UTC

head link

Re: [Xen-devel] VM start time slows with large number of VMs

On 6/9/07 20:25, "Carb, Brian A" <Brian.Carb@unisys.com> wrote:
>   killall xensored xenconsoled
> 
>   rm -rf /var/lib/xenstored   <- has the tdb
>   rm -rf /var/lib/xend <- has domains, state and storage dirs
> 
> 
> but when i startup xend again, xenstore-ls still shows all the old
> domains. I must be missing something.
Do you actually spell xenstored properly in your killall line and the typo
is restricted to this email?

Killing xenstored, deleting /var/lib/xenstored/*, and restarting xenstored
should suffice.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Mats Petersson

2007-Sep-06 21:32 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

At 20:25 06/09/2007, Carb, Brian A wrote:>I can''t seem to clear out this information. I tried
>
>   /etc/init.d/xend stop
>   killall xensored xenconsoled
I take it that was with the CORRECT spelling of xenstore?

Verify that there are no xen processes running.

>   rm -rf /var/lib/xenstored   <- has the tdb
>   rm -rf /var/lib/xend <- has domains, state and storage dirs
>
>
>but when i startup xend again, xenstore-ls still shows all the old
>domains. I must be missing something.

I have successfully removed this file, and the when starting xend 
afterwards, the system has no "history" of xenstore data.

--
Mats



>brian carb
>unisys corporation - malvern, pa
>
>-----Original Message-----
>From: mats petersson Sent: Thursday, September 06, 2007 1:30 PM
>To: Carb, Brian A; Keir Fraser; xen-devel@lists.xensource.com
>Subject: RE: [Xen-devel] VM start time slows with large number of VMs
>
>At 17:24 06/09/2007, Carb, Brian A wrote:
> >Content-class: urn:content-classes:message
> >Content-Type: multipart/alternative;
> >         boundary="----_=_NextPart_001_01C7F0A2.55DB0EB0"
> >
> >keir
> >
> >I''ll try timing this as I load up...
> >
> >but when I restart my test system, with only dom0 running, xenstore-ls
> >is already slow before any VMs are started:
> >
> >   time xenstore-ls >/dev/null
> >   real    0m21.725s
> >   user    0m0.010s
> >   sys     0m0.090s
> >
> >If I save the output from the xenstore-ls, the file has 61,318 lines
> >with information from all the VMs run yesterday. I conclude that
> >xenstore does not get cleared on startup. Is there a way to clear it?
>
>
>It is perfectly fine to remove the xenstore.tdb [filename from memory]
>in /var/xen/lib [again, from memory]. Unless of course you actually NEED
>some of your old Xenstrore entries for soemthing, but in general, they
>are fine to be removed completely.
>
>--
>Mats
>
> >
> >brian carb
> >unisys corporation - malvern, pa
> >
> >
> >
> >----------
> >From: Keir Fraser
> >Sent: Wednesday, September 05, 2007 1:11 PM
> >To: Carb, Brian A; xen-devel@lists.xensource.com
> >Subject: Re: [Xen-devel] VM start time slows with large number of VMs
> >
> >xenstore slowness seems a reasonable hypothesis. How does
> >performance of ''time xenstore-ls >/dev/null''
change as you increase
> >the number of guests?
> >
> >  -- Keir
> >
> >On 4/9/07 17:51, "Carb, Brian A" wrote:
> >
> >Hello,
> >
> >We''re running Xen Unstable changeset 15468 with SLES10 on a
Unisys
> >ES7000/one 56x cpu and 160GB memory.  Xen is built with
> >NR_DYNIRQS=1024 and max_phys_cpus=64, and booted with dom0_mem=1024
> >and xenheap_megabytes=64. We have 320 para-virtualized SLES10 VMs on
> >external storage.
> >
> >Our script starts a VM, waits for the start to complete, then starts
> >the next: "for i in *.cfg; do xm create $i; done". As more
VMs
> >start, the time to start each one increases. I''ve attached a
config
> >file for one of the VMs, as well as a portion of the xend.log which
> >shows timestamps of the events for 2 VMS, vm002 and vm263.
> >
> >The total elapsed time of the start from "create" to
"unpaused" for
> >vm002 is 13 seconds, but by the time 262 VMs are started, the start
> >of vm263 has slowed to 238 seconds. The host itself is not sluggish,
> >as we can open putty sessions to and interact with dom0.  However,
> >if we run "xm top", its interface is very unresponsive. 
Could
> >writing to the xenstore be taking longer and longer? Could there be
> >possible contention for vmlinuz and initrd since they are being used
> >by every VM to boot? Would it be better to use bootloader/bootentry
> >and store the vmlinuz and initrd within each VM?
> >
> >Any insight would be appreciated.
> >
> >brian carb
> >unisys corporation, malvern pa
> >
> >----- domU config file 002.cfg -----
> >kernel = "/boot/vmlinuz-xen"
> >ramdisk = "/boot/initrd-xen"
> >root = "/dev/sda2 ro"
> >disk = [ ''phy:/tmp/xendevs/xen002,sda,w'' ]
> >memory = 512
> >vcpus = 4
> >name = ''xen002''
> >vif = [ ''mac=00:16:3e:0d:a0:01'' ]
> >ip = ''10.0.27.2''
> >netmask = ''255.254.0.0''
> >extra = '' xencons=tty1 3''
> >----- xend.log -----
> >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:78)
> >XendDomainInfo.create([''vm'',
[''name'', ''xen002''],
[''memory'', 512],
> >[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
> >''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
> >[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
> >''10.0.27.2:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
> >ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
> >''phy:/tmp/xendevs/xen002''], [''dev'',
''sda''], [''mode'', ''w'']]],
> >[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
> >[2007-09-02 05:26:42 8597] DEBUG (XendDomainInfo:1418)
> >XendDomainInfo.constructDomain
> >[2007-09-02 05:26:42 8597] DEBUG (balloon:114) Balloon: 169819872
> >KiB free; need 2048; done.
> >[2007-09-02 05:26:43 8597] DEBUG (XendDomain:434) Adding Domain: 16
> >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1472)
> >XendDomainInfo.initDomain: 16 256
> >[2007-09-02 05:26:43 8597] DEBUG (XendDomainInfo:1503)
> >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
> >memory_static_min=0x0.
> >[2007-09-02 05:26:43 8597] DEBUG (balloon:114) Balloon: 169819864
> >KiB free; need 524288; done.
> >[2007-09-02 05:26:43 8597] INFO (image:128) buildDomain os=linux
> >dom=16 vcpus=4
> >[2007-09-02 05:26:43 8597] DEBUG (image:197) domid          = 16
> >[2007-09-02 05:26:43 8597] DEBUG (image:198) memsize        = 512
> >[2007-09-02 05:26:43 8597] DEBUG (image:199) image          >
>/boot/vmlinuz-xen
> >[2007-09-02 05:26:43 8597] DEBUG (image:200) store_evtchn   = 1
> >[2007-09-02 05:26:43 8597] DEBUG (image:201) console_evtchn = 2
> >[2007-09-02 05:26:43 8597] DEBUG (image:202) cmdline        >
>root=/dev/sda2 ro ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off
>xencons=tty1 3
> >[2007-09-02 05:26:43 8597] DEBUG (image:203) ramdisk       
>/boot/initrd-xen
> >[2007-09-02 05:26:43 8597] DEBUG (image:204) vcpus          = 4
> >[2007-09-02 05:26:43 8597] DEBUG (image:205) features       >
>[2007-09-02 05:26:43 8597] INFO (XendDomainInfo:1321) createDevice:
> >vbd : {''uuid'':
''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''bootable'':
> >1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
> >''phy:/tmp/xendevs/xen002'', ''mode'':
''w''}
> >[2007-09-02 05:26:43 8597] DEBUG (DevController:115) DevController:
> >writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
> >''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
> >''backend'':
''/local/domain/0/backend/vbd/16/2048''} to
> >/local/domain/16/device/vbd/2048.
> >[2007-09-02 05:26:43 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen002'',
''frontend'':
> >''/local/domain/16/device/vbd/2048'',
''uuid'':
> >''8d79c4ed-4ddc-afee-9725-f5c2d388a02e'',
''dev'': ''sda'', ''state'':
''1'',
> >''params'': ''/tmp/xendevs/xen002'',
''mode'': ''w'', ''online'':
''1'',
> >''frontend-id'': ''16'',
''type'': ''phy''} to
>/local/domain/0/backend/vbd/16/2048.
> >[2007-09-02 05:26:44 8597] INFO (XendDomainInfo:1321) createDevice:
> >vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
> >''bb43e84a-a9a1-7488-1ba4-3b713eff82aa''}
> >[2007-09-02 05:26:45 8597] DEBUG (DevController:115) DevController:
> >writing {''mac'':
''00:16:3e:0d:a0:01'', ''handle'':
''0'', ''protocol'':
> >''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
> >''/local/domain/0/backend/vif/16/0''} to
/local/domain/16/device/vif/0.
> >[2007-09-02 05:26:45 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen002'',
''handle'': ''0'', ''uuid'':
> >''bb43e84a-a9a1-7488-1ba4-3b713eff82aa'',
''script'':
> >''/etc/xen/scripts/vif-bridge'',
''state'': ''1'', ''frontend'':
> >''/local/domain/16/device/vif/0'',
''mac'': ''00:16:3e:0d:a0:01'',
> >''online'': ''1'',
''frontend-id'': ''16'',
''type'': ''netfront''} to
> >/local/domain/0/backend/vif/16/0.
> >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:1978) Storing VM
> >details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
> >''f146fdfc-e624-f7cb-4baf-0471f01f6014'',
''on_reboot'': ''restart'',
> >''start_time'': ''1188725206.15'',
''on_poweroff'': ''destroy'',
> >''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
> >''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
> >''image'': "(linux (kernel /boot/vmlinuz-xen)
(ramdisk
> >/boot/initrd-xen) (args ''root=/dev/sda2 ro
> >ip=10.0.27.2:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'')
(notes
> >(FEATURES
>
>''writable_page_tables|writable_descriptor_tables|auto_translated_physma
>p|pae_pgdir_above_4gb|supervisor_mode_kernel'')
> >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET
> >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER
> >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION
> >xen-3.0)))", ''name'': ''xen002''}
> >[2007-09-02 05:26:46 8597] DEBUG (XendDomainInfo:823) Storing domain
> >details: {''console/ring-ref'':
''34194913'', ''image/entry'':
> >''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
> >''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
> >''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
> >''control/platform-feature-multiprocessor-suspend'':
''1'',
> >''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
> >''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> >''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
> >''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
> >''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
> >''image/paddr-offset'':
''-2147483648'', ''image/hypercall-page'':
> >''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'':
> >''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> >''image/features/writable-page-tables'':
''1'',
> >''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
> >''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
> >[2007-09-02 05:26:50 8597] DEBUG (DevController:115) DevController:
> >writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
> >''backend'':
''/local/domain/0/backend/console/16/0''} to
> >/local/domain/16/device/console/0.
> >[2007-09-02 05:26:50 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen002'',
''protocol'': ''vt100'',
''uuid'':
> >''8dc5f3eb-5c78-35e1-7876-17c26fc6ad6e'',
''frontend'':
> >''/local/domain/16/device/console/0'',
''state'': ''1'', ''location'':
''2'',
> >''online'': ''1'',
''frontend-id'': ''16''} to
>/local/domain/0/backend/console/16/0.
> >[2007-09-02 05:26:52 8597] DEBUG (DevController:148) Waiting for
>devices vif.
> >[2007-09-02 05:26:53 8597] DEBUG (DevController:153) Waiting for 0.
> >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:823) Storing domain
> >details: {''console/ring-ref'':
''34194913'', ''image/entry'':
> >''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
> >''store/ring-ref'': ''34194914'',
''image/loader'': ''generic'',
''vm'':
> >''/vm/f146fdfc-e624-f7cb-4baf-0471f01f6014'',
> >''control/platform-feature-multiprocessor-suspend'':
''1'',
> >''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
> >''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> >''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
> >''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
> >''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
> >''image/paddr-offset'':
''-2147483648'', ''image/hypercall-page'':
> >''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'':
> >''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> >''image/features/writable-page-tables'':
''1'',
> >''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen002'',
> >''domid'': ''16'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
> >[2007-09-02 05:26:53 8597] DEBUG (XendDomainInfo:907)
> >XendDomainInfo.handleShutdownWatch
> >[2007-09-02 05:26:53 8597] DEBUG (DevController:541)
> >hotplugStatusCallback /local/domain/0/backend/vif/16/0/hotplug-status.
> >[2007-09-02 05:26:53 8597] DEBUG (DevController:555)
>hotplugStatusCallback 1.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
>devices usb.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
>devices vbd.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:153) Waiting for 2048.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:541)
> >hotplugStatusCallback
>/local/domain/0/backend/vbd/16/2048/hotplug-status.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:555)
>hotplugStatusCallback 1.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
>devices irq.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
>devices vkbd.
> >[2007-09-02 05:26:54 8597] DEBUG (DevController:148) Waiting for
>devices vfb.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
> >devices console.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:153) Waiting for 0.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
>devices pci.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
> >devices ioports.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
>devices tap.
> >[2007-09-02 05:26:55 8597] DEBUG (DevController:148) Waiting for
>devices vtpm.
> >[2007-09-02 05:26:55 8597] INFO (XendDomain:1117) Domain xen002 (16)
>unpaused.
> >...
> >... [vms xen003 to xen262 deleted]
> >...
> >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:78)
> >XendDomainInfo.create([''vm'',
[''name'', ''xen263''],
[''memory'', 512],
> >[''vcpus'', 4], [''on_xend_start'',
''ignore''], [''on_xend_stop'',
> >''ignore''], [''image'',
[''linux'', [''kernel'',
''/boot/vmlinuz-xen''],
> >[''ramdisk'', ''/boot/initrd-xen''],
[''ip'',
> >''10.0.29.63:1.2.3.4::255.254.0.0::eth0:off''],
[''root'', ''/dev/sda2
> >ro''], [''args'', '' xencons=tty1
3'']]], [''device'', [''vbd'',
[''uname'',
> >''phy:/tmp/xendevs/xen263''], [''dev'',
''sda''], [''mode'', ''w'']]],
> >[''device'', [''vif'',
[''mac'', ''00:16:3e:0d:a0:01'']]]])
> >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1418)
> >XendDomainInfo.constructDomain
> >[2007-09-02 17:44:28 8597] DEBUG (balloon:114) Balloon: 32978616 KiB
> >free; need 2048; done.
> >[2007-09-02 17:44:28 8597] DEBUG (XendDomain:434) Adding Domain: 277
> >[2007-09-02 17:44:28 8597] DEBUG (XendDomainInfo:1472)
> >XendDomainInfo.initDomain: 277 256
> >[2007-09-02 17:44:29 8597] DEBUG (XendDomainInfo:1503)
> >_initDomain:shadow_memory=0x0, memory_static_max=0x20000000,
> >memory_static_min=0x0.
> >[2007-09-02 17:44:29 8597] DEBUG (balloon:114) Balloon: 32978608 KiB
> >free; need 524288; done.
> >[2007-09-02 17:44:29 8597] INFO (image:128) buildDomain os=linux
> >dom=277 vcpus=4
> >[2007-09-02 17:44:29 8597] DEBUG (image:197) domid          = 277
> >[2007-09-02 17:44:29 8597] DEBUG (image:198) memsize        = 512
> >[2007-09-02 17:44:29 8597] DEBUG (image:199) image          >
>/boot/vmlinuz-xen
> >[2007-09-02 17:44:29 8597] DEBUG (image:200) store_evtchn   = 1
> >[2007-09-02 17:44:29 8597] DEBUG (image:201) console_evtchn = 2
> >[2007-09-02 17:44:29 8597] DEBUG (image:202) cmdline        >
>root=/dev/sda2 ro ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off
>xencons=tty1 3
> >[2007-09-02 17:44:29 8597] DEBUG (image:203) ramdisk       
>/boot/initrd-xen
> >[2007-09-02 17:44:29 8597] DEBUG (image:204) vcpus          = 4
> >[2007-09-02 17:44:29 8597] DEBUG (image:205) features       >
>[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice:
> >vbd : {''uuid'':
''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''bootable'':
> >1, ''driver'': ''paravirtualised'',
''dev'': ''sda'', ''uname'':
> >''phy:/tmp/xendevs/xen263'', ''mode'':
''w''}
> >[2007-09-02 17:44:30 8597] DEBUG (DevController:115) DevController:
> >writing {''virtual-device'': ''2048'',
''device-type'': ''disk'',
> >''protocol'': ''x86_64-abi'',
''backend-id'': ''0'',
''state'': ''1'',
> >''backend'':
''/local/domain/0/backend/vbd/277/2048''} to
> >/local/domain/277/device/vbd/2048.
> >[2007-09-02 17:44:30 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen263'',
''frontend'':
> >''/local/domain/277/device/vbd/2048'',
''uuid'':
> >''9d639202-73af-64ec-451d-3abe0fc57bc7'',
''dev'': ''sda'', ''state'':
''1'',
> >''params'': ''/tmp/xendevs/xen263'',
''mode'': ''w'', ''online'':
''1'',
> >''frontend-id'': ''277'',
''type'': ''phy''} to
>/local/domain/0/backend/vbd/277/2048.
> >[2007-09-02 17:44:30 8597] INFO (XendDomainInfo:1321) createDevice:
> >vif : {''mac'': ''00:16:3e:0d:a0:01'',
''uuid'':
> >''a126d815-fd4a-332b-c881-235a7e003cfb''}
> >[2007-09-02 17:45:02 8597] DEBUG (DevController:115) DevController:
> >writing {''mac'':
''00:16:3e:0d:a0:01'', ''handle'':
''0'', ''protocol'':
> >''x86_64-abi'', ''backend-id'':
''0'', ''state'': ''1'',
''backend'':
> >''/local/domain/0/backend/vif/277/0''} to
/local/domain/277/device/vif/0.
> >[2007-09-02 17:45:02 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen263'',
''handle'': ''0'', ''uuid'':
> >''a126d815-fd4a-332b-c881-235a7e003cfb'',
''script'':
> >''/etc/xen/scripts/vif-bridge'',
''state'': ''1'', ''frontend'':
> >''/local/domain/277/device/vif/0'',
''mac'': ''00:16:3e:0d:a0:01'',
> >''online'': ''1'',
''frontend-id'': ''277'',
''type'': ''netfront''} to
> >/local/domain/0/backend/vif/277/0.
> >[2007-09-02 17:45:56 8597] DEBUG (XendDomainInfo:1978) Storing VM
> >details: {''on_xend_stop'': ''ignore'',
''shadow_memory'': ''0'',
''uuid'':
> >''8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
''on_reboot'': ''restart'',
> >''start_time'': ''1188769556.11'',
''on_poweroff'': ''destroy'',
> >''on_xend_start'': ''ignore'',
''on_crash'': ''restart'',
> >''xend/restart_count'': ''0'',
''vcpus'': ''4'',
''vcpu_avail'': ''15'',
> >''image'': "(linux (kernel /boot/vmlinuz-xen)
(ramdisk
> >/boot/initrd-xen) (args ''root=/dev/sda2 ro
> >ip=10.0.29.63:1.2.3.4::255.254.0.0::eth0:off  xencons=tty1 3'')
> >(notes (FEATURES
>
>''writable_page_tables|writable_descriptor_tables|auto_translated_physma
>p|pae_pgdir_above_4gb|supervisor_mode_kernel'')
> >(VIRT_BASE -2147483648) (GUEST_VERSION 2.6) (PADDR_OFFSET
> >-2147483648) (GUEST_OS linux) (HYPERCALL_PAGE -2145361920) (LOADER
> >generic) (SUSPEND_CANCEL 1) (ENTRY -2145386496) (XEN_VERSION
> >xen-3.0)))", ''name'': ''xen263''}
> >[2007-09-02 17:45:58 8597] DEBUG (XendDomainInfo:823) Storing domain
> >details: {''console/ring-ref'':
''5227479'', ''image/entry'':
> >''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
> >''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
> >''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
> >''control/platform-feature-multiprocessor-suspend'':
''1'',
> >''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
> >''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> >''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
> >''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
> >''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
> >''image/paddr-offset'':
''-2147483648'', ''image/hypercall-page'':
> >''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'':
> >''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> >''image/features/writable-page-tables'':
''1'',
> >''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
> >''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
> >[2007-09-02 17:47:25 8597] DEBUG (DevController:115) DevController:
> >writing {''protocol'': ''x86_64-abi'',
''state'': ''1'',
''backend-id'': ''0'',
> >''backend'':
''/local/domain/0/backend/console/277/0''} to
> >/local/domain/277/device/console/0.
> >[2007-09-02 17:47:25 8597] DEBUG (DevController:117) DevController:
> >writing {''domain'': ''xen263'',
''protocol'': ''vt100'',
''uuid'':
> >''d0da45dd-1a00-da65-61dd-66dadd6e9897'',
''frontend'':
> >''/local/domain/277/device/console/0'',
''state'': ''1'', ''location'':
''2'',
> >''online'': ''1'',
''frontend-id'': ''277''} to
>/local/domain/0/backend/console/277/0.
> >[2007-09-02 17:47:26 8597] DEBUG (XendDomainInfo:823) Storing domain
> >details: {''console/ring-ref'':
''5227479'', ''image/entry'':
> >''-2145386496'', ''console/port'':
''2'', ''cpu/3/availability'':
''online'',
> >''store/ring-ref'': ''5227480'',
''image/loader'': ''generic'',
''vm'':
> >''/vm/8f8f9839-dd61-b9cd-89a6-f66bb3d41726'',
> >''control/platform-feature-multiprocessor-suspend'':
''1'',
> >''cpu/2/availability'': ''online'',
''cpu/1/availability'': ''online'',
> >''image/features/writable-descriptor-tables'':
''1'', ''image/virt-base'':
> >''-2147483648'', ''memory/target'':
''524288'', ''image/guest-version'':
> >''2.6'',
''image/features/supervisor-mode-kernel'':
''1'',
> >''image/guest-os'': ''linux'',
''console/limit'': ''1048576'',
> >''image/paddr-offset'':
''-2147483648'', ''image/hypercall-page'':
> >''-2145361920'',
''image/suspend-cancel'': ''1'',
''cpu/0/availability'':
> >''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
> >''image/features/writable-page-tables'':
''1'',
> >''image/features/auto-translated-physmap'':
''1'', ''name'': ''xen263'',
> >''domid'': ''277'',
''image/xen-version'': ''xen-3.0'',
''store/port'': ''1''}
> >[2007-09-02 17:47:27 8597] DEBUG (XendDomainInfo:907)
> >XendDomainInfo.handleShutdownWatch
> >[2007-09-02 17:48:23 8597] DEBUG (DevController:148) Waiting for
>devices vif.
> >[2007-09-02 17:48:23 8597] DEBUG (DevController:153) Waiting for 0.
> >[2007-09-02 17:48:23 8597] DEBUG (DevController:541)
> >hotplugStatusCallback /local/domain/0/backend/vif/277/0/hotplug-status.
> >[2007-09-02 17:48:23 8597] DEBUG (DevController:555)
>hotplugStatusCallback 1.
> >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
>devices usb.
> >[2007-09-02 17:48:24 8597] DEBUG (DevController:148) Waiting for
>devices vbd.
> >[2007-09-02 17:48:24 8597] DEBUG (DevController:153) Waiting for 2048.
> >[2007-09-02 17:48:25 8597] DEBUG (DevController:541)
> >hotplugStatusCallback
>/local/domain/0/backend/vbd/277/2048/hotplug-status.
> >[2007-09-02 17:48:25 8597] DEBUG (DevController:555)
>hotplugStatusCallback 1.
> >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
>devices irq.
> >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
>devices vkbd.
> >[2007-09-02 17:48:25 8597] DEBUG (DevController:148) Waiting for
>devices vfb.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
> >devices console.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:153) Waiting for 0.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
>devices pci.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
> >devices ioports.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
>devices tap.
> >[2007-09-02 17:48:26 8597] DEBUG (DevController:148) Waiting for
>devices vtpm.
> >[2007-09-02 17:48:26 8597] INFO (XendDomain:1117) Domain xen263
> >(277) unpaused.
> >
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
>
><http://lists.xensource.com/xen-devel>http://lists.xensource.com/xen-de
>vel
> >
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Carb, Brian A

2007-Sep-11 16:27 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

I tried timing xenstore-ls and then tried it again with the shared memory idea
(adding the 2 lines to the /etc/init.d/rc3.0/S10xend script before it starts
xend). Here are the average times for xenstore-ls:

 # vms     std    shm
001-020    .67    .59
021-040   1.70   1.54
041-060   2.41   2.36
061-080   3.26   3.14
081-100   4.06   3.95
101-120   4.96   4.90
121-140   6.56   6,02
141-160   7.33   7.21
161-180   8.00   8.62
181-200   9.90   9.61

Without using shared memory, the total elapsed time to start 200 VMs is about 95
minutes, vs 81 minutes with the shm statements. Time to start the 200th vm, from
create to unpaused, is about 55 seconds, and with xenstore in shared memory that
time is about 43 seconds. I''m not seeing the 200+ seconds I was seeing
before, but that time was for vm263 and I''ve only been able to test
with 200 VMs. This week I''ll try to test with more.

Since the shared and non-shared memory results are so close, how can I verify
that the xenstore data is actually going in and out of shared memory? It seems
as if the start times became reduced once I cleared out the old info from
xenstore, regardless of shared memory.

Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well?

brian carb
unisys corporation - malvern, pa

-----Original Message-----
From: Daniel P. Berrange
Sent: Wednesday, September 05, 2007 1:38 PM
Subject: Re: [Xen-devel] VM start time slows with large number of VMs

On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser
wrote:> xenstore slowness seems a reasonable hypothesis. How does performance 
> of Œtime xenstore-ls >/dev/null¹ change as you increase the number of
guests?
There''s another issue with the way hotplug scripts do locking which
exhbits fairly pathelogical behaviour when a large number of guests are started
concurrently. We''ve got a patch we''re evaluating which changes
the use of lock files + looping + sleeps, to instead use the
''flock'' shell util.

I suspect the performance issues will be a combination of both the locking stuff
and the xenstore I/O. You might also want to try putting /var/lib/xenstored onto
tmpfs eg in  initscripts before xenstored starts ...

  mkdir /dev/shm/xenstored
  mount --bind /dev/shm/xenstored /var/lib/xenstored


Dan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Mats Petersson

2007-Sep-11 17:42 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

At 17:27 11/09/2007, Carb, Brian A wrote:
>  I tried timing xenstore-ls and then tried it again with the shared 
> memory idea (adding the 2 lines to the /etc/init.d/rc3.0/S10xend 
> script before it starts xend). Here are the average times for xenstore-ls:
>
>  # vms     std    shm
>001-020    .67    .59
>021-040   1.70   1.54
>041-060   2.41   2.36
>061-080   3.26   3.14
>081-100   4.06   3.95
>101-120   4.96   4.90
>121-140   6.56   6,02
>141-160   7.33   7.21
>161-180   8.00   8.62
>181-200   9.90   9.61
>
>Without using shared memory, the total elapsed time to start 200 VMs 
>is about 95 minutes, vs 81 minutes with the shm statements. Time to 
>start the 200th vm, from create to unpaused, is about 55 seconds, 
>and with xenstore in shared memory that time is about 43 seconds. 
>I''m not seeing the 200+ seconds I was seeing before, but that time 
>was for vm263 and I''ve only been able to test with 200 VMs. This 
>week I''ll try to test with more.
>
>Since the shared and non-shared memory results are so close, how can 
>I verify that the xenstore data is actually going in and out of 
>shared memory? It seems as if the start times became reduced once I 
>cleared out the old info from xenstore, regardless of shared memory.

That would make sense.

I''m not sure how well the xenstore structure is designed, with 
respect to holding hundred and thousands of different entries - I had 
a quick look at it, but not really looking at it deeply, just very 
briefly - the code I was looking at seemed to do a lot of small 
function calls to dig data out of the database a byte or some few 
bytes at a time, which didn''t make me say "Wow, that''s
clever".

--
Mats

>Is it worth trying bootloader/bootentry instead of kernel/ramdisk as well?
>
>brian carb
>unisys corporation - malvern, pa
>
>-----Original Message-----
>From: Daniel P. Berrange
>Sent: Wednesday, September 05, 2007 1:38 PM
>Subject: Re: [Xen-devel] VM start time slows with large number of VMs
>
>On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser wrote:
> > xenstore slowness seems a reasonable hypothesis. How does performance
> > of time xenstore-ls >/dev/null¹ change as you increase the 
> number of guests?
>
>There''s another issue with the way hotplug scripts do locking which
>exhbits fairly pathelogical behaviour when a large number of guests 
>are started concurrently. We''ve got a patch we''re
evaluating which
>changes  the use of lock files + looping + sleeps, to instead use 
>the ''flock'' shell util.
>
>I suspect the performance issues will be a combination of both the 
>locking stuff and the xenstore I/O. You might also want to try 
>putting /var/lib/xenstored onto tmpfs eg in  initscripts before 
>xenstored starts ...
>
>   mkdir /dev/shm/xenstored
>   mount --bind /dev/shm/xenstored /var/lib/xenstored
>
>
>Dan.
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Carb, Brian A

2007-Sep-24 14:18 UTC

head link

RE: [Xen-devel] VM start time slows with large number of VMs

We compared total elapsed time to start 320 para VMs on ES7000 (56x 160gb) where
xenstore was on physical disk vs ram disk:

physical:  5 hours 19 minutes
ram disk:  4 hours  2 minutes


brian carb
unisys corporation - malvern, pa


-----Original Message-----
From: Daniel P. Berrange [mailto:berrange@redhat.com] 
Sent: Wednesday, September 05, 2007 1:38 PM
To: Keir Fraser
Cc: Carb, Brian A; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] VM start time slows with large number of VMs

On Wed, Sep 05, 2007 at 06:11:06PM +0100, Keir Fraser
wrote:> xenstore slowness seems a reasonable hypothesis. How does performance 
> of Œtime xenstore-ls >/dev/null¹ change as you increase the number of
guests?
There''s another issue with the way hotplug scripts do locking which
exhbits fairly pathelogical behaviour when a large number of guests are started
concurrently. We''ve got a patch we''re evaluating which changes
the use of lock files + looping + sleeps, to instead use the
''flock'' shell util.

I suspect the performance issues will be a combination of both the locking stuff
and the xenstore I/O. You might also want to try putting /var/lib/xenstored onto
tmpfs eg in  initscripts before xenstored starts ...

  mkdir /dev/shm/xenstored
  mount --bind /dev/shm/xenstored /var/lib/xenstored


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Seemingly Similar Threads

Search for more seemingly similar threads

Xen devel - Sep 2007 - VM start time slows with large number of VMs

[Xen-devel] VM start time slows with large number of VMs

Re: [Xen-devel] VM start time slows with large number of VMs

Re: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

Re: [Xen-devel] VM start time slows with large number of VMs

Re: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

Re: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

RE: [Xen-devel] VM start time slows with large number of VMs

Seemingly Similar Threads