Pierre POMES
2010-May-08 02:47 UTC
[Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Hi all,
I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy
kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am
writing this email). This kernel is also used in my domu.
I can save a domu without any problem, but restoring it may need from 2
to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There also
errors in /var/log/xen/xend.log :
[2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain
details: {''image/entry'':
''18446744071587529216'', ''console/port'':
''2'',
''image/loader'': ''generic'',
''vm'':
''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''image/hv-start-low'':
''18446603336221196288'', ''image/guest-os'':
''linux'',
''image/virt-base'': ''18446744071562067968'',
''memory/target'': ''1048576'',
''image/guest-version'': ''2.6'',
''image/pae-mode'': ''yes'',
''description'':
'''', ''console/limit'':
''1048576'', ''image/paddr-offset'':
''0'',
''image/hypercall-page'':
''18446744071578882048'',
''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
''image/features/writable-page-tables'': ''0'',
''console/type'':
''xenconsoled'', ''name'':
''domusample'', ''domid'':
''10'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
[2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286)
restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0,
[2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]:
/usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0
[2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore
start: p2m_size = 40000
[2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory
pages: 0%
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal
error: Error when reading batch size
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal
error: error when buffering batch, finishing
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423)
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100%
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded (0
pages)
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0
Live migration has the same problem, it may need several minutes to
complete. Please note that restore and migration do not fail, but there
are very long
My domu is on top of DRBD, and the config file is :
-------------
kernel = ''/boot/vmlinuz-2.6.32.12-it-xen''
ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen''
memory = ''1024''
#
# Disk device(s).
#
root = ''/dev/xvda2 ro''
disk = [
''drbd:domusampleswap,xvda1,w'',
''drbd:domusampleslash,xvda2,w'',
]
#
# Hostname
#
name = ''domusample''
#
# Networking
#
vif = [ ''mac=00:16:3E:58:FC:F9'' ]
#
# Behaviour
#
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
extra = ''2 console=hvc0''
----------
I do not have any idea here.
Did somebody already have (and solved ?) this issue ?
Thanks.
Pierre
*
*
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Pierre POMES
2010-May-08 11:29 UTC
[Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Hi all,
I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy
kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am
writing this email). This kernel is also used in my domu.
I can save a domu without any problem, but restoring it may need from 2
to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There also
errors in /var/log/xen/xend.log, "Error when reading batch size" and
"Error when reading batch size":
[2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain
details: {''image/entry'':
''18446744071587529216'', ''console/port'':
''2'',
''image/loader'': ''generic'',
''vm'':
''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'',
''control/platform-feature-multiprocessor-suspend'':
''1'',
''image/hv-start-low'':
''18446603336221196288'', ''image/guest-os'':
''linux'',
''image/virt-base'': ''18446744071562067968'',
''memory/target'': ''1048576'',
''image/guest-version'': ''2.6'',
''image/pae-mode'': ''yes'',
''description'':
'''', ''console/limit'':
''1048576'', ''image/paddr-offset'':
''0'',
''image/hypercall-page'':
''18446744071578882048'',
''image/suspend-cancel'':
''1'', ''cpu/0/availability'':
''online'',
''image/features/pae-pgdir-above-4gb'': ''1'',
''image/features/writable-page-tables'': ''0'',
''console/type'':
''xenconsoled'', ''name'':
''domusample'', ''domid'':
''10'', ''image/xen-version'':
''xen-3.0'', ''store/port'':
''1''}
[2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286)
restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0,
[2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]:
/usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0
[2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore
start: p2m_size = 40000
[2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory
pages: 0%
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal
error: Error when reading batch size
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal
error: error when buffering batch, finishing
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423)
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100%
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded (0
pages)
[2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0
Live migration has the same problem, it may need several minutes to
complete. Please note that restore and migration do not fail, but there
are very long.
My domu is on top of DRBD, and the config file is:
-------------
kernel = ''/boot/vmlinuz-2.6.32.12-it-xen''
ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen''
memory = ''1024''
#
# Disk device(s).
#
root = ''/dev/xvda2 ro''
disk = [
''drbd:domusampleswap,xvda1,w'',
''drbd:domusampleslash,xvda2,w'',
]
#
# Hostname
#
name = ''domusample''
#
# Networking
#
vif = [ ''mac=00:16:3E:58:FC:F9'' ]
#
# Behaviour
#
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
extra = ''2 console=hvc0''
----------
I do not have any idea here.
Did somebody already have (and solve ?) this issue ?
Thanks.
Pierre
*
*
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Pierre POMES
2010-May-12 21:28 UTC
Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Hi, First sorry for the double posting... I just built a 2.6.32.10 kernel with Andrew Lyon patches (so it is a "xenlinux" kernel, not a "pvops" kernel). Live migration and restore operations are between 4 and 10 times faster with this kernel . Furthermore, during live migration, hangs time in domu are shorter (1-2 seconds versus 1 to 15 seconds for a domu with 256M RAM). Error messages "Error when reading batch size" / "error when buffering batch, finishing" are still in my logs. Regarding times, all is now similar to what I had with Xen 3.x on top of xenlinux kernels. Regards, Pierre> Hi all, > > I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy > kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am > writing this email). This kernel is also used in my domu. > > I can save a domu without any problem, but restoring it may need from > 2 to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There > also errors in /var/log/xen/xend.log, "Error when reading batch size" > and "Error when reading batch size": > > [2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain > details: {''image/entry'': ''18446744071587529216'', ''console/port'': ''2'', > ''image/loader'': ''generic'', ''vm'': > ''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'', > ''control/platform-feature-multiprocessor-suspend'': ''1'', > ''image/hv-start-low'': ''18446603336221196288'', ''image/guest-os'': > ''linux'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': > ''1048576'', ''image/guest-version'': ''2.6'', ''image/pae-mode'': ''yes'', > ''description'': '''', ''console/limit'': ''1048576'', ''image/paddr-offset'': > ''0'', ''image/hypercall-page'': ''18446744071578882048'', > ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', > ''image/features/pae-pgdir-above-4gb'': ''1'', > ''image/features/writable-page-tables'': ''0'', ''console/type'': > ''xenconsoled'', ''name'': ''domusample'', ''domid'': ''10'', > ''image/xen-version'': ''xen-3.0'', ''store/port'': ''1''} > [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286) > restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, > [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]: > /usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0 > [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore > start: p2m_size = 40000 > [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory > pages: 0% > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal > error: Error when reading batch size > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal > error: error when buffering batch, finishing > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100% > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded > (0 pages) > [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0 > > Live migration has the same problem, it may need several minutes to > complete. Please note that restore and migration do not fail, but > there are very long. > > My domu is on top of DRBD, and the config file is: > > > ------------- > kernel = ''/boot/vmlinuz-2.6.32.12-it-xen'' > ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen'' > memory = ''1024'' > > # > # Disk device(s). > # > root = ''/dev/xvda2 ro'' > disk = [ > ''drbd:domusampleswap,xvda1,w'', > ''drbd:domusampleslash,xvda2,w'', > ] > > > > # > # Hostname > # > name = ''domusample'' > > # > # Networking > # > vif = [ ''mac=00:16:3E:58:FC:F9'' ] > > # > # Behaviour > # > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > > extra = ''2 console=hvc0'' > ---------- > > I do not have any idea here. > > Did somebody already have (and solve ?) this issue ? > > Thanks. > Pierre > > > * > * > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-May-13 12:32 UTC
Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
On Wed, May 12, 2010 at 05:28:26PM -0400, Pierre POMES wrote:> Hi, > > First sorry for the double posting... > > I just built a 2.6.32.10 kernel with Andrew Lyon patches (so it is a > "xenlinux" kernel, not a "pvops" kernel). > > Live migration and restore operations are between 4 and 10 times faster > with this kernel . Furthermore, during live migration, hangs time in > domu are shorter (1-2 seconds versus 1 to 15 seconds for a domu with > 256M RAM). >Interesting.. Do you happen to have all the debugging options enabled in the pvops dom0 kernel? There was some network performance problem reports earlier, and those were caused by having debugging options enabled in the pvops dom0 kernel. After disabling the debug options (ie. having the same kernel options with xenlinux and pvops) the performance was roughly the same for both kernel types. -- Pasi> Error messages "Error when reading batch size" / "error when buffering > batch, finishing" are still in my logs. > > Regarding times, all is now similar to what I had with Xen 3.x on top of > xenlinux kernels. > > Regards, > Pierre > > > >> Hi all, >> >> I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy >> kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am >> writing this email). This kernel is also used in my domu. >> >> I can save a domu without any problem, but restoring it may need from >> 2 to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There >> also errors in /var/log/xen/xend.log, "Error when reading batch size" >> and "Error when reading batch size": >> >> [2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain >> details: {''image/entry'': ''18446744071587529216'', ''console/port'': ''2'', >> ''image/loader'': ''generic'', ''vm'': >> ''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'', >> ''control/platform-feature-multiprocessor-suspend'': ''1'', >> ''image/hv-start-low'': ''18446603336221196288'', ''image/guest-os'': >> ''linux'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': >> ''1048576'', ''image/guest-version'': ''2.6'', ''image/pae-mode'': ''yes'', >> ''description'': '''', ''console/limit'': ''1048576'', ''image/paddr-offset'': >> ''0'', ''image/hypercall-page'': ''18446744071578882048'', >> ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', >> ''image/features/pae-pgdir-above-4gb'': ''1'', >> ''image/features/writable-page-tables'': ''0'', ''console/type'': >> ''xenconsoled'', ''name'': ''domusample'', ''domid'': ''10'', >> ''image/xen-version'': ''xen-3.0'', ''store/port'': ''1''} >> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286) >> restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, >> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]: >> /usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0 >> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore >> start: p2m_size = 40000 >> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory >> pages: 0% >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >> error: Error when reading batch size >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >> error: error when buffering batch, finishing >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100% >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded >> (0 pages) >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0 >> >> Live migration has the same problem, it may need several minutes to >> complete. Please note that restore and migration do not fail, but >> there are very long. >> >> My domu is on top of DRBD, and the config file is: >> >> >> ------------- >> kernel = ''/boot/vmlinuz-2.6.32.12-it-xen'' >> ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen'' >> memory = ''1024'' >> >> # >> # Disk device(s). >> # >> root = ''/dev/xvda2 ro'' >> disk = [ >> ''drbd:domusampleswap,xvda1,w'', >> ''drbd:domusampleslash,xvda2,w'', >> ] >> >> >> >> # >> # Hostname >> # >> name = ''domusample'' >> >> # >> # Networking >> # >> vif = [ ''mac=00:16:3E:58:FC:F9'' ] >> >> # >> # Behaviour >> # >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> >> extra = ''2 console=hvc0'' >> ---------- >> >> I do not have any idea here. >> >> Did somebody already have (and solve ?) this issue ? >> >> Thanks. >> Pierre >> >> >> * >> * >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2010-May-13 13:31 UTC
Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Forwarding to xen-devel .. Jeremy: Have you heard of this before? save/restore/migration takes 4-10x longer on pvops 2.6.32 compared to xenlinux 2.6.32. He verified the DEBUG options in kernel .configs are the same. -- Pasi On Wed, May 12, 2010 at 05:28:26PM -0400, Pierre POMES wrote:> Hi, > > First sorry for the double posting... > > I just built a 2.6.32.10 kernel with Andrew Lyon patches (so it is a > "xenlinux" kernel, not a "pvops" kernel). > > Live migration and restore operations are between 4 and 10 times faster > with this kernel . Furthermore, during live migration, hangs time in > domu are shorter (1-2 seconds versus 1 to 15 seconds for a domu with > 256M RAM). > > Error messages "Error when reading batch size" / "error when buffering > batch, finishing" are still in my logs. > > Regarding times, all is now similar to what I had with Xen 3.x on top of > xenlinux kernels. > > Regards, > Pierre > > > >> Hi all, >> >> I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy >> kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am >> writing this email). This kernel is also used in my domu. >> >> I can save a domu without any problem, but restoring it may need from >> 2 to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There >> also errors in /var/log/xen/xend.log, "Error when reading batch size" >> and "Error when reading batch size": >> >> [2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain >> details: {''image/entry'': ''18446744071587529216'', ''console/port'': ''2'', >> ''image/loader'': ''generic'', ''vm'': >> ''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'', >> ''control/platform-feature-multiprocessor-suspend'': ''1'', >> ''image/hv-start-low'': ''18446603336221196288'', ''image/guest-os'': >> ''linux'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': >> ''1048576'', ''image/guest-version'': ''2.6'', ''image/pae-mode'': ''yes'', >> ''description'': '''', ''console/limit'': ''1048576'', ''image/paddr-offset'': >> ''0'', ''image/hypercall-page'': ''18446744071578882048'', >> ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', >> ''image/features/pae-pgdir-above-4gb'': ''1'', >> ''image/features/writable-page-tables'': ''0'', ''console/type'': >> ''xenconsoled'', ''name'': ''domusample'', ''domid'': ''10'', >> ''image/xen-version'': ''xen-3.0'', ''store/port'': ''1''} >> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286) >> restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, >> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]: >> /usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0 >> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore >> start: p2m_size = 40000 >> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory >> pages: 0% >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >> error: Error when reading batch size >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >> error: error when buffering batch, finishing >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100% >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded >> (0 pages) >> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0 >> >> Live migration has the same problem, it may need several minutes to >> complete. Please note that restore and migration do not fail, but >> there are very long. >> >> My domu is on top of DRBD, and the config file is: >> >> >> ------------- >> kernel = ''/boot/vmlinuz-2.6.32.12-it-xen'' >> ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen'' >> memory = ''1024'' >> >> # >> # Disk device(s). >> # >> root = ''/dev/xvda2 ro'' >> disk = [ >> ''drbd:domusampleswap,xvda1,w'', >> ''drbd:domusampleslash,xvda2,w'', >> ] >> >> >> >> # >> # Hostname >> # >> name = ''domusample'' >> >> # >> # Networking >> # >> vif = [ ''mac=00:16:3E:58:FC:F9'' ] >> >> # >> # Behaviour >> # >> on_poweroff = ''destroy'' >> on_reboot = ''restart'' >> on_crash = ''restart'' >> >> extra = ''2 console=hvc0'' >> ---------- >> >> I do not have any idea here. >> >> Did somebody already have (and solve ?) this issue ? >> >> Thanks. >> Pierre >> >> >> * >> * >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeremy Fitzhardinge
2010-May-13 17:08 UTC
Re: [Xen-devel] Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
On 05/13/2010 06:31 AM, Pasi Kärkkäinen wrote:> Forwarding to xen-devel .. > > Jeremy: Have you heard of this before? save/restore/migration > takes 4-10x longer on pvops 2.6.32 compared to xenlinux 2.6.32. > > He verified the DEBUG options in kernel .configs are the same. >No, I wasn''t aware of any big save/restore performance differences. Is the difference caused by a pvops dom0 or domU or both? One materially different thing is that pvops kernels support preemption, which requires all processes to be frozen before a suspend. I wonder if disabling preemption makes a difference (assuming that it is the domU which is causing the slowdown). Ah, but the report is that its the restore which is very slow. Which suggests that it is the dom0 environment which is causing problems. Does "top" show a particular process is very cpu-bound during the restore? Or is it IO bound? J> -- Pasi > > On Wed, May 12, 2010 at 05:28:26PM -0400, Pierre POMES wrote: > >> Hi, >> >> First sorry for the double posting... >> >> I just built a 2.6.32.10 kernel with Andrew Lyon patches (so it is a >> "xenlinux" kernel, not a "pvops" kernel). >> >> Live migration and restore operations are between 4 and 10 times faster >> with this kernel . Furthermore, during live migration, hangs time in >> domu are shorter (1-2 seconds versus 1 to 15 seconds for a domu with >> 256M RAM). >> >> Error messages "Error when reading batch size" / "error when buffering >> batch, finishing" are still in my logs. >> >> Regarding times, all is now similar to what I had with Xen 3.x on top of >> xenlinux kernels. >> >> Regards, >> Pierre >> >> >> >> >>> Hi all, >>> >>> I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy >>> kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am >>> writing this email). This kernel is also used in my domu. >>> >>> I can save a domu without any problem, but restoring it may need from >>> 2 to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There >>> also errors in /var/log/xen/xend.log, "Error when reading batch size" >>> and "Error when reading batch size": >>> >>> [2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain >>> details: {''image/entry'': ''18446744071587529216'', ''console/port'': ''2'', >>> ''image/loader'': ''generic'', ''vm'': >>> ''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'', >>> ''control/platform-feature-multiprocessor-suspend'': ''1'', >>> ''image/hv-start-low'': ''18446603336221196288'', ''image/guest-os'': >>> ''linux'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': >>> ''1048576'', ''image/guest-version'': ''2.6'', ''image/pae-mode'': ''yes'', >>> ''description'': '''', ''console/limit'': ''1048576'', ''image/paddr-offset'': >>> ''0'', ''image/hypercall-page'': ''18446744071578882048'', >>> ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', >>> ''image/features/pae-pgdir-above-4gb'': ''1'', >>> ''image/features/writable-page-tables'': ''0'', ''console/type'': >>> ''xenconsoled'', ''name'': ''domusample'', ''domid'': ''10'', >>> ''image/xen-version'': ''xen-3.0'', ''store/port'': ''1''} >>> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286) >>> restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, >>> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]: >>> /usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0 >>> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore >>> start: p2m_size = 40000 >>> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory >>> pages: 0% >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >>> error: Error when reading batch size >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >>> error: error when buffering batch, finishing >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100% >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded >>> (0 pages) >>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0 >>> >>> Live migration has the same problem, it may need several minutes to >>> complete. Please note that restore and migration do not fail, but >>> there are very long. >>> >>> My domu is on top of DRBD, and the config file is: >>> >>> >>> ------------- >>> kernel = ''/boot/vmlinuz-2.6.32.12-it-xen'' >>> ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen'' >>> memory = ''1024'' >>> >>> # >>> # Disk device(s). >>> # >>> root = ''/dev/xvda2 ro'' >>> disk = [ >>> ''drbd:domusampleswap,xvda1,w'', >>> ''drbd:domusampleslash,xvda2,w'', >>> ] >>> >>> >>> >>> # >>> # Hostname >>> # >>> name = ''domusample'' >>> >>> # >>> # Networking >>> # >>> vif = [ ''mac=00:16:3E:58:FC:F9'' ] >>> >>> # >>> # Behaviour >>> # >>> on_poweroff = ''destroy'' >>> on_reboot = ''restart'' >>> on_crash = ''restart'' >>> >>> extra = ''2 console=hvc0'' >>> ---------- >>> >>> I do not have any idea here. >>> >>> Did somebody already have (and solve ?) this issue ? >>> >>> Thanks. >>> Pierre >>> >>> >>> * >>> * >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pierre POMES
2010-May-13 19:44 UTC
Re: [Xen-devel] Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Hi Jeremy,> No, I wasn''t aware of any big save/restore performance differences. Is > the difference caused by a pvops dom0 or domU or both? >On the domu, I tried a 2.6.32.12 pvops kernel and the standard "2.6.32-22-server" from Ubuntu Lucid. It makes no difference. What is making the difference is using a "xenlinux" kernel in dom0 (2.6.32.10 with Andrew Lyon patches)> One materially different thing is that pvops kernels support preemption, > which requires all processes to be frozen before a suspend. I wonder if > disabling preemption makes a difference (assuming that it is the domU > which is causing the slowdown). > > Ah, but the report is that its the restore which is very slow. Which > suggests that it is the dom0 environment which is causing problems. > Does "top" show a particular process is very cpu-bound during the > restore? Or is it IO bound? >I tried a restore a little domu with 256 MB RAM. It tooks around 30s with pvops kernel : root@narbonne:~# time xm restore pp real 0m26.905s user 0m0.070s sys 0m0.020s Here is the result of "iostat -c 5 100" during the time of the restore: avg-cpu: %user %nice %system %iowait %steal %idle 0.15 0.00 1.00 0.20 0.15 98.51 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.27 0.00 0.14 99.58 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.94 0.00 0.35 98.70 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.88 0.00 0.46 98.67 avg-cpu: %user %nice %system %iowait %steal %idle 0.01 0.00 0.38 0.09 0.15 99.38 avg-cpu: %user %nice %system %iowait %steal %idle 0.06 0.00 0.97 0.44 0.30 98.23 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.08 99.92 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.01 0.00 0.00 99.99 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 So it seems to be neither CPU bound not I/O bound. I rebooted the server with my xenlinux kernel, and the same restore took : root@narbonne:~# time xm restore pp real 0m10.006s user 0m0.080s sys 0m0.000s My domu is on top of DRBD, using the drbd block script in config file. Live migration can show larger difference between both dom0 kernels : 10s versus 1 minute for the migration of the same domu. Hardware: Dell R610 with 32GB RAM, bi quad core, one RAID10 container (hard). I will compile again my dom0 pvops kernel, and triple check kernel config. I will also try without DRBD backends. If I have new results, I will ket you know. Many thanks, Pierre> J > > >> -- Pasi >> >> On Wed, May 12, 2010 at 05:28:26PM -0400, Pierre POMES wrote: >> >> >>> Hi, >>> >>> First sorry for the double posting... >>> >>> I just built a 2.6.32.10 kernel with Andrew Lyon patches (so it is a >>> "xenlinux" kernel, not a "pvops" kernel). >>> >>> Live migration and restore operations are between 4 and 10 times faster >>> with this kernel . Furthermore, during live migration, hangs time in >>> domu are shorter (1-2 seconds versus 1 to 15 seconds for a domu with >>> 256M RAM). >>> >>> Error messages "Error when reading batch size" / "error when buffering >>> batch, finishing" are still in my logs. >>> >>> Regarding times, all is now similar to what I had with Xen 3.x on top of >>> xenlinux kernels. >>> >>> Regards, >>> Pierre >>> >>> >>> >>> >>> >>>> Hi all, >>>> >>>> I am using Xen 4.0.0 on top of Ubuntu Lucid (amd64), with the Jeremy >>>> kernel taken from git (xen/stable-2.6.32.x branch, 2.6.32.12 when I am >>>> writing this email). This kernel is also used in my domu. >>>> >>>> I can save a domu without any problem, but restoring it may need from >>>> 2 to 5 minutes, from a 1G checkpoint file (domu has 1GB RAM). There >>>> also errors in /var/log/xen/xend.log, "Error when reading batch size" >>>> and "Error when reading batch size": >>>> >>>> [2010-05-08 04:23:16 9497] DEBUG (XendDomainInfo:1804) Storing domain >>>> details: {''image/entry'': ''18446744071587529216'', ''console/port'': ''2'', >>>> ''image/loader'': ''generic'', ''vm'': >>>> ''/vm/156ea44d-6707-cbe6-2d58-7bea4792dff4'', >>>> ''control/platform-feature-multiprocessor-suspend'': ''1'', >>>> ''image/hv-start-low'': ''18446603336221196288'', ''image/guest-os'': >>>> ''linux'', ''image/virt-base'': ''18446744071562067968'', ''memory/target'': >>>> ''1048576'', ''image/guest-version'': ''2.6'', ''image/pae-mode'': ''yes'', >>>> ''description'': '''', ''console/limit'': ''1048576'', ''image/paddr-offset'': >>>> ''0'', ''image/hypercall-page'': ''18446744071578882048'', >>>> ''image/suspend-cancel'': ''1'', ''cpu/0/availability'': ''online'', >>>> ''image/features/pae-pgdir-above-4gb'': ''1'', >>>> ''image/features/writable-page-tables'': ''0'', ''console/type'': >>>> ''xenconsoled'', ''name'': ''domusample'', ''domid'': ''10'', >>>> ''image/xen-version'': ''xen-3.0'', ''store/port'': ''1''} >>>> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:286) >>>> restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, >>>> [2010-05-08 04:23:16 9497] DEBUG (XendCheckpoint:305) [xc_restore]: >>>> /usr/lib/xen/bin/xc_restore 22 10 1 2 0 0 0 0 >>>> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) xc_domain_restore >>>> start: p2m_size = 40000 >>>> [2010-05-08 04:23:16 9497] INFO (XendCheckpoint:423) Reloading memory >>>> pages: 0% >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >>>> error: Error when reading batch size >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ERROR Internal >>>> error: error when buffering batch, finishing >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) ^H^H^H^H100% >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) Memory reloaded >>>> (0 pages) >>>> [2010-05-08 04:25:53 9497] INFO (XendCheckpoint:423) read VCPU 0 >>>> >>>> Live migration has the same problem, it may need several minutes to >>>> complete. Please note that restore and migration do not fail, but >>>> there are very long. >>>> >>>> My domu is on top of DRBD, and the config file is: >>>> >>>> >>>> ------------- >>>> kernel = ''/boot/vmlinuz-2.6.32.12-it-xen'' >>>> ramdisk = ''/boot/initrd.img-2.6.32.12-it-xen'' >>>> memory = ''1024'' >>>> >>>> # >>>> # Disk device(s). >>>> # >>>> root = ''/dev/xvda2 ro'' >>>> disk = [ >>>> ''drbd:domusampleswap,xvda1,w'', >>>> ''drbd:domusampleslash,xvda2,w'', >>>> ] >>>> >>>> >>>> >>>> # >>>> # Hostname >>>> # >>>> name = ''domusample'' >>>> >>>> # >>>> # Networking >>>> # >>>> vif = [ ''mac=00:16:3E:58:FC:F9'' ] >>>> >>>> # >>>> # Behaviour >>>> # >>>> on_poweroff = ''destroy'' >>>> on_reboot = ''restart'' >>>> on_crash = ''restart'' >>>> >>>> extra = ''2 console=hvc0'' >>>> ---------- >>>> >>>> I do not have any idea here. >>>> >>>> Did somebody already have (and solve ?) this issue ? >>>> >>>> Thanks. >>>> Pierre >>>> >>>> >>>> * >>>> * >>>> >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >>>> >>>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >>> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pierre POMES
2010-May-13 21:37 UTC
Re: [Xen-devel] Re: [Xen-users] Problem with restore/migration with Xen 4.0.0 and Jeremy kernel (2.6.32.12)
Le 2010-05-13 15:44, Pierre POMES a écrit :> I will compile again my dom0 pvops kernel, and triple check kernel > config. I will also try without DRBD backends. > > If I have new results, I will ket you know.I compiled a new kernel. This time, I started from the Ubuntu lucid "-server" config, on which I set specific CONFIG_XEN* options. My previous kernel was build starting from "-generic" config, with the same CONFIG_XEN* settings. I can send you both config if you wish. Config diff is (left is new config) : root@narbonne:~# diff /boot/config-2.6.32.12-it-xen /root/config-2.6.32.12-it-xen_2.0 4c4 < # Thu May 13 22:31:10 2010 --- > # Tue May 11 22:27:57 2010 215,216c215,216 < CONFIG_DEFAULT_DEADLINE=y < # CONFIG_DEFAULT_CFQ is not set --- > # CONFIG_DEFAULT_DEADLINE is not set > CONFIG_DEFAULT_CFQ=y 218c218 < CONFIG_DEFAULT_IOSCHED="deadline" --- > CONFIG_DEFAULT_IOSCHED="cfq" 312,313c312,313 < CONFIG_PREEMPT_NONE=y < # CONFIG_PREEMPT_VOLUNTARY is not set --- > # CONFIG_PREEMPT_NONE is not set > CONFIG_PREEMPT_VOLUNTARY=y 343d342 < CONFIG_ARCH_MEMORY_PROBE=y 354,356c353 < CONFIG_MEMORY_HOTPLUG=y < CONFIG_MEMORY_HOTPLUG_SPARSE=y < CONFIG_MEMORY_HOTREMOVE=y --- > # CONFIG_MEMORY_HOTPLUG is not set 400d396 < CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y 447d442 < CONFIG_ACPI_HOTPLUG_MEMORY=m 1385c1380 < CONFIG_VIRTIO_BLK=y --- > CONFIG_VIRTIO_BLK=m 1445c1440 < CONFIG_SCSI_SPI_ATTRS=y --- > CONFIG_SCSI_SPI_ATTRS=m 1515c1510 < CONFIG_SCSI_SYM53C8XX_2=y --- > CONFIG_SCSI_SYM53C8XX_2=m 2114c2109 < CONFIG_VIRTIO_NET=y --- > CONFIG_VIRTIO_NET=m 4621a4617 > # CONFIG_DEBUG_KMEMLEAK is not set 4887,4889c4883,4885 < CONFIG_VIRTIO=y < CONFIG_VIRTIO_RING=y < CONFIG_VIRTIO_PCI=y --- > CONFIG_VIRTIO=m > CONFIG_VIRTIO_RING=m > CONFIG_VIRTIO_PCI=m My domu is still on top of DRBD. This time, live migration and restore are really better. They are still slower than xenlinux kernel : from several tests, what is taking 10s in xenlinux kernel now needs around 15s (instead of 30/40s or more initially with my previous opvs config) BUT : During live migration, at the end of the process, domu is still freezing (domu is 256MB RAM, there is no activity on it), from 10 to 20 seconds, and clock inside the domy may vary a few seconds (up to 5). I also have some hangs during migration with xenlinux kernel, but they are really shorter, and clock variation seems to be smallers (around 1 second). About clock with Xen 4, is it still possible to have the same clock for dom0 and all domus ? (independant_clock=0 in Xen 3.x) Pierre * * _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users