Trying to migrate a 64bit PV guest with 64GB running medium to heavy load on xen 3.4.0, it is showing lot of soft lockups. The softlockups are causing dom0 reboot by the cluster FS. The hardware has 256GB and 32 CPUs. Looking into the hypervisor thru kdb, I see one cpu in sh_resync_all() while all other 31 appear spinning on the shadow_lock. I vaguely remember seeing some thread on this while ago, but just can''t seem to google find it now. I''m trying to figure what could be done in the short run. Now that guests are getting bigger in memory, bugs of this nature are slowly popping up under medium/heavy load. I''ve been thinking of what could be done to adderss those in the long run. May be create a certain class of pages, that once migrated, are ''w'' protected, and any write faults on them are resolved on the target system, is one idea. Incidentally, IBM took the reverse approach. The (VCPU) contexts are migrated and pages are pulled in. thanks, Mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Oct 22, 2009 at 09:21:49PM -0700, Mukesh Rathor wrote:> > > Trying to migrate a 64bit PV guest with 64GB running medium to heavy load > on xen 3.4.0, it is showing lot of soft lockups. The softlockups are > causing dom0 reboot by the cluster FS. The hardware has 256GB and 32 > CPUs. >Did you try with Xen 3.4.1 or the latest xen-3.4-testing.hg ? There are a lot of fixes after 3.4.0 .. -- Pasi> Looking into the hypervisor thru kdb, I see one cpu in sh_resync_all() > while all other 31 appear spinning on the shadow_lock. I vaguely remember > seeing some thread on this while ago, but just can''t seem to google find > it now. I''m trying to figure what could be done in the short run. > > Now that guests are getting bigger in memory, bugs of this nature are slowly > popping up under medium/heavy load. I''ve been thinking of what could be > done to adderss those in the long run. May be create a certain class of > pages, that once migrated, are ''w'' protected, and any write faults on them > are resolved on the target system, is one idea. Incidentally, IBM took > the reverse approach. The (VCPU) contexts are migrated and pages are > pulled in. > > > thanks, > Mukesh > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote:> Trying to migrate a 64bit PV guest with 64GB running medium to heavy load > on xen 3.4.0, it is showing lot of soft lockups. The softlockups are > causing dom0 reboot by the cluster FS. The hardware has 256GB and 32 > CPUs. > > Looking into the hypervisor thru kdb, I see one cpu in sh_resync_all() > while all other 31 appear spinning on the shadow_lock.How many vcpus does the guest have? Scalability issues in the OOS shadow code are more related to number of VCPUs than amount of RAM.> I vaguely remember > seeing some thread on this while ago, but just can''t seem to google find > it now. I''m trying to figure what could be done in the short run.The solution (for BS2000) was to plumb in a flag that disabled the OOS code for particular domains. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 11:09 +0100 on 23 Oct (1256296176), Tim Deegan wrote:> At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote: > > Trying to migrate a 64bit PV guest with 64GB running medium to heavy load > > on xen 3.4.0, it is showing lot of soft lockups. The softlockups are > > causing dom0 reboot by the cluster FS. The hardware has 256GB and 32 > > CPUs. > > > > Looking into the hypervisor thru kdb, I see one cpu in sh_resync_all() > > while all other 31 appear spinning on the shadow_lock. > > How many vcpus does the guest have? Scalability issues in the OOS > shadow code are more related to number of VCPUs than amount of RAM.With my brain in gear, I realise that it must have 32. :) Yes, that''s likely to be well past the point of diminishing returns for OOS. Tim.> > I vaguely remember > > seeing some thread on this while ago, but just can''t seem to google find > > it now. I''m trying to figure what could be done in the short run. > > The solution (for BS2000) was to plumb in a flag that disabled the OOS > code for particular domains. > > Tim. > > -- > Tim Deegan <Tim.Deegan@citrix.com> > Principal Software Engineer, Citrix Systems (R&D) Ltd. > [Company #02300071, SL9 0DZ, UK.]-- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 23 Oct 2009 11:09:36 +0100 Tim Deegan <Tim.Deegan@citrix.com> wrote:> At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote: > > Trying to migrate a 64bit PV guest with 64GB running medium to > > heavy load on xen 3.4.0, it is showing lot of soft lockups. The > > softlockups are causing dom0 reboot by the cluster FS. The hardware > > has 256GB and 32 CPUs. > > > > Looking into the hypervisor thru kdb, I see one cpu in > > sh_resync_all() while all other 31 appear spinning on the > > shadow_lock. > > How many vcpus does the guest have? Scalability issues in the OOS > shadow code are more related to number of VCPUs than amount of RAM.Actually, things are fine with 32GB/32vcpus. Problem happens with 64GB/32vcpus. Trying the unstable version now.> > I vaguely remember > > seeing some thread on this while ago, but just can''t seem to google > > find it now. I''m trying to figure what could be done in the short > > run. > > The solution (for BS2000) was to plumb in a flag that disabled the OOS > code for particular domains. > > Tim. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 23 Oct 2009 15:16:51 -0700 Mukesh Rathor <mukesh.rathor@oracle.com> wrote:> On Fri, 23 Oct 2009 11:09:36 +0100 > Tim Deegan <Tim.Deegan@citrix.com> wrote: > > > At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote: > > > Trying to migrate a 64bit PV guest with 64GB running medium to > > > heavy load on xen 3.4.0, it is showing lot of soft lockups. The > > > softlockups are causing dom0 reboot by the cluster FS. The > > > hardware has 256GB and 32 CPUs. > > > > > > Looking into the hypervisor thru kdb, I see one cpu in > > > sh_resync_all() while all other 31 appear spinning on the > > > shadow_lock. > > > > How many vcpus does the guest have? Scalability issues in the OOS > > shadow code are more related to number of VCPUs than amount of RAM. > > Actually, things are fine with 32GB/32vcpus. Problem happens with > 64GB/32vcpus. Trying the unstable version now.Nah, with c/s 20365 and oos=0 in vm.cfg, it fails right away: [root@OVM_EL5U3_X86_64_PVM_4GB]# xm migrate -l 3 vega7183 Error: /usr/lib/xen/bin/xc_save 82 3 0 0 1 failed On source xend.log: [2009-09-23 16:22:33 16993] DEBUG (balloon:181) Balloon: 199147540 KiB free; need 16384; done. [2009-09-23 16:22:34 16993] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib/xen/bin/xc_save 82 3 0 0 1 [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) xc_save: failed to get the suspend evtchn port [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) ERROR Internal error: xc_get_m2p_mfns [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) ERROR Internal error: Failed to map live M2P table [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) Save exit rc=1 [2009-09-23 16:22:34 16993] ERROR (XendCheckpoint:164) Save failed on domain OVM_EL5U3_X86_64_PVM_4GB (3) - resuming. on TARGET looks pretty screwy: domain'', [''domid'', ''3''], [''on_crash'', ''restart''], [''uuid'', ''b990db11-57f4-a553-5ee0-c022234f3dd5''], [''bootloader_args'', ''-q''], [''vcpus'', ''32''], [''name'', ''OVM_EL5U3_X86_64_PVM_4GB''], [''on_poweroff'', ''destroy''], [''on_reboot'', ''restart''], [''cpus'', [[''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''! 1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''], [''0'', ''1'', ''2'', ''3'', ''4'', ''5'', ''6'', ''7'', ''8'', ''9'', ''10'', ''11'', ''12'', ''13'', ''14'', ''15'', ''16'', ''17'', ''18'', ''19'', ''20'', ''21'', ''22'', ''23'', ''24'', ''25'', ''26'', ''27'', ''28'', ''29'', ''30'', ''31''] ....... .......... [2009-10-23 16:21:15 11795] DEBUG (image:319) No VNC passwd configured for vfb access [2009-10-23 16:21:15 11795] DEBUG (XendCheckpoint:261) restore:shadow=0x0, _static_max=0xfa0000000, _static_min=0x0, [2009-10-23 16:21:15 11795] DEBUG (balloon:181) Balloon: 264942044 KiB free; need 65536000; done. [2009-10-23 16:21:15 11795] DEBUG (XendCheckpoint:278) [xc_restore]: /usr/lib/xen/bin/xc_restore 4 3 1 2 0 0 0 [2009-10-23 16:21:15 11795] INFO (XendCheckpoint:418) ERROR Internal error: read: p2m_size [2009-10-23 16:21:15 11795] INFO (XendCheckpoint:418) Restore exit with rc=1 [2009-10-23 16:21:15 11795] DEBUG (XendDomainInfo:2748) XendDomainInfo.destroy: domid=3 [2009-10-23 16:21:15 11795] ERROR (XendDomainInfo:2762) XendDomainInfo.destroy: domain destruction failed. Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2755, in destroy _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 23:16 +0100 on 23 Oct (1256339811), Mukesh Rathor wrote:> On Fri, 23 Oct 2009 11:09:36 +0100 > Tim Deegan <Tim.Deegan@citrix.com> wrote: > > > At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote: > > > Trying to migrate a 64bit PV guest with 64GB running medium to > > > heavy load on xen 3.4.0, it is showing lot of soft lockups. The > > > softlockups are causing dom0 reboot by the cluster FS. The hardware > > > has 256GB and 32 CPUs. > > > > > > Looking into the hypervisor thru kdb, I see one cpu in > > > sh_resync_all() while all other 31 appear spinning on the > > > shadow_lock. > > > > How many vcpus does the guest have? Scalability issues in the OOS > > shadow code are more related to number of VCPUs than amount of RAM. > > Actually, things are fine with 32GB/32vcpus. Problem happens with > 64GB/32vcpus. Trying the unstable version now.Interesting. Have you tried increading the amount of shadow memeory you give to the guest? IIRC xend tries to pick a sensible default but if it''s too low and you start thrashing things can get very slow indeed. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, At 02:04 +0100 on 24 Oct (1256349879), Mukesh Rathor wrote:> Nah, with c/s 20365 and oos=0 in vm.cfg, it fails right away: > > [root@OVM_EL5U3_X86_64_PVM_4GB]# xm migrate -l 3 vega7183 > Error: /usr/lib/xen/bin/xc_save 82 3 0 0 1 failed > On source xend.log: > > [2009-09-23 16:22:33 16993] DEBUG (balloon:181) Balloon: 199147540 KiB free; need 16384; done. > [2009-09-23 16:22:34 16993] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib/xen/bin/xc_save 82 3 0 0 1 > [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) xc_save: failed to get the suspend evtchn portHmm. Are you sure your xen and tools match? Does save/restore work with oos=1?> on TARGET looks pretty screwy: > > [2009-10-23 16:21:15 11795] INFO (XendCheckpoint:418) ERROR Internal error: read: p2m_sizeThat rather unhelpful message is what you get if the sending side aborts before sending any data (p2m_size is the first field in the save format). Cheers, Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 26 Oct 2009 09:28:03 +0000 Tim Deegan <Tim.Deegan@citrix.com> wrote:> Hi, > > At 02:04 +0100 on 24 Oct (1256349879), Mukesh Rathor wrote: > > Nah, with c/s 20365 and oos=0 in vm.cfg, it fails right away: > > > > [root@OVM_EL5U3_X86_64_PVM_4GB]# xm migrate -l 3 vega7183 > > Error: /usr/lib/xen/bin/xc_save 82 3 0 0 1 failed > > On source xend.log: > > > > [2009-09-23 16:22:33 16993] DEBUG (balloon:181) Balloon: 199147540 > > KiB free; need 16384; done. [2009-09-23 16:22:34 16993] DEBUG > > (XendCheckpoint:110) [xc_save]: /usr/lib/xen/bin/xc_save 82 3 0 0 1 > > [2009-09-23 16:22:34 16993] INFO (XendCheckpoint:418) xc_save: > > failed to get the suspend evtchn port > > Hmm. Are you sure your xen and tools match? Does save/restore work > with oos=1?Grrr... the tools compatibility breaks soo easly... yeah, 3.4 tools with unstable xen... sorry, i missed that... will try again..> > on TARGET looks pretty screwy: > > > > [2009-10-23 16:21:15 11795] INFO (XendCheckpoint:418) ERROR > > Internal error: read: p2m_size > > That rather unhelpful message is what you get if the sending side > aborts before sending any data (p2m_size is the first field in the > save format). > > Cheers, > > Tim. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 23 Oct 2009 11:09:36 +0100 Tim Deegan <Tim.Deegan@citrix.com> wrote:> At 05:21 +0100 on 23 Oct (1256275309), Mukesh Rathor wrote: > > Trying to migrate a 64bit PV guest with 64GB running medium to > > heavy load on xen 3.4.0, it is showing lot of soft lockups. The > > softlockups are causing dom0 reboot by the cluster FS. The hardware > > has 256GB and 32 CPUs. > > > > Looking into the hypervisor thru kdb, I see one cpu in > > sh_resync_all() while all other 31 appear spinning on the > > shadow_lock. > > How many vcpus does the guest have? Scalability issues in the OOS > shadow code are more related to number of VCPUs than amount of RAM. > > > I vaguely remember > > seeing some thread on this while ago, but just can''t seem to google > > find it now. I''m trying to figure what could be done in the short > > run. > > The solution (for BS2000) was to plumb in a flag that disabled the OOS > code for particular domains.Ok, I''m confused. It appears oos disable is relevant for hvm only... I''m running PV. sh_update_paging_modes() .... if ( is_hvm_domain(d) && !d->arch.paging.shadow.oos_off ) <--- ... Also,>> Actually, things are fine with 32GB/32vcpus. Problem happens with >> 64GB/32vcpus. Trying the unstable version now.>Interesting. Have you tried increading the amount of shadow memeory >you give to the guest? IIRC xend tries to pick a sensible default but >if it''s too low and you start thrashing things can get very slow indeed.What do you recommend I start with for 32VCPUs and 64GB? thanks Mukesh _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, At 03:06 +0000 on 06 Nov (1257476764), Mukesh Rathor wrote:> Ok, I''m confused. It appears oos disable is relevant for hvm only... I''m > running PV. > > sh_update_paging_modes() > .... > if ( is_hvm_domain(d) && !d->arch.paging.shadow.oos_off ) <--- > ...Hmmm. It looks like we never unsync for PV guests. There''s probably no reason not to, but it probably wouldn''t help all that much since we already intercept all PV pagetable updates. It''s a bit surprising that sh_resync_all() is the function your CPU is stopped in. Is that consistently the case or was it just one example? I suppose for 32 VCPUs it does a lot of locking and unlocking of the shadow lock. You could try adding if ( !d->arch.paging.shadow.oos_active ) return; at the top of that function and see if it helps.> Also, > >> Actually, things are fine with 32GB/32vcpus. Problem happens with > >> 64GB/32vcpus. Trying the unstable version now. > > >Interesting. Have you tried increading the amount of shadow memeory > >you give to the guest? IIRC xend tries to pick a sensible default but > >if it''s too low and you start thrashing things can get very slow indeed. > > What do you recommend I start with for 32VCPUs and 64GB?It really depends on the workload. I think the default for a 64GiB domain will be about 128MiB, so maybe try 256MiB and 512MiB and see if it makes a difference. This bit of python will let you change it on the fly: run it with a domid and a shadow allocation in MiB. #!/usr/bin/env python import sys import xen.lowlevel.xc xc = xen.lowlevel.xc.xc() print "%i" % xc.shadow_mem_control(dom=int(sys.argv[1]), mb=int(sys.argv[2])) Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 6 Nov 2009 10:03:44 +0000 Tim Deegan <Tim.Deegan@citrix.com> wrote:> > >Interesting. Have you tried increading the amount of shadow > > >memeory you give to the guest? IIRC xend tries to pick a sensible > > >default but if it''s too low and you start thrashing things can get > > >very slow indeed. > > > > What do you recommend I start with for 32VCPUs and 64GB? > > It really depends on the workload. I think the default for a 64GiB > domain will be about 128MiB, so maybe try 256MiB and 512MiB and see if > it makes a difference. This bit of python will let you change it on > the fly: run it with a domid and a shadow allocation in MiB.Setting shadow mem to 256MB fixes the problem for 64GB guest... now, it''s turn for 200GB+ guest... _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Possibly Parallel Threads
- [PATCH]: PVH: specify xen features strings cleany for PVH
- [help]: VPID tagged TLBs question.
- [PATCH 4/5] xen: arm: implement remap interfaces needed for privcmd mappings.
- xen debugger (kdb/xdb/hdb) patch for c/s 25467
- [PATCH RFC v13 00/20] Introduce PVH domU support