Hi, I''m running xen 4.2 unstable (last commit e605d60923bf128dab6bd567d9fde54d001b0c24) and try to do some cpu pining. But my domUs changed the cpus from time to time like: --snip-- xen:~# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 -b- 2833.2 any cpu Domain-0 0 1 1 r-- 1772.3 any cpu gateway 1 0 0 --- 421.6 0 appserver 2 0 1 -b- 57.7 0 filer 3 0 1 -b- 1819.5 0 backup 5 0 1 -b- 2082.5 0 dvdconvert 6 0 0 --- 3546.1 any cpu dvdconvert 6 1 0 --- 3320.6 any cpu vdrserver 7 0 0 -b- 14.7 1 --snip-- I''ve pinned vdrserver to cpu 1 and it runs an cpu 0 for example. Is this a bug, or did I something wrong ? Btw: I use the credit2 scheduler, because it is better for my dvb broadcasting stuff. Here the output of xl dmesg: --snip-- host : xen release : 3.3.8-grsec version : #1-Alpine SMP Tue Jun 5 12:56:58 UTC 2012 machine : x86_64 nr_cpus : 2 max_cpu_id : 1 nr_nodes : 1 cores_per_socket : 2 threads_per_core : 1 cpu_mhz : 1781 hw_caps : 178bf3ff:2fd3fbff:00000000:00001710:00802201:00000000:000035ff:00000000 virt_caps : hvm total_memory : 7789 free_memory : 3089 sharing_freed_memory : 0 sharing_used_memory : 0 free_cpus : 0 xen_major : 4 xen_minor : 2 xen_extra : -unstable xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit2 xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable xen_commandline : dom0_mem=1024M sched=credit2 loglvl=all guest_loglvl=all cc_compiler : gcc version 4.6.3 (Alpine 4.6.3-r0) cc_compile_by : root cc_compile_domain : intern.loc cc_compile_date : Wed Jun 13 15:54:36 CEST 2012 xend_config_format : 4 --snip-- -- best regards Dieter Bloms -- I do not get viruses because I do not use MS software. If you use Outlook then please do not put my email address in your address-book so that WHEN you get a virus it won''t use my address in the From field.
George Dunlap
2012-Jun-22 11:18 UTC
Re: cpu pinning not working correctly in xen-4.2 unstable
On Fri, Jun 22, 2012 at 10:37 AM, Dieter Bloms <xensource.com@bloms.de> wrote:> Hi, > > I''m running xen 4.2 unstable (last commit e605d60923bf128dab6bd567d9fde54d001b0c24) > and try to do some cpu pining.Hey Deiter, did you mean to post this to xen-devel instead?> > But my domUs changed the cpus from time to time like: > > --snip-- > xen:~# xl vcpu-list > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 -b- 2833.2 any cpu > Domain-0 0 1 1 r-- 1772.3 any cpu > gateway 1 0 0 --- 421.6 0 > appserver 2 0 1 -b- 57.7 0 > filer 3 0 1 -b- 1819.5 0 > backup 5 0 1 -b- 2082.5 0 > dvdconvert 6 0 0 --- 3546.1 any cpu > dvdconvert 6 1 0 --- 3320.6 any cpu > vdrserver 7 0 0 -b- 14.7 1 > --snip-- > > I''ve pinned vdrserver to cpu 1 and it runs an cpu 0 for example. > > Is this a bug, or did I something wrong ? > > Btw: I use the credit2 scheduler, because it is better for my dvb broadcasting stuff.Glad to know it works well for you -- that''s exactly the kind of workload it''s aimed at. Unfortunately it''s still experimental: pinning is not implemented yet in credit2. You should be able to implement something similar using cpupools. There''s a blog post on that here: http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/ Another feature introduced in 4.2 is the ability to set credit1''s credit timeslice. The default is still 30ms, which is *far* too long for real-time; you might try setting it to something between 1 and 5 and see if that works well enough for you. There''s a blog post on that here: http://blog.xen.org/index.php/2012/04/10/xen-4-2-new-scheduler-parameters-2/ -George
Dieter Bloms
2012-Jun-22 11:59 UTC
Re: cpu pinning not working correctly in xen-4.2 unstable
Hi George, thank you for the quick answer. On Fri, Jun 22, George Dunlap wrote:> Hey Deiter, did you mean to post this to xen-devel instead?No, I have sent this to xen-users, because I thought I made a mistake, or missed some config option.> > Btw: I use the credit2 scheduler, because it is better for my dvb broadcasting stuff. > > Glad to know it works well for you -- that''s exactly the kind of > workload it''s aimed at. Unfortunately it''s still experimental: > pinning is not implemented yet in credit2. You should be able to > implement something similar using cpupools. There''s a blog post on > that here: http://blog.xen.org/index.php/2012/04/23/xen-4-2-cpupools/my cpu has only 2 cores without hyperthreading, so I think cpupools are a little oversized. But the possibility of two complete different schedulers is cool ;)> Another feature introduced in 4.2 is the ability to set credit1''s > credit timeslice. The default is still 30ms, which is *far* too long > for real-time; you might try setting it to something between 1 and 5 > and see if that works well enough for you. There''s a blog post on > that here: http://blog.xen.org/index.php/2012/04/10/xen-4-2-new-scheduler-parameters-2/I''ve played a little with these in the past, for example sched_credit_tslice_ms=5 and sched_ratelimit_us=1000, but hadn''t good results (artifacts on my television from time to time). Which values do you recommened for sched_ratelimit_us, when I use credit1 ? Is it possible to see the overhead from the hypervisor (if I set this two parameters to small for example). I tried the sedf scheduler, but this scheduler slow down all of my domUs, so I think credit* schedulers are the best ones for my workload. -- Best regards Dieter -- I do not get viruses because I do not use MS software. If you use Outlook then please do not put my email address in your address-book so that WHEN you get a virus it won''t use my address in the From field.
Florian Heigl
2012-Jun-23 16:18 UTC
Re: cpu pinning not working correctly in xen-4.2 unstable
Hi, [cut the bits i didn''t know even the smallest tiny fraction of]> Is it possible to see the overhead from the hypervisor (if I set > this two parameters to small for example).If you can sink some time into it, there is libxenstat. The python bindings are a bit broken and it didnt manage to compile it on 64bit. It gives fine grained results though. I think the HV time is not in there, but if you''re under load you should be able to calculate it simply by looking at the timers.> I tried the sedf scheduler, but this scheduler slow down all of my > domUs, so I think credit* schedulers are the best ones for my workload.I think sedf had a limit in that it only works as intended with one core. Anyone remember if thats true?
Dario Faggioli
2012-Jun-26 15:28 UTC
Re: cpu pinning not working correctly in xen-4.2 unstable
On Sat, 2012-06-23 at 18:18 +0200, Florian Heigl wrote:> > I tried the sedf scheduler, but this scheduler slow down all of my > > domUs, so I think credit* schedulers are the best ones for my workload. > > I think sedf had a limit in that it only works as intended with one > core. Anyone remember if thats true? >It is. Well, you can use all the cores/cpus but you have to manually move the vcpus there, the scheduler itself won''t balance the workload automatically... Which sucks and needs fixing... as soon as someone will find some time to do it! :-P Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Florian Heigl
2012-Jun-26 20:14 UTC
Re: cpu pinning not working correctly in xen-4.2 unstable
2012/6/26 Dario Faggioli <raistlin.df@gmail.com>:> On Sat, 2012-06-23 at 18:18 +0200, Florian Heigl wrote: >> > I tried the sedf scheduler, but this scheduler slow down all of my >> > domUs, so I think credit* schedulers are the best ones for my workload. >> >> I think sedf had a limit in that it only works as intended with one >> core. Anyone remember if thats true? >> > It is. > > Well, you can use all the cores/cpus but you have to manually move the > vcpus there, the scheduler itself won''t balance the workload > automatically... Which sucks and needs fixing... as soon as someone will > find some time to do it! :-PI bet that would make a great summer of code project for someone really skilled. :)) Probably schedulers are like mail clients? All suck, one sucks less. :> Florian -- the purpose of libvirt is to provide an abstraction layer hiding all xen features added since 2006 until they were finally understood and copied by the kvm devs.