I have a multi-processor, multi-core environment where I will be running an application on one HVM domain A (Windows 2003) and another app in another HVM domain A'' (Windows as well). There will be multiple instances of this pair combination. For performance reasons, I would like to find out if there is any way to control the scheduling of the paired domains, such that 1) if domain A is scheduled on a physical CPU 1, domain A'' is also scheduled at the same time on cpu 2. or 2) if domain A is scheduled on physical CPU, domain A'' is the next domain to be scheduled on the CPU, even if domain B was the next legal owner of the time slice. Assigning a higher priority to a domain type (A or A'') may not work because multiple domains of one type may be available simultaneously. I need to guarantee that once a domain is scheduled, its corresponding pair will run simultaneously (in another CPU) or immediately after. I''m new to Xen so forgive me if this is trivial question to which I have not found the answer yet. Roger Cruz Principal SW Engineer Marathon Technologies Corp. 978-489-1153 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''m resending this to the list because I don''t believe it made it through the first time. ------------------- I have a multi-processor, multi-core environment where I will be running an application on one HVM domain A (Windows 2003) and another app in another HVM domain A'' (Windows as well). There will be multiple instances of this pair combination. For performance reasons, I would like to find out if there is any way to control the scheduling of the paired domains, such that 1) if domain A is scheduled on a physical CPU 1, domain A'' is also scheduled at the same time on cpu 2. or 2) if domain A is scheduled on physical CPU, domain A'' is the next domain to be scheduled on the CPU, even if domain B was the next legal owner of the time slice. Assigning a higher priority to a domain type (A or A'') may not work because multiple domains of one type may be available simultaneously. I need to guarantee that once a domain is scheduled, its corresponding pair will run simultaneously (in another CPU) or immediately after. I''m new to Xen so forgive me if this is trivial question to which I have not found the answer yet. Roger R Cruz Principal SW Engineer Marathon Technologies Corp. 978-489-1153 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I''m resending this to the list because I don''t believe it made it > through the first time.If you''re not subscribed there''s a delay before somebody comes along and manually allows your mail.> I have a multi-processor, multi-core environment where I will be running > an application on one HVM domain A (Windows 2003) and another app in > another HVM domain A'' (Windows as well). There will be multiple > instances of this pair combination. For performance reasons, I would > like to find out if there is any way to control the scheduling of the > paired domains, such that > > > > 1) if domain A is scheduled on a physical CPU 1, domain A'' is also > scheduled at the same time on cpu 2. orThere''s not a simple way of arranging this.> 2) if domain A is scheduled on physical CPU, domain A'' is the next > domain to be scheduled on the CPU, even if domain B was the next legal > owner of the time slice.If you can dedicate a CPU per domain pair then you can just pin each domain pair to a different CPU. This should approximate the behaviour you want. Even if you pin several pairs to a CPU it should bound the latency somewhat... however... What''s you''re application? If requests between your domains can be batched up somehow then I''d expect performance to be OK without special configuration. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark, 0) You are right, I was not subscribed to the list when I sent the first email. 1) I''ve heard of something called stub domains that may do something similar to case 1. I''m trying to get more info on it. Don''t know if it even works for Windows HVMs. 2) We''ll have a lot more domain pairs than CPUs. They (the pairs) run independent applications of each other so batching requests is not a choice at this point. But we do know that the two pairs will communicate (server-client app) and will use a shared memory construct to pass the messages back (no IP connection, just a proprietary comm. protocol using shared memory). There are latency issues involved in getting to those packets so if we could guarantee that when the client makes a request, the server will get it (almost) immediately, then we''re golden. So, I''m interested in tweaking the scheduling algorithm to guarantee that the server can get on the CPU after the client runs. Thank you Roger R. Cruz> I''m resending this to the list because I don''t believe it made it > through the first time.If you''re not subscribed there''s a delay before somebody comes along and manually allows your mail.> I have a multi-processor, multi-core environment where I will berunning> an application on one HVM domain A (Windows 2003) and another app in > another HVM domain A'' (Windows as well). There will be multiple > instances of this pair combination. For performance reasons, I would > like to find out if there is any way to control the scheduling of the > paired domains, such that > > > > 1) if domain A is scheduled on a physical CPU 1, domain A'' isalso> scheduled at the same time on cpu 2. orThere''s not a simple way of arranging this.> 2) if domain A is scheduled on physical CPU, domain A'' is thenext> domain to be scheduled on the CPU, even if domain B was the next legal > owner of the time slice.If you can dedicate a CPU per domain pair then you can just pin each domain pair to a different CPU. This should approximate the behaviour you want. Even if you pin several pairs to a CPU it should bound the latency somewhat... however... What''s you''re application? If requests between your domains can be batched up somehow then I''d expect performance to be OK without special configuration. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tuesday 27 March 2007, Roger Cruz wrote:> 2) We''ll have a lot more domain pairs than CPUs. They (the pairs) run > independent applications of each other so batching requests is not a > choice at this point. But we do know that the two pairs will > communicate (server-client app) and will use a shared memory construct > to pass the messages back (no IP connection, just a proprietary comm. > protocol using shared memory). There are latency issues involved in > getting to those packets so if we could guarantee that when the client > makes a request, the server will get it (almost) immediately, then we''re > golden. So, I''m interested in tweaking the scheduling algorithm to > guarantee that the server can get on the CPU after the client runs.have you managed to get that shared memory working on Xen? i guess that would be your first problem, CPU scheduling a very distant second. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Roger Cruz > Sent: 27 March 2007 21:46 > To: xen-users@lists.xensource.com > Subject: RE: [Xen-users] RE: Co-scheduling HVM domains... > > Mark, > > 0) You are right, I was not subscribed to the list when I > sent the first > email. > > 1) I''ve heard of something called stub domains that may do something > similar to case 1. I''m trying to get more info on it. Don''t > know if it > even works for Windows HVMs.Stub domains has nothing to do with the problem you''re trying to solve.> > 2) We''ll have a lot more domain pairs than CPUs. They (the pairs) run > independent applications of each other so batching requests is not a > choice at this point. But we do know that the two pairs will > communicate (server-client app) and will use a shared memory construct > to pass the messages back (no IP connection, just a proprietary comm. > protocol using shared memory). There are latency issues involved in > getting to those packets so if we could guarantee that when the client > makes a request, the server will get it (almost) immediately, > then we''re > golden. So, I''m interested in tweaking the scheduling algorithm to > guarantee that the server can get on the CPU after the client runs.This will not be easy to solve on Xen. The current scheduler(s) available for Xen are not aware of any relationships between guests, so there''s not going to be any "gang scheduling" in Xen. It''s probably possible to rewrite the scheduler to understand groups of guests (and if you do this, you should do it so that it''s able to understand groups of not just 2 guests, but any number you like to group together - or you''ll have NO chance to get the scheduler accepted by the Xen folks, I would think, and then you''re forced to maintain your own scheduler forever forwards). I think that would be a very interesting challenge. -- Mats> > Thank you > Roger R. Cruz > > > > > I''m resending this to the list because I don''t believe it made it > > through the first time. > > If you''re not subscribed there''s a delay before somebody > comes along and > > manually allows your mail. > > > I have a multi-processor, multi-core environment where I will be > running > > an application on one HVM domain A (Windows 2003) and another app in > > another HVM domain A'' (Windows as well). There will be multiple > > instances of this pair combination. For performance > reasons, I would > > like to find out if there is any way to control the > scheduling of the > > paired domains, such that > > > > > > > > 1) if domain A is scheduled on a physical CPU 1, domain A'' is > also > > scheduled at the same time on cpu 2. or > > There''s not a simple way of arranging this. > > > 2) if domain A is scheduled on physical CPU, domain A'' is the > next > > domain to be scheduled on the CPU, even if domain B was the > next legal > > owner of the time slice. > > If you can dedicate a CPU per domain pair then you can just pin each > domain > pair to a different CPU. This should approximate the behaviour you > want. > Even if you pin several pairs to a CPU it should bound the latency > somewhat... however... > > What''s you''re application? If requests between your domains can be > batched up > somehow then I''d expect performance to be OK without special > configuration. > > Cheers, > Mark > > -- > Dave: Just a question. What use is a unicyle with no seat? And no > pedals! > Mark: To answer a question with a question: What use is a skateboard? > Dave: Skateboards have wheels. > Mark: My wheel has a wheel! > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>> 1) I''ve heard of something called stub domains that may do something >> similar to case 1. I''m trying to get more info on it. Don''t >> know if it >> even works for Windows HVMs.> Stub domains has nothing to do with the problem you''re trying tosolve. My limited understanding of stubs domains is that it allows a mini-domain to be scheduled with the main domain as a pair. The mini-domain would have easy access to the domain''s memory space. So they are basically two CPU contexts using the same page tables, etc. That is all I know so far since the info on this is rather limited. On the issue of the scheduler, I would agree with you on making "bonding" of domains generic so it can be accepted. I just wanted to get these questions out there to make sure I was not missing something obvious or support coming in a near-future release. Javier, You asked about whether we have gotten domains to shared memory and the answer is not yet. We have not even tried. Our understanding is that there are grant tables that are supposed to do this and the code is available for HVM domains as well. If someone has more accurate info, please correct my understanding. Thank you all. R. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Roger Cruz > Sent: 28 March 2007 17:05 > To: xen-users@lists.xensource.com > Subject: RE: [Xen-users] RE: Co-scheduling HVM domains... > > >> 1) I''ve heard of something called stub domains that may do > something > >> similar to case 1. I''m trying to get more info on it. Don''t > >> know if it > >> even works for Windows HVMs. > > > Stub domains has nothing to do with the problem you''re trying to > solve. > > My limited understanding of stubs domains is that it allows a > mini-domain to be scheduled with the main domain as a pair. The > mini-domain would have easy access to the domain''s memory space. So > they are basically two CPU contexts using the same page tables, etc. > That is all I know so far since the info on this is rather limited.Yes, but the stub domain isn''t a "normal domain". It''s more like a "monitor domain" or "hypervisor extension" that is loaded into the same/similar memory space and runs on the SAME VCPU as the actual domain. It''s still not going to solve your problem. If stub domains where implemented today, your two windows domains would each have a stub-domain - which would be used whenever the guest-domain needs to perform emulation of hardware. The purpose of stub-domains is twofold: 1. To remove the "bottleneck" of Dom0 to perform the emulation of hardware, instead (a large portion) of this work could be done by the stub-domain. 2. To make accounting for emulation of hardware be more easily connected to the domain that is responsible for that work. I don''t think it''s necessary for the scheduler to change to make this work (aside from perhaps to keep the stub-domain and "real" domain account together).> > > On the issue of the scheduler, I would agree with you on making > "bonding" of domains generic so it can be accepted. > > I just wanted to get these questions out there to make sure I was not > missing something obvious or support coming in a near-future release.I don''t think you''re missing anything important.> > Javier, > > You asked about whether we have gotten domains to shared > memory and the > answer is not yet. We have not even tried. Our understanding is that > there are grant tables that are supposed to do this and the code is > available for HVM domains as well. If someone has more accurate info, > please correct my understanding.Yup, there''s a way to call hypercalls from HVM domains (check out the "unmodified-driver" directory in the xen distribution - that contains the Linux Para-virtual drivers for HVM). -- Mats> > Thank you all. > R. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users