On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote: >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote:>> > Here''s the domU boot log:>> > memory = 4096 > > And also there is a bug where we can''t do PCI in PV guests when guest > has more than 3GB. >Is there any expectation that this 3GB restriction will be resolved in the short term ? Does anyone have any pointers to the specific problem ? Thanks -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Jun-11 16:42 UTC
Re: [Xen-devel] 3GB memory restriction for pv domU with PCI
On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote:> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote: > >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote: > > >> > Here''s the domU boot log: > > >> > memory = 4096 > > > > And also there is a bug where we can''t do PCI in PV guests when guest > > has more than 3GB. > > > > Is there any expectation that this 3GB restriction will be resolved in > the short term ?What is "short-term" ? This week = no. This month = likely.> > Does anyone have any pointers to the specific problem ?http://old.nabble.com/-PATCH-More-PCIFront--td26273977.html http://lists.xensource.com/archives/html/xen-devel/2009-12/msg01112.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2010-Oct-14 00:10 UTC
Re: [Xen-devel] 3GB memory restriction for pv domU with PCI
On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote: >> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote: >> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote: >> >> >> > Here''s the domU boot log: >> >> >> > memory = 4096 >> > >> > And also there is a bug where we can''t do PCI in PV guests when guest >> > has more than 3GB. >> > >> >> Is there any expectation that this 3GB restriction will be resolved in >> the short term ? > > What is "short-term" ? This week = no. This month = likely.It looks like this is still a restriction. Is there any plan to allow pvops domU''s to use more than 4GB ? How much and who do we have to pay to get it in sooner? If this ends up being our only show stopper for pvops, I''ll start selling lab equipment out the back door to pay for it. :-) -Bruce> >> >> Does anyone have any pointers to the specific problem ? > > http://old.nabble.com/-PATCH-More-PCIFront--td26273977.html > http://lists.xensource.com/archives/html/xen-devel/2009-12/msg01112.html >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-14 14:23 UTC
Re: [Xen-devel] 3GB memory restriction for pv domU with PCI
On Wed, Oct 13, 2010 at 05:10:32PM -0700, Bruce Edge wrote:> On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > > On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote: > >> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk > >> <konrad.wilk@oracle.com> wrote: > >> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote: > >> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote: > >> > >> >> > Here''s the domU boot log: > >> > >> >> > memory = 4096 > >> > > >> > And also there is a bug where we can''t do PCI in PV guests when guest > >> > has more than 3GB. > >> > > >> > >> Is there any expectation that this 3GB restriction will be resolved in > >> the short term ? > > > > What is "short-term" ? This week = no. This month = likely. > > It looks like this is still a restriction. Is there any plan to allow > pvops domU''s to use more than 4GB ?Jeremy wrote up some new shinny code where the memory can be fragmented (so that we can now stick in a big PCI hole). His patches are in xen/balloon. But I haven''t taken those and fiddled with them and changed the toolstack (xm or xl) to actually take advantage of this.> > How much and who do we have to pay to get it in sooner? > If this ends up being our only show stopper for pvops, I''ll start > selling lab equipment out the back door to pay for it. :-)Sheeshh. you are missing the mark. Instead of selling the lab equipment you _send_ the equipment - that is like a X-mas present for geeks: shiny new toy! :-) But jokes aside, I haven''t touched this since this last email. And I am not going to get to it this week nor the next. Then there is one week where (25th->29th) where I can play with this. After that, Linux Kernel Summit and the Linux Plumbers Conference is in - so it will be quiet. If this is a show-stopper right now, you could come with a hack where the resource check is just ignored. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Bruce Edge
2010-Oct-14 14:39 UTC
Re: [Xen-devel] 3GB memory restriction for pv domU with PCI
On Thu, Oct 14, 2010 at 7:23 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Wed, Oct 13, 2010 at 05:10:32PM -0700, Bruce Edge wrote: >> On Fri, Jun 11, 2010 at 9:42 AM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >> > On Fri, Jun 11, 2010 at 08:54:05AM -0700, Bruce Edge wrote: >> >> On Fri, Jun 11, 2010 at 6:38 AM, Konrad Rzeszutek Wilk >> >> <konrad.wilk@oracle.com> wrote: >> >> > On Fri, Jun 11, 2010 at 09:09:36AM +0300, Pasi Kärkkäinen wrote: >> >> >> On Thu, Jun 10, 2010 at 04:25:54PM -0700, Bruce Edge wrote: >> >> >> >> >> > Here''s the domU boot log: >> >> >> >> >> > memory = 4096 >> >> > >> >> > And also there is a bug where we can''t do PCI in PV guests when guest >> >> > has more than 3GB. >> >> > >> >> >> >> Is there any expectation that this 3GB restriction will be resolved in >> >> the short term ? >> > >> > What is "short-term" ? This week = no. This month = likely. >> >> It looks like this is still a restriction. Is there any plan to allow >> pvops domU''s to use more than 4GB ? > > Jeremy wrote up some new shinny code where the memory can be fragmented > (so that we can now stick in a big PCI hole). His patches are in > xen/balloon. But I haven''t taken those and fiddled with them and changed > the toolstack (xm or xl) to actually take advantage of this. > >> >> How much and who do we have to pay to get it in sooner? >> If this ends up being our only show stopper for pvops, I''ll start >> selling lab equipment out the back door to pay for it. :-) > > Sheeshh. you are missing the mark. Instead of selling the lab equipment > you _send_ the equipment - that is like a X-mas present for geeks: shiny > new toy! :-) > > But jokes aside, I haven''t touched this since this last email. And I am > not going to get to it this week nor the next. Then there is one week > where (25th->29th) where I can play with this. After that, Linux Kernel > Summit and the Linux Plumbers Conference is in - so it will be quiet.It''s not the last nail in the coffin right now, but it will be if it''s not in there by the end of the year at the latest. We''ll have to fallback to hvm which no one wants to do because we''ll lose all the new goodies that come along with pvops. We are in a pv-ops vs. hvm showdown mode right now. The remaining sticking points are this and the 2MB contig alloc for the tachyon 2 MB block for the FC SEST. I feel like if we don''t get pvops in now, it''ll be a hard sell later on. Anyway, thanks for the info and I''ll keep my fingers crossed that 5 days is all it takes.> > If this is a show-stopper right now, you could come with a hack where > the resource check is just ignored. >I assume one could do this only if not using PCI passthrough? -Bruce _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-14 17:58 UTC
Re: [Xen-devel] 3GB memory restriction for pv domU with PCI
> sticking points are this and the 2MB contig alloc for the tachyon 2 MB > block for the FC SEST. I feel like if we don''t get pvops in now, it''ll > be a hard sell later on.The 2MB contig alloc ought to work with the pci_alloc_coherent function.> > Anyway, thanks for the info and I''ll keep my fingers crossed that 5 > days is all it takes. > > > > > If this is a show-stopper right now, you could come with a hack where > > the resource check is just ignored. > > > I assume one could do this only if not using PCI passthrough?Well, it is basically relaxing the criteria. In the olden days (2.6.18) the System RAM region wa not accounted for. Nowadays it is. So when a PCI device requests a region that the kernel thinks is System RAM it tells it to buzz of. In 2.6.18, the kernel would have no trouble giving you that region. So the hack is just to relax the criteria and maybe just do: if (strncmp(r->name,"System RAM") == 0) { return OK; } Or something like that in the resource.c file. I do''t remember the details of that file, but that looked like one of those quick-and-fast ways of doing this. The right way is to actually make the guest aware of the PCI hole the Dom0 has and just shift the memory as required. It sounds easy :-) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel