Anthony Wright
2011-Jun-16 12:58 UTC
[Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
I have a fairly old machine with 512M of RAM that I use for testing. Under Xen 3.4 with the 2.6.18 linux kernel it would happily work with dom0_mem=50M, leaving 450M for VMs. I am in the process of upgrading to Xen 4.1 with a stock 3.0.0-rc3 linux kernel (with konrad''s vga-support patch). This combination however requires dom0_mem=250M to work leaving only 250M for the VMs (if I try to set dom0_mem any lower I get a kernel panic early on in the kernel boot phase). This is an increase of 5x between the two versions, and I wondered if there''s an explanation why. Is it possible to reduce the memory requirements, or do I have to learn to live with it. thanks, Anthony Wright _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jun-16 14:56 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On Thu, Jun 16, 2011 at 01:58:25PM +0100, Anthony Wright wrote:> I have a fairly old machine with 512M of RAM that I use for testing. Under Xen 3.4 with the 2.6.18 linux kernel it would happily work with dom0_mem=50M, leaving 450M for VMs. I am in the process of upgrading to Xen 4.1 with a stock 3.0.0-rc3 linux kernel (with konrad''s vga-support patch). This combination however requires dom0_mem=250M to work leaving only 250M for the VMs (if I try to set dom0_mem any lower I get a kernel panic early on in the kernel boot phase).> > This is an increase of 5x between the two versions, and I wondered if there''s an explanation why. Is it possible to reduce the memory requirements, or do I have to learn to live with it.The SWIOTLB is taking a 64MB chunk out of it. You could lower that by ''swiotlb=2048'' or so. Also make sure to compile the CONFIG_XEN_MAX_DOMAIN_MEMORY from 128GB down to say 1. Besides that .. run with ''memblock=debug loglevel=10 initcall_debug debug'' and see what other regions memblock reserves that could be shrunk.> > thanks, > > Anthony Wright > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Wright
2011-Jun-17 15:31 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
Lowering swiotlb helped, and got me down to 200M for dom0. What is the effect of reducing this value? I set CONFIG_XEN_MAX_DOMAIN_MEMORY down to 8, but that didn''t seem to have any effect on dom0''s memory requirement. What is this value? Does it only apply to a domU''s memory usage? I tried the memblock=debug options, and while I got lots of output, I could see very little on the subject of memory usage. I''m at 4x memory usage, which is better, but I''d really like to understand where all the memory''s going. thanks, Anthony. ----- Original Message ----- From: "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> To: "Anthony Wright" <anthony@overnetdata.com> Cc: xen-devel@lists.xensource.com Sent: Thursday, 16 June, 2011 3:56:11 PM Subject: Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0 On Thu, Jun 16, 2011 at 01:58:25PM +0100, Anthony Wright wrote:> I have a fairly old machine with 512M of RAM that I use for testing. Under Xen 3.4 with the 2.6.18 linux kernel it would happily work with dom0_mem=50M, leaving 450M for VMs. I am in the process of upgrading to Xen 4.1 with a stock 3.0.0-rc3 linux kernel (with konrad''s vga-support patch). This combination however requires dom0_mem=250M to work leaving only 250M for the VMs (if I try to set dom0_mem any lower I get a kernel panic early on in the kernel boot phase).> > This is an increase of 5x between the two versions, and I wondered if there''s an explanation why. Is it possible to reduce the memory requirements, or do I have to learn to live with it.The SWIOTLB is taking a 64MB chunk out of it. You could lower that by ''swiotlb=2048'' or so. Also make sure to compile the CONFIG_XEN_MAX_DOMAIN_MEMORY from 128GB down to say 1. Besides that .. run with ''memblock=debug loglevel=10 initcall_debug debug'' and see what other regions memblock reserves that could be shrunk.> > thanks, > > Anthony Wright > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jun-20 12:45 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On Fri, Jun 17, 2011 at 04:31:11PM +0100, Anthony Wright wrote:> Lowering swiotlb helped, and got me down to 200M for dom0. What is the effect of reducing this value?Less amount of bounce buffer. But you don''t need the bounce buffer for PCI devices b/c you don''t have more than 4GB of physical memory in the machine.> > I set CONFIG_XEN_MAX_DOMAIN_MEMORY down to 8, but that didn''t seem to have any effect on dom0''s memory requirement. What is this value? Does it only apply to a domU''s memory usage?It makes some internal datastructures (P2M) smaller. They are set up for 128GB or so machines initially.> > I tried the memblock=debug options, and while I got lots of output, I could see very little on the subject of memory usage.The numbers are what amount of memory is reserved. You can find out which are are is eating the most by computing the difference.> > I''m at 4x memory usage, which is better, but I''d really like to understand where all the memory''s going.Me too.> > thanks, > > Anthony. > > > ----- Original Message ----- > From: "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> > To: "Anthony Wright" <anthony@overnetdata.com> > Cc: xen-devel@lists.xensource.com > Sent: Thursday, 16 June, 2011 3:56:11 PM > Subject: Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0 > > On Thu, Jun 16, 2011 at 01:58:25PM +0100, Anthony Wright wrote: > > I have a fairly old machine with 512M of RAM that I use for testing. Under Xen 3.4 with the 2.6.18 linux kernel it would happily work with dom0_mem=50M, leaving 450M for VMs. I am in the process of upgrading to Xen 4.1 with a stock 3.0.0-rc3 linux kernel (with konrad''s vga-support patch). This combination however requires dom0_mem=250M to work leaving only 250M for the VMs (if I try to set dom0_mem any lower I get a kernel panic early on in the kernel boot phase). > > > > > This is an increase of 5x between the two versions, and I wondered if there''s an explanation why. Is it possible to reduce the memory requirements, or do I have to learn to live with it. > > The SWIOTLB is taking a 64MB chunk out of it. You could lower that by ''swiotlb=2048'' or so. > Also make sure to compile the CONFIG_XEN_MAX_DOMAIN_MEMORY from 128GB down to say 1. > > Besides that .. run with ''memblock=debug loglevel=10 initcall_debug debug'' and see what other > regions memblock reserves that could be shrunk. > > > > thanks, > > > > Anthony Wright > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Wright
2011-Jun-22 13:15 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On 20/06/2011 13:45, Konrad Rzeszutek Wilk wrote:> On Fri, Jun 17, 2011 at 04:31:11PM +0100, Anthony Wright wrote: >> Lowering swiotlb helped, and got me down to 200M for dom0. What is the effect of reducing this value? > Less amount of bounce buffer. But you don''t need the bounce buffer for PCI devices b/c you don''t > have more than 4GB of physical memory in the machine.Do I only need bounce buffers if I have > 4GB of physical memory? In this case should I allocate the 64M, or is it a sliding memory requirement?>> I set CONFIG_XEN_MAX_DOMAIN_MEMORY down to 8, but that didn''t seem to have any effect on dom0''s memory requirement. What is this value? Does it only apply to a domU''s memory usage? > It makes some internal datastructures (P2M) smaller. They are set up for 128GB or so machines initially.It sounds like this value applies to DomUs, does this config variables set the maximum amount of memory 128GB per DomU or across all DomUs, i.e. if I have 16 DomUs and a CONFIG_XEN_MAX_DOMAIN_MEMORY of 16, do they each have a maximum of 16GB, or do the get 1GB each?>> I tried the memblock=debug options, and while I got lots of output, I could see very little on the subject of memory usage. > The numbers are what amount of memory is reserved. You can find out which are are is eating the most > by computing the difference.Maybe I''m misreading the output, but I couldn''t see any numbers that look like memory being assigned. I''ve attached the dmesg output. Do I need to enable a CONFIG variable to get the output I need or am I missing something. thanks, Anthony. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jun-22 13:32 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On Wed, Jun 22, 2011 at 02:15:35PM +0100, Anthony Wright wrote:> On 20/06/2011 13:45, Konrad Rzeszutek Wilk wrote: > > On Fri, Jun 17, 2011 at 04:31:11PM +0100, Anthony Wright wrote: > >> Lowering swiotlb helped, and got me down to 200M for dom0. What is the effect of reducing this value? > > Less amount of bounce buffer. But you don''t need the bounce buffer for PCI devices b/c you don''t > > have more than 4GB of physical memory in the machine. > Do I only need bounce buffers if I have > 4GB of physical memory? InCorrect.> this case should I allocate the 64M, or is it a sliding memory requirement?It really is unknown. At some point the maintainer was thinking of adding dynamic code so that it would increase as neccessary - but it never got further than: "this would be neat". I would say try some small numbers and when the machine is on full load it panics (or gives you a nasty kernel message).> >> I set CONFIG_XEN_MAX_DOMAIN_MEMORY down to 8, but that didn''t seem to have any effect on dom0''s memory requirement. What is this value? Does it only apply to a domU''s memory usage? > > It makes some internal datastructures (P2M) smaller. They are set up for 128GB or so machines initially. > It sounds like this value applies to DomUs, does this config variables > set the maximum amount of memory 128GB per DomU or across all DomUs, > i.e. if I have 16 DomUs and a CONFIG_XEN_MAX_DOMAIN_MEMORY of 16, do > they each have a maximum of 16GB, or do the get 1GB each?I mistyped this. The internal datastructures just set the limits to what they can go to. But they don''t consume any real RAM - just virtual addresses and during boot time that gets resolved (shrunk appropiately). So don''t worry about it.> >> I tried the memblock=debug options, and while I got lots of output, I could see very little on the subject of memory usage. > > The numbers are what amount of memory is reserved. You can find out which are are is eating the most > > by computing the difference.> Maybe I''m misreading the output, but I couldn''t see any numbers that > look like memory being assigned. I''ve attached the dmesg output. Do I > need to enable a CONFIG variable to get the output I need or am I > missing something.The memblock=debug should give you some idea of what is Reserved. The Reserved includes memory that is allocated by boot-time services (P2M, pagetables, NUMA) and by real reservations (for example ACPI space). Using the ''memblock=debug'' can give you an idea of what services are reserving the most. Then we can narrow down who or what is eating the gobs of memory. see the ''Memory: ".. numbers. Also you might want to eliminate the balloon usage space algother by doing two things: Xen command line: dom0_mem=max:512MB Linux command line: mem=512MB That will effectivly remove any balloon space (so your Dom0 will _never_ grow up). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Wright
2011-Jun-22 14:22 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On 22/06/2011 14:32, Konrad Rzeszutek Wilk wrote:> On Wed, Jun 22, 2011 at 02:15:35PM +0100, Anthony Wright wrote: >> Maybe I''m misreading the output, but I couldn''t see any numbers that >> look like memory being assigned. I''ve attached the dmesg output. Do I >> need to enable a CONFIG variable to get the output I need or am I >> missing something. > The memblock=debug should give you some idea of what is Reserved. The > Reserved includes memory that is allocated by boot-time services (P2M, > pagetables, NUMA) and by real reservations (for example ACPI space). > Using the ''memblock=debug'' can give you an idea of what services are > reserving the most. Then we can narrow down who or what is eating the gobs > of memory. > > see the ''Memory: ".. numbers. Also you might want to eliminate > the balloon usage space algother by doing two things: > > Xen command line: dom0_mem=max:512MB > > Linux command line: mem=512MB > > That will effectivly remove any balloon space (so your Dom0 will _never_ > grow up).The problem is I can''t see any lines in the kernel dmesg output (attached to previous email) that start "Memory: ", or anything else that looks hopeful. Is there anything else I should add to the command line, is there a kernel CONFIG option I should turn on or am I missing something else? thanks, Anthony. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jun-22 14:39 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On Wed, Jun 22, 2011 at 03:22:30PM +0100, Anthony Wright wrote:> On 22/06/2011 14:32, Konrad Rzeszutek Wilk wrote: > > On Wed, Jun 22, 2011 at 02:15:35PM +0100, Anthony Wright wrote: > >> Maybe I''m misreading the output, but I couldn''t see any numbers that > >> look like memory being assigned. I''ve attached the dmesg output. Do I > >> need to enable a CONFIG variable to get the output I need or am I > >> missing something. > > The memblock=debug should give you some idea of what is Reserved. The > > Reserved includes memory that is allocated by boot-time services (P2M, > > pagetables, NUMA) and by real reservations (for example ACPI space). > > Using the ''memblock=debug'' can give you an idea of what services are > > reserving the most. Then we can narrow down who or what is eating the gobs > > of memory. > > > > see the ''Memory: ".. numbers. Also you might want to eliminate > > the balloon usage space algother by doing two things: > > > > Xen command line: dom0_mem=max:512MB > > > > Linux command line: mem=512MB > > > > That will effectivly remove any balloon space (so your Dom0 will _never_ > > grow up). > The problem is I can''t see any lines in the kernel dmesg output > (attached to previous email) that start "Memory: ", or anything else > that looks hopeful. Is there anything else I should add to the commanddmesg | grep Memory: ? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Wright
2011-Jun-22 15:53 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On 22/06/2011 15:39, Konrad Rzeszutek Wilk wrote:> On Wed, Jun 22, 2011 at 03:22:30PM +0100, Anthony Wright wrote: >> On 22/06/2011 14:32, Konrad Rzeszutek Wilk wrote: >>> On Wed, Jun 22, 2011 at 02:15:35PM +0100, Anthony Wright wrote: >>>> Maybe I''m misreading the output, but I couldn''t see any numbers that >>>> look like memory being assigned. I''ve attached the dmesg output. Do I >>>> need to enable a CONFIG variable to get the output I need or am I >>>> missing something. >>> The memblock=debug should give you some idea of what is Reserved. The >>> Reserved includes memory that is allocated by boot-time services (P2M, >>> pagetables, NUMA) and by real reservations (for example ACPI space). >>> Using the ''memblock=debug'' can give you an idea of what services are >>> reserving the most. Then we can narrow down who or what is eating the gobs >>> of memory. >>> >>> see the ''Memory: ".. numbers. Also you might want to eliminate >>> the balloon usage space algother by doing two things: >>> >>> Xen command line: dom0_mem=max:512MB >>> >>> Linux command line: mem=512MB >>> >>> That will effectivly remove any balloon space (so your Dom0 will _never_ >>> grow up). >> The problem is I can''t see any lines in the kernel dmesg output >> (attached to previous email) that start "Memory: ", or anything else >> that looks hopeful. Is there anything else I should add to the command > dmesg | grep Memory: ?Ok, I''m being really thick here, but I can''t get the Memory: lines to be output by the kernel. It looks like I can''t get any KERN_INFO or KERN_DEBUG lines to be output no matter what I do to log level. I''m not much of a kernel hacker, but is there some debug code that I need to enable to get these messages? I''ve wandered through google, and the kernel code but I can''t find anything obvious. thanks, Anthony. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Jun-23 13:02 UTC
Re: [Xen-devel] 5x dom0 memory increase from Xen/Linux 3.4/2.6.18 to 4.1/3.0.0
On Wed, Jun 22, 2011 at 04:53:40PM +0100, Anthony Wright wrote:> On 22/06/2011 15:39, Konrad Rzeszutek Wilk wrote: > > On Wed, Jun 22, 2011 at 03:22:30PM +0100, Anthony Wright wrote: > >> On 22/06/2011 14:32, Konrad Rzeszutek Wilk wrote: > >>> On Wed, Jun 22, 2011 at 02:15:35PM +0100, Anthony Wright wrote: > >>>> Maybe I''m misreading the output, but I couldn''t see any numbers that > >>>> look like memory being assigned. I''ve attached the dmesg output. Do I > >>>> need to enable a CONFIG variable to get the output I need or am I > >>>> missing something. > >>> The memblock=debug should give you some idea of what is Reserved. The > >>> Reserved includes memory that is allocated by boot-time services (P2M, > >>> pagetables, NUMA) and by real reservations (for example ACPI space). > >>> Using the ''memblock=debug'' can give you an idea of what services are > >>> reserving the most. Then we can narrow down who or what is eating the gobs > >>> of memory. > >>> > >>> see the ''Memory: ".. numbers. Also you might want to eliminate > >>> the balloon usage space algother by doing two things: > >>> > >>> Xen command line: dom0_mem=max:512MB > >>> > >>> Linux command line: mem=512MB > >>> > >>> That will effectivly remove any balloon space (so your Dom0 will _never_ > >>> grow up). > >> The problem is I can''t see any lines in the kernel dmesg output > >> (attached to previous email) that start "Memory: ", or anything else > >> that looks hopeful. Is there anything else I should add to the command > > dmesg | grep Memory: ? > Ok, I''m being really thick here, but I can''t get the Memory: lines to be > output by the kernel. It looks like I can''t get any KERN_INFO or > KERN_DEBUG lines to be output no matter what I do to log level. I''m not > much of a kernel hacker, but is there some debug code that I need to > enable to get these messages? I''ve wandered through google, and the > kernel code but I can''t find anything obvious.''debug loglevel=10'' ? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel