Hello, this is a question I wanted to ask you since a long time because there is something not too much clear to me, as regard the memory management in XEN. I don''t want to go too much in deep with such argument, but It''s just to understand what is the best setup for my environment. I know about memory ballooning (http://wiki.xensource.com/xenwiki/XenBestPractices) and the dom0_mem parameter, but I would like to know what such parameter implies, in practise. Let''s go with my XEN setup now. I have 8GB of RAM and I have seen that if I put dom0_mem=1024M option, the dom0 has 1GB of RAM for itself. What does this really mean? This is confusing me, because I don''t understand how to decide the amount of RAM to assign to the dom0. For example, what if I assign 256M? What if 8192M? For sure you can say to try, but it is not so clear to me what I am doing. As far as I understand, this would be the situation: dom0_mem=1024M implies that ''xl list'' shows that the dom0 has got 1024MB of RAM reserved for itself. Are the remaining 7GB available for the domUs and the host operating system? i.e.: this could be the case study: 1024M for the dom0 1024M for domU1 2048M for domU2 2048M for domU3 Total: 6144MB "occupied" So, the 2 remaining GB would be available for the System or other 1/2 domUs with 1/2GB of RAM each. Of course memory ballooning helps to not crash the dom0, so the total amount of assigned memory would hopefully be never at saturation point, but what does such dom0 parameter imply in practise? This doubt has arisen to me even because of the memory management issue reported here: http://www.gossamer-threads.com/lists/xen/users/226845 Regards, -- Flavio
Flavio wrote:>Let''s go with my XEN setup now. >I have 8GB of RAM and I have seen that if I put dom0_mem=1024M >option, the dom0 >has 1GB of RAM for itself. >What does this really mean?The default is that when you boot, all the system RAM (less a bit used by Xen itself) will be ''available'' to Dom0. As you start guests, then Dom0 will balloon down it''s memory availability to free memory for the guest. This all works fine, and as long as you never force memory down too low then Dom0 won''t complain - you can set a minimum below which Dom0 will not balloon and it would be a good idea to tune this to your requirements. Now, for a couple of reasons this isn''t ideal. The main reason is that certain buffers and structures used to keep track of "stuff" are sized at boot time based on the amount of memory available. So in your case, you start with 8G of RAM but potentially balloon down to as little as 1/4G (256M) - just 1/32 of the size. So you now have a system running in a small amount of RAM but with buffers and structures sized for a lot of RAM (about 32 times what it actually has available in this case). Also, each time you start a guest (or alter a guest''s memory allocation), then the host''s memory configuration must be changed which takes a little time. Lastly, these changes in memory allocation will (or may) affect performance in some areas - giving a variable system performance depending on how many guests are running and how much RAM they use. So it''s a good idea to specify how much memory to give to Dom0. It will then size things to suit (which in itself will reduce it''s memory footprint). You can still balloon between high and low limits, or you can set a fixed allocation and have deterministic performance and minimum memory usage.>This is confusing me, because I don''t understand how >to decide the amount of RAM to assign to the dom0. For example, what >if I assign >256M? What if 8192M?8G isn''t a good choise - see above about sizing of buffers and structures. 256M may or may not be enough - only you can only determine that for your setup, it depends on what is running on the system. If you run just the bare minimum of software/services on Dom0 then it may well be enough, but if you run other things (particularly memory hungry programs) it might not be enough. You need to pick a value (not too low, try starting at 512M and work down) and then look at how the system is running - look at how much memory is used for different things (cat /proc/memory) and particularly if it is swapping (very bad for performance). Normally, you don''t tend to do much in Dom0, so not having large caches isn''t really an issue (it just delays things when stuff has to read from disk instead of cache when you issue a command) as long as all the stuff that''s running "all the time" or "frequently" can stay in RAM. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books.
On 14 April 2012 20:57, Simon Hobson <linux@thehobsons.co.uk> wrote: Hello Simon, thanks for replying.> The default is that when you boot, all the system RAM (less a bit used by > Xen itself) will be ''available'' to Dom0. As you start guests, then Dom0 will > balloon down it''s memory availability to free memory for the guest. This all > works fine, and as long as you never force memory down too low then Dom0 > won''t complain - you can set a minimum below which Dom0 will not balloon and > it would be a good idea to tune this to your requirements.Good. This is clear to me now.> > Now, for a couple of reasons this isn''t ideal. The main reason is that > certain buffers and structures used to keep track of "stuff" are sized at > boot time based on the amount of memory available. So in your case, you > start with 8G of RAM but potentially balloon down to as little as 1/4G > (256M) - just 1/32 of the size. So you now have a system running in a small > amount of RAM but with buffers and structures sized for a lot of RAM (about > 32 times what it actually has available in this case).This still confuses me a little bit. I don''t know if the memory issue I''ve reported in my previous message makes things hard to be understood by me, but what I can see when I use dom0_mem=1024M, is that the dom0 has 1024MB of RAM for itself and the ''free'' command shows that I only have 600MB of total RAM. If I''m not wrong, I should see about 7GB of free RAM, since 1GB is assigned to dom0. Now, if things would work fine, I should have 1GB for the dom0, that could potentially balloon down for the reason you said above, and about 7GB for every domU I am running. Now, since I only have 600MB RAM free, this would mean that if I start a domU, this would completely freeze my system, due to memory exhaustion. Isn''t it? Or the memory is simply hidden, and I can start whatever domU I want? To be honest I think that the last option shouldn''t be true, since I notice a slowdown.> Lastly, these changes in memory allocation will (or may) affect performance > in some areas - giving a variable system performance depending on how many > guests are running and how much RAM they use.OK> > So it''s a good idea to specify how much memory to give to Dom0. It will then > size things to suit (which in itself will reduce it''s memory footprint). You > can still balloon between high and low limits, or you can set a fixed > allocation and have deterministic performance and minimum memory usage. > > > 8G isn''t a good choise - see above about sizing of buffers and structures.OK. 1024M should be ok, in my case apart of the problem caused by the memory issue mentioned above.> > 256M may or may not be enough - only you can only determine that for your > setup, it depends on what is running on the system. If you run just the bare > minimum of software/services on Dom0 then it may well be enough, but if you > run other things (particularly memory hungry programs) it might not be > enough.Yes, actually if I run KDE, this sucks a lot. You may wonder why I run KDE on a dom0, but I usually use this installation for testing purposes and I need X to run graphical sessions of domUs.> You need to pick a value (not too low, try starting at 512M and work > down) and then look at how the system is running - look at how much memory > is used for different things (cat /proc/memory) and particularly if it is > swapping (very bad for performance). > Normally, you don''t tend to do much in Dom0, so not having large caches > isn''t really an issue (it just delays things when stuff has to read from > disk instead of cache when you issue a command) as long as all the stuff > that''s running "all the time" or "frequently" can stay in RAM.Great. Thank you so much for the explanation. I would ask you if you know some book where all this memory issue is explained. It would be great to see some scheme, to understand in deep the memory management. I don''t know if The definitive guide to the XEN hypervisor of Devid Chisnall would deal about this, and is good for the new XEN support into the vanilla kernel. I don''t know how many things have changed with respect to the "old" xen-sources. Regards! Flavio
Flavio wrote:> > Now, for a couple of reasons this isn''t ideal. The main reason is that >> certain buffers and structures used to keep track of "stuff" are sized at >> boot time based on the amount of memory available. So in your case, you >> start with 8G of RAM but potentially balloon down to as little as 1/4G >> (256M) - just 1/32 of the size. So you now have a system running in a small >> amount of RAM but with buffers and structures sized for a lot of RAM (about >> 32 times what it actually has available in this case). >This still confuses me a little bit. I don''t know if the memory issue >I''ve reported >in my previous message makes things hard to be understood by me, but what >I can see when I use dom0_mem=1024M, is that the dom0 has 1024MB of RAM >for itself and the ''free'' command shows that I only have 600MB of total RAM. >If I''m not wrong, I should see about 7GB of free RAM, since 1GB is assigned to >dom0.That sounds about right. Dom0 will only "see" the RAM that''s assigned to it, you need to use the Xen tools (eg xentop) to see how much memory is assigned to each domain and how much is free. What you are probably missing is that Dom0 is itself a guest virtual machine running under the Xen hypervisor - it differs from a DomU PV guest mainly in having privileged access to communicate with the hypervisor and hardware. But since it is itself a guest, it only has access to resources (in this discussion, RAM) given to it by the hypervisor. The dom0 memory allocation statements are instructions to the hypervisor, not Dom0.>I would ask you if you know some book where all this memory issue is >explained.Sorry, like a lot of FOSS, documentation tends to lag development - writing documentation (or at least documentation that''s any good) is a) really hard, and b) a different skill to writing code. There tend to be less volunteers for this task. I don''t personally know of any good books - but then books are a tricky thing in this sort of area as they have a limited shelf life before the software moves on. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books.
On 15 April 2012 23:56, Simon Hobson <linux@thehobsons.co.uk> wrote:> That sounds about right. Dom0 will only "see" the RAM that's assigned to it, > you need to use the Xen tools (eg xentop) to see how much memory is assigned > to each domain and how much is free. > What you are probably missing is that Dom0 is itself a guest virtual machine > running under the Xen hypervisor - it differs from a DomU PV guest mainly in > having privileged access to communicate with the hypervisor and hardware. > But since it is itself a guest, it only has access to resources (in this > discussion, RAM) given to it by the hypervisor. The dom0 memory allocation > statements are instructions to the hypervisor, not Dom0.OK Simon. Thanks a lot. This clarified me the doubt I had on this topic. To be honest I did know that the dom0 is a guest VM, but I was confusing between the hypervisor and the dom0 itself. It should be totally clear to me now. So, now the problem is that until the memory issue problem won't be solved, it will be a sort of trouble to manage XEN with its domUs. I think everyone would have some pain if its system shows 600MB for free, when it has at least 7GB.> I don't personally know of any good books - but then books are a tricky > thing in this sort of area as they have a limited shelf life before the > software moves on.I perfectly agree with you. This is true in most of the field within computer research and IT technologies, but it was worth to ask! Thank you very much, Flavio _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users