Hi All, First let me say that I am a big fan of Solaris x86 (even after the 2.5 fiasco) and having been ''solaris'' (inc SunOS) for 16 years I am glad to see such effort being placed into the product. Because of this I am taking time to post some comments about my first play with Xen (err... hVM) within OpenSolaris. My goal was to move from two separate hosts (opensolaris b54 & CentOS running VMWare) to a single opensolaris hosts, I was staring to feel guilty about having a fileserver and vmware host running 24x7; and it was making the room hot! Aside from the finding a Pentium D 9xx (Can''t ship a 2 year old CPU from the US to Australia), waiting for SXCE b81 to be released and the metadb must be in s7 upgrade issue I got a functional B81 environment, at which point I discovered the following issues (and help/advise welcome): #1. The ugen driver seems a little broken, native boot I can see and manage my UPS, under hVM (Dom0) the ugen driver can''t even set device permissions correctly or allow read/write access to cntlr0 device (it does make the device tree correctly however). #2. A 4GB machine loses 700MB to the hypervisor!?!!? huh? #3. virt-install of a linux PV is generally broken, might be the install media not having a PV kernel, but bitching about ''rpm'' packages is just wrong, and I can''t believe that this isn''t documented any where. #4. virt-install and HVM just hangs for openSUSE 10.2.... once again, nice. #5 Doco? this python config file? all you can find are examples, no documentation about it seems to exist, for something which has been in the works since b27 (or something) this is poor. In short you can probably tell I am a little disappointed and I am about to ditch hVM and aggressively using zones and vbox... At least I''ll be able to protect the storage when the power goes out (i.e shutdown). I can see a lot of work has been done on integrating Xen & OpenSolaris. It is unfortunate that the core components of the opensolaris builds have been rock stable (first build I used was B33) which has set the bar quite high for all the other projects. I understand I am opening the door for negative comments (i.e flames), but if I have made simple stupid mistakes I would like to know, If I have not then having someone highlight some issues can only make the product better. This message posted from opensolaris.org
On Fri, Feb 15, 2008 at 05:06:00AM -0800, Paul Miach wrote:> #1. The ugen driver seems a little broken, native boot I can see and > manage my UPS, under hVM (Dom0) the ugen driver can''t even set device > permissions correctly or allow read/write access to cntlr0 device (it > does make the device tree correctly however).Filing a bug (with what you specifically tried) seems like a good idea.> #2. A 4GB machine loses 700MB to the hypervisor!?!!? huh?On what do you base this? It doesn''t seem right.> #3. virt-install of a linux PV is generally broken, might be the > install media not having a PV kernel, but bitching about ''rpm'' > packages is just wrong, and I can''t believe that this isn''t documented > any where.Too vague to be useful I''m afraid. I''ve done many Linux virt-install''s on Solaris dom0 without problems (and some with - it very much depends on exactly what you''re doing, and whether you''re doing it right). Are you asking for an example Linux install in virt-install(1m) ? That''d be reasonable; please file an RFE. Or are you complaining about the poor error reporting of virt-install? If so, you''re quite right - we''re working to improve this.> #4. virt-install and HVM just hangs for openSUSE 10.2.... once again, nice.I suspect this might be a known bug, fixed in a more recent build, but I''m not sure.> #5 Doco? this python config file? all you can find are examples, no > documentation about it seems to exist, for something which has been in > the works since b27 (or something) this is poor.You''re not "supposed" to use the .py file approach which is why it''s not documented. virt-install is the documented way to create domUs. If you have specific issues with the relevant man pages, please file bugs.> I understand I am opening the door for negative comments (i.e flames)There''s no problem with negative comments, but we need a bit more detail to be able to help. regards, john
> #2. A 4GB machine loses 700MB to the hypervisor!?!!? > huh?How do you come up with that number?> #4. virt-install and HVM just hangs for openSUSE > 10.2.... once again, nice.This is a known bug in the realmode emulation in the Xen hypervisor. OpenSUSE 10 hangs on every dom0 - including the openSUSE dom0. It was fixed in the upstream Xen source code within the past week or two.> #5 Doco? this python config file? all you can find > are examples, no documentation about it seems to > exist, for something which has been in the works > since b27 (or something) this is poor.It''s not documented because you''re not supposed to be using it. Nils This message posted from opensolaris.org
John Levon wrote:> On Fri, Feb 15, 2008 at 05:06:00AM -0800, Paul Miach wrote:>> #2. A 4GB machine loses 700MB to the hypervisor!?!!? huh? > > On what do you base this? It doesn''t seem right.If you create a domU, memory will be taken away from dom0. If you looked at ''virsh dominfo 0'' or ''xm list'' after starting a domU with 500MB, that might explain the fact that 700MB is missing from dom0. On my 4GB Ultra 20, Xen takes less than 200MB. -Ryan
Hi Ryan, My ~700MB came from ''prtconf -v'' and looking at the difference between physical and reported; amup# prtconf | head -2 System Configuration: Sun Microsystems i86pc Memory size: 3396 Megabytes amup# Doing a little more digging with xm the numbers look a lot worse and don''t seem to make much sense; sense in that prtconf is reporting more memory than xm dmesg is reporting as being allocated... An extract from ''xm dmesg'' xVM version 3.0.4-1-xvm Latest ChangeSet: Tue Dec 04 09:56:10 2007 +0000 13231:f6074ad033f3 (XEN) Command line: /boot/amd64/xen.gz (XEN) Physical RAM map: (XEN) 0000000000000000 - 00000000000a0000 (usable) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 00000000dfe8cc00 (usable) (XEN) 00000000dfe8cc00 - 00000000dfe8ec00 (ACPI NVS) (XEN) 00000000dfe8ec00 - 00000000dfe90c00 (ACPI data) (XEN) 00000000dfe90c00 - 00000000e4000000 (reserved) (XEN) 00000000fec00000 - 00000000fed00400 (reserved) (XEN) 00000000fed20000 - 00000000feda0000 (reserved) (XEN) 00000000fee00000 - 00000000fef00000 (reserved) (XEN) 00000000ffb00000 - 0000000100000000 (reserved) (XEN) System RAM: 3582MB (3668144kB) Aside from the non-contiguous memory ranges (where did they go?) I have 0xDFE2CC00 usable memory, or 3582MB. 4096 - 3582 = 514MB lost before anything is allocated. A little later: (XEN) Dom0 alloc.: 000000000c000000->0000000010000000 (852864 pages to be allocated) So that means Dom0 only has 3331MB allocated to it (another 150MB gone). As I only have a Dom0 (couldn''t get my linux of choice installed as a domU) all memory should be allocated to it. virsh dominfo 0 shows: Max memory: no limit Used memory: 3065856 kB This is 2994MB out of 4GB allocated to a single Dom0; it is interesting that prtconf and virsh don''t agree on how much memory dom0 is using. I agree that I shouldn''t be losing 700MB. Tomorrow given some time I''ll re-test the ugen issue and log as a bug if it still holds. Is this memory issue something which should be reported as a bug or am I missing a subtlety with opensolaris & Xen''s reporting & usage of memory? Thanks. This message posted from opensolaris.org
> My ~700MB came from ''prtconf -v'' and looking at the > difference between physical and reported; > > amup# prtconf | head -2 > System Configuration: Sun Microsystems i86pc > Memory size: 3396 Megabytes > amup#The "3396 Megabytes" is reported when booted as a xVM / Xen dom0? How much is reported when you boot without xVM / Xen?> Doing a little more digging with xm the numbers look > a lot worse and don''t seem to make much sense; sense > in that prtconf is reporting more memory than xm > dmesg is reporting as being allocated... > > An extract from ''xm dmesg'' > > xVM version 3.0.4-1-xvm > Latest ChangeSet: Tue Dec 04 09:56:10 2007 +0000 13231:f6074ad033f3 > > (XEN) Command line: /boot/amd64/xen.gz > (XEN) Physical RAM map: > (XEN) 0000000000000000 - 00000000000a0000 (usable) > (XEN) 00000000000f0000 - 0000000000100000 (reserved) > (XEN) 0000000000100000 - 00000000dfe8cc00 (usable) > (XEN) 00000000dfe8cc00 - 00000000dfe8ec00 (ACPI NVS) > (XEN) 00000000dfe8ec00 - 00000000dfe90c00 (ACPI data) > (XEN) 00000000dfe90c00 - 00000000e4000000 (reserved) > (XEN) 00000000fec00000 - 00000000fed00400 (reserved) > (XEN) 00000000fed20000 - 00000000feda0000 (reserved) > (XEN) 00000000fee00000 - 00000000fef00000 (reserved) > (XEN) 00000000ffb00000 - 0000000100000000 (reserved) > (XEN) System RAM: 3582MB (3668144kB) > > Aside from the non-contiguous memory ranges (where > did they go?)A part of the 4G address space is reserved for PCI devices.> I have 0xDFE2CC00 usable memory, or > 3582MB. 4096 - 3582 = 514MB lost before anything is > allocated.Hmm, is there a BIOS setup option to remap the memory that is lost due to PCI device address space at an address >= 4GB? The BIOS should have something like "Memory Hole Remapping" and it should be enabled so that the OS can use all of the available physical ram. Of cause this also depends on the chipset used on the mainboard, there are chipsets that can''t access memory at addresses >= 4GB. This message posted from opensolaris.org
> > I have 0xDFE2CC00 usable memory, or > > 3582MB. 4096 - 3582 = 514MB lost before anything > is > > allocated. > > Hmm, is there a BIOS setup option to remap the > memory that is lost due to PCI device address space > at an address >= 4GB? The BIOS should have > something > like "Memory Hole Remapping" and it should be > enabled > so that the OS can use all of the available physical > ram.Boards ASUS P5B Deluxe (P965+ICH8R , 4 GB MAX ) & ASUS P5K Premium/WIFI-AP (P35+ICH9R, 8 GB MAX) have option in BIOS kind of :- Advanced ->ChipsetSettings->NorthBridge->Memory Remapp : Enabled (4GB & 8GB corespondently) or Disabled (3 GB avalilable on P5B Deluxe, ability to load CentOS 5.0 (x86_64), F7 (x86_64) with usual kernels ) I never disabled it on P5K Premium/WIFI. The most recent F7(64 bit) install ran smoothly with this mentioned option enabled.> > Of cause this also depends on the chipset used on > the > mainboard, there are chipsets that can''t access > memory > at addresses >= 4GB.This message posted from opensolaris.org