Greetings,
I''m having problems with nevada on Xvm. Looking back, I''ve
had the problem all along. But I didn''t really dig into it until now.
Specifically, I got very interested in liveupgrade once the zfs root became
available in snv_90. However, looking back, I believe I''ve had the
problem since the snv_79a/SXDE 01/08 release.
I had hacked together a number of workarounds, thinking I had bitten off a new
Motherboard/CPU combo that wasn''t quite stable. But, I''ve
convinced myself that is not the problem, and the Mobo has been fully patched.
I only dug into things as the problem I''ll describe prevents luupgrade
from completing.
Problem manifests itself under a variety of circumstances, and at first I
didn''t associate it with Xen. The problem is that a system under
moderate load will no longer fork(), and will not send any errors to syslog or
stderr.
I''ve a number of processes that attest to this:
''ntpq -p xenhost'' from another system shows reasonable
jitter
ssh to the box produces a "password:" prompt (PAM interactive
login), but no shell,
a console login accepts userid, but never prompts for password.
All indicitave of a system that is running but cannot fork().
System in question is an ASUS M3A78-EMH + AMD Phenom 9600 + 8 GBytes RAM.
After flailing about a bit, I backed off, installed snv_91, as there are some
release notes which I thought might apply, and reset the BIOS to factory
defaults, and began the one change per test debug cycle on the snv_91 install.
The test operation was luupgrade (options using a local .iso image, and a zfs
clone BE)
The only common factor associated with the failure was running Solaris as Dom0.
Solaris on bare metal does not exhibit the problem (though I can''t say
for certain for snv_79a, snv_86, snv_90, as I wasn''t quite so rigorous
in testing).
luupgrade completes (as do a number of other large compiles, etc) only when
Solaris runs on the bare metal, regardless of SVM, ECC, ACPI, or other settings.
I can''t find a report of the same problem, though this thread may be
close:
http://www.opensolaris.org/jive/search.jspa?q=fork%20hangs&objID=f53&dateRange=thisyear&searchID=1108736&forumID=53&rankBy=10001&threadID=58022
I don''t grok the xen code at all, so I''m not going to go
diving for the fix, but if this rings a bell, and someone wants me to try
something, I''ll be happy to. The system is my test system after all.
Cheers!
-sam
This message posted from opensolaris.org
> The problem is that a system under moderate load will no longer fork()...> System in question is an ASUS M3A78-EMH + AMD Phenom > 9600 + 8 GBytes RAM....> I can''t find a report of the same problem, though > this thread may be close: > http://www.opensolaris.org/jive/search.jspa?q=fork%20hangs&objID=f53&dateRange=thisyear&searchID=1108736&forumID=53&rankBy=10001&threadID=58022Is that the " xvm 32-bit: excessive number of pagefaults after fork() / copy-on-write fault" thread? (or bug 6702228, "full tlb flush required when changing L2 PDPTR entries under 32bit PAE dom0/domU", http://bugs.opensolaris.org/view_bug.do?bug_id=6702228 ) Are you forcing the dom0 to boot into 32-bit mode? This message posted from opensolaris.org
Yeah, I think that''s the one. What I''m seeing looks like mapping failures. Things already mapped continue to work, but new ones fail to be created. Thus my zeroing in on that report. But, you point out what I missed. I''m not using 32 bit at all. What I''m seeing is similar, but different. Cheers! -sam This message posted from opensolaris.org
On Mon, Jul 28, 2008 at 03:57:45PM -0700, Sam Nicholson wrote:> Problem manifests itself under a variety of circumstances, and at > first I didn''t associate it with Xen. The problem is that a system > under moderate load will no longer fork(), and will not send any > errors to syslog or stderr.Are you running out of memory perhaps (vmstat 1) ? regards john
Well, with zfs on-board, vmstat''s going to show it all used up. If I''m running out of memory, I''m really running out of places to hold references to memory. I''ve got 8GB and that much more swap. And the system is performing nothing harder than a series of pkgadds. (luupgrade, as it were, but other loads, pax -rw /net/server/dir . will do it also.) (Pause while I go look for something...Seem to recall some forum msgs regarding zfs) Sigh, I now know far more than I ever wanted to about Xen. OK, I''m suspicious that it''s a ZFS/Xen interaction. I''m off to test (best as I can) the following hypothesis: Solaris/x86xpv reports the same memory values to ZFS/arc as does Solaris x86pc. But ZFZ/arc is designed to let Solaris have enough memory for Solaris. But perhaps not enough for Solaris+Xen. My tool will be mdb -k ::memstat I''m adding the zfs->discuss list to the distro. But it will be a few days before I can pull this system down to do that. Hopefully someone already knows the answer... Cheers! Cheers! -sam This message posted from opensolaris.org
Sam Nicholson wrote:> Well, with zfs on-board, vmstat''s going to show it all used up. > > If I''m running out of memory, I''m really running out of places to hold references to memory. I''ve got 8GB and that much more swap. And the system is performing nothing harder than a series of pkgadds. (luupgrade, as it were, but other loads, pax -rw /net/server/dir . will do it also.) > > (Pause while I go look for something...Seem to recall some forum msgs regarding zfs) > > Sigh, I now know far more than I ever wanted to about Xen. > > OK, I''m suspicious that it''s a ZFS/Xen interaction. I''m off to test (best as I can) the following hypothesis: > > Solaris/x86xpv reports the same memory values to ZFS/arc as does Solaris x86pc. But ZFZ/arc is designed to let Solaris have enough memory for Solaris. But perhaps not enough for Solaris+Xen. My tool will be mdb -k ::memstat > > I''m adding the zfs->discuss list to the distro. > > But it will be a few days before I can pull this system down to do that. Hopefully someone already knows the answer...If your using zfs on dom0, you need to do some tuning. Here are some zfs tuning suggestions.. MRJ -- On dom0, you should limit the amount of memory uses by adding dom0_mem to grubs menu.lst. Normally, if you have zfs, you should set dom0_mem to ~ 2G. kernel$ /boot/$ISADIR/xen.gz com1=9600,8n1 console=com1 dom0_mem=2g You should also limit the amount of memory can give away. svccfg -s xvm/xend setprop config/dom0-min-mem=2000 svcadm refresh xvm/xend;svcadm restart xvm/xend If your using zfs in dom0, you should limit the size of the arc. I would also stay away from debug bits on dom0 when your using zfs. echo "set zfs:zfs_arc_max = 0x10000000" >> /etc/system If your using a disk file (vs a zvol) on a zfs filesystem, you should set the recordsize for that fs to 8k. zvols already default to 8k. zfs set recordsize=8k rpool/guests
Yep.>dom0_memThat''s what I''ve been searching for. The docs tell how to limit dom_Us, but I haven''t seen the bits for dom_0. Is there documentation ? I''ve been able to find this thread: http://www.opensolaris.org/jive/click.jspa?searchID=1112653&messageID=99031 Re the zfs arc limits. That ought to do the trick for lots of issues I have. Thanks!!! This message posted from opensolaris.org
Silly me!>Is there documentation ?http://www.opensolaris.org/os/community/xen/docs/sysadmin/ I thought that was just how to install, turn on, off, etc. Well, the etc is important!!! This message posted from opensolaris.org
So, booting with dom0_mem=ANYTHING fails. 1g , 2g 4g I''ve spent the last day or so going through the change one thing, reboot cycles. booting xen with solaris as dom0 with -v shows the system hung just after cmdk0 loads; motherboard disk light on solid. And without the dom0_mem argument, it boots fine. As to the other two bits, svccfg -s xvm/xend setprop config/dom0-min-mem=2000 and echo "set zfs:zfs_arc_max = 0x10000000" >> /etc/system... Well obviously the first won''t have much effect if solaris/i86pv won''t load as dom0. But the second ones effect is immediate. Under bar metal solaris, there is lots of memory available, and things are a bit snappier. Interestingly, I determined that one didn''t need to luupgrade, a simple copy from an nfs mount will do the trick, if the source is large enough. but I don''t think it''s nfs related. It''s when cp mmaps the origin file and the file is very large. scp, even local to local, works fine. The last two can be illustrated by this snippet of top output: load averages: 0.16, 0.28, 0.16; up 0+00:08:15 18:40:01 49 processes: 48 sleeping, 1 on cpu CPU states: 99.7% idle, 0.0% user, 0.3% kernel, 0.0% iowait, 0.0% swap Memory: 7678M phys mem, 5777M free mem, 4032M total swap, 4032M free swap PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND 692 scion 1 60 0 9872K 9348K sleep 0:19 0.00% cp Notice the large amount of free mem. This was just under 1100M before restricting the zfs arc size. Also notice the cp (other processes not relevant). As it started it was using about 19% of a cpu core. After a short bit, it''s usage fell to near 0, and the system did its hangup. No new forks, but extant processes continue just fine. Again, this is xen/solaris_dom0. bare metal solaris runs just fine. I really hope this is helpful to someone. I''ve reached the practical limits of my ability to debug this any further. VirtualBox seems to do what I need, so I''ve really got to get on to building systems. I''ll be happy to try any suggestions if I''m at a reasonable reboot spot. Cheers! This message posted from opensolaris.org