Hi all,
I am experiencing the below error systematically on all the servers
where I have installed Xen and reverted back to a redhat kernel.
I managed to fix it but unfortunately I can''t understand the reason
(not
yet at least ;) ).
I suspect some strange interaction between Xen kernel and prelink that
is carried on even after the xen kernel is gone.
The servers are HP ProLiant BL465c G1
cpus: 2xDual-Core AMD Opteron(tm) Processor 2216 HE
os: RHEL 4.6 32bit version (Linux server-0007 2.6.9-67.0.15.ELsmp #1 SMP
Tue Apr 22 13:50:33 EDT 2008 i686 athlon i386 GNU/Linux)
ram: 8GB
I have several java apps that start with -Xmx2048m option and they work
just fine on 12 blades, but on some of them that were used as dom0s.
[root@server-0007 jdk1.5.0_11]# ./bin/java -Xmx2048M -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
the limit seems to be at 1538m:
sh-3.00# ./bin/java -Xmx1538m -version
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)
sh-3.00# ./bin/java -Xmx1539m -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
These are the steps that make my servers act this way:
- Enabled Virtualization support on the CPU
- Install Xen and a Xen kernel ( I had 3.1.0-1 installed with the rhel4
rpms from xen.org, not sure if the version matters )
- moved /lib/tls /lib/tls.disabled
- Booted an HVM domU
- Uninstalled Xen Xen Kernel
- Disabled Virtualization support on CPU
- moved /lib/tls.disabled /lib/tls
- Rebooted to a redhat kernel (2.6.9-67.0.15.ELsmp on my case)
Now, there''s seems to be plenty of free memory but java fails as above
[massimo@server-0007 ~]$ free -m
total used free shared buffers cached
Mem: 8114 169 7944 0 15 74
-/+ buffers/cache: 79 8035
Swap: 0 0 0
ulimit is not guilty in this:
[massimo@server-0007 ~]$ ulimit -m -v
max memory size (kbytes, -m) unlimited
virtual memory (kbytes, -v) unlimited
selinux is set in permissive mode
To fix this problem I did:
(since I spent several hours and many commands on this I am not sure if
some steps can be skipped. Maybe on the next server that I will switch
from xen to rh kernel I''ll try to shorten the list and maybe give it
more sense!)
- reboot in "emergency" mode and with selinux=0, (not single user. It
didn''t work in single user mode) ( emergency = nothing is mounted
except
/ in ro)
- mount -t proc none /proc
- mount -o remount,rw /
- prelink -au (undo any prelinking)
- rm -f /etc/prelink.cache
- ldconfig -v
- prelink -avmR (redo prelinking)
- ldconfig -v
[root@(none) jdk1.5.0_11]# ./bin/java -Xmx2048m -version
java version "1.5.0_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_11-b03)
Java HotSpot(TM) Server VM (build 1.5.0_11-b03, mixed mode)
## ypeee!
I am sorry if some commands are probably redundant here, I just copied
and pasted my notes...
So my questions are:
Why is this happening after Xen touched my system?
and
Why in emergency mode?
and ....
Can anyone share some light on this or perhaps try and reproduce it?
Maybe there are not so many people switching from Xen kernel to Redhat
kernels that need to run this sort of java applications but I have the
feeling that there''s something buggish behind this...
Regards,
Massimo
--
Massimo Mongardini
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
echo
''Jg!J!hjwf!zpv!bo!bqqmf!boe!zpv!hjwf!nf!bo!bqqmf-!uifo!xf!xjmm!ibwf!bo!bqqmf!fbdi/!Cvu!jg!J!hjwf!zpv!bo!jefb!boe!zpv!hjwf!nf!bo!jefb-!xf!xjmm!ibwf!uxp!jefbt!fbdi!''
| perl -pe ''s/(.)/chr(ord($1)-1)/ge''
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
http://massimo.mongardini.it
http://www.getthefacts.it
http://www.mongardini.it/pizza-howto
~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~
Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
On Fri, 2008-07-25 at 17:35 +0100, Massimo Mongardini wrote:> The servers are HP ProLiant BL465c G1 > cpus: 2xDual-Core AMD Opteron(tm) Processor 2216 HE > os: RHEL 4.6 32bit version (Linux server-0007 2.6.9-67.0.15.ELsmp #1 > SMP > Tue Apr 22 13:50:33 EDT 2008 i686 athlon i386 GNU/Linux) > ram: 8GB > > I have several java apps that start with -Xmx2048m option and they > work > just fine on 12 blades, but on some of them that were used as dom0s.Really? With a 32-bit JVM? I''ve never been able to get it to start with any more than the ~1530MB limit and I found docs on Sun''s site to back that up -- different OS''s place different limits under 2048MB for various overheads. I''d say you should count yourself very, very lucky that the 2048MB''s never ran into OOM conditions. Still, odd that the JVM won''t even *start* under Xen... John -- John Madden Sr. UNIX Systems Engineer Ivy Tech Community College of Indiana jmadden@ivytech.edu _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Massimo Mongardini
2008-Jul-25 21:32 UTC
Re: [Xen-users] Xen, Memory and prelink interaction
John Madden wrote:> On Fri, 2008-07-25 at 17:35 +0100, Massimo Mongardini wrote: > >> The servers are HP ProLiant BL465c G1 >> cpus: 2xDual-Core AMD Opteron(tm) Processor 2216 HE >> os: RHEL 4.6 32bit version (Linux server-0007 2.6.9-67.0.15.ELsmp #1 >> SMP >> Tue Apr 22 13:50:33 EDT 2008 i686 athlon i386 GNU/Linux) >> ram: 8GB >> >> I have several java apps that start with -Xmx2048m option and they >> work >> just fine on 12 blades, but on some of them that were used as dom0s. >> > > Really? With a 32-bit JVM? I''ve never been able to get it to start > with any more than the ~1530MB limit and I found docs on Sun''s site to > back that up -- different OS''s place different limits under 2048MB for > various overheads. I''d say you should count yourself very, very lucky > that the 2048MB''s never ran into OOM conditions. Still, odd that the > JVM won''t even *start* under Xen... > > John > > > >Sorry my post wasn''t clear enough... the problem arises on a non-xen kernel (redhat kernel 2.6.9-67.0.15.ELsmp) after the server was "xenified" and then reverted back to a plain redhat kernel. Not sure if and how this can matter but when running as a dom0 I gave the single domU that it had running 6GB of memory out of 8GB. On my site I have various java instances like that one happily running on various non dom0/U servers, for instance [massimo@server-0006 ~]$ ps auxw | grep Xmx2048M | grep -v grep | wc -l 6 I believe the java heap size limit for a 32-bit is 4GB if not more constrained by the os. http://java.sun.com/performance/reference/whitepapers/tuning.html#section4.1.2 I still have 5 servers that are running a xen kernel and since this is the second server that has this issue in the same circumstances, I am quite confident that if I switch those back to a rh kernel I will face the same issue.... If any of you wants to do some debugging I am more than willing to switch one of them back to run "plain" kernels. cheers Massimo -- Massimo Mongardini ~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~ echo ''Jg!J!hjwf!zpv!bo!bqqmf!boe!zpv!hjwf!nf!bo!bqqmf-!uifo!xf!xjmm!ibwf!bo!bqqmf!fbdi/!Cvu!jg!J!hjwf!zpv!bo!jefb!boe!zpv!hjwf!nf!bo!jefb-!xf!xjmm!ibwf!uxp!jefbt!fbdi!'' | perl -pe ''s/(.)/chr(ord($1)-1)/ge'' ~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~ http://massimo.mongardini.it http://www.getthefacts.it http://www.mongardini.it/pizza-howto ~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~-.-~ Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users