Dear Xen-users at Xen Discuss, I started this thread on Xen-users@lists.xensource.com but would like to carry on the discussion here. (as it''s a 9.06 based platform) Original discussion (I''ll only repeat the main bits) http://lists.xensource.com/archives/html/xen-users/2009-11/msg00338.html My Setup: very recent quad core Xeon 2Ghz, Xen 3.1 from OpenSolaris 09/06 OpenSolaris 09/06 Dom0 Ubuntu 8.04 with 2.6.24-25-xen kernel image OR debian 2.6.26-2-xen-amd64 kernel JVMs: ANY (1.5.0 - 1.7, either openjdk or sun) My Problem: ubuntu (8.04) / Debian (5) hosts cannot run a JBoss instance when the virtual machine is configured with more than 1791MB of memory. Symptoms: JVM Seg fault.> Current thread (0x000000000061e800): GCTaskThread [stack: > 0x0000000041166000,0x0000000041267000] [id=7238] > > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), > si_addr=0x0000000000000018 >Notes: If I set the RAM to 1791 it''s as stable as normal, if I set the RAM to 1792 or above, the JVM segfaults within 10secs. If I use an HVM vm, it works fine. Any pointers/input or help with this would be greatly appreciated. A great many thanks Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd
Rob Shepherd wrote:> Dear Xen-users at Xen Discuss, > > I started this thread on Xen-users@lists.xensource.com > but would like to carry on the discussion here. (as it''s a 9.06 based > platform) > > Original discussion (I''ll only repeat the main bits) > http://lists.xensource.com/archives/html/xen-users/2009-11/msg00338.html > > My Setup: > very recent quad core Xeon 2Ghz, > Xen 3.1 from OpenSolaris 09/06 > OpenSolaris 09/06 Dom0 > Ubuntu 8.04 with 2.6.24-25-xen kernel image OR debian 2.6.26-2-xen-amd64 > kernel > JVMs: ANY (1.5.0 - 1.7, either openjdk or sun) > > > My Problem: > > ubuntu (8.04) / Debian (5) hosts cannot run a JBoss instance when the > virtual machine is configured with more than 1791MB of memory. > > Symptoms: > JVM Seg fault. >> Current thread (0x000000000061e800): GCTaskThread [stack: >> 0x0000000041166000,0x0000000041267000] [id=7238] >> >> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), >> si_addr=0x0000000000000018 >> > > > Notes: If I set the RAM to 1791 it''s as stable as normal, if I set the > RAM to 1792 or above, the JVM segfaults within 10secs. > If I use an HVM vm, it works fine. > > > Any pointers/input or help with this would be greatly appreciated. > > A great many thanks > > Rob >Some more info, this taken on a PV VM system (above) 1792MB memory. By adding -XX:+PrintGCDetails I can confirm that garbage collection is the crash trigger. A snippet from the JBoss startup log. We can see (in this instance) the "Full GC" never completes.... (it''s not always Full GCs)> 12:38:13,791 INFO [ServerInfo] Java version: 1.6.0_0,Sun Microsystems Inc. > 12:38:13,791 INFO [ServerInfo] Java Runtime: OpenJDK Runtime Environment (build 1.6.0_0-b16) > 12:38:13,791 INFO [ServerInfo] Java VM: OpenJDK 64-Bit Server VM 14.0-b16,Sun Microsystems Inc. > 12:38:13,791 INFO [ServerInfo] OS-System: Linux 2.6.26-2-xen-amd64,amd64 > 12:38:13,792 INFO [ServerInfo] VM arguments: -Dprogram.name=run.sh -Xms512m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -XX:+PrintGCDetails -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/usr/local/jboss-5.1.0.GA/lib/endorsed > 12:38:13,823 INFO [JMXKernel] Legacy JMX core initialized > [GC [PSYoungGen: 131072K->16975K(152896K)] 131072K->16975K(502464K), 0.0590160 secs] [Times: user=0.06 sys=0.03, real=0.06 secs] > [GC [PSYoungGen: 148047K->20094K(152896K)] 148047K->20094K(502464K), 0.1033480 secs] [Times: user=0.07 sys=0.03, real=0.11 secs] > 12:38:15,607 INFO [ProfileServiceBootstrap] Loading profile: ProfileKey@30f49e8f[domain=default, server=default, name=default] > [GC [PSYoungGen: 151166K->21817K(152896K)] 151166K->43690K(502464K), 0.1092050 secs] [Times: user=0.13 sys=0.03, real=0.11 secs] > 12:38:16,984 INFO [WebService] Using RMI server codebase: http://127.0.0.1:8083/ > [GC [PSYoungGen: 53080K->21796K(152896K)] 74952K->48044K(502464K), 0.0478300 secs] [Times: user=0.08 sys=0.01, real=0.04 secs] > [Full GC (System)# > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x00007f9d6fb96b30, pid=5681, tid=1091848528 > # > # JRE version: 6.0-b16Again, any insight into this, and any thoughts on solving this would be appreciated. A great many thanks Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd
Rob Shepherd wrote:> Rob Shepherd wrote: >> Dear Xen-users at Xen Discuss, >> >> I started this thread on Xen-users@lists.xensource.com >> but would like to carry on the discussion here. (as it''s a 9.06 based >> platform) >> >> Original discussion (I''ll only repeat the main bits) >> http://lists.xensource.com/archives/html/xen-users/2009-11/msg00338.html >> >> My Setup: >> very recent quad core Xeon 2Ghz, >> Xen 3.1 from OpenSolaris 09/06 >> OpenSolaris 09/06 Dom0 >> Ubuntu 8.04 with 2.6.24-25-xen kernel image OR debian >> 2.6.26-2-xen-amd64 kernel >> JVMs: ANY (1.5.0 - 1.7, either openjdk or sun) >> >> >> My Problem: >> >> ubuntu (8.04) / Debian (5) hosts cannot run a JBoss instance when the >> virtual machine is configured with more than 1791MB of memory. >> >> Symptoms: >> JVM Seg fault. >>> Current thread (0x000000000061e800): GCTaskThread [stack: >>> 0x0000000041166000,0x0000000041267000] [id=7238] >>> >>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), >>> si_addr=0x0000000000000018 >>> >> >> >> Notes: If I set the RAM to 1791 it''s as stable as normal, if I set the >> RAM to 1792 or above, the JVM segfaults within 10secs. >> If I use an HVM vm, it works fine. >> >> >> Any pointers/input or help with this would be greatly appreciated. >> >> A great many thanks >> >> Rob >> > > Some more info, this taken on a PV VM system (above) 1792MB memory. > > By adding -XX:+PrintGCDetails I can confirm that garbage collection is > the crash trigger. > > A snippet from the JBoss startup log. We can see (in this instance) the > "Full GC" never completes.... (it''s not always Full GCs) > >> 12:38:13,791 INFO [ServerInfo] Java version: 1.6.0_0,Sun Microsystems >> Inc. >> 12:38:13,791 INFO [ServerInfo] Java Runtime: OpenJDK Runtime >> Environment (build 1.6.0_0-b16) >> 12:38:13,791 INFO [ServerInfo] Java VM: OpenJDK 64-Bit Server VM >> 14.0-b16,Sun Microsystems Inc. >> 12:38:13,791 INFO [ServerInfo] OS-System: Linux 2.6.26-2-xen-amd64,amd64 >> 12:38:13,792 INFO [ServerInfo] VM arguments: -Dprogram.name=run.sh >> -Xms512m -Xmx512m -XX:MaxPermSize=256m >> -Dorg.jboss.resolver.warning=true -XX:+PrintGCDetails >> -Djava.net.preferIPv4Stack=true >> -Djava.endorsed.dirs=/usr/local/jboss-5.1.0.GA/lib/endorsed >> 12:38:13,823 INFO [JMXKernel] Legacy JMX core initialized >> [GC [PSYoungGen: 131072K->16975K(152896K)] 131072K->16975K(502464K), >> 0.0590160 secs] [Times: user=0.06 sys=0.03, real=0.06 secs] [GC >> [PSYoungGen: 148047K->20094K(152896K)] 148047K->20094K(502464K), >> 0.1033480 secs] [Times: user=0.07 sys=0.03, real=0.11 secs] >> 12:38:15,607 INFO [ProfileServiceBootstrap] Loading profile: >> ProfileKey@30f49e8f[domain=default, server=default, name=default] >> [GC [PSYoungGen: 151166K->21817K(152896K)] 151166K->43690K(502464K), >> 0.1092050 secs] [Times: user=0.13 sys=0.03, real=0.11 secs] >> 12:38:16,984 INFO [WebService] Using RMI server codebase: >> http://127.0.0.1:8083/ >> [GC [PSYoungGen: 53080K->21796K(152896K)] 74952K->48044K(502464K), >> 0.0478300 secs] [Times: user=0.08 sys=0.01, real=0.04 secs] [Full GC >> (System)# >> # A fatal error has been detected by the Java Runtime Environment: >> # >> # SIGSEGV (0xb) at pc=0x00007f9d6fb96b30, pid=5681, tid=1091848528 >> # >> # JRE version: 6.0-b16 > > Again, any insight into this, and any thoughts on solving this would be > appreciated. > > A great many thanks > > Rob > > >More info. It works fine with Oracle/BEA''s JRockit VM, which according to a google can operate with a non-contiguous heap. Is heap coniguity a problem for Xen? Many Thanks Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd
Rob Shepherd wrote:> Rob Shepherd wrote: >> Rob Shepherd wrote: > More info. It works fine with Oracle/BEA''s JRockit VM, which according > to a google can operate with a non-contiguous heap. > > Is heap coniguity a problem for Xen? > > Many ThanksI believe this is probably the following bug 6822619 Linux Hotspot server VM crashes when running specJvm2008 and specJbb2005 in RHEL5-U3 domU which was marked incomplete waiting for someone else to run into it... I''ve moved it back to dispatched. Do you see it on a Solaris domU? Do you set dom0_mem? If not, do you see it if you set dom0_mem? MRJ
Mark Johnson wrote:> > Rob Shepherd wrote: >> Rob Shepherd wrote: >>> Rob Shepherd wrote: >> More info. It works fine with Oracle/BEA''s JRockit VM, which >> according to a google can operate with a non-contiguous heap. >> >> Is heap coniguity a problem for Xen? >> >> Many Thanks > > > I believe this is probably the following bug > 6822619 Linux Hotspot server VM crashes when running specJvm2008 and > specJbb2005 in RHEL5-U3 domU > > which was marked incomplete waiting for someone else to run > into it... I''ve moved it back to dispatched. > > Do you see it on a Solaris domU? > > Do you set dom0_mem? If not, do you see it if you > set dom0_mem? >I am about to try a Solaris DomU. Will post the results. dom0_mem is (and has always been) set to 1024M A great many thanks for your input Rob -- Rob Shepherd BEng PhD - Director / Senior Engineer - DataCymru Ltd