生产服务器宕机，初步分析发现遇到JVM崩溃，jboss服务器日志和GC日志看起来没问题，jvm错误日志提到“Problematic frame：第五章【libjvm.so】
配置详细信息：-
服务器为64位计算机，具有32位RHEL-5.2。JAVA：jdk1.6.0_11内存（RAM）中：2GB Java虚拟机的堆大小为-Xmx 1024 m。
'Java运行时环境检测到一个意外错误：在pc= 0x 06546720，pid=5127，tid=1822063504时的信号SEGV（0xb）
Java VM：Java HotSpot（TM）服务器VM（11.0-b16混合模式linux-x86）有问题的框架：V [libjvm.so +0x546720]（中文）
如果您想提交错误报告，请访问：http://java.sun.com/webapps/bugreport/crash.jsp

当前线程（0x 09 a05400）：GCT任务线程[堆栈：[ID=5130]的数据类型
签名信息：si_signo=SIGSEGV：系统信息错误号=0，系统信息代码=1（SEGV_MAPERR），系统信息地址= 0x 00000008
寄存器：EAX= 0x 0000008，EBX= 0xa 60239 c 0，ECX= 0x 00000000 c，EDX= 0x 0000000 c电子稳定程序= 0x 6c 9a 6 ea 0，电子稳定程序= 0x 6c 9a 6 f18，电子服务接口= 0x 00003 ece，电子数据交换接口= 0x 2c 30913，电子信息处理接口= 0x 06546720，CR2= 0x 00000008，状态标志= 0x 00010202
堆栈顶部：（sp= 0x 6c 9a 6 ea 0）0x 6c 9a 6 ea 0的最大值：004AC889 0000000 004B7FF4 6cb57c56 0x6c9a6eb0：00000000 09a06300 0000000400 1c00dc04 0x6c9a6ec0：0000000000000000000000000000009a06300 00000000000400 1c000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000 00 a039 c8 000000008 000000001505 e98 0x 6c 9a 6 ed 0：在一个数据库中，每个数据库都有一个09 a06328 09 a06328我的天啊！0x6C9A6EF0：0x6C9A6F1800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0x 6c 9a 6 f00的第一个字符串：
0x 6c 9a 6 f10的最大值，则该值为零。00003 ec 09 a5 fc 40 6c 9a 6 f58 065462 c3的电子邮件地址
使用说明：（pc= 0x 06546720）0x 06546710：数据类型00184C00F84160500008B45AC83E0FD0x06546720：8b 00 c1 e8 03 83 e0 0f 3b 05 cc 28 6b 06 0f 8d
堆栈：[0x 6c 927000，0x 6c 9a 8000]，sp= 0x 6c 9a 6 ea 0，可用空间= 511 k本机帧：（J=编译的Java代码，j=解释的，Vv=VM代码，C=本机代码libjvm.so）
（英文）
Java执行绪：（=〉当前线程）0x 0a 745000 JavaThread“保持活动计时器”守护进程[线程阻塞，id=9822，堆栈（0x 68 fe 4000，0x 69005000）] 0x 6ae 59000 Java线程“Java 2D处理器”守护进程[线程阻塞，id=5496，堆栈（0x 6c 2a 7000，0x 6c 2c 8000）] 0x 6 b245800 Java线程“TP监视器”守护进程[线程阻塞，id=5493，堆栈（0x 69 ca 6000，0x 69 cc 7000）]
0x 6 b244000 Java线程“TP-处理器4”守护进程[本机中的线程，id=5492，堆栈[0x 69 cc 7000，0x 69 ce 8000）] 0x 6 b276 c 00 Java线程“TP处理器3”守护进程[线程阻塞，id=5491，堆栈[线程阻塞，id=5490，堆栈（0x 69 d 09000，0x 69 d2 a000）]

0x 6 b276000 Java线程“TP-处理器1”守护进程[线程阻塞，id=5489，堆栈在一个示例中，可以使用一个或多个Java线程来创建一个或多个Java线程。（0x 69 d4 b 000，0x 69 d 6c 000）] 0x 0ad 38400 Java线程“http-000,0 -8080-处理器25”守护进程[线程阻塞，id=5487，堆栈在一个示例中，可以使用一个或多个Java线程守护进程来阻止线程阻塞。在一个示例中，如果您的请求是一个请求，则将其发送到服务器，并在服务器上运行。（0x 69 dae 000，0x 69 dcf 000）] 0x 0 bd 03 c 00 Java线程“http-000,0 -8080-处理器22”守护进程[线程阻塞，id=5484，堆栈在一个示例中，如果您的请求是一个请求，则将其发送到服务器，并在服务器上运行。在本机中运行的Java线程在本机中运行的Java线程（0x 69 e32000，0x 69 e53000）] 0x 0 bcfa 800 Java线程“http-0.0.0.0-8080-处理器18”守护程序[线程阻塞，id=5480，堆栈（0x 69 e53000，0x 69 e74000）] 0x 0 bcf 9000 Java线程“http-0.0.0.0-8080-处理器17”守护进程[线程阻塞，id=5479，堆栈[0x 0aaa 8400 Java线程“http-0.0.0.0-8080-处理器16”守护进程[线程阻塞，id=5478，堆栈在一个示例中，如果您的请求是一个请求，那么您可以使用以下代码来调用该请求：在一个示例中，如果您的请求是一个请求，那么您可以使用以下代码来调用该请求：在一个示例中，如果您的请求是一个请求，那么您可以使用以下代码来调用该请求：在一个示例中，如果您的请求是一个请求，则将其发送到服务器，并在服务器上运行。在一个示例中，如果您的请求是一个请求，那么您可以使用以下代码来调用该请求：（0x 69 f3 a000，0x 69 f5 b 000）] 0x 0a 93 bc 00 Java线程“http-000,0 -8080-处理器10”守护进程[线程阻塞，id=5472，堆栈[0x 69 f5 b 000，0x 69 f7 c 000）] 0x 0a 93 a400 Java线程“http-000,0 -8080-处理器9”守护进程[线程阻塞，id=5471，堆栈（0x 69 f7 c 000，0x 69 f9 d 000）] 0x 0a 9 e1 c 00 Java线程“http-000,0 -8080-处理器8”守护进程[线程阻塞，id=5470，堆栈（0x 69 f9 d 000，0x 69 fbe 000）] 0x 0aa 7 bc 00 Java线程“http-000,0 -8080-处理器7”守护进程[线程阻塞，id=5469，堆栈（0x 69 fbe 000，0x 69 fdf 000）] 0x 0a 213 c 00 Java线程“http-000,0 -8080-处理器6”守护进程[线程阻塞，id=5468，堆栈（0x 69 fdf 000，0x 6a 000000）] 0x 0aa 0a 400 Java线程“http-000,0 -8080-处理器5”守护进程[线程阻塞，id=5467，堆栈（0x 6a 519000，0x 6a 53 a000）] 0x 0aa 09400 Java线程“http-0.0.0.0-8080-处理器4”守护进程[线程阻塞，id=5466，堆栈2000 Java线程“http-000,0 -8080-处理器3”守护进程[线程阻塞，id=5465，堆栈（0x 6a 55 b 000，0x 6a 57 c 000）] 0x 0a 10 bc 00 Java线程“http-000,0 -8080-处理器2”守护进程[线程阻塞，id=5464，堆栈在一个示例中，如果您的请求是“HTTP-000,0 -8080-处理器1”，则该请求将被阻止。（0x 6a 59 d 000，0x 6a 5 be 000）] 0x 0a 006 c 00 Java线程“JBossLifeThread”[已阻塞的线程，id=5462，堆栈（0x 6a 5 be 000，0x 6a 5df 000）] 0x 6 b652400 Java线程“定时器-4”守护进程[线程阻塞，id=5461，堆栈“测试调度程序_QuartzSchedulerThread”[线程阻塞，id=5460，堆栈“测试调度器工作线程-11”[线程阻塞，id=5459，堆栈“测试调度器工作线程-10”[线程阻塞，id=5458，堆栈“测试调度器工作线程-9”[阻塞的线程，id=5457，堆栈“测试调度器工作线程-8”[阻塞的线程，id=5456，堆栈“测试调度器工作线程-7”[线程阻塞，id=5455，堆栈“测试调度器工作线程-6”[线程阻塞，id=5454，堆栈“测试调度器工作线程-5”[线程阻塞，id=5453，堆栈“测试调度器工作线程-4”[线程阻塞，id=5452，堆栈“测试调度器工作线程-3”[阻塞的线程，id=5451，堆栈“测试调度器工作线程-2”[阻塞的线程，id=5450，堆栈“测试调度器工作线程-1”[阻塞的线程，id=5449，堆栈“测试调度器工作线程-0”[线程阻塞，id=5448，堆栈（0x 6a 9 be 000，0x 6a 9df 000）] 0x 0a 2c 1c 00 Java线程“线程-12”守护程序[线程阻塞，id=5447，堆栈（0x 6a 9df 000，0x 6aa 00000）]

0x0a85cc00 JavaThread "Thread-11" daemon [_thread_in_native, id=5446, stack(0x6ab19000,0x6ab3a000)] 0x0a7cb800 JavaThread "ElementEventQueue.QProcessor-1" daemon [_thread_blocked, id=5445, stack(0x6ab5b000,0x6ab7c000)] 0x0a504800 JavaThread "Thread-9" [_thread_blocked, id=5442, stack(0x6ab3a000,0x6ab5b000)] 0x0ad5c400 JavaThread "Timer-3" daemon [_thread_blocked, id=5437, stack(0x6ab7c000,0x6ab9d000)] 0x0b284000 JavaThread "Timer-2" daemon [_thread_blocked, id=5436, stack(0x6ab9d000,0x6abbe000)] 0x0aef7c00 JavaThread "Thread-5" [_thread_blocked, id=5435, stack(0x6abbe000,0x6abdf000)] 0x0a1dd400 JavaThread "Thread-4" [_thread_blocked, id=5434, stack(0x6b06d000,0x6b08e000)] 0x0a7df400 JavaThread "Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:2/b744868be8f2f16c9ebefdb987f81c71 destination=TOPIC.MastersSubscriberMDB messageSelector=null Local Create] id=2" [_thread_blocked, id=5433, stack(0x6b08e000,0x6b0af000)] 0x0b26fc00 JavaThread "Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:1/fcba95a2a4e911261ee0d2731536dba2 destination=TOPIC.EPrescriptionMDB messageSelector=null Local Create] id=1" [_thread_blocked, id=5432, stack(0x6b0af000,0x6b0d0000)]
0x6b62f000 JavaThread "UILServerILService Accept Thread" [_thread_in_native, id=5431, stack(0x6b35b000,0x6b37c000)]
0x6b1e8400 JavaThread "JCA PoolFiller" [_thread_blocked, id=5430, stack(0x6b37c000,0x6b39d000)] 0x6bbf4400 JavaThread "IdleRemover" [_thread_blocked, id=5429, stack(0x6b39d000,0x6b3be000)] 0x6b6aec00 JavaThread "TimeoutFactory" daemon [_thread_blocked, id=5428, stack(0x6b3be000,0x6b3df000)] 0x6bc8f400 JavaThread "Timer-1" daemon [_thread_blocked, id=5427, stack(0x6abdf000,0x6ac00000)] 0x6bccd400 JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=5426, stack(0x6a800000,0x6a821000)] 0x0a514400 JavaThread "HSQLDB Timer @1ab5dae" daemon [_thread_blocked, id=5424, stack(0x6b3df000,0x6b400000)] 0x0a411400 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=5422, stack(0x6b788000,0x6b7a9000)] 0x09f58400 JavaThread "PooledInvokerAcceptor#0-4445" [_thread_in_native, id=5421, stack(0x6b7f8000,0x6b819000)] 0x09f59000 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=5420, stack(0x6b819000,0x6b83a000)] 0x09f46800 JavaThread "ClassLoadingPool(2)-1" daemon [_thread_in_native, id=5419, stack(0x6b83a000,0x6b85b000)] 0x09f3b000 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=5418, stack(0x6b85b000,0x6b87c000)] 0x09f87400 JavaThread "GC Daemon" daemon [_thread_blocked, id=5417, stack(0x6b87c000,0x6b89d000)]
0x09f88c00 JavaThread "RMI Reaper" [_thread_blocked, id=5416, stack(0x6b89d000,0x6b8be000)] 0x09f88400 JavaThread "RMI TCP Accept-1098" daemon [_thread_in_native, id=5415, stack(0x6b8be000,0x6b8df000)] 0x09f78c00 JavaThread "ScannerThread" daemon [_thread_blocked, id=5414, stack(0x6b8df000,0x6b900000)]
0x09f3e800 JavaThread "Timer-0" daemon [_thread_blocked, id=5413, stack(0x6ba39000,0x6ba5a000)] 0x099fcc00 JavaThread "DestroyJavaVM" [_thread_blocked, id=5128, stack(0xb7f24000,0xb7f45000)] 0x09a9f000 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=5137, stack(0x6c349000,0x6c36a000)] 0x09a9d400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=5136, stack(0x6c36a000,0x6c3eb000)] 0x09a9a400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5135, stack(0x6c3eb000,0x6c46c000)] 0x09a98c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5134, stack(0x6c46c000,0x6c48d000)] 0x09a85c00 JavaThread "Finalizer" daemon [_thread_blocked, id=5133, stack(0x6c68d000,0x6c6ae000)]
0x09a84400 JavaThread "Reference Handler" daemon [_thread_blocked, id=5132, stack(0x6c6ae000,0x6c6cf000)]
Other Threads:
0x09a80c00 VMThread [stack: 0x6c6cf000,0x6c750000] [id=5131]
0x09aa1000 WatcherThread [stack: 0x6c2c8000,0x6c349000] [id=5138]
=>0x09a05400 (exited) GCTaskThread [stack: 0x6c927000,0x6c9a8000] [id=5130]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event]) [0x099faff8] UNKNOWN - owner thread: 0x09a80c00 [0x099fb408] UNKNOWN - owner thread: 0x0bd03c00
Heap PSYoungGen total 328704K, used 313529K [0x9f8b0000, 0xb4e00000, 0xb4e00000) eden space 307904K, 100% used [0x9f8b0000,0xb2560000,0xb2560000) from space 20800K, 27% used [0xb2560000,0xb2ade6b0,0xb39b0000) to space 20352K, 25% used [0xb3a20000,0xb3f2e800,0xb4e00000) PSOldGen total 699072K, used 489011K [0x74e00000, 0x9f8b0000, 0x9f8b0000) object space 699072K, 69% used [0x74e00000,0x92b8cca8,0x9f8b0000) PSPermGen
total 131072K, used 102017K [0x6ce00000, 0x74e00000, 0x74e00000)
object space 131072K, 77% used [0x6ce00000,0x731a0488,0x74e00000)
Dynamic libraries:

00312000-0032c000 r-xp 00000000 03:03 20776307 /lib/ld-2.5.so 0032c000-0032d000 r-xp 00019000 03:03 20776307 /lib/ld-2.5.so 0032d000-0032e000 rwxp 0001a000 03:03 20776307 /lib/ld-2.5.so 00330000-0046d000 r-xp 00000000 03:03 20776308 /lib/libc-2.5.so 0046d000-0046f000 r-xp 0013d000 03:03 20776308 /lib/libc-2.5.so 0046f000-00470000 rwxp 0013f000 03:03 20776308 /lib/libc-2.5.so 00470000-00473000 rwxp 00470000 00:00 0 00475000-0049a000 r-xp 00000000 03:03 20776309 /lib/libm-2.5.so 0049a000-0049b000 r-xp 00024000 03:03 20776309 /lib/libm-2.5.so 0049b000-0049c000 rwxp 00025000 03:03 20776309 /lib/libm-2.5.so 0049e000-004a0000 r-xp 00000000 03:03 20776314 /lib/libdl-2.5.so 004a0000-004a1000 r-xp 00001000 03:03 20776314 /lib/libdl-2.5.so 004a1000-004a2000 rwxp 00002000 03:03 20776314 /lib/libdl-2.5.so 004a4000-004b7000 r-xp 00000000 03:03 20776311 /lib/libpthread-2.5.so 004b7000-004b8000 r-xp 00012000 03:03 20776311 /lib/libpthread-2.5.so 004b8000-004b9000 rwxp 00013000 03:03 20776311
/lib/libpthread-2.5.so 004b9000-004bb000 rwxp 004b9000 00:00 0 008da000-008e1000 r-xp 00000000 03:03 20776312 /lib/librt-2.5.so 008e1000-008e2000 r-xp 00006000 03:03 20776312 /lib/librt-2.5.so 008e2000-008e3000 rwxp 00007000 03:03 20776312 /lib/librt-2.5.so 008eb000-008fa000 r-xp 00000000 03:03 20776327 /lib/libresolv-2.5.so 008fa000-008fb000 r-xp 0000e000 03:03 20776327 /lib/libresolv-2.5.so 008fb000-008fc000 rwxp 0000f000 03:03 20776327 /lib/libresolv-2.5.so 008fc000-008fe000 rwxp 008fc000 00:00 0 00c74000-00c87000 r-xp 00000000 03:03 20776313 /lib/libnsl-2.5.so 00c87000-00c88000 r-xp 00012000 03:03 20776313 /lib/libnsl-2.5.so 00c88000-00c89000 rwxp 00013000 03:03 20776313 /lib/libnsl-2.5.so 00c89000-00c8b000 rwxp 00c89000 00:00 0 06000000-06676000 r-xp 00000000 03:03 4620339
/usr/local/java/jdk1.6.0_11/jre/lib/i386/server/libjvm.so 06676000-066bb000 rwxp 00675000 03:03 4620339
/usr/local/java/jdk1.6.0_11/jre/lib/i386/server/libjvm.so 066bb000-06add000 rwxp 066bb000 00:00 0 08048000-08052000 r-xp 00000000 03:03 4685976 /usr/local/java/jdk1.6.0_11/bin/java 08052000-08053000 rwxp 00009000 03:03 4685976
/usr/local/java/jdk1.6.0_11/bin/java 099f7000-0bd51000 rwxp 099f7000 00:00 0 68c00000-68cd8000 rwxp 68c00000 00:00 0 68cd8000-68d00000 --xp 68cd8000 00:00 0 68e00000-68efc000 rwxp 68e00000 00:00 0 68efc000-68f00000 --xp 68efc000 00:00 0 68fe4000-68fe7000 --xp 68fe4000 00:00 0 68fe7000-69005000 rwxp 68fe7000 00:00 0 69005000-69008000 rwxp 69005000 00:00 0 69008000-69026000 rwxp 69008000 00:00 0 69026000-69045000 r-xs 00000000 03:03 4166654
/usr/share/X11/fonts/Type1/courb.pfa 69045000-69059000 r-xs 00000000 03:03 4166663 /usr/share/X11/fonts/Type1/l047016t.pfa ............... b7f59000-b7f5a000 rwxp b7f59000 00:00 0 b7f5a000-b7f5b000 --xp b7f5a000 00:00 0 b7f5b000-b7f5c000 rwxp b7f5b000 00:00 0 b7f5c000-b7f5d000 r-xp b7f5c000 00:00 0
[vdso] bfb27000-bfb3d000 rwxp bfb27000 00:00 0 [stack]
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Xmx1024m -Xms1024m -Xss128k -Dsalmon.props.path=/usr/local/jboss/jboss-3.2.7/server/default/deploy/jbossweb-tomcat50.sar/salmonprops -Djava.awt.headless=true -verbose:gc -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:PermSize=128m -XX:MaxPermSize=128m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.io.tmpdir=/tmp -Xloggc:/usr/local/jboss/jboss-3.2.7/server/default/log/gclog.log -XX:+DisableExplicitGC -Djava.endorsed.dirs=/usr/local/jboss/jboss-3.2.7/lib/endorsed java_command: org.jboss.Main Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/jboss/bin LD_LIBRARY_PATH=/usr/local/java/jdk1.6.0_11/jre/lib/i386/server:/usr/local/java/jdk1.6.0_11/jre/lib/i386:/usr/local/java/jdk1.6.0_11/jre/../lib/i386 SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x606fd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGBUS: [libjvm.so+0x606fd0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE: [libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGPIPE: [libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGXFSZ: [libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL: [libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2: [libjvm.so+0x506b60], sa_mask[0]=0x00000000, sa_flags=0x10000004 SIGHUP: [libjvm.so+0x506900], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGTERM: [libjvm.so+0x506900], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGQUIT: [libjvm.so+0x506900], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.2 (Tikanga)
uname:Linux 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:12 EDT 2008 i686 libc:glibc 2.5 NPTL 2.5 rlimit: STACK 10240k, CORE 0k, NPROC 32601, NOFILE 1024, AS infinity load average:0.10 0.06 0.02

CPU：共2个（每个CPU 2个内核，每个内核1个线程）系列6型号15步进13、cmov、cx 8、fxsr、mmx、sse、sse 2、sse 3、sse 3
内存：4k页，物理内存2065048 k（434208 k可用），交换内存4192956 k（4192956 k可用）
vm_info：用于linux-x86 JRE（1.6.0_11-b 03）的Java HotSpot（TM）服务器虚拟机（11.0-b16），由“java_re”使用gcc 3.2.1-7a（J2 SE发行版）于2008年11月10日01：21：35构建
时间：2011年6月14日星期二15：50：47经过时间：17909秒