OS: CentOS 6.4 x86_64<br />
Kernel: Linux version 2.6.32-358.6.2.el6.x86_64<br />
Memory: 16GB<br />
Processors: 8<br />
Virtualisation: Xen 4.0.1<br />
Hypervisor Platform: Debian Squeeze<br />
NFS: Yes<br />
<br />
Summary:<br />
crash> sys<br />
KERNEL: /usr/lib/debug/lib/modules/2.6.32-358.6.2.el6.x86_64/vmlinux<br />
DUMPFILE: hidden.core<br />
CPUS: 16<br />
DATE: Thu Jun 6 06:41:57 2013<br />
UPTIME: 5 days, 14:06:57<br />
LOAD AVERAGE: 4.20, 3.60, 4.17<br />
TASKS: 754<br />
NODENAME: hidden<br />
RELEASE: 2.6.32-358.6.2.el6.local.x86_64<br />
VERSION: <a href="http://bugs.centos.org/view.php?id=1">0000001</a> SMP Fri May 31 14:44:02 PDT 2013<br />
MACHINE: x86_64 (2992 Mhz)<br />
MEMORY: 16 GB<br />
PANIC: "Kernel panic - not syncing: Fatal exception"<br />
<br />
The CentOS 6.4 is a guest domU running in a virtual environment. The guest VM is running both Apache and Ruby and during peak times of load both system and memory pressure, the OS will kernel panic and crash. This typically occurs within a few days (3, 5 days).<br />
<br />
The web server itself has NFS mounts where the content is located. The server gets a fair amount of load. Based on stack trace data it appears that at some point it attempts to read in a page that is dirty and causes a panic. I am unsure what can be corrupting this page.<br />
<br />
Backtrace<br />
crash> bt<br />
PID: 31356 TASK: ffff8802356acae0 CPU: 9 COMMAND: "ruby"<br />
#0 [ffff880228ec17b0] xen_panic_event at ffffffff810033c2<br />
<a href="http://bugs.centos.org/view.php?id=1">0000001</a> [ffff880228ec17d0] notifier_call_chain at ffffffff81513645<br />
<a href="http://bugs.centos.org/view.php?id=2">0000002</a> [ffff880228ec1810] notifier_call_chain at ffffffff815136aa<br />
#3 [ffff880228ec1830] _spin_lock at ffffffff8151048c<br />
#4 [ffff880228ec18a0] oops_end at ffffffff81511664<br />
#5 [ffff880228ec18d0] die at ffffffff8100f19b<br />
#6 [ffff880228ec1900] do_general_protection at ffffffff81511142<br />
<a href="http://bugs.centos.org/view.php?id=7">0000007</a> [ffff880228ec1930] xen_stack_segment at ffffffff81510915<br />
[exception RIP: list_sort+923]<br />
RIP: ffffffff81288edb RSP: ffff880228ec19e8 RFLAGS: 00010046<br />
RAX: dead000000100100 RBX: ffffea0005ad97f0 RCX: 0000000000000000<br />
RDX: 0000000000000030 RSI: ffff8803fa2df870 RDI: ffffea0005ad97f0<br />
RBP: ffff880228ec19f8 R8: ffffea0005ad97f0 R9: 0000000000000000<br />
R10: 0000000000000000 R11: 0000000000000003 R12: ffff8803fa2df840<br />
R13: ffff880000018d80 R14: 0000000000000ee5 R15: ffffea0005ad97c8<br />
ORIG_RAX: ffffffffffffffff CS: 10000e030 SS: e02b<br />
<a href="http://bugs.centos.org/view.php?id=8">0000008</a> [ffff880228ec19e0] bad_page at ffffffff81128dc2<br />
<a href="http://bugs.centos.org/view.php?id=9">0000009</a> [ffff880228ec1a00] get_page_from_freelist at ffffffff8112a1d8<br />
<a href="http://bugs.centos.org/view.php?id=10">0000010</a> [ffff880228ec1b20] __alloc_pages_nodemask at ffffffff8112bb83<br />
<a href="http://bugs.centos.org/view.php?id=11">0000011</a> [ffff880228ec1b58] xen_mc_flush at ffffffff810046b6<br />
#12 [ffff880228ec1c60] alloc_pages_vma at ffffffff81160a1a<br />
<a href="http://bugs.centos.org/view.php?id=13">0000013</a> [ffff880228ec1cb0] handle_pte_fault at ffffffff81143d9b<br />
<a href="http://bugs.centos.org/view.php?id=14">0000014</a> [ffff880228ec1d48] __raw_callee_save_xen_pmd_val at ffffffff81004a49<br />
<a href="http://bugs.centos.org/view.php?id=15">0000015</a> [ffff880228ec1d90] handle_mm_fault at ffffffff811443ba<br />
#16 [ffff880228ec1e00] __do_page_fault at ffffffff810474c9<br />
#17 [ffff880228ec1f20] arch_prepare_kprobe at ffffffff8151358e<br />
#18 [ffff880228ec1f50] general_protection at ffffffff81510945<br />
RIP: 00007f0ad13cf97b RSP: 00007f0ac067be18 RFLAGS: 00010202<br />
RAX: 00007f0ab82851b8 RBX: 00007fffd66437f0 RCX: 0000000000000116<br />
RDX: 00000000000016f8 RSI: 000000001ab43910 RDI: 00007f0ab8286000<br />
RBP: 00000000000001c1 R8: 000000001ab441c0 R9: 0000000015620100<br />
R10: 000000000000017f R11: 00000000000016f8 R12: 00000000000001c4<br />
R13: 000000001ab42ac8 R14: 00007f0ab82851b8 R15: 0000000000000000<br />
ORIG_RAX: ffffffffffffffff CS: e033 SS: e02b
↧