I have gotten a bug soft lockup on this system 4 times in the last few weeks. The lockup causes the load to slowly rise and the system is completely unresponsive (can't log in via ssh or console). I had updated to latest version of kernel (2.6.32-358.23.2.el6.x86_64) last week and it happened again today. The system is a Dell Poweredge M710. Every time it happens on this system, it references the pid for sshd. sshd is also at the latest version:<br />
<br />
# rpm -qa | grep openssh<br />
openssh-clients-5.3p1-84.1.el6.x86_64<br />
openssh-5.3p1-84.1.el6.x86_64<br />
openssh-server-5.3p1-84.1.el6.x86_64<br />
<br />
Here is the latest stack trace:<br />
<br />
Nov 20 05:59:38 stage kernel: BUG: soft lockup - CPU#2 stuck for 67s! [sshd:24074]<br />
Nov 20 05:59:38 stage kernel: Modules linked in: bridge stp llc hcpdriver(P)(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu be2iscsi iscsi_boot_sy<br />
sfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ext<br />
4 jbd2 power_meter ses enclosure sg bnx2 dcdbas microcode iTCO_wdt iTCO_vendor_support i7core_edac edac_core sr_mod cdrom ext3 jbd mbcache dm_round_robin usb_storage sd_mod crc_<br />
t10dif pata_acpi ata_generic ata_piix wmi megaraid_sas dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]<br />
Nov 20 05:59:38 stage kernel: CPU 2<br />
Nov 20 05:59:38 stage kernel: Modules linked in: bridge stp llc hcpdriver(P)(U) mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf dell_rbu be2iscsi iscsi_boot_sy<br />
sfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ext<br />
4 jbd2 power_meter ses enclosure sg bnx2 dcdbas microcode iTCO_wdt iTCO_vendor_support i7core_edac edac_core sr_mod cdrom ext3 jbd mbcache dm_round_robin usb_storage sd_mod crc_<br />
t10dif pata_acpi ata_generic ata_piix wmi megaraid_sas dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]<br />
Nov 20 05:59:38 stage kernel: <br />
Nov 20 05:59:38 stage kernel: Pid: 24074, comm: sshd Tainted: P --------------- 2.6.32-358.23.2.el6.x86_64 <a href="http://bugs.centos.org/view.php?id=1">0000001</a> Dell Inc. PowerEdge M710/0X3X22<br />
Nov 20 05:59:38 stage kernel: RIP: 0010:[<ffffffff810d4313>] [<ffffffff810d4313>] audit_log_start+0xf3/0x430<br />
Nov 20 05:59:38 stage kernel: RSP: 0018:ffff880011a599a8 EFLAGS: 00000282<br />
Nov 20 05:59:38 stage kernel: RAX: ffffffffffef6fcd RBX: ffff880011a59a58 RCX: 0000000000000141<br />
Nov 20 05:59:38 stage kernel: RDX: 000000000000ea60 RSI: 000000000000ea60 RDI: 0000000000000140<br />
Nov 20 05:59:38 stage kernel: RBP: ffffffff8100bb8e R08: fffffffed24ca858 R09: 00000000ffffffff<br />
Nov 20 05:59:38 stage kernel: R10: 0000000000000000 R11: 0000000000000000 R12: dead000000200200<br />
Nov 20 05:59:38 stage kernel: R13: 0000000000000000 R14: 0000000000000286 R15: ffff880000000430<br />
Nov 20 05:59:38 stage kernel: FS: 00007fc1ea0e27c0(0000) GS:ffff88043e420000(0000) knlGS:0000000000000000<br />
Nov 20 05:59:38 stage kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br />
Nov 20 05:59:38 stage kernel: CR2: 00007f9bd4740000 CR3: 00000004271ad000 CR4: 00000000000007e0<br />
Nov 20 05:59:38 stage kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br />
Nov 20 05:59:38 stage kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br />
Nov 20 05:59:38 stage kernel: Process sshd (pid: 24074, threadinfo ffff880011a58000, task ffff880427135540)<br />
Nov 20 05:59:38 stage kernel: Stack:<br />
Nov 20 05:59:38 stage kernel: ffff880011a599c8 000000000000ea60 000000d000000964 0000000000000000<br />
Nov 20 05:59:38 stage kernel: <d> ffff880011a599e8 ffffffff81471485 0000000000000000 ffff880427135540<br />
Nov 20 05:59:38 stage kernel: <d> ffffffff81063990 dead000000100100 dead000000200200 ffffffff81b165c0<br />
Nov 20 05:59:38 stage kernel: Call Trace:<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81471485>] ? netlink_sendskb+0x35/0x60<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81063990>] ? default_wake_function+0x0/0x20<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810d485a>] ? audit_log_common_recv_msg+0x6a/0xf0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81510fb5>] ? page_fault+0x25/0x30<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810d4ca8>] ? audit_receive+0x3c8/0xd90<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81055ad3>] ? __wake_up+0x53/0x70<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81472386>] ? netlink_unicast+0x2e6/0x300<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81472d10>] ? netlink_sendmsg+0x200/0x2e0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81435f33>] ? sock_sendmsg+0x123/0x150<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8112bc43>] ? __alloc_pages_nodemask+0x113/0x8d0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff811685ac>] ? __kmalloc+0x20c/0x220<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8121ca06>] ? security_sk_alloc+0x16/0x20<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8143a2cc>] ? sk_prot_alloc+0x6c/0x1c0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81436249>] ? sys_sendto+0x139/0x190<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8117e037>] ? fd_install+0x47/0x90<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810dc937>] ? audit_syscall_entry+0x1d7/0x200<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b<br />
Nov 20 05:59:38 stage kernel: Code: cf 0f 83 5f 01 00 00 45 85 ff 0f 84 98 00 00 00 85 d2 0f 84 90 00 00 00 48 8b 05 b9 a5 b3 00 49 89 f0 49 29 c0 4c 89 c0 4c 01 e0 <48> 85 c0 7<br />
e cf 48 c7 45 80 00 00 00 00 48 c7 45 98 00 00 00 00<br />
Nov 20 05:59:38 stage kernel: Call Trace:<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810d438b>] ? audit_log_start+0x16b/0x430<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81471485>] ? netlink_sendskb+0x35/0x60<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81063990>] ? default_wake_function+0x0/0x20<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810d485a>] ? audit_log_common_recv_msg+0x6a/0xf0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81510fb5>] ? page_fault+0x25/0x30<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810d4ca8>] ? audit_receive+0x3c8/0xd90<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81055ad3>] ? __wake_up+0x53/0x70<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81472386>] ? netlink_unicast+0x2e6/0x300<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81472d10>] ? netlink_sendmsg+0x200/0x2e0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81435f33>] ? sock_sendmsg+0x123/0x150<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8112bc43>] ? __alloc_pages_nodemask+0x113/0x8d0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff811685ac>] ? __kmalloc+0x20c/0x220<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8121ca06>] ? security_sk_alloc+0x16/0x20<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8143a2cc>] ? sk_prot_alloc+0x6c/0x1c0<br />
Nov 20 05:59:38 stage kernel: [<ffffffff81436249>] ? sys_sendto+0x139/0x190<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8117e037>] ? fd_install+0x47/0x90<br />
Nov 20 05:59:38 stage kernel: [<ffffffff810dc937>] ? audit_syscall_entry+0x1d7/0x200<br />
Nov 20 05:59:38 stage kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b<br />
Nov 20 05:59:40 stage abrtd: Directory 'oops-2013-11-20-05:59:40-8709-1' creation detected<br />
Nov 20 05:59:40 stage abrt-dump-oops: Reported 1 kernel oopses to Abrt
↧