0003192: list_del corruption

Hi guys,<br /> <br /> I run a CentOS-5.2 (i686) Samba server (samba-3.0.28-1.el5_2.1) for around 600 users. The Server seems to randomly lock up. I enabled kdump which has the following output results. The lock ups are totally random but do seem to occur when the server is under more Samba load.<br /> <br /> KERNEL: /usr/lib/debug/lib/modules/2.6.18-92.1.6.el5PAE/vmlinux<br /> DUMPFILE: /var/crash/2008-10-14-12:34/vmcore<br /> CPUS: 8<br /> DATE: Tue Oct 14 12:32:37 2008<br /> UPTIME: 25 days, 04:52:30<br /> LOAD AVERAGE: 0.82, 0.70, 0.61<br /> TASKS: 339<br /> NODENAME: tux.hiltoncollege.com<br /> RELEASE: 2.6.18-92.1.10.el5PAE<br /> VERSION: <a href="http://bugs.centos.org/view.php?id=1">0000001</a> SMP Tue Aug 5 08:14:05 EDT 2008<br /> MACHINE: i686 (1995 Mhz)<br /> MEMORY: 5.5 GB<br /> PANIC: "kernel BUG at lib/list_debug.c:65!"<br /> PID: 25206<br /> COMMAND: "smbd"<br /> TASK: f4530000 [THREAD_INFO: e0e50000]<br /> CPU: 0<br /> STATE: TASK_RUNNING (PANIC)<br /> <br /> PID: 25206 TASK: f4530000 CPU: 0 COMMAND: "smbd"<br /> #0 [e0e50d5c] die at c04064a9<br /> <a href="http://bugs.centos.org/view.php?id=1">0000001</a> [e0e50d88] do_invalid_op at c0406beb<br /> <a href="http://bugs.centos.org/view.php?id=2">0000002</a> [e0e50e38] error_code (via invalid_op) at c0405a6f<br /> EAX: 00000048 EBX: e67fd828 ECX: 00200092 EDX: 00200000 EBP: e2c47748<br /> DS: 007b ESI: e67fd800 ES: 007b EDI: e0e50f34<br /> CS: 0060 EIP: c04e6ae4 ERR: ffffffff EFLAGS: 00210046<br /> #3 [e0e50e6c] crc32_le at c04e6ae4<br /> #4 [e0e50e80] free_uid at c042e444<br /> #5 [e0e50e8c] rm_from_queue at c042e8c4<br /> #6 [e0e50e94] __dequeue_signal at c042ea96<br /> <a href="http://bugs.centos.org/view.php?id=7">0000007</a> [e0e50eb8] dequeue_signal at c042fe22<br /> <a href="http://bugs.centos.org/view.php?id=8">0000008</a> [e0e50ecc] get_signal_to_deliver at c0430143<br /> <a href="http://bugs.centos.org/view.php?id=9">0000009</a> [e0e50eec] do_notify_resume at c0404566<br /> <a href="http://bugs.centos.org/view.php?id=10">0000010</a> [e0e50fb8] system_call at c0404f89<br /> EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: ffffffff<br /> DS: 007b ESI: bffabff8 ES: 007b EDI: 00ae3ff4<br /> SS: 007b ESP: bffabfc4 EBP: bffabfec<br /> CS: 0073 EIP: 00f0c410 ERR: 000000d0 EFLAGS: 00200246<br /> <br /> <br /> <br /> ------------[ cut here ]------------<br /> kernel BUG at lib/list_debug.c:65!<br /> invalid opcode: 0000 [<a href="http://bugs.centos.org/view.php?id=1">0000001</a>]<br /> SMP<br /> last sysfs file: /devices/pci0000:00/0000:00:00.0/irq<br /> Modules linked in: autofs4 i2c_isa hidp l2cap bluetooth sunrpc xfs(U) dm_multipath video sbs backlight i2c_ec button battery asus_acpi ac ipv6 xfrm_nalgo cry<br /> pto_api parport_pc lp parport e1000e i2c_i801 sg ide_cd i2c_core cdrom pcspkr i5000_edac serio_raw edac_mc dm_snapshot dm_zero dm_mirror dm_mod ata_piix liba<br /> ta mptspi mptscsih scsi_transport_spi mptbase aacraid sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd<br /> CPU: 0<br /> EIP: 0060:[<c04e6ae4>] Tainted: G VLI<br /> EFLAGS: 00210046 (2.6.18-92.1.10.el5PAE <a href="http://bugs.centos.org/view.php?id=1">0000001</a>)<br /> EIP is at list_del+0x18/0x5c<br /> eax: 00000048 ebx: e67fd828 ecx: 00200092 edx: 00200000<br /> esi: e67fd800 edi: e0e50f34 ebp: e2c47748 esp: e0e50e70<br /> ds: 007b es: 007b ss: 0068<br /> Process smbd (pid: 25206, ti=e0e50000 task=f4530000 task.ti=e0e50000)<br /> Stack: c06370e3 e67fd828 0000ffff 00200086 c042e449 e2c47748 e2c47774 c042e8c9<br /> ef250ad8 c042ea9b 0000000a 0000000a 00000009 00000000 00000000 e0e50f14<br /> f4530000 f4530454 c042fe27 00000000 bffabff8 00ae3ff4 e0e50fbc c0430148<br /> Call Trace:<br /> [<c042e449>] free_uid+0x21/0x50<br /> [<c042e8c9>] __sigqueue_free+0x1e/0x2d<br /> [<c042ea9b>] __dequeue_signal+0x101/0x150<br /> [<c042fe27>] dequeue_signal+0x2d/0xa8<br /> [<c0430148>] get_signal_to_deliver+0x114/0x39f<br /> [<c040456b>] do_notify_resume+0x77/0x67d<br /> [<c041ee78>] __wake_up_common+0x2f/0x53<br /> [<c060741c>] schedule+0x920/0x9cd<br /> [<c060741c>] schedule+0x920/0x9cd<br /> [<c0608be8>] _spin_unlock_irqrestore+0x8/0x9<br /> [<c044938b>] audit_syscall_exit+0x2cc/0x2e2<br /> [<c0404f8e>] work_notifysig+0x13/0x19<br /> =======================<br /> Code: 51 04 8d 46 0c 5b 5e 5f e9 62 00 00 00 89 c3 eb eb 90 90 53 89 c3 8b 40 04 8b 00 39 d8 74 17 50 53 68 e3 70 63 c0 e8 b7 ff f3 ff <0f> 0b 41 00 20 71 63<br /> c0 83 c4 0c 8b 03 8b 40 04 39 d8 74 17 50<br /> EIP: [<c04e6ae4>] list_del+0x18/0x5c SS:ESP 0068:e0e50e70