0006891: CentOS 6.5: ext4 lockdep stack trace

After updating CentOS 6.5 with new kernel vmlinuz-2.6.32-431.3.1.el6.x86_64, server does not boot.<br /> Installing kernel debug package, i can see this trace:<br /> <br /> =======================================================<br /> [ INFO: possible circular locking dependency detected ]<br /> 2.6.32-431.3.1.el6.x86_64.debug <a href="http://bugs.centos.org/view.php?id=1">0000001</a><br /> -------------------------------------------------------<br /> dd/739 is trying to acquire lock:<br /> (&s->s_dquot.dqio_mutex){+.+...}, at: [<ffffffff8120997b>] dquot_commit+0x3b/0xf0<br /> <br /> but task is already holding lock:<br /> (&s->s_dquot.dqptr_sem){++++..}, at: [<ffffffff8120a9a3>] __dquot_free_space+0x83/0x2f0<br /> <br /> which lock already depends on the new lock.<br /> <br /> <br /> the existing dependency chain (in reverse order) is:<br /> <br /> -> <a href="http://bugs.centos.org/view.php?id=2">0000002</a> (&s->s_dquot.dqptr_sem){++++..}:<br /> [<ffffffff810bc5de>] __lock_acquire+0xede/0x1560<br /> [<ffffffff810bcd04>] lock_acquire+0xa4/0x120<br /> [<ffffffff8155cf21>] down_read+0x51/0xa0<br /> [<ffffffff8120a9a3>] __dquot_free_space+0x83/0x2f0<br /> [<ffffffff8120ac40>] dquot_free_space+0x10/0x20<br /> [<ffffffffa00928ce>] ext4_free_blocks+0x79e/0x900 [ext4]<br /> [<ffffffffa008a15a>] ext4_ext_remove_space+0xa9a/0xdc0 [ext4]<br /> [<ffffffffa008aa30>] ext4_ext_truncate+0x190/0x1f0 [ext4]<br /> [<ffffffffa0067d48>] ext4_truncate+0x4d8/0x6b0 [ext4]<br /> [<ffffffff8114ec96>] vmtruncate+0x56/0x70<br /> [<ffffffff811c5240>] inode_setattr+0x30/0x60<br /> [<ffffffffa006802c>] ext4_setattr+0x10c/0x330 [ext4]<br /> [<ffffffff811c5628>] notify_change+0x168/0x340<br /> [<ffffffff811a47d6>] do_truncate+0x66/0xa0<br /> [<ffffffff811b9751>] do_filp_open+0x861/0xd30<br /> [<ffffffff811a3489>] do_sys_open+0x69/0x140<br /> [<ffffffff811a35a0>] sys_open+0x20/0x30<br /> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br /> <br /> -> <a href="http://bugs.centos.org/view.php?id=1">0000001</a> (&ei->i_data_sem){++++..}:<br /> [<ffffffff810bc5de>] __lock_acquire+0xede/0x1560<br /> [<ffffffff810bcd04>] lock_acquire+0xa4/0x120<br /> [<ffffffff8155cf21>] down_read+0x51/0xa0<br /> [<ffffffffa0064f06>] ext4_get_blocks+0x56/0x2a0 [ext4]<br /> [<ffffffffa0066bc9>] ext4_getblk+0x79/0x1f0 [ext4]<br /> [<ffffffffa0066d58>] ext4_bread+0x18/0x90 [ext4]<br /> [<ffffffffa007bc58>] ext4_quota_read+0xd8/0x140 [ext4]<br /> [<ffffffff8120dc4b>] v2_read_header+0x1b/0x40<br /> [<ffffffff8120e09c>] v2_read_file_info+0x2c/0x1a0<br /> [<ffffffff8120cb36>] vfs_load_quota_inode+0x236/0x710<br /> [<ffffffff8120d3e8>] vfs_quota_on_path+0x78/0x80<br /> [<ffffffffa007985d>] ext4_quota_on+0x12d/0x1c0 [ext4]<br /> [<ffffffff8120fc16>] do_quotactl+0x286/0x4c0<br /> [<ffffffff81210064>] sys_quotactl+0x214/0x410<br /> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br /> <br /> -> #0 (&s->s_dquot.dqio_mutex){+.+...}:<br /> [<ffffffff810bcb5a>] __lock_acquire+0x145a/0x1560<br /> [<ffffffff810bcd04>] lock_acquire+0xa4/0x120<br /> [<ffffffff8155c5ec>] __mutex_lock_common+0x4c/0x400<br /> [<ffffffff8155cb0a>] mutex_lock_nested+0x4a/0x60<br /> [<ffffffff8120997b>] dquot_commit+0x3b/0xf0<br /> [<ffffffffa007b91e>] ext4_write_dquot+0x5e/0x80 [ext4]<br /> [<ffffffffa007b979>] ext4_mark_dquot_dirty+0x39/0x60 [ext4]<br /> [<ffffffff8120ab69>] __dquot_free_space+0x249/0x2f0<br /> [<ffffffff8120ac40>] dquot_free_space+0x10/0x20<br /> [<ffffffffa00928ce>] ext4_free_blocks+0x79e/0x900 [ext4]<br /> [<ffffffffa008a15a>] ext4_ext_remove_space+0xa9a/0xdc0 [ext4]<br /> [<ffffffffa008aa30>] ext4_ext_truncate+0x190/0x1f0 [ext4]<br /> [<ffffffffa0067d48>] ext4_truncate+0x4d8/0x6b0 [ext4]<br /> [<ffffffff8114ec96>] vmtruncate+0x56/0x70<br /> [<ffffffff811c5240>] inode_setattr+0x30/0x60<br /> [<ffffffffa006802c>] ext4_setattr+0x10c/0x330 [ext4]<br /> [<ffffffff811c5628>] notify_change+0x168/0x340<br /> [<ffffffff811a47d6>] do_truncate+0x66/0xa0<br /> [<ffffffff811b9751>] do_filp_open+0x861/0xd30<br /> [<ffffffff811a3489>] do_sys_open+0x69/0x140<br /> [<ffffffff811a35a0>] sys_open+0x20/0x30<br /> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b<br /> <br /> other info that might help us debug this:<br /> <br /> 6 locks held by dd/739:<br /> #0: (sb_writers){.+.+.+}, at: [<ffffffff811c91a4>] mnt_want_write+0x24/0x50<br /> <a href="http://bugs.centos.org/view.php?id=1">0000001</a>: (&sb->s_type->i_mutex_key#11){+.+.+.}, at: [<ffffffff811a47ca>] do_truncate+0x5a/0xa0<br /> <a href="http://bugs.centos.org/view.php?id=2">0000002</a>: (&sb->s_type->i_alloc_sem_key#5){+.+...}, at: [<ffffffff811c570d>] notify_change+0x24d/0x340<br /> #3: (jbd2_handle){+.+...}, at: [<ffffffffa003849e>] start_this_handle+0x3ee/0x5f0 [jbd2]<br /> #4: (&ei->i_data_sem){++++..}, at: [<ffffffffa008a9e6>] ext4_ext_truncate+0x146/0x1f0 [ext4]<br /> #5: (&s->s_dquot.dqptr_sem){++++..}, at: [<ffffffff8120a9a3>] __dquot_free_space+0x83/0x2f0<br /> <br /> stack backtrace:<br /> Pid: 739, comm: dd Not tainted 2.6.32-431.3.1.el6.x86_64.debug <a href="http://bugs.centos.org/view.php?id=1">0000001</a><br /> Call Trace:<br /> [<ffffffff810b9cf3>] ? print_circular_bug+0xf3/0x100<br /> [<ffffffff810bcb5a>] ? __lock_acquire+0x145a/0x1560<br /> [<ffffffff810bcd04>] ? lock_acquire+0xa4/0x120<br /> [<ffffffff8120997b>] ? dquot_commit+0x3b/0xf0<br /> [<ffffffff8155c5ec>] ? __mutex_lock_common+0x4c/0x400<br /> [<ffffffff8120997b>] ? dquot_commit+0x3b/0xf0<br /> [<ffffffff810a8bd5>] ? sched_clock_local+0x25/0x90<br /> [<ffffffff8120997b>] ? dquot_commit+0x3b/0xf0<br /> [<ffffffff810b74dd>] ? trace_hardirqs_off+0xd/0x10<br /> [<ffffffff810a8e3f>] ? cpu_clock+0x6f/0x80<br /> [<ffffffff8155cb0a>] ? mutex_lock_nested+0x4a/0x60<br /> [<ffffffff8120997b>] ? dquot_commit+0x3b/0xf0<br /> [<ffffffffa007b91e>] ? ext4_write_dquot+0x5e/0x80 [ext4]<br /> [<ffffffffa007b979>] ? ext4_mark_dquot_dirty+0x39/0x60 [ext4]<br /> [<ffffffff8120ab69>] ? __dquot_free_space+0x249/0x2f0<br /> [<ffffffff8120ac40>] ? dquot_free_space+0x10/0x20<br /> [<ffffffffa00928ce>] ? ext4_free_blocks+0x79e/0x900 [ext4]<br /> [<ffffffffa008a15a>] ? ext4_ext_remove_space+0xa9a/0xdc0 [ext4]<br /> [<ffffffffa008a9e6>] ? ext4_ext_truncate+0x146/0x1f0 [ext4]<br /> [<ffffffffa008aa30>] ? ext4_ext_truncate+0x190/0x1f0 [ext4]<br /> [<ffffffff810a8cf8>] ? sched_clock_cpu+0xb8/0x110<br /> [<ffffffff810b74dd>] ? trace_hardirqs_off+0xd/0x10<br /> [<ffffffffa0067d48>] ? ext4_truncate+0x4d8/0x6b0 [ext4]<br /> [<ffffffff8155e51b>] ? _spin_unlock+0x2b/0x40<br /> [<ffffffff81160bd2>] ? unmap_mapping_range+0xb2/0x150<br /> [<ffffffff8114ec96>] ? vmtruncate+0x56/0x70<br /> [<ffffffff811c5240>] ? inode_setattr+0x30/0x60<br /> [<ffffffffa006802c>] ? ext4_setattr+0x10c/0x330 [ext4]<br /> [<ffffffff811c570d>] ? notify_change+0x24d/0x340<br /> [<ffffffff811c5628>] ? notify_change+0x168/0x340<br /> [<ffffffff811a47d6>] ? do_truncate+0x66/0xa0<br /> [<ffffffff811b9751>] ? do_filp_open+0x861/0xd30<br /> [<ffffffff8155e51b>] ? _spin_unlock+0x2b/0x40<br /> [<ffffffff811c654b>] ? alloc_fd+0xab/0x160<br /> [<ffffffff811a3489>] ? do_sys_open+0x69/0x140<br /> [<ffffffff8155df72>] ? trace_hardirqs_on_thunk+0x3a/0x3f<br /> [<ffffffff811a35a0>] ? sys_open+0x20/0x30<br /> [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b<br /> Adding 1048568k swap on /dev/sda2. Priority:-1 extents:1 across:1048568k <br /> Adding 1048568k swap on /dev/sdb2. Priority:-2 extents:1 across:1048568k