0005280: RAM hotplug under VMWare sometimes works, sometimes results in kernel stack trace

I have two machines that are clones on VMWare, but running on different ESX clusters. On both, I go to increment the RAM by 2G for a hotadd.<br /> <br /> Works:<br /> VMCIUtil: Updating context id from 0xffffffff to 0xd22b5ac5 on event 0.<br /> Hotplug Mem Device <br /> init_memory_mapping: 0000000240000000-00000002c0000000<br /> 0240000000 - 02c0000000 page 2M<br /> [ffffea0007e00000-ffffea0007ffffff] PMD -> [ffff880234c00000-ffff880234dfffff] on node 0<br /> [ffffea0008000000-ffffea00081fffff] PMD -> [ffff880234400000-ffff8802345fffff] on node 0<br /> [ffffea0008200000-ffffea00083fffff] PMD -> [ffff880234000000-ffff8802341fffff] on node 0<br /> [ffffea0008400000-ffffea00085fffff] PMD -> [ffff880233c00000-ffff880233dfffff] on node 0<br /> [ffffea0008600000-ffffea00089fffff] PMD -> [ffff8801ea400000-ffff8801ea7fffff] on node 0<br /> [ffffea0008a00000-ffffea0008dfffff] PMD -> [ffff8801ea000000-ffff8801ea3fffff] on node 0<br /> [ffffea0008e00000-ffffea00091fffff] PMD -> [ffff8801e9c00000-ffff8801e9ffffff] on node 0<br /> [ffffea0009200000-ffffea00095fffff] PMD -> [ffff8801e9800000-ffff8801e9bfffff] on node 0<br /> <br /> <br /> Does Not Work:<br /> VMCIUtil: Updating context id from 0xffffffff to 0x23bccbd9 on event 0.<br /> Hotplug Mem Device <br /> init_memory_mapping: 00000002c0000000-0000000340000000<br /> 02c0000000 - 0340000000 page 2M<br /> [ffffea0009a00000-ffffea0009bfffff] PMD -> [ffff88000a200000-ffff88000a3fffff] on node 0<br /> [ffffea0009c00000-ffffea0009dfffff] PMD -> [ffff88000ac00000-ffff88000adfffff] on node 0<br /> [ffffea0009e00000-ffffea0009ffffff] PMD -> [ffff880100800000-ffff8801009fffff] on node 0<br /> [ffffea000a000000-ffffea000a1fffff] PMD -> [ffff880100000000-ffff8801001fffff] on node 0<br /> [ffffea000a200000-ffffea000a5fffff] PMD -> [ffff880100a00000-ffff880100dfffff] on node 0<br /> [ffffea000a600000-ffffea000a7fffff] PMD -> [ffff880101600000-ffff8801017fffff] on node 0<br /> kacpi_notify: page allocation failure. order:9, mode:0x80d0<br /> Pid: 23, comm: kacpi_notify Not tainted 2.6.32-71.29.1.el6.x86_64 <a href="http://bugs.centos.org/view.php?id=1">0000001</a><br /> Call Trace:<br /> [<ffffffff8111eab6>] __alloc_pages_nodemask+0x706/0x850<br /> [<ffffffff814c844e>] vmemmap_alloc_block+0x6c/0xec<br /> [<ffffffff814c6c40>] vmemmap_populate+0x135/0x27c<br /> [<ffffffff814c836f>] sparse_mem_map_populate+0x31/0x43<br /> [<ffffffff814c822d>] sparse_add_one_section+0x51/0x162<br /> [<ffffffff814b40ce>] __add_pages+0xbe/0x280<br /> [<ffffffff81045671>] arch_add_memory+0x71/0x100<br /> [<ffffffff812b19b3>] ? acpi_os_signal_semaphore+0x28/0x2c<br /> [<ffffffff814b446b>] add_memory+0xcb/0x210<br /> [<ffffffff812bbe61>] ? acpi_get_pxm+0x4a/0x5c<br /> [<ffffffff812de00d>] acpi_memory_enable_device+0x95/0x131<br /> [<ffffffff812de52d>] acpi_memory_device_add+0x160/0x169<br /> [<ffffffff812b4874>] acpi_device_probe+0x50/0x122<br /> [<ffffffff8132fa90>] driver_probe_device+0xa0/0x2a0<br /> [<ffffffff8132fd40>] ? __device_attach+0x0/0x60<br /> [<ffffffff8132fd40>] ? __device_attach+0x0/0x60<br /> [<ffffffff8132fd93>] __device_attach+0x53/0x60<br /> [<ffffffff8132e974>] bus_for_each_drv+0x64/0x90<br /> [<ffffffff8132fe5c>] device_attach+0x8c/0xa0<br /> [<ffffffff8132e71d>] bus_probe_device+0x2d/0x50<br /> [<ffffffff8132cb77>] device_add+0x527/0x640<br /> [<ffffffff81335dfb>] ? pm_runtime_init+0xcb/0xe0<br /> [<ffffffff8132ccae>] device_register+0x1e/0x30<br /> [<ffffffff812b5727>] acpi_add_single_object+0x84f/0x9fa<br /> [<ffffffff812d64f3>] ? acpi_ut_release_mutex+0x63/0x67<br /> [<ffffffff812b59b2>] acpi_bus_check_add+0xe0/0x138<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812b5a44>] acpi_bus_scan+0x3a/0x71<br /> [<ffffffff812b5ada>] acpi_bus_add+0x2a/0x2e<br /> [<ffffffff812de1f0>] acpi_memory_device_notify+0xa6/0x24f<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812b3d43>] ? acpi_bus_get_device+0x2a/0x3d<br /> [<ffffffff812b3de2>] ? acpi_bus_notify+0x4b/0x82<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812c1f1a>] acpi_ev_notify_dispatch+0x64/0x72<br /> [<ffffffff812b1b05>] acpi_os_execute_deferred+0x29/0x36<br /> [<ffffffff8108c7d0>] worker_thread+0x170/0x2a0<br /> [<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40<br /> [<ffffffff8108c660>] ? worker_thread+0x0/0x2a0<br /> [<ffffffff81091b36>] kthread+0x96/0xa0<br /> [<ffffffff810141ca>] child_rip+0xa/0x20<br /> [<ffffffff81091aa0>] ? kthread+0x0/0xa0<br /> [<ffffffff810141c0>] ? child_rip+0x0/0x20<br /> Mem-Info:<br /> Node 0 DMA per-cpu:<br /> CPU 0: hi: 0, btch: 1 usd: 0<br /> CPU 1: hi: 0, btch: 1 usd: 0<br /> Node 0 DMA32 per-cpu:<br /> CPU 0: hi: 186, btch: 31 usd: 0<br /> CPU 1: hi: 186, btch: 31 usd: 0<br /> Node 0 Normal per-cpu:<br /> CPU 0: hi: 186, btch: 31 usd: 0<br /> CPU 1: hi: 186, btch: 31 usd: 46<br /> active_anon:13656 inactive_anon:4107 isolated_anon:0<br /> active_file:1124402 inactive_file:1125173 isolated_file:268<br /> unevictable:0 dirty:482 writeback:0 unstable:0<br /> free:106110 slab_reclaimable:82911 slab_unreclaimable:48236<br /> mapped:1139350 shmem:101 pagetables:5087 bounce:0<br /> Node 0 DMA free:15672kB min:96kB low:120kB high:144kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15276kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no<br /> lowmem_reserve[]: 0 3000 10070 10070<br /> Node 0 DMA32 free:145108kB min:20104kB low:25128kB high:30156kB active_anon:0kB inactive_anon:648kB active_file:1250732kB inactive_file:1251716kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:380kB writeback:0kB mapped:1340100kB shmem:0kB slab_reclaimable:117156kB slab_unreclaimable:592kB kernel_stack:0kB pagetables:3104kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no<br /> lowmem_reserve[]: 0 0 7070 7070<br /> Node 0 Normal free:263660kB min:47376kB low:59220kB high:71064kB active_anon:54624kB inactive_anon:15780kB active_file:3246876kB inactive_file:3248976kB unevictable:0kB isolated(anon):0kB isolated(file):1072kB present:7239680kB mlocked:0kB dirty:1548kB writeback:0kB mapped:3217128kB shmem:404kB slab_reclaimable:214488kB slab_unreclaimable:192352kB kernel_stack:1576kB pagetables:17244kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:268 all_unreclaimable? no<br /> lowmem_reserve[]: 0 0 0 0<br /> Node 0 DMA: 2*4kB 2*8kB 2*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15672kB<br /> Node 0 DMA32: 12843*4kB 3327*8kB 1069*16kB 501*32kB 169*64kB 61*128kB 26*256kB 7*512kB 3*1024kB 1*2048kB 0*4096kB = 145108kB<br /> Node 0 Normal: 20809*4kB 8943*8kB 2467*16kB 509*32kB 412*64kB 69*128kB 26*256kB 14*512kB 4*1024kB 0*2048kB 0*4096kB = 263660kB<br /> 2250432 total pagecache pages<br /> 533 pages in swap cache<br /> Swap cache stats: add 2301, delete 1768, find 5462/5556<br /> Free swap = 4091248kB<br /> Total swap = 4095992kB<br /> 2981872 pages RAM<br /> 449549 pages reserved<br /> 3144977 pages shared<br /> 374870 pages non-shared<br /> ------------[ cut here ]------------<br /> WARNING: at arch/x86/mm/init_64.c:694 arch_add_memory+0xe2/0x100() (Not tainted)<br /> Hardware name: VMware Virtual Platform<br /> Modules linked in: dm_snapshot nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc autofs4 ipv6 vsock(U) xfs exportfs dm_mirror dm_region_hash dm_log ppdev parport_pc parport pcnet32 vmxnet(U) mii vmware_balloon sg i2c_piix4 i2c_core vmci(U) shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom mptsas mptscsih mptbase scsi_transport_sas vmw_pvscsi pata_acpi ata_generic ata_piix dm_mod [last unloaded: scsi_wait_scan]<br /> Pid: 23, comm: kacpi_notify Not tainted 2.6.32-71.29.1.el6.x86_64 <a href="http://bugs.centos.org/view.php?id=1">0000001</a><br /> Call Trace:<br /> [<ffffffff8106b947>] warn_slowpath_common+0x87/0xc0<br /> [<ffffffff8106b99a>] warn_slowpath_null+0x1a/0x20<br /> [<ffffffff810456e2>] arch_add_memory+0xe2/0x100<br /> [<ffffffff814b446b>] add_memory+0xcb/0x210<br /> [<ffffffff812bbe61>] ? acpi_get_pxm+0x4a/0x5c<br /> [<ffffffff812de00d>] acpi_memory_enable_device+0x95/0x131<br /> [<ffffffff812de52d>] acpi_memory_device_add+0x160/0x169<br /> [<ffffffff812b4874>] acpi_device_probe+0x50/0x122<br /> [<ffffffff8132fa90>] driver_probe_device+0xa0/0x2a0<br /> [<ffffffff8132fd40>] ? __device_attach+0x0/0x60<br /> [<ffffffff8132fd40>] ? __device_attach+0x0/0x60<br /> [<ffffffff8132fd93>] __device_attach+0x53/0x60<br /> [<ffffffff8132e974>] bus_for_each_drv+0x64/0x90<br /> [<ffffffff8132fe5c>] device_attach+0x8c/0xa0<br /> [<ffffffff8132e71d>] bus_probe_device+0x2d/0x50<br /> [<ffffffff8132cb77>] device_add+0x527/0x640<br /> [<ffffffff81335dfb>] ? pm_runtime_init+0xcb/0xe0<br /> [<ffffffff8132ccae>] device_register+0x1e/0x30<br /> [<ffffffff812b5727>] acpi_add_single_object+0x84f/0x9fa<br /> [<ffffffff812d64f3>] ? acpi_ut_release_mutex+0x63/0x67<br /> [<ffffffff812b59b2>] acpi_bus_check_add+0xe0/0x138<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812b5a44>] acpi_bus_scan+0x3a/0x71<br /> [<ffffffff812b5ada>] acpi_bus_add+0x2a/0x2e<br /> [<ffffffff812de1f0>] acpi_memory_device_notify+0xa6/0x24f<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812b3d43>] ? acpi_bus_get_device+0x2a/0x3d<br /> [<ffffffff812b3de2>] ? acpi_bus_notify+0x4b/0x82<br /> [<ffffffff812b1adc>] ? acpi_os_execute_deferred+0x0/0x36<br /> [<ffffffff812c1f1a>] acpi_ev_notify_dispatch+0x64/0x72<br /> [<ffffffff812b1b05>] acpi_os_execute_deferred+0x29/0x36<br /> [<ffffffff8108c7d0>] worker_thread+0x170/0x2a0<br /> [<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40<br /> [<ffffffff8108c660>] ? worker_thread+0x0/0x2a0<br /> [<ffffffff81091b36>] kthread+0x96/0xa0<br /> [<ffffffff810141ca>] child_rip+0xa/0x20<br /> [<ffffffff81091aa0>] ? kthread+0x0/0xa0<br /> [<ffffffff810141c0>] ? child_rip+0x0/0x20<br /> ---[ end trace 773400a64c048684 ]---<br /> ACPI:memory_hp:add_memory failed<br /> ACPI:memory_hp:Error in acpi_memory_enable_device<br /> acpi_memhotplug: probe of PNP0C80:00 failed with error -22<br /> <br /> driver data not found<br /> ACPI:memory_hp:Cannot find driver data