PDA

View Full Version : SLES 11 SP3 SP3 3.0.101-0.47.79 kernel deadlock (dump_trace+0x75/0x300)



hernan_fernandez
28-Jul-2016, 20:57
Hi friends,

I have a SLES sp3 kernel version "Linux sepcomtas01 3.0.101-0.47.79-default #1 SMP Fri Apr 22 06:30:00 UTC 2016 (9d3a4e8) x86_64 x86_64 x86_64 GNU/Linux". It doesn't seem to be different from any other sles that I have installed. Any ideas?


server installed on vmware

thanks!





Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540574] Call Trace:
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540593] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540603] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540613] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540621] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540627] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540634] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540641] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540648] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540657] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540665] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540674] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540682] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540690] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540716] [<00007f3c2669fee0>] 0x7f3c2669fedf
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900272] Pid: 20048, comm: sapcimb Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900276] Call Trace:
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900297] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900307] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900317] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900325] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900331] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900338] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900345] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900353] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900362] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900370] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900379] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900387] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900395] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900421] [<00007ff19ab017e0>] 0x7ff19ab017df
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817007] Pid: 20060, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817011] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817031] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817041] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817051] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817059] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817065] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817072] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817079] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817086] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817095] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817103] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817112] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817119] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817128] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817168] [<00007fcb4d03eee0>] 0x7fcb4d03eedf
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825429] Pid: 20070, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825433] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825446] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825455] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825463] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825470] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825476] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825483] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825490] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825497] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825505] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825512] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825525] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825550] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825557] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825577] [<00007fada5ef1ee0>] 0x7fada5ef1edf
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849429] Pid: 20076, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849433] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849445] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849454] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849461] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849468] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849474] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849481] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849487] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849494] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849502] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849509] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849516] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849524] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849530] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849545] [<00007f6f0c8afee0>] 0x7f6f0c8afedf
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606921] Pid: 6954, comm: saposcol Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606924] Call Trace:
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606943] [<ffffffff81004b95>] dump_trace+0x75/0x300
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606953] [<ffffffff814641c3>] dump_stack+0x69/0x6f
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606964] [<ffffffff81120372>] print_bad_pte+0x1b2/0x270
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606971] [<ffffffff8112048d>] vm_normal_page+0x5d/0x70
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606977] [<ffffffff8112073e>] follow_page+0x29e/0x4a0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606984] [<ffffffff81125634>] __get_user_pages+0x174/0x560
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606992] [<ffffffff81125bd0>] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606999] [<ffffffff81125d09>] access_process_vm+0x59/0x90
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607009] [<ffffffff811bdd36>] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607016] [<ffffffff811be8a2>] proc_info_read+0xb2/0xf0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607026] [<ffffffff8115f0c7>] vfs_read+0xc7/0x130
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607034] [<ffffffff8115f233>] sys_read+0x53/0xa0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607042] [<ffffffff8146f172>] system_call_fastpath+0x16/0x1b
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607067] [<00007f3c2669fee0>] 0x7f3c2669fedf

jmozdzen
02-Aug-2016, 17:06
Hi hernan_fernandez,

if you have (long-term) support, it might be a good idea to open a service request to have some SUSE engineer pin-point the actual cause of the problem.

Do you have more in the logs than this (ok, the actual deadlock messages are missing, but I'm referring to further errors/warnings reported prior to the deadlock, since last reboot). Maybe the VMware log offers more details as well.

Other than that? I have nothing to point at. You might try to "fix" things by rebooting the VM (although 19 days isn't much uptime, for sure) and/or verify the file systems and installed packages for corruption. But all that is nothing more than a shot in the dark, as would be asking for an extended hardware check to rule out memory errors.

Regards,
J