hrtimer버그로 인한 재부팅 이슈 해결 방법
페이지 정보
본문
OSP10 Compute 노드의 알 수 없는 reboot이 발생.
원인으로 파악된 관련 vmcore 내용
-------------------------------------------------------------
[68179929.277605] kernel BUG at kernel/hrtimer.c:1236!
[68179929.283060] invalid opcode: 0000 [#1] SMP
[68179929.287949] Modules linked in: binfmt_misc sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag nf_conntrack_netlink vhost_net vhost macvtap macvlan tun nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip_set_hash_net ip6table_raw xt_CT xt_mac xt_comment xt_physdev veth iptable_raw drbg ansi_cprng cmac rmd160 crypto_null ip_vti ip_tunnel af_key ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate cts gcm ccm serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64
[68179929.368313] serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_avx_x86_64 twofish_x86_64_3way xts twofish_x86_64 twofish_common xcbc sha256_mb sha512_ssse3 sha512_generic sha512_mb mcryptd des_generic tpm_rng timeriomem_rng br_netfilter bridge stp llc virtio_rng virtio_ring virtio ebtable_filter ebtables ip6table_filter ip6_tables devlink bonding openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter nls_utf8 isofs sb_edac iTCO_wdt iTCO_vendor_support intel_powerclamp dcdbas coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass ipmi_si pcspkr sg ipmi_devintf mei_me ipmi_msghandler acpi_power_meter mei shpchp lpc_ich nfsd auth_rpcgss nfs_acl lockd
[68179929.446684] nf_conntrack grace ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic mxm_wmi mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common fb_sys_fops crc32_pclmul crc32c_intel ttm ghash_clmulni_intel ahci drm aesni_intel libahci lrw gf128mul glue_helper scsi_transport_iscsi i40e ablk_helper cryptd libata tg3 megaraid_sas i2c_core ptp pps_core wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod
[68179929.492177] CPU: 26 PID: 0 Comm: swapper/26 Kdump: loaded Not tainted 3.10.0-862.3.2.el7.x86_64 #1
[68179929.502489] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.7.1 001/22/2018
[68179929.511247] task: ffff93cee6fcbf40 ti: ffff93cee6ffc000 task.ti: ffff93cee6ffc000
[68179929.519907] RIP: 0010:[<ffffffff898bf4ac>] [<ffffffff898bf4ac>] __hrtimer_run_queues+0x25c/0x260
[68179929.530153] RSP: 0018:ffff940d3e943f28 EFLAGS: 00010002
[68179929.536383] RAX: 0000000000000001 RBX: ffff944d3238d010 RCX: 0000000000000001
[68179929.544656] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff940d3e9539a0
[68179929.552928] RBP: ffff940d3e943f70 R08: 0000000000000101 R09: 0000000000000018
[68179929.561200] R10: 0000000000000518 R11: 7fffffffffffffff R12: ffff940d3e9539a0
[68179929.569471] R13: ffff940d3e9539e0 R14: 0000000000000001 R15: ffff940d3e953ad8
[68179929.577758] FS: 0000000000000000(0000) GS:ffff940d3e940000(0000) knlGS:0000000000000000
[68179929.587100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[68179929.593809] CR2: 00007fa83f4cb000 CR3: 00000042fae0e000 CR4: 00000000003627e0
[68179929.602080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[68179929.610351] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[68179929.618622] Call Trace:
[68179929.621648] <IRQ>
[68179929.624093] [<ffffffff898bf8bf>] hrtimer_interrupt+0xaf/0x1d0
[68179929.631117] [<ffffffff89857c8b>] local_apic_timer_interrupt+0x3b/0x60
[68179929.638703] [<ffffffff89f25503>] smp_apic_timer_interrupt+0x43/0x60
[68179929.646102] [<ffffffff89f2189c>] apic_timer_interrupt+0x17c/0x190
[68179929.653306] <EOI>
[68179929.655750] [<ffffffff89d69807>] ? cpuidle_enter_state+0x57/0xd0
[68179929.663060] [<ffffffff89d6995e>] cpuidle_idle_call+0xde/0x230
[68179929.670738] [<ffffffff898353de>] arch_cpu_idle+0xe/0x40
[68179929.677849] [<ffffffff898f2e9a>] cpu_startup_entry+0x14a/0x1e0
[68179929.685612] [<ffffffff89855772>] start_secondary+0x1f2/0x270
[68179929.693204] [<ffffffff898000d5>] start_cpu+0x5/0x14
[68179929.699889] Code: d8 fe ff ff 0f 1f 00 48 8b 4b 28 48 8b 53 48 4c 8d 43 50 8b 73 40 45 31 c9 48 89 df e8 ce 38 04 00 e9 64 fe ff ff e8 44 22 fd ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 55 48 8d 75 d8 41 54
[68179929.723677] RIP [<ffffffff898bf4ac>] __hrtimer_run_queues+0x25c/0x260
[68179929.732137] RSP <ffff940d3e943f28>
-------------------------------------------------------------
vmcore 분석한 결과 hrtimer 버그로 재기동됨.
errata RHSA-2018:2748 또는 RHSA-2018:3083 에 따라
커널을 3.10.0-862.14.4.el7 이상으로 업데이트하여 해결.
관련 문서
https://access.redhat.com/solutions/3432391
원인으로 파악된 관련 vmcore 내용
-------------------------------------------------------------
[68179929.277605] kernel BUG at kernel/hrtimer.c:1236!
[68179929.283060] invalid opcode: 0000 [#1] SMP
[68179929.287949] Modules linked in: binfmt_misc sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag nf_conntrack_netlink vhost_net vhost macvtap macvlan tun nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip_set_hash_net ip6table_raw xt_CT xt_mac xt_comment xt_physdev veth iptable_raw drbg ansi_cprng cmac rmd160 crypto_null ip_vti ip_tunnel af_key ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate cts gcm ccm serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64
[68179929.368313] serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_avx_x86_64 twofish_x86_64_3way xts twofish_x86_64 twofish_common xcbc sha256_mb sha512_ssse3 sha512_generic sha512_mb mcryptd des_generic tpm_rng timeriomem_rng br_netfilter bridge stp llc virtio_rng virtio_ring virtio ebtable_filter ebtables ip6table_filter ip6_tables devlink bonding openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat xt_set ip_set nfnetlink xt_multiport xt_conntrack iptable_filter nls_utf8 isofs sb_edac iTCO_wdt iTCO_vendor_support intel_powerclamp dcdbas coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass ipmi_si pcspkr sg ipmi_devintf mei_me ipmi_msghandler acpi_power_meter mei shpchp lpc_ich nfsd auth_rpcgss nfs_acl lockd
[68179929.446684] nf_conntrack grace ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic mxm_wmi mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common fb_sys_fops crc32_pclmul crc32c_intel ttm ghash_clmulni_intel ahci drm aesni_intel libahci lrw gf128mul glue_helper scsi_transport_iscsi i40e ablk_helper cryptd libata tg3 megaraid_sas i2c_core ptp pps_core wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod
[68179929.492177] CPU: 26 PID: 0 Comm: swapper/26 Kdump: loaded Not tainted 3.10.0-862.3.2.el7.x86_64 #1
[68179929.502489] Hardware name: Dell Inc. PowerEdge R730/0WCJNT, BIOS 2.7.1 001/22/2018
[68179929.511247] task: ffff93cee6fcbf40 ti: ffff93cee6ffc000 task.ti: ffff93cee6ffc000
[68179929.519907] RIP: 0010:[<ffffffff898bf4ac>] [<ffffffff898bf4ac>] __hrtimer_run_queues+0x25c/0x260
[68179929.530153] RSP: 0018:ffff940d3e943f28 EFLAGS: 00010002
[68179929.536383] RAX: 0000000000000001 RBX: ffff944d3238d010 RCX: 0000000000000001
[68179929.544656] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff940d3e9539a0
[68179929.552928] RBP: ffff940d3e943f70 R08: 0000000000000101 R09: 0000000000000018
[68179929.561200] R10: 0000000000000518 R11: 7fffffffffffffff R12: ffff940d3e9539a0
[68179929.569471] R13: ffff940d3e9539e0 R14: 0000000000000001 R15: ffff940d3e953ad8
[68179929.577758] FS: 0000000000000000(0000) GS:ffff940d3e940000(0000) knlGS:0000000000000000
[68179929.587100] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[68179929.593809] CR2: 00007fa83f4cb000 CR3: 00000042fae0e000 CR4: 00000000003627e0
[68179929.602080] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[68179929.610351] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[68179929.618622] Call Trace:
[68179929.621648] <IRQ>
[68179929.624093] [<ffffffff898bf8bf>] hrtimer_interrupt+0xaf/0x1d0
[68179929.631117] [<ffffffff89857c8b>] local_apic_timer_interrupt+0x3b/0x60
[68179929.638703] [<ffffffff89f25503>] smp_apic_timer_interrupt+0x43/0x60
[68179929.646102] [<ffffffff89f2189c>] apic_timer_interrupt+0x17c/0x190
[68179929.653306] <EOI>
[68179929.655750] [<ffffffff89d69807>] ? cpuidle_enter_state+0x57/0xd0
[68179929.663060] [<ffffffff89d6995e>] cpuidle_idle_call+0xde/0x230
[68179929.670738] [<ffffffff898353de>] arch_cpu_idle+0xe/0x40
[68179929.677849] [<ffffffff898f2e9a>] cpu_startup_entry+0x14a/0x1e0
[68179929.685612] [<ffffffff89855772>] start_secondary+0x1f2/0x270
[68179929.693204] [<ffffffff898000d5>] start_cpu+0x5/0x14
[68179929.699889] Code: d8 fe ff ff 0f 1f 00 48 8b 4b 28 48 8b 53 48 4c 8d 43 50 8b 73 40 45 31 c9 48 89 df e8 ce 38 04 00 e9 64 fe ff ff e8 44 22 fd ff <0f> 0b 66 90 0f 1f 44 00 00 55 48 89 e5 41 55 48 8d 75 d8 41 54
[68179929.723677] RIP [<ffffffff898bf4ac>] __hrtimer_run_queues+0x25c/0x260
[68179929.732137] RSP <ffff940d3e943f28>
-------------------------------------------------------------
vmcore 분석한 결과 hrtimer 버그로 재기동됨.
errata RHSA-2018:2748 또는 RHSA-2018:3083 에 따라
커널을 3.10.0-862.14.4.el7 이상으로 업데이트하여 해결.
관련 문서
https://access.redhat.com/solutions/3432391
- 이전글OSP10 RabbitMQ Failed issue 해결 방법 20.10.13
댓글목록
등록된 댓글이 없습니다.