OSP10 RabbitMQ Failed issue 해결 방법 > OpenStack 자료실

OSP10 controller node의 RabbitMQ fail issue 발생

분석 내용
---------------------------------------------------------
=ERROR REPORT==== 21-Sep-2020::15:25:47 ===
Error on AMQP connection <0.15590.0> (10.1.76.25:59842 -> 10.1.76.12:5672, vhost: '/', user: 'guest', state: running), channel 0:
operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"

=ERROR REPORT==== 21-Sep-2020::15:25:47 ===
Error on AMQP connection <0.15604.0> (10.1.76.25:59848 -> 10.1.76.12:5672, vhost: '/', user: 'guest', state: running), channel 0:
operation none caused a connection exception connection_forced: "broker forced connection closure with reason 'shutdown'"

$ grep -i rabbit ../../../installed-rpms
puppet-rabbitmq-5.6.0-1.057a013git.el7ost.noarch            Fri Jul 28 04:38:52 2017
rabbitmq-server-3.6.3-6.el7ost.noarch                      Fri Jul 28 04:34:02 2017
$ cat pcs_status
~~~
Failed Actions:
* rabbitmq_start_0 on ggcbsctr-0 'unknown error' (1): call=934, status=Timed Out, exitreason='none',
    last-rc-change='Mon Sep 21 18:26:33 2020', queued=0ms, exec=200211ms



Almost of all memory is used by user applications. there is little memory left.

DMIDECODE
  Memory:
    Total: 65536 MiB (64 GiB)
OS
  Hostname: ggcbsctr-0
  Distro:  [redhat-release] Red Hat Enterprise Linux Server release 7.4 (Maipo)
  Kernel:
    Booted kernel:  3.10.0-862.3.2.el7.x86_64
SYSCTLS
  vm.
    min_free_kbytes =  "90112"
PS CHECK
  Top users of CPU & MEM:
    ceilome+  21.1%  0.9%  0.74 GiB
    neutron  14.0%  16.5%  13.33 GiB
    nova      4.8%  22.1%  16.51 GiB
    mysql    4.7%  10.4%  6.54 GiB
    manila    3.0%  0.4%  0.43 GiB
    gnocchi  1.4%  1.2%  1.09 GiB

$ cat free
              total        used        free      shared  buff/cache  available
Mem:      65687452    55707384      356736      56788    9623332    8969216
Swap:            0          0          0

MEMORY
  RAM:
    62.6 GiB total ram
    62.3 GiB (99%) used
    54.3 GiB (87%) used excluding Buffers/Cached
    0.02 GiB (0%) dirty
  LowMem/Slab/PageTables/Shmem:
    1.17 GiB (2%) of total ram used for Slab
    0.92 GiB (1%) of total ram used for PageTables
    0.05 GiB (0%) of total ram used for Shmem

Linux 3.10.0-862.3.2.el7.x86_64 (ggcbsctr-0)    09/21/20        _x86_64_        (32 CPU)

00:00:02    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit  %commit  kbactive  kbinact  kbdirty
18:40:03      397236  65290216    99.40        0  8294832  85251828    129.78  57025792  4164548      1492                                                                                                   
Average:      505818  65181634    99.23        80  11259615  84316662    128.36  55395639  5657391    24510

------------------------------------------------------------------
로그 및 버전 분석 결과 rabbitmq 버그로 파악
관련문서
 [1] RabbitMQ failed on one controller: auth issues
    https://access.redhat.com/solutions/3546911
 [2] How to disable gnocchi, aodh and ceilometer
    https://access.redhat.com/solutions/3024461
 [3] How to purge rabbitmq messages from notification queue
    https://access.redhat.com/solutions/3363661
에 따라
rabbitmq-server-3.6.3-9.el7ost 이상 버전 업데이트로 해결
권고 받은 rabbitmq-server-3.6.3-9.el7ost로 업데이트 하여 RabbitMQ Queue 목록 초기화 및 MongoDB 서비스 정상화 확인