The total file descriptors number of rabbitmq is 1024 by default. This value is not big enough obviously when your openstack cluster grows. If the file descriptor or socket descriptor reaches its limit, the rabbitmq server will close incoming connections. and you will see the error below:
AMQP server on controller1:5672 is unreachable: timed out. Trying again in 1 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 2 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 4 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 6 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 8 seconds. AMQP server controller1:5672 closed the connection. Check login credentials: Socket closed AMQP server on controller1:5672 is unreachable: timed out. Trying again in 12 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 14 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 16 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 18 seconds. AMQP server on controller1:5672 is unreachable: timed out. Trying again in 20 seconds. AMQP server controller1:5672 closed the connection. Check login credentials: Socket closed
To check the current status of file descriptors used, you can run command ‘service rabbitmq-server status’
the result is