Resource shortage in MQ V7.5
Resource shortage in MQ V7.5 Queue manager in Linux RHEL 6.5 when running large number of clients. Linux machine running with Message queues gets hung and the user that is running the mqm process shows a message frequently “-bash: fork: retry: Resource temporarily unavailable”.
MQ FDC logs will show below message.
| Probe Description :- AMQ6119: An internal WebSphere MQ error has
occurred
| (’11 – Resource temporarily unavailable’ from pthread_create.)
| Arith1 :- 11 (0xb)
| Comment1 :- ’11 – Resource temporarily unavailable’ from
| pthread_create.
.
And it might be followed by:
| Probe Id :- ZS401010
| Component :- zstStartAsyncConsumeThread
| Major Errorcode :- xecP_E_PROC_LIMIT
| Probe Description :- AMQ6026: A resource shortage prevented the creation of a WebSphere MQ process.
Resolution
One of the common troubleshooting point here is to check what is the exact reason in the FDC log file and also based on the reason code proceed with the solution. In my case Kernel level settings were not properly defined and this was the problem here.
As mentioned in the Technote and also in my earlier Blog entry, please update Linux Kernal level settings accordingly and the issue will get resolved.
As mentioned in infocenter,
The minimum configuration for WebSphere MQ for these resources is as follows:
kernel.msgmni = 1024
kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 268435456
kernel.sem = 500 256000 250 1024
fs.file-max = 524288
net.ipv4.tcp_keepalive_time = 300
In addition to the above, there are some other kernel parameters that might be useful, but they are not documented in the Infocenter, such as:
a) kernel.pid_max
By default the Linux kernel has a maximum process identifier, which may also be used with threads and may limit the allowed number of threads. This parameter is mentioned in the following technical documents. The value used was the same in both documents:
kernel.pid_max = 120000
b) kernel.threads-max
According to the Linux manual page, the Operating System reports when the system lacks the necessary resources to create another thread, or the system-imposed limit on the total number of threads in a process {PTHREAD_THREADS_MAX} would be exceeded. If the value of kernel.threads-max is low, increase its value. The command “cat /proc/sys/kernel/threads-max” displays the current value.
The number of threads created by MQ depends on number of application connecting to the queue manager. MQ creates one thread for each local connection (in amqzlaa0) or two threads (one in amqzlaa0 and the other one in amqrmppa) for each client connection. If there are 100 connections active to the queue manager, then the queue manager may need about 200 (amqzlla0 and amqrmppa) + approximately 100 threads (for other MQ processes).
In Linux, the number of active threads can be calculated from the output of the command: ps -elfL
The value used is: kernel.threads-max = 48000
In my case I have used the same value for both a and b as 120000.