Memory leak related to org.jboss.threads.JBossThreadFactory

[dmlloyd]

We are using jboss-cli-client.jar inside of Jolokia to extract metrics from several JBoss server instances using JMX.

We use Jolokia in proxy mode, it means that we connect to Jolokia using a HTTP/JSON API and Jolokia uses jmx-remoting to obtain the info from the JBoss servers.

After some days running and several thousands queries the Jolokia server runs out of heap memory (java.lang.OutOfMemoryError: GC overhead limit exceeded).

Looking at the different memory spaces, PS Old Gen reaches the max value and did not release.

Analyzing the heap dump with Eclipse MAT: attached image img-2019-03-27-203310.png

One instance of "java.lang.ThreadGroup" loaded by "<system class loader>" occupies 20.846.512 (41,08%) bytes. The instance is referenced by org.jboss.threads.JBossThreadFactory @ 0xe093f8e8 , loaded by "org.jboss.modules.ModuleClassLoader @ 0xe0814138". The memory is accumulated in one instance of "java.lang.ThreadGroup[]" loaded by "<system class loader>".

We were using jboss-threads 2.1.2.Final-redhat-1 (which comes with jboss-cli-client.jar in JBoss EAP 6). After seeing this fix we upgraded jboss-cli-client.jar to use version 2.3.2.Final-redhat-1 (we take that lib from the newer jboss-cli-client.jar of JBoss EAP 7 and put into the JBoss EAP 6 client, maybe that surgery is not really updating the lib used?)

I tried to compile jboss eap 6 from the sources but got a bunch of different errors.

I have also open an issue with Jolokia but as the problem comes from the jboss lib, looks like nothing can be done in that part.

Thanks!

This issue was imported from JIRA. The original issue is JBTHR-72.

[dmlloyd]

Could you please share the JDK version you are using?

This comment was imported from JIRA. It was originally authored by @dmlloyd.

[dmlloyd]

Also - is it possible that your application has a very large number of active threads?

This comment was imported from JIRA. It was originally authored by @dmlloyd.

[dmlloyd]

java version "1.7.0_80"

Java(TM) SE Runtime Environment (build 1.7.0_80-b15)

Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

The application you mean Jolokia?

From the comment on the issue I wrote on Jolokia:

Jolokia itself does not start any threads

This comment was imported from JIRA.

[dmlloyd]

No, Jolokia doesn't matter here, or it shouldn't matter at least. But this problem can only occur if there is a thread pool which is growing without bound, with the threads not exiting (because entries are automatically deleted from the thread group when the corresponding thread exits). This might be due to a thread pool misconfiguration, but it would be within the Remoting or (more likely) in the JBoss management client configuration, and I'm not sure how this would be configured in your case.

You might try upgrading to a newer client JAR, though I think your options will be limited due to using Java 7.

This comment was imported from JIRA. It was originally authored by @dmlloyd.

[dmlloyd]

Jolokia uses the remoting libs from the client JAR to connect to the JMX endpoint of the JBoss servers.

In our case, Jolokia is running alone in one JBoss server.

As far as I understand, Jolokia only want to start a connection to a JMX endpoint using the remoting protocol, then some of the jboss/remoting libs executes some code to make this happen. This code starts some threads, using jboss-threads, and don't release them.

So maybe this issue should go to the remoting area.

I tried to use the client JAR from JBoss EAP 7.x, but in that version the remoting protocol goes over HTTP, so it's not working with EAP 6 (maybe I'm missing some configuration bits to allow this)

This comment was imported from JIRA.

[dmlloyd]

Closed the wrong issue 😉

This comment was imported from JIRA. It was originally authored by @dmlloyd.