We have two jmx config files that match (via autodiscovery) a single JVM. During startup, we sometime see errors in the datadog agent log like (note that there are two different checks, check1 and check2):
2019-10-22 15:05:31 UTC | CORE | INFO | (pkg/jmxfetch/jmxfetch.go:216 in func1) | 2019-10-22 15:05:31,684 | INFO | App | Could not initialize instance: check1-x.x.x.x-yyyy: java.util.concurrent.ExecutionException: java.io.IOException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: x.x.x.x; nested exception is:
2019-10-22 15:05:31 UTC | CORE | INFO | (pkg/jmxfetch/jmxfetch.go:216 in func1) | 2019-10-22 15:05:31,684 | INFO | App | Could not initialize instance: check2-x.x.x.x-yyyy: java.util.concurrent.ExecutionException: java.io.IOException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: x.x.x.x; nested exception is:
Looking at App.java, two lines after that message, the instance is put in brokenInstanceMap:
|
log.info( |
|
"Could not initialize instance: " + instance.getName() |
|
+ ": " + e.toString()); |
|
instance.cleanUpAsync(); |
|
brokenInstanceMap.put(instance.toString(), instance); |
In our case, instance.toString() resolves to ${host}:${port}, which is not unique across the two checks:
|
return this.instanceMap.get("host") + ":" + this.instanceMap.get("port"); |
We tried to set Instance.getName() to something unique (see below), but that didn't solve the problem because Instance.toString() doesn't take into account the name...
instances:
- host: '%%host%%'
port: '9999'
name: 'check1-%%host%%-9999'A workaround is to combine the two JMX check configs into a single file.
We have two jmx config files that match (via autodiscovery) a single JVM. During startup, we sometime see errors in the datadog agent log like (note that there are two different checks,
check1andcheck2):Looking at App.java, two lines after that message, the instance is put in
brokenInstanceMap:jmxfetch/src/main/java/org/datadog/jmxfetch/App.java
Lines 1032 to 1036 in f59dace
In our case,
instance.toString()resolves to${host}:${port}, which is not unique across the two checks:jmxfetch/src/main/java/org/datadog/jmxfetch/Instance.java
Line 395 in f59dace
We tried to set
Instance.getName()to something unique (see below), but that didn't solve the problem becauseInstance.toString()doesn't take into account the name...A workaround is to combine the two JMX check configs into a single file.