Not able to fetch JMX Metrics from SDC

Problem

After enabling External JMX Tools for SDC, you are unable to fetch any JMX data from SDC’s JMX socket. For instance, you enabled the JMX socket in your SDC_JAVA_OPTS for 0.0.0.0:3333:

export SDC_JAVA_OPTS="-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=3333 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false"

But connecting to 0.0.0.0:3333 via jconsole, you don’t get any metric data.

Solution

This behavior typically indicates that your Java Security Policy is not granting SDC’s JMX Metrics library access to query the JVM’s runtime information or to connect to the JMX Socket.

At SDC startup, you will see if some Java Security Policy is denying some SDC library from loading due to insufficient permissions (java.security.AccessControlException: access denied):

Exception in thread "Attach Listener" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
Caused by: java.io.IOError: java.security.AccessControlException: access denied ("javax.management.MBeanPermission" "sun.manage
ment.OperatingSystemImpl#-[java.lang:type=OperatingSystem]" "isInstanceOf")
        at jvmmon.core.GcMonitor.<init>(GcMonitor.java:35)
        at jvmmon.core.JvmMon.<init>(JvmMon.java:23)
        at jvmmon.Agent.<init>(Agent.java:23)
        at jvmmon.Agent.agentmain(Agent.java:31)
        ... 6 more
Caused by: java.security.AccessControlException: access denied ("javax.management.MBeanPermission" "sun.management.OperatingSys
temImpl#-[java.lang:type=OperatingSystem]" "isInstanceOf")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.security.AccessController.checkPermission(AccessController.java:886)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1830)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.isInstanceOf(DefaultMBeanServerInterceptor.java:1402)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.isInstanceOf(JmxMBeanServer.java:1091)
        at java.lang.management.ManagementFactory.newPlatformMXBeanProxy(ManagementFactory.java:617)
        at jvmmon.core.GcMonitor.<init>(GcMonitor.java:30)

If you see these java.security.AccessControlException: access denied exceptions in your SDC log during startup, this means your JDK installation or global Java Security Policy requires you to explicitly grant access the following permissions.

Adding the following lines to your SDC Security Policy ($SDC_CONF/sdc-security.policy or in the Java Policy tab in your SCH DataOps Deployment Advanced Configuration):

grant {
  permission javax.management.MBeanServerPermission "createMBeanServer";
  permission javax.management.MBeanPermission "*", "registerMBean,unregisterMBean";
  permission javax.management.AttributePermission "*", "read,write";
  permission javax.management.QueryPermission "*";
  permission javax.management.MBeanPermission "sun.management.OperatingSystemImpl#-[java.lang:type=OperatingSystem]", "isInstanceOf";
  permission javax.management.MBeanPermission "sun.management.OperatingSystemImpl#-","getAttribute";
  permission javax.management.MBeanPermission "sun.management.OperatingSystemImpl#-","setAttribute";
  permission javax.management.MBeanPermission "sun.management.OperatingSystemImpl#-","invoke";
  permission java.net.SocketPermission "0.0.0.0:3333", "connect,resolve";
};

After adding these grants in your SDC Security Policy and restarting SDC, you should be able to connect to the SDC’s JMX socket and receive JVM metrics with your JMX Client.