wiki:RemoteJavaMonitoring
Last modified 14 years ago Last modified on 08/02/07 14:04:40

Remote Java Monitoring

Introduction

Since Version 1.5 (aka Java 5.0), Java comes with a technology on board for monitoring and instrumenting the virtual machine, which is known as the Java Management Extensions (JMX). This allows (authenticated) clients to attach themselves to a running JVM and grab some detailed information (such as memory consumption, number of threads etc. ). The Java Development Kit also provides a client tool to connect to and observe a locally- or remote-running JVM.

Preparing a dCache Domain

Each dCache domain corresponds to a Java virtual machine instance. To enable monitoring for one particular domain, you have to add a java option to the corresponding <domainName>Setup-file in $dCacheHome/config/. Unfortunately, this will affect all domains on that host, because the setup-files are just links to the $dCacheHome/config/dCacheSetup. If you are not intending to instrument all domains simultaneously, delete the link and create a copy of dCacheSetup:

rm -rf $dCacheHome/config/<domainName>Setup
cp $dCacheHome/config/dCacheSetup $dCacheHome/config/<domainName>Setup

In order to recreate the links once your are done with monitoring, move away or delete all modfied Setup-files and then

cd $dCacheHome/jobs
./initPackage.sh

Enabling local monitoring

Add the following line to the section java_options in $dCacheHome/config/<domainName>Setup

              -Dcom.sun.management.jmxremote 

Make sure that there appear no whitespaces or other characters after the backslash. It should now look like

java_options="-server -Xmx512m -XX:MaxDirectMemorySize=512m 
              -Dsun.net.inetaddr.ttl=1800 
              -Dorg.globus.tcp.port.range=20000,25000 
              -Djava.net.preferIPv4Stack=true 
              -Dorg.dcache.dcap.port=0 
              -Dorg.dcache.net.tcp.portrange=33115:33145 
              -Dcom.sun.management.jmxremote 
"

After restarting that particular domain, you should be able to connect locally to the instrumented JVM by using JConsole, which ships with the JDK.

Execute

$JAVA_HOME/bin/jconsole <pid>

where <pid> must be the proccess id of your java process you want to monitor.

Please note that JConsole is a GUI application, so X-forwarding is needed to do it over SSH (ssh -X user@dcacheHost).

Enabling remote monitoring

Remote monitoring means that the instrumented JVM additionally starts a RMI server, where clients can then connect to over the network. To allow only trusted users, password as well as SSL-based authentication is available. The following describes only the password method.

To enable JMX and let remote clients monitor your domain, you have to specify an unused port where the monitoring server is going to listen on. Add to the section java_options in $dCacheHome/config/<domainName>Setup the lines

              -Dcom.sun.management.jmxremote.port=<portNumber> 
              -Dcom.sun.management.jmxremote.ssl=false 
              -Dcom.sun.management.jmxremote.authenticate=true 

Please make sure that you edit the real <domainName>Setup-file as described above (and not a link)! Otherwise you well get all Domains trying to start a monitoring server next time you restart them'''

Now you have to edit the password file to change the default passwords. First find out the home directory of the java virtual machine which is used for running your dCache. The java=.. in $dCacheHome/config/dCacheSetup should give you a hint on this!

Now the monitoring role ( = username ) and the according password has to be set:

cd $JAVA_HOME/lib/management
cp jmxremote.password.template jmxremote.password
echo "monitorRole <somePassword>" >> jmxremote.password

After a restart of that domain, you should be able to connect with JConsole over the network. Just start JConsole on a client machine and connect to the serverhost:port, using monitorRole as username and the password specified.

Monitoring CPU-per-thread usage

This functionality is provided by an additional plugin for JConsole (See Links section). Note that this is only working with Java 6 on the client side (JConsole). You have to connect as controlRole in order to use the plugin. In order to do this, append a line to jmxremote.password saying 'controlRole' <yourPassword>'. It is neither necessary to restart the monitored Domain nor to install the plugin on the server JVM.

  • Monitoring and Management Using JMX : link
  • Using JConsole : link
  • Article: Using JConsole to Monitor Applications : link
  • JConsole Plugin for monitoring CPU-per-thread usage : link

last modified Fri Feb 26 14:34:08 2021, by Martin