Connecting Visual VM to Tomcat 7

Connecting Visual VM to a remote instance of Tomcat 7 is surprisingly easy. All you have to do is add some options to JAVA_OPTS turning on JMX, specifying how you want to handle security and setting the hostname. While it is easy to get it up and running, there are quite a few steps to go through if you want to make it work with authentication and behind a firewall.

My goal with this post is to walk through the basics of getting it running and then modifying the installation to support common configuration needs.

Here are instructions for how to set it up using Ubuntu 11.10:

First lets install Tomcat 7 if you don’t have it.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install tomcat7

Now we need to set the JAVA_OPTS. We will do that by creating a setenv.sh file in /usr/share/tomcat7/bin/ and putting the options in there. setenv.sh gets called before Tomcat starts to set any environmental variables you may want.

export JAVA_OPTS="-Dcom.sun.management.jmxremote=true \
                  -Dcom.sun.management.jmxremote.port=9090 \
                  -Dcom.sun.management.jmxremote.ssl=false \
                  -Dcom.sun.management.jmxremote.authenticate=false \
                  -Djava.rmi.server.hostname=50.112.22.47"

Line 1 enables jmxremote. Line 2 specifies the port. Line 3 says that we don’t need to use ssl. Line 4 says to leave it wide open and not use any type of authentication. Line 5 specifies the ip address of the server where you are running Tomcat. (Don’t use my ip address of 50.112.22.47, substitute your own.) This is left out of many instructions on the web, so it might work in some circumstances without it, but I wasn’t able to connect with VisualVM unless this configuration points to itself.

I believe this has to do with the fact that JMX is going to open another connection on a random port (discussed below). If you don’t tell it what its hostname (or ip) is, JMX doesn’t know how to tell the client how to connect back to that other port.

Now open VisualVM. On OS X you just run:

jvisualvm

Add the connection by clicking on File > Add JMX Connection… and fill out the dialog box as shown (but using the ip address of your server).

Once you add it, you should see the server in the list on the left hand side. Double click on the JMX connection to the server. (The JMX connection has a JMX icon and should show port 9090.)

You should then be able to view the following screens of information showing what is going on inside of Tomcat.

Firewall

One problem people run into in getting this to work is that they open port 9090 (or whatever they have specified) and VisualVM is unable to connect. This is because JMX appears to accept connections port 9090, but then opens at least one other random port and instructs the client to connect to this port as well.

If we run

sudo netstat -ntlp

We should see something like this:

ubuntu@ip-10-252-22-93:~$ sudo netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      494/sshd        
tcp6       0      0 :::8080                 :::*                    LISTEN      2650/java       
tcp6       0      0 :::36851                :::*                    LISTEN      2650/java       
tcp6       0      0 :::22                   :::*                    LISTEN      494/sshd        
tcp6       0      0 :::35543                :::*                    LISTEN      2650/java       
tcp6       0      0 :::9090                 :::*                    LISTEN      2650/java 

Line 4 shows ssh running on port 22. 5 is where Tomcat is serving HTTP. 9 shows the JMX connection. However 6 & 8 appear to be part of the JMX process. If you have firewall that is blocking access to these ports, VisualVM won’t be able to connect. You can’t just add those specific ports because they are random and can change every time Tomcat is restarted. So you have to leave your machine wide open to connect or use the Listener that will be explained a few sections below.

Authentication

Now lets look at how to secure the connection a bit and require a username and password. We can change the settings we put into setenv.sh to tell it to require authentication by changing false to true.

export JAVA_OPTS="-Dcom.sun.management.jmxremote=true \
                  -Dcom.sun.management.jmxremote.port=9090 \
                  -Dcom.sun.management.jmxremote.ssl=false \
                  -Dcom.sun.management.jmxremote.authenticate=true \
                  -Djava.rmi.server.hostname=50.112.22.47"

By default this should look for two files. One is called jmxremote.access and the other is jmxremote.password. It will probably look for the files in /usr/lib/jvm/java-6-openjdk/jre/lib/management/ but this may be different depending on which JDK you have installed and in some cases it will look for the files in the CATALINA_HOME directory.

We need to specify where the files should be found with the following options. Here we specify the files will be in the tomcat7/conf directories. So now our /usr/share/tomcat7/setenv.sh file should look like:

export JAVA_OPTS="-Dcom.sun.management.jmxremote=true \
 -Dcom.sun.management.jmxremote.port=9090 \
 -Dcom.sun.management.jmxremote.ssl=false \
 -Dcom.sun.management.jmxremote.authenticate=true \
 -Djava.rmi.server.hostname=50.112.22.47 \
 -Dcom.sun.management.jmxremote.password.file=/var/lib/tomcat7/conf/jmxremote.password \
 -Dcom.sun.management.jmxremote.access.file=/var/lib/tomcat7/conf/jmxremote.access"

jmxremote.password should look something like:

jmxadmin mysecretpassword

and jmxremote.access should have something like:

jmxadmin readwrite

Our user is jmxadmin, but could be any username. jmxremote.password tells what password is assigned to each user and jmxremote.access tells what access rights each user has. For a user to have access, they need to have an entry in both files.

Now if you try to run this setup, you will probably see something like this error in your catalina.out file:

Error: Password file read access must be restricted: /var/lib/tomcat7/conf/jmxremote.password

To fix this we need to make sure that both files are owned by the tomcat7 user:

sudo chown tomcat7:tomcat7 /var/lib/tomcat7/conf/jmxremote.*

Then we need to make sure that the tomcat7 user is the only user who has read access.

sudo chmod 0600 /var/lib/tomcat7/conf/jmxremote.*

Now you should be able to create a new connection to the server as before, but this time specifying the username and password you wish to use to connect. VisualVM wouldn’t let me just modify an existing JMX connection, so I had to create a new one rather than just adding the username and password to the existing connection.

Controlling the Ports

The only remaining inconvenience is the fact that JMX is going to choose a random port. If you aren’t dealing with a firewall this might not be a big deal, but if you are dealing with a remote server in a data center or in the cloud, it becomes more problematic. We need some way to tell Tomcat to bind the other JMX ports to a specific port number rather than choosing something at random.

We can do this by adding a listener to the /var/lib/tomcat7/conf/server.xml file like this:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
  rmiRegistryPortPlatform="9090" rmiServerPortPlatform="9091" />

Just put it below the other Listeners in server.xml. Notice the rmiRegistryPortPlatform is the 9090 that we previously specified in setenv.sh. The rmiServerPortPlatform allows us to bind the process to 9091 instead of a random port number.

Note: You can now remove the line that specifies port 9090 in setenv.sh.

In addition to adding the Listener we need to put the jar in /usr/share/tomcat7/lib/. The jar we are looking for is called catalina-jmx-remote.jar.

To locate this jar, first determine what version of Tomcat you are using by running the version script which will give us the output as shown.

ubuntu@ip-10-252-22-93:$ /usr/share/tomcat7/bin/version.sh 
Using CATALINA_BASE:   /usr/share/tomcat7
Using CATALINA_HOME:   /usr/share/tomcat7
Using CATALINA_TMPDIR: /usr/share/tomcat7/temp
Using JRE_HOME:        /usr
Using CLASSPATH:       /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar
Server version: Apache Tomcat/7.0.21
Server built:   Sep 8 2011 01:23:08
Server number:  ...0
OS Name:        Linux
OS Version:     3.0.0-14-virtual
Architecture:   amd64
JVM Version:    1.6.0_23-b23
JVM Vendor:     Sun Microsystems Inc.

In our case we are using Tomcat/7.0.21, so we want to go to http://archive.apache.org/dist/tomcat/tomcat-7/v7.0.21/bin/extras/. You can substitute your own version number in the URL to find the file.

Once the file is in /usr/share/tomcat7/lib/, restart your tomcat server and create a new JMX connection as specified above using VisualVM. You should now have a server that requires authenticated access for JMX and where you don’t have to leave all of the ports open.

There is also a way to tunnel VisualVM over an SSH connection for people who want even stronger security, but we aren’t going to address that in this post.

Other Notes

I have seen cases where VisualVM stops showing the Monitor and Threads tabs. If you delete the connection and add it again, they come back. I’m not sure why, but it is worth trying if you aren’t seeing all the data you expect.

Visual VM allows you to add plugins. They can be found under the Tools > Plugin menu option. They are supposed to download when you select them, but I kept getting a failure when trying to install the Visual GC garbage connection plugin.

I was able to get it to install by switching to JDK 1.6 (from Apple) instead of OpenJDK 1.7 on the client. However, when I tried to use Visual GC it said “Not supported for this JVM”. I wasn’t clear if that meant that the client or the server wasn’t supported, but I think it was complaining because the server is using OpenJDK 1.6 instead of the Oracle JDK.

About 

32 Replies to “Connecting Visual VM to Tomcat 7”

  1. Thank you for sharing this, one year later and still pertinent, it helped me with the concepts and the tecnical stuff.

    Merci et bonne journée

    J.

  2. Thanks. That part with IP: “Line 5 specifies the ip address of the server where you are running Tomcat.” was the GOD line.

  3. I followed yout instructions with the changes to server.xml and the fixed port (as opposed to random port) doesn’t work for me :( Tomcat is still doing its thing, opening whatever port it pleases.

  4. Hi, these are great instructions, thanks! I’m having some difficulty getting it working with Tomcat 7.0.26 though. When I add the to server.xml, I’m seeing an exception in catalina.out:

    Dec 20, 2013 7:31:48 PM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServer
    SEVERE: Unable to create the RMI registry for the Platform server using port 9998
    java.rmi.server.ExportException: internal error: ObjID already in use
    at sun.rmi.transport.ObjectTable.putTarget(ObjectTable.java:186)
    at sun.rmi.transport.Transport.exportObject(Transport.java:92)
    at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:247)
    at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
    at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
    at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207)
    at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:112)
    at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98)
    at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:203)
    at org.apache.catalina.mbeans.JmxRemoteLifecycleListener.createServer(JmxRemoteLifecycleListener.java:236)
    at org.apache.catalina.mbeans.JmxRemoteLifecycleListener.lifecycleEvent(JmxRemoteLifecycleListener.java:221)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
    at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
    at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:401)
    at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:346)
    at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:725)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:645)

    And I’m not able to connect with Visual VM. It seems like something else has already created the RMI registry, but I don’t know what. Any thoughts, ideas, or suggestions? :-)

    Thanks,

    Chuck

  5. Problem solved. This was the key:

    “Note: You can now remove the line that specifies port 9090 in setenv.sh.”

    Actually, you *must* remove that.

    Thanks!!

    Chuck

  6. Hej!

    I was accessing my server through dev-server through VPN so that I don’t have to configure JMX authentication. To make it work the “-Djava.rmi.server.hostname=192.168.240.1” was crucial! So thank you for that tip. I can see the sense in that since if there is a default ip it is probably not the VPN ip.

  7. hi ,
    i did all the config as above except, adding system properties in restart.sh since my server is tomcat 5 how do i connect..though i m not getting any error ,i couldnt get the sampler tab when i open my jmx connection.

  8. HI Mark,

    I was trying to add tomcat service in VisualVm I was able to add it but unable to get VisualGC, it is giving error “Not supported for this JVM”.
    Below are the parameters which i added

    -Dcom.sun.management.jmxremote=true
    -Dcom.sun.management.jmxremote.port=9090
    -Dcom.sun.management.jmxremote.ssl=false
    -Dcom.sun.management.jmxremote.authenticate=false

  9. Hy Mark,

    When i try to connect a VisualVM, return this error:
    “Cannot connect to : using service:jmx:rmi:///jndi/rmi://:/jmxrmi”

    Can you help me please?

  10. Hello,
    I’ve followed described steps, but when I add the Java Ops,
    the -Dcom.sun.management.jmxremote.port=8080 parameters makes tomcat does not start up.
    If I remove that parameter, then tomcat starts successfully, but then I cannot add it to VisulVM.
    What am I doing wrong?

    Thanks.

    1. I’ve corrrected the hostname and ip in the /etc/hosts file and now the Tomcat starts ok,
      but remotely from VisualVM I still get the error:
      Cannot connect to 10.116.67.216:8080 using service:jmx:rmi:///jndi/rmi://10.116.67.216:8080/jmxrmi

      PS: I have running the jstatd

      Thanks.

  11. About specifying the host port:

    “Note: You can now remove the line that specifies port 9090 in setenv.sh.”

    Actually, you MUST remove that line for the connection to work. I spent a long time debugging the connection until I found out that was the reason.

  12. For some reason when I start tomcat as a service it’s not picking up the JMX configuration in setenv.bat or even startup.bat but when I run startup.bat from the commandline it works……strange

      1. On Windows you can change the Java options on apache-tomcat-7.0.62\bin\tomcat7w.exe.
        Then in the Java tab you have the Java Options.

    1. The startup files on a Linux machine would have .sh file extensions, so they would be setenv.sh or startup.sh. The .bat file extension is only for windows.

  13. This is about the Visual GC not working. As a quick reminder, Visual GC is a plugin that you will likely need to add to your VisualVM. Visual GC requires a jstatd connection in order to monitor your garbage collector. You also need to be aware that the starting of jstatd must be run by the same user as the JVM you wish to monitor, or have super-user access.

  14. Hi, from JDK6, you can simply specify the RMI random port with the com.sun.management.jmxremote.rmi.port option.

    export CATALINA_OPTS=”-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8991 -Dcom.sun.management.jmxremote.rmi.port=8991 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false”

  15. Hi! On the last part, I don’t think VisualGC has something to do with the Java version. I was having trouble making Visual GC plugin to work too. I have 2 separate servers.

    Server 1 JVM: Java HotSpot(TM) 64-Bit Server VM (24.71-b01, mixed mode)
    Server 1 Java: version 1.7.0_71, vendor Oracle Corporation

    Server 2 JVM: OpenJDK 64-Bit Server VM (23.25-b01, mixed mode)
    Server 2 Java: version 1.6.0_33, vendor Sun Microsystems Inc

    As you can see, one server uses Java from Oracle and the other one is using OpenJDK. Both of them shows unsupported JVM in Visual GC plugin.

  16. Thanks for the nice post.

    1 question: This setting does not show the Profiler tab in Visual VM. Is it not available via JMX?

    1. I believe you are correct. The profiler isn’t available when using JMX. (Or at least it wasn’t when I wrote this post.) You should be able to use the sampler to get a pretty good idea of what is going on.

Leave a Reply

Your email address will not be published. Required fields are marked *