Monitoring Liferay with Nagios, Jolokia and JMX4Perl

/ / , /

How do I monitor Liferay? That’s a question I’ve heard a lot lately. Well the standard way of getting some information about the application is by using JMX. The downside of JMX is that it’s a Java only standard and the only remote connection is by using RMI which doesn’t really sit well with non Java monitoring software like very popular Nagios. Another hurdle might be that your network admin might not be inclined to open up RMI access to the jvm.

There’s a nice agent called Jolokia that can provide a http bridge to JMX. You can install it as java agent in pretty much any java app or deploy it as a webapp. With Jolokia installed you can query any MBeans for their values using a simple http GET and get the data as JSON objects. JMX4Perl is a perl module and scripts that provide a easy way to run those queries through Jolokia. One of those scripts is check_jmx4perl which can be used in nagios service checks.

Okay so now we know that we are going to need Nagios, Jolokia and JMX4Perl to monitor the Liferay JVM but what should we monitor? Well that depends on what information you are interested in but at minimum I would monitor ajp or http thread usage as well as heap utilization. Just by monitoring those values you’ll know when your JVM is becomes unresponsive and can also get some early warning that there’s issues for example heap usage goes over warning threashold and never returns to normal or keeps constantly going over the threshold which could indicate they you don’t have enough heap allocated.

I’m going to assume that you have  nagios installed and configured and I will only go through how to install Jolokia and configure some checks for threads and heap. So let’s start by installing JMX4Perl.

Installing JMX4Perl is pretty simple with cpan. You just launch cpan command line client and install it like this:

cpan> install JMX::Jmx4Perl

Next you’ll need to download Jolokia and deploy the jolokia.war to your app server. For this example I’m going to assume that you are using Tomcat 7. Once you’ve deployed Jolokia it’s usually good idea to restrict who can query it. For this example we are just going to restrict it to a certain IP address (the Nagios server) and limit it to read operations only. Since I don’t like modifying the war we are going to tell Jolokia where to find the policy file through a context parameter. Create a jolokia.xml in tomcat/conf/Catalina/localhost with following content:

<Context path="/jolokia">
        <Parameter name="policyLocation" value="file:///etc/jolokia/jolokia-access.xml" />
</Context>

That tells Jolokia to look for the policy file jolokia-access.xml from /etc/jolokia/jolokia-access.xml. This is great when you are running multiple tomcats in the same server and want them to share the jolokia policy file.

Now go ahead and create the jolokia-access.xml in /etc/jolokia

<?xml version="1.0" encoding="utf-8"?>
<restrict>
        <remote>
                <host>[YOUR NAGIOS SERVER IP]</host>
        </remote>
        <http>
                <method>get</method>
                <method>post</method>
        </http>
        <commands>
                <command>read</command>
        </commands>
</restrict>

Next we need to create configuration for jmx4perl. In /etc/jmx4perl/jmx4perl.cfg we are going to include some preconfigured checks extend them. Tomcat 7 you need to add quotes around the thread pool name. We also need to set warning and critical levels for alerts. You’ll also need to add a Server for each tomcat you want to monitor.

# Default definitions
include default/memory.cfg
include default/tomcat.cfg

# ==========================
# Check definitions

<Check tc7_connector_threads>
	Use = relative_base($1,$2)
	Label = Connector $0 : $BASE
	Value = Catalina:name="$0",type=ThreadPool/currentThreadCount
	Base = Catalina:name="$0",type=ThreadPool/maxThreads
	Critical 95
	Warning 90
</Check>

<Check j4p_memory_heap>
	Use memory_heap
	Critical 95
	Warning 90
</Check>

<Server tomcat>
	Url http://MY_TOMCAT_HOSTNAME:8080/jolokia
</Server>

Then in /etc/nagios3/commands.cfg we’ll need to add a check command for jmx4perl and we’ll use the check_jmx4perl script to do that.

define command {
	command_name    check_j4p_cmd
	command_line    /usr/local/bin/check_jmx4perl --unknown-is-critical --config /etc/jmx4perl/jmx4perl.cfg --server $ARG1$ --check $ARG2$ $ARG3$
}

Then we need to define a service to monitor in /etc/nagios3/conf.d/host-MY_TOMCAT_HOSTNAME.cfg

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat Heap Memory
	check_command check_j4p_cmd!tomcat!j4p_memory_heap!x
}

define service {
	use generic-service
	host_name MY_TOMCAT_HOSTNAME
	service_description Tomcat AJP Threads
	check_command check_j4p_cmd!tomcat!tc7_connector_threads!ajp-bio-8009
}

The check above is for your tomcat heap and the other one is for Tomcat 7 AJP threads.

Now you should all the pieces to implement your own monitoring using Nagios, Jolokia and JMX4Perl. You should also remember that you can apply this to any JEE application not just Liferay.