Home > GLASSFISH / J2EE / JAVA PERFORMANCE, glassfish monitoring, JASPER REPORTS, JMeter Jasper Performance Reports, Rest > One Step Monitoring of Key Indicators in Glassfish 3.1 via REST

One Step Monitoring of Key Indicators in Glassfish 3.1 via REST


Ever since upgrading Glassfish from v.3.0.1 to V3.1.2.2 i made a note to myself to redesign and simplify the active monitoring i was using in my Load Testing scripts, so that i could easily monitor things like:

  • JDBC Connections used
  • JDBC Connections timed out
  • JDBC Connections free
  • Http-threads busy

and so on.

Since the interface has changed, and my previous monitoring implementation was using also pretty much hardcoded values (of which i am absolutely no fan), it was time to redesign this in a smarter way, making use of the new interface, trying to get all my metrics in one step, or in as few steps as required.

Glassfish 3.1.2.2 Rest Monitoring

Needless to remind you how to enable/disable monitoring (sure you know it by now), but then again, it is not that hard to detail it once more.

First we check on the current status of the monitoring levels

asadmin -p 11048 –passwordfile /opt/glassfish/portal/v3.1.2.2/passwords get server.monitoring-service.module-monitoring-levels.*

*
server.monitoring-service.module-monitoring-levels.connector-connection-pool=OFF
server.monitoring-service.module-monitoring-levels.connector-service=OFF
server.monitoring-service.module-monitoring-levels.deployment=OFF
server.monitoring-service.module-monitoring-levels.ejb-container=OFF
server.monitoring-service.module-monitoring-levels.http-service=OFF
server.monitoring-service.module-monitoring-levels.jdbc-connection-pool=HIGH
server.monitoring-service.module-monitoring-levels.jersey=OFF
server.monitoring-service.module-monitoring-levels.jms-service=OFF
server.monitoring-service.module-monitoring-levels.jpa=OFF
server.monitoring-service.module-monitoring-levels.jvm=OFF
server.monitoring-service.module-monitoring-levels.orb=OFF
server.monitoring-service.module-monitoring-levels.security=OFF
server.monitoring-service.module-monitoring-levels.thread-pool=OFF
server.monitoring-service.module-monitoring-levels.transaction-service=OFF
server.monitoring-service.module-monitoring-levels.web-container=HIGH
server.monitoring-service.module-monitoring-levels.web-services-container=OFF

Now we set the desired monitoring level, let’s take the http-service as example:

asadmin -p 11048 –passwordfile /opt/glassfish/portal/v3.1.2.2/passwords set server.monitoring-service.module-monitoring-levels.http-service=HIGH
server.monitoring-service.module-monitoring-levels.http-service=HIGH
Command set executed successfully.

Let’s check it if it is really enabled (we choose xml as output format, available are html, xml and json)

curl -k -s -u admin:adminadmin -H “Accept: application/xml” https://server:11048/monitoring/domain/server/network/jk-main-listener-1/thread-pool

<?xml version=”1.0″ encoding=”UTF-8″ standalone=”no”?>
<map>
<entry key=”extraProperties”>
<map>
<entry key=”entity”>
<map>
<entry key=”corethreads”>
<map>
<entry key=”unit” value=”count”/>
<entry key=”starttime”>
<number>1392653042076</number>
</entry>
<entry key=”count”>
<number>5</number>
</entry>
<entry key=”description” value=”Core number of threads in the thread pool”/>
<entry key=”name” value=”CoreThreads”/>
<entry key=”lastsampletime”>
<number>1392722067843</number>
</entry>
</map>
</entry>
<entry key=”currentthreadsbusy“>
<map>
<entry key=”unit” value=”count”/>
<entry key=”starttime”>
<number>1392653042077</number>
</entry>
<entry key=”count”>
<number>0</number>
</entry>
<entry key=”description” value=”Provides the number of request processing threads currently in use in the listener thread pool serving requests”/>
<entry key=”name” value=”CurrentThreadsBusy”/>
<entry key=”lastsampletime”>
<number>1392738373205</number>
</entry>
</map>
</entry>
<entry key=”totalexecutedtasks“>
<map>
<entry key=”unit” value=”count”/>
<entry key=”starttime”>
<number>1392653042077</number>
</entry>
<entry key=”count”>
<number>123022</number>
</entry>
<entry key=”description” value=”Provides the total number of tasks, which were executed by the thread pool”/>
<entry key=”name” value=”TotalExecutedTasksCount”/>
<entry key=”lastsampletime”>
<number>1392738373205</number>
</entry>
</map>
</entry>
<entry key=”maxthreads“>
<map>
<entry key=”unit” value=”count”/>
<entry key=”starttime”>
<number>1392653042076</number>
</entry>
<entry key=”count”>
<number>1024</number>
</entry>
<entry key=”description” value=”Maximum number of threads allowed in the thread pool”/>
<entry key=”name” value=”MaxThreads”/>
<entry key=”lastsampletime”>
<number>1392722067843</number>
</entry>
</map>
</entry>
<entry key=”currentthreadcount“>
<map>
<entry key=”unit” value=”count”/>
<entry key=”starttime”>
<number>1392653042077</number>
</entry>
<entry key=”count”>
<number>150</number>
</entry>
<entry key=”description” value=”Provides the number of request processing threads currently in the listener thread pool”/>
<entry key=”name” value=”CurrentThreadCount”/>
<entry key=”lastsampletime”>
<number>1392737736187</number>
</entry>
</map>
</entry>
</map>
</entry>
<entry key=”childResources”>
<map/>
</entry>
</map>
</entry>
<entry key=”message” value=””/>
<entry key=”exit_code” value=”SUCCESS”/>
<entry key=”command” value=”Monitoring Data”/>
</map>

Of course you can use a browser using the same url connection and have it nicely displayed, but we need the “curled” version, in order to further extract the desired values.

Extracting xml tags with awk and xmllint under bash

Let’s say we now need to extract following metrics:

  • currentthreadsbusy
  • totalexecutedtasks
  • maxthreads
  • currentthreadcount

Since my Linux distribution did not have any xmlextractor, but had an xmlparser and i wanted this solution to be portable on any Linux machine, i decided to go the hard way, and use awk and xmllint.

xmllint : The xmllint program parses one or more XML files, specified on the command line as XML-FILE (or the standard input if the filename provided is – ). It prints various types of output, depending upon the options selected. It is useful for detecting errors both in XML code and in the XML parser itself.

I used a trick here, and reformatted the xml response displaying it as a pretty printed xml with line breaks.

curl -k -s -u admin:adminadmin -H “Accept: application/xml” https://myserver:11048/monitoring/domain/server/network/jk-main-listener-1/thread-pool | xmllint –format -

Now i need to get my monitoring items. The trick i used here is as follows:

  1. Extract everything starting with the item i am looking for, in this case currentthreadsbusy searching up to the pattern “/map”. This will return the following fragment

    currentthreadcount“>
    <map>
    <entry key=”unit” value=”count”/>
    <entry key=”starttime”>
    <number>1392653042077</number>
    </entry>
    <entry key=”count”>
    <number>150</number>
    </entry>
    <entry key=”description” value=”Provides the number of request processing threads currently in the listener thread pool”/>
    <entry key=”name” value=”CurrentThreadCount”/>
    <entry key=”lastsampletime”>
    <number>1392737736187</number>
    </entry>
    </map>

  2. Further i need to extract the monitoring value i am interested in. This can be in my case either “count” or “current”, so i will use awk once more and look for the following pattern

    awk ‘/<entry key=”count|current”>/,/<\/entry>/’

  3. This will now return the following fragment:

    <entry key=”count”>
    <number>150</number>
    </entry>

  4. The only thing left here to do is use a regular expression to extract only digits:

    grep -o ‘[0-9]*’

Let us put it all together now:

  1. Store the response of the curl request into a variable:

    http_mon_response=`curl -k -s -u admin:adminadmin -H “Accept: application/xml” https://myserver:11048/monitoring/domain/server/network/jk-main-listener-1/thread-pool`

  2. Retrieve the desired metric:

    val=`echo $http_mon_response | xmllint –nowarning –format – | awk ‘/currentthreadcount/,/\/map/’ | awk ‘/<entry key=”count|current”>/,/<\/entry>/’ | grep -o ‘[0-9]*’ `

Since we want to make this dynamic, and use one request, and extract as many metrics as possible, let’s write a small for loop that does that for us.

Suppose we want to retrieve the following monitoring metrics:

JDBC

  • numconnused
  • numconnfree
  • numconntimedout

HTTP

  • currentthreadsbusy

We will create a function called trace_gf_statistics that will post the curl request regularly, and write the outputs into an external file:

function trace_gf_statistics
{
# List of jdbc monitoring items to be retrieved

jdbc_names_short=(numconnused numconnfree numconntimedout)
# List of http thread pool monitoring items to be retrieved

http_names=(currentthreadsbusy)

# Only run the monitoring while this file exists. This file will be removed by the controlling process once the monitoring is stopped
status=`ls /tmp | grep glassfish_stats`
while [ "$status" != "" ];
do
MONITOR_TIMESTAMP=`date +%H-%M-%S`

# Store the JDBC Metrics into a variable
jdbc_mon_response=`curl -k -s -u admin:adminadmin -H “Accept: application/xml” https://myserver:11048/monitoring/domain/server/resources/EocPool`

# Store the HTTP Thread Pool Metrics into a variable
http_mon_response=`curl -k -s -u admin:adminadmin -H “Accept: application/xml” https://myserver:11048/monitoring/domain/server/network/jk-main-listener-1/thread-pool`

# Now iterate through all monitoring items we defined in the beginning and output the results
for jdbc_mon_item in ${jdbc_names_short[@]} ;
do
val=`echo $jdbc_mon_response | xmllint –nowarning –format – | awk ‘/’${jdbc_mon_item}’/,/\/map/’ | awk ‘/<entry key=”count|current”>/,/<\/entry>/’ | grep -o ‘[0-9]*’ `
echo $MONITOR_TIMESTAMP”:JDBC-“$jdbc_mon_item:$val >> ${JMETER_RESULTS}/glassfish_stats.log
done
for http_mon_item in ${http_names[@]} ;
do
val=`echo $http_mon_response | xmllint –nowarning –format – | awk ‘/’${http_mon_item}’/,/\/map/’ | awk ‘/<entry key=”count|current”>/,/<\/entry>/’ | grep -o ‘[0-9]*’ `
echo $MONITOR_TIMESTAMP”:HTTP-“$http_mon_item:$val >> ${JMETER_RESULTS}/glassfish_stats.log
done

# Post the requests every 3 seconds and then check for the existence of the status file
sleep 3
status=`ls /tmp | grep glassfish_stats`
done
}

Results

And this is how a typical monitoring output file looks like, separated by “:” delimiter.

16-32-32:JDBC-numconnused:131
16-32-32:JDBC-numconnfree:31
16-32-32:JDBC-numconntimedout:0
16-32-32:HTTP-currentthreadsbusy:58
16-32-35:JDBC-numconnused:110
16-32-35:JDBC-numconnfree:10
16-32-35:JDBC-numconntimedout:0
16-32-35:HTTP-currentthreadsbusy:40
16-32-38:JDBC-numconnused:110
16-32-38:JDBC-numconnfree:10
16-32-38:JDBC-numconntimedout:0
16-32-38:HTTP-currentthreadsbusy:36
16-32-42:JDBC-numconnused:103
16-32-42:JDBC-numconnfree:3
16-32-42:JDBC-numconntimedout:0
16-32-42:HTTP-currentthreadsbusy:27
16-32-45:JDBC-numconnused:121
16-32-45:JDBC-numconnfree:21
16-32-45:JDBC-numconntimedout:0
16-32-45:HTTP-currentthreadsbusy:43
16-32-48:JDBC-numconnused:83
16-32-48:JDBC-numconnfree:17
16-32-48:JDBC-numconntimedout:0
16-32-48:HTTP-currentthreadsbusy:7
16-32-51:JDBC-numconnused:126
16-32-51:JDBC-numconnfree:37
16-32-51:JDBC-numconntimedout:0
16-32-51:HTTP-currentthreadsbusy:64
16-32-55:JDBC-numconnused:204
16-32-55:JDBC-numconnfree:74
16-32-55:JDBC-numconntimedout:0
16-32-55:HTTP-currentthreadsbusy:127

You can now import the delimited file into whatever reporting tool you like, generating reports like this:

Glassfish Rest JDBC Monitoring Report

JasperReport-RestMonitoring-JDBC-Monitoring-Report

Glassfish Rest HTTP Thread Pool Monitoring Report

JasperReport-RestMonitoring-HTTP-Monitoring-Report

Needless to say that you can extend your monitoring items in the script above with whatever monitors you may need. It suffices to add a corresponding metric array, the curl request and iterate over the monitoring items.

This is a sample of my performance report, while using Jasper Server and Jasper Reports:

JasperReport-RestMonitoring

I will probably update this script regularly, so come back soon for a new, improved version of it.

Cheers

Alex

About these ads
  1. Dario
    February 26, 2014 at 10:48 am

    Very good stuff as usual !!!!

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 32 other followers

%d bloggers like this: