Measure performance with statsD, Graphite and Java Aspects

For some time I was looking into a way to monitor backend performance of each SQL /NoSQL query to my data store in real time. The monitoring should be asynchronous – not slowing down the application and provide good visualization with graphs. I think there are some commertial product doing this, but I have discovered statsd and Graphite – open source tools.

This post will be broken down into following parts:

  1. Meet the statsd
  2. Installing statsd and dependencies
  3. What is Graphite
  4. Installing Graphite
  5. Create Spring Aspect to log events and performance

Statsd is a network daemon that runs on the Node.js platform and listens for statistics, like counters and timers, sent over UDP and sends aggregates to one or more pluggable backend services (e.g., Graphite). It was open sourced by Etsy and can be found on gitHub
UDP was chosen because it allow sending statistics as fire-and-forget. Either StatsD gets the data, or it doesn’t. The application doesn’t care if StatsD is up, down, or on fire; it simply trusts that things will work. This allow sending a lot of data from the application without slowing it down.

Our first step will be installing node.js and statsd. I will skill how to install node.js because there are hundread of resources available on the net, depending on your platform. I did not bother installing it myself – I opted to use Amazon EC2 instances and found coummunitiy image with preinstalled node.js on Ubuntu.
Amazon EC2 community images and Marketplace are huge time savers when you need to have something up and running quickly.
Next is installation of statsd package – this is history from my console, you can modify according to your system:

 git clone git://github.com/etsy/statsd.git
cd statsd
cp exampleConfig.js myConfig.js
 
# edit config file your settings

sudo vi myConfig.js 

#run test
./run_tests.sh
#start statsd 
node stats.js myConfig.js 

Now let concentrate on Graphite. Graphite is a realtime graphic application backed by relational datastore. It come with SQLite but can be configured for any other store like mySQL or Posgres. It is a Python/ django application, so some knowledge of python will be helpful to install the thing.


Graphite also very easy to use, and has very powerful graphing and data manipulation capabilities. Most importantly for StatsD, you can create new metrics in graphite just by sending it data for that metric. That means there’s no management overhead for engineers to start tracking something new: simply tell StatsD you want to track “grue.dinners” and it’ll automagically appear in graphite. Statsd flush data to graphite every 10 seconds, so StatsD metrics are near-realtime.
Installing Graphite on Ubuntu system.

####################################
# BASIC REQUIREMENTS
# http://graphite.wikidot.com/installation
####################################
 
sudo apt-get update
sudo apt-get upgrade
 
wget https://launchpad.net/graphite/0.9/0.9.10/+download/graphite-web-0.9.10-pre4.tar.gz
wget https://launchpad.net/graphite/0.9/0.9.10/+download/carbon-0.9.10-pre3.tar.gz
wget https://launchpad.net/graphite/0.9/0.9.10/+download/whisper-0.9.10-pre3.tar.gz
wget https://launchpad.net/graphite/0.9/0.9.10/+download/check-dependencies.py

tar -zxvf graphite-web-0.9.10-pre4.tar.gz
tar -zxvf carbon-0.9.10-pre3.tar.gz
tar -zxvf whisper-0.9.10-pre3.tar.gz
mv graphite-web-0.9.10-pre4 graphite
mv carbon-0.9.10-pre3 carbon
mv whisper-0.9.10-pre3 whisper

sudo easy_install django-tagging
 
####################################
# INSTALL WHISPER
####################################
 
cd ~/whisper
sudo python setup.py install
 
####################################
# INSTALL CARBON
####################################
 
cd ~/carbon
sudo python setup.py install
# CONFIGURE CARBON
####################
cd /opt/graphite/conf
sudo cp carbon.conf.example carbon.conf
sudo cp storage-schemas.conf.example storage-schemas.conf
sudo vim storage-schemas.conf
### edited storage-schemas.conf to be the following
[stats]
priority = 110
pattern = .*
retentions = 10:2160,60:10080,600:262974
###
 
####################################
# CONFIGURE GRAPHITE (webapp)
####################################
 
cd ~/graphite
sudo python check-dependencies.py
sudo python setup.py install
#I had missing dependency 
 apt-get install libapache2-mod-python

# CONFIGURE APACHE
###################
cd ~/graphite/examples
sudo cp example-graphite-vhost.conf /etc/apache2/sites-available/default
sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
sudo vim /etc/apache2/sites-available/default
# moved 'WSGIImportScript /opt/gr..' to right before virtual host since it gave me an error saying
# WSGIImportScript cannot occur within <VirtualHost> section
# if this path does not exist make it!!!!!!
# /etc/httpd/wsgi

sudo mkdir /etc/httpd
sudo mkdir /etc/httpd/wsgi
sudo /etc/init.d/apache2 reload
 
####################################
# INITIAL DATABASE CREATION
####################################
cd /opt/graphite/webapp/graphite/
sudo python manage.py syncdb
# follow prompts to setup django admin user
sudo chown -R www-data:www-data /opt/graphite/storage/
sudo /etc/init.d/apache2 restart
cd /opt/graphite/webapp/graphite
sudo cp local_settings.py.example local_settings.py
 
####################################
# START CARBON
####################################
cd /opt/graphite/
sudo ./bin/carbon-cache.py start
 
####################################
# SEND DATA TO GRAPHITE
####################################
cd ~/graphite/examples
sudo chmod +x example-client.py
# [optional] edit example-client.py to report data faster
# sudo vim example-client.py
sudo ./example-client.py

# fix for flot graph
sudo vim /opt/graphite/webapp/graphite/graphlot 
#and changing line 6 from:
#import simplejson
#to
#import json as simplejson

Logging statistics and performance from Java – spring application.

Spring MVC one of the most popular web application framework in Java universe. Outside of MVC, spring framework provides reach functionality to develop enterprice applications. One of this capabilities is Spring AOP – Aspect Oriented Programming. for this new to the term, AOP help to modularize cross-cutting concerns. In short, a cross- cutting concern can be described as any functionality that affects multiple points of an application. Performance measurement or logging, for example, are a cross-cutting concerns, in that many methods in an application can be measured. Aspects offer an alternative to inheritance and delegation that can be cleaner in many circumstances. With AOP, you still define the common functionality in one place, but you can declaratively define how and where this functionality is applied without having to modify the class to which you’re applying the new feature. Cross- cutting concerns can now be modularized into special classes called aspects. This has two benefits. First, the logic for each concern is now in one place, as opposed to being scattered all over the code base. Second, our service modules are now cleaner since they only contain code for their primary concern (or core functionality) and second- ary concerns have been moved to aspects.
There is much more terminology to learn about AOP, but let do some coding and learn more from example.

We will create a monitoring advice that will be executed as around advise, measuring performance of the method and sending statistics and method name to statsd.

package com.statsd.aspect;

// imports omitted





/**
 * This class declares the aspects for the system.  In this example is an around advice for timing the length of methods
 * marked with the Timed annotation
 *
 *
 */
@Aspect
public class StatdPerformanceAspect {

	@Autowired 
	private StatsdClient statsdClient;
    /**
     * This around advice adds timing to any method annotated with the Timed annotation.
     * It binds the annotation to the parameter timedAnnotation so that the values are available at runtime.
     * Also note that the retention policy of the annotation needs to be RUNTIME.
     *
     * @param pjp             - the join point for this advice
     * @param timedAnnotation - the Timed annotation as declared on the method
     * @return
     * @throws Throwable
     */

    @Around("@annotation( timedAnnotation ) ")
    public Object processSystemRequest(final ProceedingJoinPoint pjp, Timed timedAnnotation) throws Throwable {
        try {
        	Log4JLogger log=new Log4JLogger("StatdPerformanceAspect.class");
            long start = System.currentTimeMillis();
            Object retVal = pjp.proceed();
            long end = System.currentTimeMillis();
            long differenceMs = end - start;

            MethodSignature methodSignature = (MethodSignature) pjp.getSignature();
            Method targetMethod = methodSignature.getMethod();
            //get the value of timing notes as declared in the method annotation
            String timingNotes = timedAnnotation.timingNotes();
            //String requestInfo = param.getRequestInfo();
            statsdClient.increment(targetMethod.getDeclaringClass().getName() + "." + targetMethod.getName());
            statsdClient.timing(targetMethod.getDeclaringClass().getName() + "." + targetMethod.getName(),(int) differenceMs) ;
            log.debug(targetMethod.getDeclaringClass().getName() + "." + targetMethod.getName() + " took " + differenceMs + " ms : timing notes: " + timingNotes + " request info : ");
            return retVal;
        } catch (Throwable t) {
            System.out.println("error occured");
            throw t;
        }

    }

}

How Spring AOP knows for which method measure performance? One way is to provide custom annotation. This annotation can be applied to both method level and type level. All method with this annotation will be measured for performance and data will be sent to statsd server for charting.

package com.statsd.aspect;



/**
 * This annotation is used to mark the methods that need timing information from the around advice in SystemTiming.
 * It is critical that the retention policy be RUNTIME so that the annotation values are available at runtime to the advice
 *
 *
 */
@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(value = RetentionPolicy.RUNTIME)
@Documented
public @interface Timed {

    //sample value passed into the advice of methods marked with this annotation
    String timingNotes();
}

And now we annotate our business method with @Timed annotation

package com.statsd.service;

import org.springframework.stereotype.Service;

import com.statsd.aspect.Timed;

@Service
public class BusinessServiceImpl implements BusinessService {
	/**
	* this method will be advised because it has the timed annotation
	*/
	@Timed(timingNotes = "this is a slow service")
	public void performTask() {
		// do something useful
		try {
			Thread.currentThread().sleep(400);
			System.out.println("Inside useful business method");
		} catch (InterruptedException e) {
			
			e.printStackTrace();
		}

	}

}

When you execute performTask() method from a test classes, it will log time into log output as well as into statsd server.
You can see working application in my github repository and use it for your projects.

References:
Measure Anything, Measure Everything

2 Comments

  1. Thanks!
    Never thought of using aspects for that.
    That is clearly an unobtrusive and clean way of logging metrics data.
    Brillant.

    Reply
  2. Simply great blog.It has helped us a lot.Actually I am new to these frameworks.It will be of really great help if you can share the methods to run all these frameworks.We need url and port to run this app.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>