Stress Testing Application in Continuous Delivery Pipeline

Everybody does stress and load testing at some point in application development lifecycle. The problem I see with many teams is that this activity is postponed
until application functionality is developed, almost last step in process. This can lead to a lot of problems, the most severe one when application architecture can’t support the number of users or performance objective it supposed to have. At this point thousands lines of code is written, many unit and integration tests are running but application is still not functional. The best approach is to start stress testing early on, when project is just started. This should be part of agile development lifecycle, where each sprint acceptance process includes some kind of stress testing. Developers should understand that unit and integration test are practically useless in context of running highly multithreaded applications like modern web applications. Stress test during each iteration will allow you to learn application and related component behavior under the load and allow to spot concurrency and robustness problem early on, when it is much cheaper to fix. We should put as much load as possible, even much more that realistic load on the production application, to discover problems.

Incorporating Load test into Continuous Deployment Pipeline.
Most effective way to run load test is to automate it and incorporate into everyday build process. This will allow development team constantly perform load test and discover areas of performance degradation before it gets out into the wild. In order to do it we need only couple of tools – Continuous Integration Server like Jenkins and load testing tools like JMeter. JMeter is popular open source tools. It works by simulating load on your application, and measuring the response time as the number of simulated users and requests increase. It effectively simulates the actions of a browser or client application, sending requests of various sorts (HTTP, SOAP, JDBC, JMS and so on) to your server. You configure a set of requests to be sent to your application, as well as random pauses, conditions and loops, and other variations designed to better imitate real user actions. If you not familiar with JMeter, don’t dispair, it is easy to learn and install. Idea here is to create tests in JMeter and execute them at least nightly.

Automation Process:
Now let take practical steps of incorporation JMeter into deployment pipeline. I use gradle as my build software, but maven will work equally well.
There is a gradle plugin for JMeter but I was not able to make it work reliable. Instead I went with proven Ant plugin, taking advantage of the fact that gradle has excellent ant support.

  • Install JMeter into the tools directory of your project. Placing tools such as JMeter within your project structure is a good habit, as it makes your build scripts more portable and easier to run on any machine, which is precisely what we need to run them on Jenkins.
  • Create a JMeter test script. This is done using JMeter simple user interface. Please check this resources if you new to JMeter
  • Deploy your application to test servers and create a job in Jenkins to do it automatically. Cargo plugins or custom deployment script can do the job. I prefer to test against real application servers, and real databases. The close your test environment is to production setup, the better result of your tests.
  • Create Gradle task to execute our JMeter tests
  • Create a Jenkins job for our gradle task and execute in deployment pipeline
  • Log results of the test and made them available as graphs
  • Monitor and troubleshot application with VisualVM

Simple gradle script running JMeter:

task JMeter << {
	 ant.taskdef( name:"jmeter",classname:
"org.programmerplanet.ant.taskdefs.jmeter.JMeterTask") {
		 classpath { fileset(dir: "$projectDir/tools/apache-jmeter-2.7/lib", includes: 'ant-jmeter-1.1.1.jar')
		 jvmarg(value: "-Xms512m")

As you can see, we load all test from $projectDir/src/test/resources/load directory and tell jMeter to run 10 thread 20 times each. We can pass differect parameter to JMeter from our gradle task for more flexibility. I usually run the same test with different number of threads for each run.
For more information about JMeter and ant visit
Now you are ready to execute the script by simply running gradle JMeter task. >gradle jMeter.
Now let put JMeter task into deployment pipeline managed by Jenkins. Here is my version of deployment pipeline up to performance test. I run performance test
after application deployed to environment and all integration tests passed.

I choose to run performance test against QA environment during CI process assuming QA resemble production environments.

Creating build job in Jenkins
For this we will use the Performance Test Jenkins plugin, which understands JMeter logs and can generate nice statistics and graphs using this data.
This build job will typically be fairly separate from your other builds. I have set up the performance build to run on a nightly basis, which is probably enough for a long-running load or performance test.

If you take your performance metrics seriously, then the build should fail if the required SLA is not met. In a Continuous Integration environment, any sort of metrics build that does not fail if minimum quality criteria are not met will tend to be ignored.

This graph gives you an overview of performance over time. You would typically use this graph to ensure that your average response times are within the expected limits, and also spot any unusually high variations in the average or maximum response times.
Using Cloud for Performance test.
Performance test requires decent amount of hardware to run JMeter. Sometimes, you even have to run distributed JMeter load tests to generate enough load to stress the server. For internet applications, it might be neces- sary to deploy the load generators into the cloud. I had a great experience running performance test from Amazon EC2 instances, but I believe any other cloud provider will do – Azure, HP e.t.c. If you running your infrastructure on the cloud as well, then it is even easier to create QA environment and testing lab on demand and tear it down when not in use. This is just my plug about cloud computing as enabler for us, developers, to do amazing thigns quickly.

Java EE applications are always executed concurrently. Even with unrealistic stress tests, you will learn a
lot about system behavior and identify some bottlenecks or concurrency bugs.
The earlier and more frequently stress tests are executed, the more you will learn about the application’s run-time behavior. Application server configuration problems, potential bottlenecks, synchronization bugs, and memory leaks can also be identified during stress tests. In next blog I will share my experience with VisualVM.

1 Comment

  1. Hi Eugene,

    Awesome post!
    We’re seeing quite a bit of interest from developers looking to add JMeter into their release cycle ourselves.

    Any chance I can get you to review BlazeMeter?
    We run JMeter in the cloud and have plugins to the leading CI tools (Jenkins, Bamboo, TeamCity). And you get real nice real time graphs and a centralized repository for all your tests and reports.

    I know you can run your own instances of JMeter on EC2, but we’ve added an extra layer on top of JMeter that let’s us run up to 300 JMeter load servers in parallel (JMeter’s own distributed solution doesn’t seem to scale well with large numbers of servers).



Submit a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>