In this article by Bayo Erinle, the author of the book Performance Testing with JMeter 3, will explore some of the options that make JMeter a great tool of choice for performance testing.
(For more resources related to this topic, see here.)
Performance testing and tuning
There is a strong relationship between performance testing and tuning, in the
sense that one often leads to the other. Often, end-to-end testing unveils system
or application bottlenecks that are regarded unacceptable with project target goals. Once those bottlenecks are discovered, the next step for most teams is a series of tuning efforts to make the application perform adequately.
Such efforts normally include, but are not limited to, the following:
- Configuring changes in system resources
- Optimizing database queries
- Reducing round trips in application calls, sometimes leading to redesigning and re-architecting problematic modules
- Scaling out application and database server capacity
- Reducing application resource footprint
- Optimizing and refactoring code, including eliminating redundancy and reducing execution time
Tuning efforts may also commence if the application has reached acceptable performance but the team wants to reduce the amount of system resources being used, decrease the volume of hardware needed, or further increase system performance.
After each change (or series of changes), the test is re-executed to see whether the performance has improved or declined due to the changes. The process will be continued with the performance results having reached acceptable goals. The outcome of these test-tuning circles normally produces a baseline.
Baselines
Baseline is a process of capturing performance metric data for the sole purpose of evaluating the efficacy of successive changes to the system or application. It is important that all characteristics and configurations, except those specifically being varied for comparison, remain the same in order to make effective comparisons as to which change (or series of changes) is driving results toward the targeted goal. Armed with such baseline results, subsequent changes can be made to the system configuration or application and testing results can be compared to see whether such changes were relevant or not. Some considerations when generating baselines include the following:
- They are application-specific
- They can be created for system, application, or modules
- They are metrics/results
- They should not be over generalized
- They evolve and may need to be redefined from time to time
- They act as a shared frame of reference
- They are reusable
- They help identify changes in performance
Load and stress testing
Load testing is the process of putting demand on a system and measuring its response, that is, determining how much volume the system can handle. Stress testing is the process of subjecting the system to unusually high loads far beyond its normal usage pattern to determine its responsiveness. These are different from performance testing, whose sole purpose is to determine the response and effectiveness of a system, that is, how fast the system is. Since load ultimately affects how a system responds, performance testing is always done in conjunction with stress testing.
JMeter to the rescue
One of the areas performance testing covers is testing tools. Which testing tool do you use to put the system and application under load? There are numerous testing tools available to perform this operation, from free to commercial solutions. However, our focus will be on Apache JMeter, a free, open source, cross-platform desktop application from the Apache Software foundation. JMeter has been around since 1998 according to historic change logs on its official site, making it a mature, robust, and reliable testing tool. Cost may also have played a role in its wide adoption. Small companies usually may not want to foot the bill for commercial end testing tools, which often place restrictions, for example, on how many concurrent users one can spin off. My first encounter with JMeter was exactly a result of this. I worked in a small shop that had paid for a commercial testing tool, but during the course of testing, we had outrun the licensing limits of how many concurrent users we needed to simulate for realistic test plans. Since JMeter was free, we explored it and were quite delighted with the offerings and the share amount of features we got for free.
Here are some of its features:
- Performance tests of different server types, including web (HTTP and HTTPS), SOAP, database, LDAP, JMS, mail, and native commands or shell scripts
- Complete portability across various operating systems
- Full multithreading framework allowing concurrent sampling by many threads and simultaneous sampling of different functions by separate thread groups
- Full featured Test IDE that allows fast Test Plan recording, building, and debugging
- Dashboard Report for detailed analysis of application performance indexes and key transactions
- In-built integration with real-time reporting and analysis tools, such as Graphite, InfluxDB, and Grafana, to name a few
- Complete dynamic HTML reports
- Graphical User Interface (GUI)
- HTTP proxy recording server
- Caching and offline analysis/replaying of test results
- High extensibility
- Live view of results as testing is being conducted
JMeter allows multiple concurrent users to be simulated on the application, allowing you to work toward most of the target goals obtained earlier, such as attaining baseline and identifying bottlenecks.
It will help answer questions, such as the following:
- Will the application still be responsive if 50 users are accessing it concurrently?
- How reliable will it be under a load of 200 users?
- How much of the system resources will be consumed under a load of
250 users? - What will the throughput look like with 1000 users active in the system?
- What will be the response time for the various components in the application under load?
JMeter, however, should not be confused with a browser. It doesn’t perform all the operations supported by browsers; in particular, JMeter does not execute JavaScript found in HTML pages, nor does it render HTML pages the way a browser does. However, it does give you the ability to view request responses as HTML through many of its listeners, but the timings are not included in any samples. Furthermore, there are limitations to how many users can be spun on a single machine. These vary depending on the machine specifications (for example, memory, processor speed, and so on) and the test scenarios being executed. In our experience, we have mostly been able to successfully spin off 250-450 users on a single machine with a 2.2 GHz processor and 8 GB of RAM.
Up and running with JMeter
Now, let’s get up and running with JMeter, beginning with its installation.
Installation
JMeter comes as a bundled archive, so it is super easy to get started with it. Those working in corporate environments behind a firewall or machines with non-admin privileges appreciate this more. To get started, grab the latest binary release by pointing your browser to http://jmeter.apache.org/download_jmeter.cgi. At the time of writing this, the current release version is 3.1. The download site offers the bundle as both a .zip file and a .tgz file. We go with the .zip file option, but feel free to download the .tgz file if that’s your preferred way of grabbing archives.
Once downloaded, extract the archive to a location of your choice. The location you extracted the archive to will be referred to as JMETER_HOME.
Provided you have a JDK/JRE correctly installed and a JAVA_HOME environment variable set, you are all set and ready to run!
The following screenshot shows a trimmed down directory structure of a vanilla JMeter install:
The following are some of the folders in Apache-JMeter-3.2, as shown in the preceding screenshot:
- bin: This folder contains executable scripts to run and perform other operations in JMeter
- docs: This folder contains a well-documented user guide
- extras: This folder contains miscellaneous items, including
samples illustrating the usage of the Apache Ant build tool
(http://ant.apache.org/) with JMeter, and bean shell scripting - lib: This folder contains utility JAR files needed by JMeter (you may add additional JARs here to use from within JMeter; we will cover this in detail later)
- printable_docs: This is the printable documentation
Installing Java JDK
Follow these steps to install Java JDK:
- Go to http://www.oracle.com/technetwork/java/javase/downloads/index.html.
- Download Java JDK (not JRE) compatible with the system that you will use to test. At the time of writing, JDK 1.8 (update 131) was the latest.
- Double-click on the executable and follow the onscreen instructions.
On Windows systems, the default location for the JDK is under Program Files. While there is nothing wrong with this, the issue is that the folder name contains a space, which can sometimes be problematic when attempting to set PATH and run programs, such as JMeter, depending on the JDK from the command line. With this in mind, it is advisable to change the default location to something like C:toolsjdk.
Setting up JAVA_HOME
Here are the steps to set up the JAVA_HOME environment variable on Windows and Unix operating systems.
On Windows
For illustrative purposes, assume that you have installed Java JDK at C:toolsjdk:
- Go to Control Panel.
- Click on System.
- Click on Advance System settings.
- Add Environment to the following variables:
- Value: JAVA_HOME
- Path: C:toolsjdk
- Locate Path (under system variables, bottom half of the screen).
- Click on Edit.
- Append %JAVA_HOME%/bin to the end of the existing path value (if any).
On Unix
For illustrative purposes, assume that you have installed Java JDK at /opt/tools/jdk:
- Open up a Terminal window.
- Export JAVA_HOME=/opt/tools/jdk.
- Export PATH=$PATH:$JAVA_HOME.
It is advisable to set this in your shell profile settings, such as .bash_profile
(for bash users) or .zshrc (for zsh users), so that you won’t have to set it for each
new Terminal window you open.
Running JMeter
Once installed, the bin folder under the JMETER_HOME folder contains all the executable scripts that can be run. Based on the operating system that you installed JMeter on, you either execute the shell scripts (.sh file) for operating systems that are Unix/Linux flavored, or their batch (.bat file) counterparts on operating systems that are Windows flavored.
JMeter files are saved as XML files with a .jmx extension. We refer to them as test scripts or JMX files.
These scripts include the following:
- jmeter.sh: This script launches JMeter GUI (the default)
- jmeter-n.sh: This script launches JMeter in non-GUI mode (takes a JMX file as input)
- jmeter-n-r.sh: This script launches JMeter in non-GUI mode remotely
- jmeter-t.sh: This opens a JMX file in the GUI
- jmeter-server.sh: This script starts JMeter in server mode (this will be kicked off on the master node when testing with multiple machines remotely)
- mirror-server.sh: This script runs the mirror server for JMeter
- shutdown.sh: This script gracefully shuts down a running non-GUI instance
- stoptest.sh: This script abruptly shuts down a running non-GUI instance
To start JMeter, open a Terminal shell, change to the JMETER_HOME/bin folder, and run the following command on Unix/Linux:
./jmeter.shAlternatively, run the following command on Windows:
jmeter.batTake a moment to explore the GUI. Hover over each icon to see a short description of what it does. The Apache JMeter team has done an excellent job with the GUI. Most icons are very similar to what you are used to, which helps ease the learning curve for new adapters. Some of the icons, for example, stop and shutdown, are disabled for now till a scenario/test is being conducted.
The JVM_ARGS environment variable can be used to override JVM settings in the jmeter.bat or jmeter.sh script. Consider the following example:
export JVM_ARGS="-Xms1024m -Xmx1024m -Dpropname=propvalue".
Command-line options
To see all the options available to start JMeter, run the JMeter executable with the -? command. The options provided are as follows:
.
./jmeter.sh -?
-?
print command line options and exit
-h, --help
print usage information and exit
-v, --version
print the version information and exit
-p, --propfile <argument>
the jmeter property file to use
-q, --addprop <argument>
additional JMeter property file(s)
-t, --testfile <argument>
the jmeter test(.jmx) file to run
-l, --logfile <argument>
the file to log samples to
-j, --jmeterlogfile <argument>
jmeter run log file (jmeter.log)
-n, --nongui
run JMeter in nongui mode
...
-J, --jmeterproperty <argument>=<value>
Define additional JMeter properties
-G, --globalproperty <argument>=<value>
Define Global properties (sent to servers)
e.g. -Gport=123
or -Gglobal.properties
-D, --systemproperty <argument>=<value>
Define additional system properties
-S, --systemPropertyFile <argument>
additional system property file(s)This is a snippet (non-exhaustive list) of what you might see if you did the same.
Summary
In this article we have learnt relationship between performance testing and tuning, and how to install and run JMeter.
Resources for Article:
Further resources on this subject:
- Functional Testing with JMeter [article]
- Creating an Apache JMeter™ test workbench [article]
- Getting Started with Apache Spark DataFrames [article]
