Should I use JMeter or Selenium for load and performance testing of Web applications?

Making sure that your software performs well under expected (or anticipated) peak loads is critical.  Load and performance testing can provide valuable data to identify bottlenecks and problem areas in your software. 

The main options available for load and performance testing of Web applications are:

  • HTTP or Protocol based testing
  • Selenium or browser based testing
  • Hybrid option using a combination of protocol and browser based testing

These options are very different in the way they work and the results they deliver.

HTTP or Protocol based testing

Protocol based testing solutions (like the ones built with JMeter, Gatling etc.) load the servers with traffic at the protocol (HTTP) level and assess the response of the server to that traffic.  So if you are looking to see how your servers perform under load conditions this works really well.  Additionally, since you are driving protocol (HTTP/S) level traffic, it is easy to generate really large volumes of traffic with relative ease.

Protocol based testing solutions however don’t do a good job of providing a measure of the user experience under these load conditions, especially for Web applications that have a rich UI. 

Today’s digital applications place a great deal of emphasis on a rich user experience and rely heavily on technologies like AJAX and jQuery that asynchronously update the UI without requiring a complete page refresh.  These UI with their asynchronous (and sometimes continuous) traffic between the client and servers, place demands on the servers that cannot be easily simulated by the traditional protocol based load-testing solutions like JMeter and Gatling.  If you want to do so, you will have to write a lot of code and jump through a number of hoops to get to something that is even remotely close to the experience of a user on an actual browser.

Selenium or Browser based testing

Running load tests with browsers remains the easiest way to simulate and measure user experience for digital applications.  Tests that use tools like Selenium to control the browser can realistically simulate the user journey and measure the response times to update various UI/content elements on the screen. Since the browser handles these updates (via AJAX or jQuery or similar technologies), it eliminates the need for custom coding making these tests a lot easier to build. 

The metrics gathered by tests run on browsers also provide a more accurate representation of the user experience under load conditions – just what you want from your performance testing solution.

The common complaint against using Selenium/browser based performance testing has been:

  • Browsers have a larger footprint (CPU and memory usage) compared to solutions like JMeter or Gatling, which places a greater demand on the machine simulating the load
  • The cost of provisioning large numbers of browsers can be high
  • The effort involved in coordinating tests running across these browsers. 

These may have been valid reasons years ago, but they are no longer the case today.

Headless versions of leading browsers like Chrome have a negligible footprint, and can be scaled without placing undue loads on the machines generating the loads.  Further, the ability to massively scale and control tests running on these browsers on the cloud with platforms like eureQa have made using Selenium/browser based testing a viable, easy to use and cost-effective option.

Hybrid approaches that use HTTP or protocol based tests as a primary means to load the server, while using a few Selenium or browser based tests to measure user experience are also being use.  These have the advantage of loading the server with simple protocol based traffic, while measuring the user experience on the browser based tests.  The effectiveness of this approach for rich UI based application however depends on the ratio of the two types of traffic (i.e. virtual users) used in the test.  Higher the ratio of browser based tests to protocol based tests, the better the quality of performance test results.  In reality, most such approaches limit the browser based test traffic to less than 10% of the total virtual users.  This delivers a very limited sample set to get realistic performance metrics.

The Upshot

If your application has a relatively simple user interface with updates primarily performed via page loads, HTTP or protocol based performance testing solution should work well for you.  However, if you are testing a digital application that provides a rich, content heavy user experience, you should seriously consider a Selenium or browser based performance testing approach.

This article does not cover other factors like ease of test development and the role of test data in performance testing.  These are important considerations that will be addressed in another post.