performance testing

What tests should I automate for digital commerce?

We are seeing a great deal of interest amongst online merchants in using automation to increase the amount of testing on their digital commerce applications.  One question that keeps coming up – “I believe automation can help me with my testing, but I am not sure what I should automate.  How do I get started?

We will try to answer this question here.

Automation can help improve the testing process for all types of software including digital commerce applications.  Automation works best for tests that must be run frequently, in a highly scalable manner where large numbers of tests have to be completed in a short period of time. Automation also has the advantage of ensuring consistency by eliminating ‘human error’ in testing.  

Testing helps identify errors introduced when ‘something changes’ in an application. While code/feature change is the frequent driver of tests, digital commerce applications have ‘change’ introduced by many factors.  Focusing the testing on these factors will provide the best return on investment in automation.

  • Regression testing application features: When you release new features you have to make sure that you don’t break something that already works. Automating tests that cover all features in the user journey and path to purchase workflows in your eCommerce web store or mobile app helps you quickly identify regression errors early in the development process. Your regression test should include product search, PLP, PDP, shopping cart, checkout, payment, registration, order status, approvals (for B2B) and any other key revenue producing workflows.

  • Content testing: Content changes, especially those relating to product information occur frequently as merchants add, remove or update their catalogs. If these changes are being applied to multiple sites in different countries and languages, the permutations of tests to be performed can go up significantly. Verifying that content is being updated accurately on all these channels, and is done so in a timely manner requires fast testing at a scale that is best done with automation.

  • Cross browser & device testing: Your customer can get to your web storefront using a myriad of mobile or desktop devices running different operating systems and browsers. You have to test every workflow of the user journey and path to purchase on this ever-changing list of device/OS/browser combinations to make sure that your users have a consistent experience on your applications. Using automation for this cross browser and cross device testing is an efficient way to solve this problem. You can avoid the hassle, effort and expense of building out the infrastructure to run these tests by using automation platforms that provide access to a large set of device/OS/browsers combinations on the Cloud.

  • Promotions, Coupons and Discounts: Promotions, coupons, discounts, daily deals and a number of other mechanisms to drive eCommerce sales are gaining popularity. Most of these are ‘configured’ in the eCommerce applications as rules or data, which means that changing them does not require a ‘code release’. Given the frequency of these changes and the fact that multiple promotions or discounts can be applied at the same time, these changes can easily give rise to errors. Automated tests that leverage data driven testing approaches can help validate the large volume of scenarios required to make sure that these promotions work the way they are designed, in a very short period of time.

  • Personalization & Localization: Personalization of user experience, user loyalty programs, wishlists, favorites and many features designed to improve a customer’s experience on an eCommerce app, place great demands on testing. B2B commerce sites should include testing custom catalogs and customer specific contracts as well. Automated tests that use data driven testing to simulate different experiences for users as they navigate the application can ensure adequate test coverage in a cost and time efficient manner.

  • Integrations: Integrations to external and back end systems like payment gateway, shipping/logistics systems, PIM, OMS, ERP, CMS and many other are integral parts of most eCommerce applications. Automation suites for eCommerce applications must include tests to make sure that these integrations are working correctly and reliably. These automation suites must include tests that validate both the integration points and end-to-end transaction workflows.

  • Load/Performance Testing: The performance of your eCommerce web store has direct correlation to customer conversion on your eCommerce app. Measuring application response times under load conditions for critical user actions like displaying results of a product search, rendering PDP, adding/removing items from cart, placing an order etc., is important. And doing so on a regular basis (as content, configurations, data and features change) is critical to identifying performance bottlenecks and eliminating problems that reduce conversion and bleed revenue.

  • Monitoring: Once you have launched your eCommerce site, making sure it continues to work and perform as desired requires continuous monitoring. You can do this by running a synthetic transaction, at regular intervals (once an hour is a good start) that simulates the User Journey from the the time a user gets to your site all the way to making a purchase. Make sure that you run multiple transactions that cover all journeys to identify any issues that may arise in infrequently used paths. You should also collect metrics for key usage steps like, search, PDP access, placing an item in the cart, checkout and placing an order. Continuously monitoring these metrics can help you proactively identify issues that adversely impact customer experience. Finally make sure that you can automatically notify the appropriate teams when issues are identified. This can help you fix problems before they hurt your revenue.

This list is based on our experience testing over a 100 B2C and B2B eCommerce sites, and is a great start to get the most benefit out of test automation for digital commerce applications.

Should I use JMeter or Selenium for load and performance testing of Web applications?

Making sure that your software performs well under expected (or anticipated) peak loads is critical.  Load and performance testing can provide valuable data to identify bottlenecks and problem areas in your software. 

The main options available for load and performance testing of Web applications are:

  • HTTP or Protocol based testing
  • Selenium or browser based testing
  • Hybrid option using a combination of protocol and browser based testing

These options are very different in the way they work and the results they deliver.

HTTP or Protocol based testing

Protocol based testing solutions (like the ones built with JMeter, Gatling etc.) load the servers with traffic at the protocol (HTTP) level and assess the response of the server to that traffic.  So if you are looking to see how your servers perform under load conditions this works really well.  Additionally, since you are driving protocol (HTTP/S) level traffic, it is easy to generate really large volumes of traffic with relative ease.

Protocol based testing solutions however don’t do a good job of providing a measure of the user experience under these load conditions, especially for Web applications that have a rich UI. 

Today’s digital applications place a great deal of emphasis on a rich user experience and rely heavily on technologies like AJAX and jQuery that asynchronously update the UI without requiring a complete page refresh.  These UI with their asynchronous (and sometimes continuous) traffic between the client and servers, place demands on the servers that cannot be easily simulated by the traditional protocol based load-testing solutions like JMeter and Gatling.  If you want to do so, you will have to write a lot of code and jump through a number of hoops to get to something that is even remotely close to the experience of a user on an actual browser.

Selenium or Browser based testing

Running load tests with browsers remains the easiest way to simulate and measure user experience for digital applications.  Tests that use tools like Selenium to control the browser can realistically simulate the user journey and measure the response times to update various UI/content elements on the screen. Since the browser handles these updates (via AJAX or jQuery or similar technologies), it eliminates the need for custom coding making these tests a lot easier to build. 

The metrics gathered by tests run on browsers also provide a more accurate representation of the user experience under load conditions – just what you want from your performance testing solution.

The common complaint against using Selenium/browser based performance testing has been:

  • Browsers have a larger footprint (CPU and memory usage) compared to solutions like JMeter or Gatling, which places a greater demand on the machine simulating the load
  • The cost of provisioning large numbers of browsers can be high
  • The effort involved in coordinating tests running across these browsers. 

These may have been valid reasons years ago, but they are no longer the case today.

Headless versions of leading browsers like Chrome have a negligible footprint, and can be scaled without placing undue loads on the machines generating the loads.  Further, the ability to massively scale and control tests running on these browsers on the cloud with platforms like eureQa have made using Selenium/browser based testing a viable, easy to use and cost-effective option.

Hybrid approaches that use HTTP or protocol based tests as a primary means to load the server, while using a few Selenium or browser based tests to measure user experience are also being use.  These have the advantage of loading the server with simple protocol based traffic, while measuring the user experience on the browser based tests.  The effectiveness of this approach for rich UI based application however depends on the ratio of the two types of traffic (i.e. virtual users) used in the test.  Higher the ratio of browser based tests to protocol based tests, the better the quality of performance test results.  In reality, most such approaches limit the browser based test traffic to less than 10% of the total virtual users.  This delivers a very limited sample set to get realistic performance metrics.

The Upshot

If your application has a relatively simple user interface with updates primarily performed via page loads, HTTP or protocol based performance testing solution should work well for you.  However, if you are testing a digital application that provides a rich, content heavy user experience, you should seriously consider a Selenium or browser based performance testing approach.

This article does not cover other factors like ease of test development and the role of test data in performance testing.  These are important considerations that will be addressed in another post.