What tests should I automate for digital commerce?

We are seeing a great deal of interest amongst online merchants in using automation to increase the amount of testing on their digital commerce applications.  One question that keeps coming up – “I believe automation can help me with my testing, but I am not sure what I should automate.  How do I get started?

We will try to answer this question here.

Automation can help improve the testing process for all types of software including digital commerce applications.  Automation works best for tests that must be run frequently, in a highly scalable manner where large numbers of tests have to be completed in a short period of time. Automation also has the advantage of ensuring consistency by eliminating ‘human error’ in testing.  

Testing helps identify errors introduced when ‘something changes’ in an application. While code/feature change is the frequent driver of tests, digital commerce applications have ‘change’ introduced by many factors.  Focusing the testing on these factors will provide the best return on investment in automation.

  • Regression testing application features: When you release new features you have to make sure that you don’t break something that already works. Automating tests that cover all features in the user journey and path to purchase workflows in your eCommerce web store or mobile app helps you quickly identify regression errors early in the development process. Your regression test should include product search, PLP, PDP, shopping cart, checkout, payment, registration, order status, approvals (for B2B) and any other key revenue producing workflows.

  • Content testing: Content changes, especially those relating to product information occur frequently as merchants add, remove or update their catalogs. If these changes are being applied to multiple sites in different countries and languages, the permutations of tests to be performed can go up significantly. Verifying that content is being updated accurately on all these channels, and is done so in a timely manner requires fast testing at a scale that is best done with automation.

  • Cross browser & device testing: Your customer can get to your web storefront using a myriad of mobile or desktop devices running different operating systems and browsers. You have to test every workflow of the user journey and path to purchase on this ever-changing list of device/OS/browser combinations to make sure that your users have a consistent experience on your applications. Using automation for this cross browser and cross device testing is an efficient way to solve this problem. You can avoid the hassle, effort and expense of building out the infrastructure to run these tests by using automation platforms that provide access to a large set of device/OS/browsers combinations on the Cloud.

  • Promotions, Coupons and Discounts: Promotions, coupons, discounts, daily deals and a number of other mechanisms to drive eCommerce sales are gaining popularity. Most of these are ‘configured’ in the eCommerce applications as rules or data, which means that changing them does not require a ‘code release’. Given the frequency of these changes and the fact that multiple promotions or discounts can be applied at the same time, these changes can easily give rise to errors. Automated tests that leverage data driven testing approaches can help validate the large volume of scenarios required to make sure that these promotions work the way they are designed, in a very short period of time.

  • Personalization & Localization: Personalization of user experience, user loyalty programs, wishlists, favorites and many features designed to improve a customer’s experience on an eCommerce app, place great demands on testing. B2B commerce sites should include testing custom catalogs and customer specific contracts as well. Automated tests that use data driven testing to simulate different experiences for users as they navigate the application can ensure adequate test coverage in a cost and time efficient manner.

  • Integrations: Integrations to external and back end systems like payment gateway, shipping/logistics systems, PIM, OMS, ERP, CMS and many other are integral parts of most eCommerce applications. Automation suites for eCommerce applications must include tests to make sure that these integrations are working correctly and reliably. These automation suites must include tests that validate both the integration points and end-to-end transaction workflows.

  • Load/Performance Testing: The performance of your eCommerce web store has direct correlation to customer conversion on your eCommerce app. Measuring application response times under load conditions for critical user actions like displaying results of a product search, rendering PDP, adding/removing items from cart, placing an order etc., is important. And doing so on a regular basis (as content, configurations, data and features change) is critical to identifying performance bottlenecks and eliminating problems that reduce conversion and bleed revenue.

  • Monitoring: Once you have launched your eCommerce site, making sure it continues to work and perform as desired requires continuous monitoring. You can do this by running a synthetic transaction, at regular intervals (once an hour is a good start) that simulates the User Journey from the the time a user gets to your site all the way to making a purchase. Make sure that you run multiple transactions that cover all journeys to identify any issues that may arise in infrequently used paths. You should also collect metrics for key usage steps like, search, PDP access, placing an item in the cart, checkout and placing an order. Continuously monitoring these metrics can help you proactively identify issues that adversely impact customer experience. Finally make sure that you can automatically notify the appropriate teams when issues are identified. This can help you fix problems before they hurt your revenue.

This list is based on our experience testing over a 100 B2C and B2B eCommerce sites, and is a great start to get the most benefit out of test automation for digital commerce applications.