How do I test eCommerce platform upgrades

Internet Retailer reports that 56% of eCommerce retailers will change their eCommerce platforms this year.  While the reasons for this change are many, a key takeaway is that retailers are investing in their eCommerce applications to improve revenue growth and customer experience. 

Any change to eCommerce applications brings with it the need for testing, and as the pace and volume of change increase, so does the volume of testing.  As they adopt new eCommerce platforms, retailers are being faced with a new dimension of change, one that is driven by the vendor of the eCommerce platform rather than the retailer.

Platform Upgrades

As eCommerce platform vendors compete for retailer’s business, they are pushing new platform features and capabilities at a fever pace. Until recently, an eCommerce retailer that ran their eCommerce application in-house or within their data center (‘premises install’) could choose if they wanted to ‘apply’ a specific platform upgrade.  Applying an upgrade meant that the retailer (or their development team or partner) has to reconcile the customizations and integrations that are built on top of the platform with the new version of the platform.  This process requires extensive testing, and can be easier to put off.  So many retailers kick the proverbial can down the road, until they really need to upgrade, to leverage a feature available on a newer version of the platform.  This procrastination often comes at a cost because retailers find that they have accumulated so many changes and customizations, that the upgrade process is as complex as a whole new project, or some times involves the same effort as a complete change in platform. So changing platforms soon becomes a viable option to applying the upgrades.

Platform vendors are trying to manage this customer churn by mandating that their customers apply the upgrades in a timely manner.  Cloud and SaaS eCommerce platform vendors have a greater deal of control in this regard as they are able to ‘force’ retailers to apply the upgrade, since the eCommerce application is being run/hosted in a ‘controlled environment’. 

Plan of action for eCommerce retailers

Believe it or not, applying platform upgrades in a timely manner can actually be beneficial to the eCommerce retailer.  It helps them stay up to date on bug fixes, and provides them with access to new product features that may provide a competitive advantage.  The biggest challenge for retailers is to identify the impact of applying an upgrade, and any changes that have to be made to the application to resolve conflicts between customizations and the new version of the platform.  A robust test automation solution is the right way to address this challenge. 

What should it include?

A good test automation solution for eCommerce must include the following types of tests – smoke tests or sanity tests to verify that critical workflows are not impacted by any changes, regression tests that thoroughly validate all application features and integrations, cross browser/device tests that make sure that the user experience on all browsers and devices used by a large majority of the retailer’s customers is consistent, and performance testing and monitoring to make sure that the site can handle the expected user loads, and continue to perform within acceptable limits.

Tests to certify a platform upgrade must include all the above types of tests.  These can be performed sequentially with subsequent tests being executed only if the preceding ones succeed.  The specific criteria for success and certification can be developed upfront with involvement of all stakeholders. 

And don’t forget these

There are some other critical factors to consider when building or implementing an automated testing solution for platform upgrades.

Keep the automation current

The solution must also enable the retailer to keep the automation up to date.  This applies to both the test workflow and the test data.  For eCommerce applications, maintaining the latter can be a challenge as product, content and other data changes frequently, the data in different environments are not usually kept in sync. Add to this the fact that not all environments may support the integrations to external systems available on the production applications.

Baseline, baseline, baseline!

Having a clean baseline to compare the results of tests run against the application after the platform upgrade has been applied is important to understand the ‘actual impact’ of the upgrade.  Since the versions of the application and data can be different on different environment (to support new feature development), appropriate baselines must be developed and maintained.  Baselines must be updated frequently to avoid backlogs.

Keep the tests clean

Do not combine testing new product features, or integrations with testing a platform upgrade.  It can be difficult to identify root cause of issues, and often there could be many issues from multiple sources. Also keeping the test environment ‘clean’ between the development of a baseline, and application of the upgrade will speed up the testing process, and will keep the results accurate.


Keeping your eCommerce platform up-to-date can really help deliver a high quality customer experience and increase revenues.Platform updates introduce change to your eCommerce infrastructure and require testing to make sure that your operations are not adversely impacted.Applying platform updates in a timely manner requires an efficient, scalable and reliable testing that can be accomplished through a robust automation solution. eureQa has helped our customers cut down the time taken for testing platform upgrades from weeks to less than 2 days.