Root Cause Analysis – The Importance of Asking ‘Why’?

Automating your tests provides you with the convenience of being able to run them more frequently, which is the key to identifying issues with your app or enterprise software implementation as early as possible, so problems can be fixed quicker. Take a look at any automation tool or framework on the market and you will see the focus on building automation scripts faster, running automation on more browsers and devices, and using AI to reduce the maintenance of tests.

But while all these will help you determine ‘what’ failed, and ‘where’ the failure occurred, there is one area that seems to be often ignored – ‘why’ did the failure occur?  Because the ‘why’ identifies the root cause of the failure, this is the most important part of the equation that needs to be known. 

Knowing what failed and where it failed, helps you fix the failure, but knowing why a test failed, i.e. knowing the root cause of failure, helps you prevent a failure in the future.

So, ask yourself the following questions:

  • How often do you capture this root cause information? 

  • Do you capture this information in an organized manner? 

  • Are you able to report on it, look for patterns in these failures across different tests and different software releases?

  • What efforts are being employed to identify this root cause and track this information?

 Wouldn’t it be great if your test automation tools were able to automatically figure out the root cause of failure as well as provide support for capturing and tracking the information? Imagine how much time and money it could save you, and how much more stable and dependable your apps or software could be.  

This is the potential that’s made available to you when you use the eureQa automation platform. The eureQa platform is specially designed to track and report the root causes of test failures. Our AI and ML capabilities operate automatically, always looking for patterns in failure to help you identify the root cause of failure of tests, so your team can make sure they never happen again.

Remember, if you don’t know ‘why’ your tests are failing, you will never solve the problem that’s preventing your app or software from being as good as it can be. Try eureQa today and take advantage of our innovative suite of features, each geared to help you build, run, and manage an effective test automation solution. 

Let us know what you think. 

How do I test eCommerce platform upgrades

Internet Retailer reports that 56% of eCommerce retailers will change their eCommerce platforms this year.  While the reasons for this change are many, a key takeaway is that retailers are investing in their eCommerce applications to improve revenue growth and customer experience. 

Any change to eCommerce applications brings with it the need for testing, and as the pace and volume of change increase, so does the volume of testing.  As they adopt new eCommerce platforms, retailers are being faced with a new dimension of change, one that is driven by the vendor of the eCommerce platform rather than the retailer.

Platform Upgrades

As eCommerce platform vendors compete for retailer’s business, they are pushing new platform features and capabilities at a fever pace. Until recently, an eCommerce retailer that ran their eCommerce application in-house or within their data center (‘premises install’) could choose if they wanted to ‘apply’ a specific platform upgrade.  Applying an upgrade meant that the retailer (or their development team or partner) has to reconcile the customizations and integrations that are built on top of the platform with the new version of the platform.  This process requires extensive testing, and can be easier to put off.  So many retailers kick the proverbial can down the road, until they really need to upgrade, to leverage a feature available on a newer version of the platform.  This procrastination often comes at a cost because retailers find that they have accumulated so many changes and customizations, that the upgrade process is as complex as a whole new project, or some times involves the same effort as a complete change in platform. So changing platforms soon becomes a viable option to applying the upgrades.

Platform vendors are trying to manage this customer churn by mandating that their customers apply the upgrades in a timely manner.  Cloud and SaaS eCommerce platform vendors have a greater deal of control in this regard as they are able to ‘force’ retailers to apply the upgrade, since the eCommerce application is being run/hosted in a ‘controlled environment’. 

Plan of action for eCommerce retailers

Believe it or not, applying platform upgrades in a timely manner can actually be beneficial to the eCommerce retailer.  It helps them stay up to date on bug fixes, and provides them with access to new product features that may provide a competitive advantage.  The biggest challenge for retailers is to identify the impact of applying an upgrade, and any changes that have to be made to the application to resolve conflicts between customizations and the new version of the platform.  A robust test automation solution is the right way to address this challenge. 

What should it include?

A good test automation solution for eCommerce must include the following types of tests – smoke tests or sanity tests to verify that critical workflows are not impacted by any changes, regression tests that thoroughly validate all application features and integrations, cross browser/device tests that make sure that the user experience on all browsers and devices used by a large majority of the retailer’s customers is consistent, and performance testing and monitoring to make sure that the site can handle the expected user loads, and continue to perform within acceptable limits.

Tests to certify a platform upgrade must include all the above types of tests.  These can be performed sequentially with subsequent tests being executed only if the preceding ones succeed.  The specific criteria for success and certification can be developed upfront with involvement of all stakeholders. 

And don’t forget these

There are some other critical factors to consider when building or implementing an automated testing solution for platform upgrades.

Keep the automation current

The solution must also enable the retailer to keep the automation up to date.  This applies to both the test workflow and the test data.  For eCommerce applications, maintaining the latter can be a challenge as product, content and other data changes frequently, the data in different environments are not usually kept in sync. Add to this the fact that not all environments may support the integrations to external systems available on the production applications.

Baseline, baseline, baseline!

Having a clean baseline to compare the results of tests run against the application after the platform upgrade has been applied is important to understand the ‘actual impact’ of the upgrade.  Since the versions of the application and data can be different on different environment (to support new feature development), appropriate baselines must be developed and maintained.  Baselines must be updated frequently to avoid backlogs.

Keep the tests clean

Do not combine testing new product features, or integrations with testing a platform upgrade.  It can be difficult to identify root cause of issues, and often there could be many issues from multiple sources. Also keeping the test environment ‘clean’ between the development of a baseline, and application of the upgrade will speed up the testing process, and will keep the results accurate.


Keeping your eCommerce platform up-to-date can really help deliver a high quality customer experience and increase revenues.Platform updates introduce change to your eCommerce infrastructure and require testing to make sure that your operations are not adversely impacted.Applying platform updates in a timely manner requires an efficient, scalable and reliable testing that can be accomplished through a robust automation solution. eureQa has helped our customers cut down the time taken for testing platform upgrades from weeks to less than 2 days.

What tests should I automate for digital commerce?

We are seeing a great deal of interest amongst online merchants in using automation to increase the amount of testing on their digital commerce applications.  One question that keeps coming up – “I believe automation can help me with my testing, but I am not sure what I should automate.  How do I get started?

We will try to answer this question here.

Automation can help improve the testing process for all types of software including digital commerce applications.  Automation works best for tests that must be run frequently, in a highly scalable manner where large numbers of tests have to be completed in a short period of time. Automation also has the advantage of ensuring consistency by eliminating ‘human error’ in testing.  

Testing helps identify errors introduced when ‘something changes’ in an application. While code/feature change is the frequent driver of tests, digital commerce applications have ‘change’ introduced by many factors.  Focusing the testing on these factors will provide the best return on investment in automation.

  • Regression testing application features: When you release new features you have to make sure that you don’t break something that already works. Automating tests that cover all features in the user journey and path to purchase workflows in your eCommerce web store or mobile app helps you quickly identify regression errors early in the development process. Your regression test should include product search, PLP, PDP, shopping cart, checkout, payment, registration, order status, approvals (for B2B) and any other key revenue producing workflows.

  • Content testing: Content changes, especially those relating to product information occur frequently as merchants add, remove or update their catalogs. If these changes are being applied to multiple sites in different countries and languages, the permutations of tests to be performed can go up significantly. Verifying that content is being updated accurately on all these channels, and is done so in a timely manner requires fast testing at a scale that is best done with automation.

  • Cross browser & device testing: Your customer can get to your web storefront using a myriad of mobile or desktop devices running different operating systems and browsers. You have to test every workflow of the user journey and path to purchase on this ever-changing list of device/OS/browser combinations to make sure that your users have a consistent experience on your applications. Using automation for this cross browser and cross device testing is an efficient way to solve this problem. You can avoid the hassle, effort and expense of building out the infrastructure to run these tests by using automation platforms that provide access to a large set of device/OS/browsers combinations on the Cloud.

  • Promotions, Coupons and Discounts: Promotions, coupons, discounts, daily deals and a number of other mechanisms to drive eCommerce sales are gaining popularity. Most of these are ‘configured’ in the eCommerce applications as rules or data, which means that changing them does not require a ‘code release’. Given the frequency of these changes and the fact that multiple promotions or discounts can be applied at the same time, these changes can easily give rise to errors. Automated tests that leverage data driven testing approaches can help validate the large volume of scenarios required to make sure that these promotions work the way they are designed, in a very short period of time.

  • Personalization & Localization: Personalization of user experience, user loyalty programs, wishlists, favorites and many features designed to improve a customer’s experience on an eCommerce app, place great demands on testing. B2B commerce sites should include testing custom catalogs and customer specific contracts as well. Automated tests that use data driven testing to simulate different experiences for users as they navigate the application can ensure adequate test coverage in a cost and time efficient manner.

  • Integrations: Integrations to external and back end systems like payment gateway, shipping/logistics systems, PIM, OMS, ERP, CMS and many other are integral parts of most eCommerce applications. Automation suites for eCommerce applications must include tests to make sure that these integrations are working correctly and reliably. These automation suites must include tests that validate both the integration points and end-to-end transaction workflows.

  • Load/Performance Testing: The performance of your eCommerce web store has direct correlation to customer conversion on your eCommerce app. Measuring application response times under load conditions for critical user actions like displaying results of a product search, rendering PDP, adding/removing items from cart, placing an order etc., is important. And doing so on a regular basis (as content, configurations, data and features change) is critical to identifying performance bottlenecks and eliminating problems that reduce conversion and bleed revenue.

  • Monitoring: Once you have launched your eCommerce site, making sure it continues to work and perform as desired requires continuous monitoring. You can do this by running a synthetic transaction, at regular intervals (once an hour is a good start) that simulates the User Journey from the the time a user gets to your site all the way to making a purchase. Make sure that you run multiple transactions that cover all journeys to identify any issues that may arise in infrequently used paths. You should also collect metrics for key usage steps like, search, PDP access, placing an item in the cart, checkout and placing an order. Continuously monitoring these metrics can help you proactively identify issues that adversely impact customer experience. Finally make sure that you can automatically notify the appropriate teams when issues are identified. This can help you fix problems before they hurt your revenue.

This list is based on our experience testing over a 100 B2C and B2B eCommerce sites, and is a great start to get the most benefit out of test automation for digital commerce applications.

Should I use JMeter or Selenium for load and performance testing of Web applications?

Making sure that your software performs well under expected (or anticipated) peak loads is critical.  Load and performance testing can provide valuable data to identify bottlenecks and problem areas in your software. 

The main options available for load and performance testing of Web applications are:

  • HTTP or Protocol based testing
  • Selenium or browser based testing
  • Hybrid option using a combination of protocol and browser based testing

These options are very different in the way they work and the results they deliver.

HTTP or Protocol based testing

Protocol based testing solutions (like the ones built with JMeter, Gatling etc.) load the servers with traffic at the protocol (HTTP) level and assess the response of the server to that traffic.  So if you are looking to see how your servers perform under load conditions this works really well.  Additionally, since you are driving protocol (HTTP/S) level traffic, it is easy to generate really large volumes of traffic with relative ease.

Protocol based testing solutions however don’t do a good job of providing a measure of the user experience under these load conditions, especially for Web applications that have a rich UI. 

Today’s digital applications place a great deal of emphasis on a rich user experience and rely heavily on technologies like AJAX and jQuery that asynchronously update the UI without requiring a complete page refresh.  These UI with their asynchronous (and sometimes continuous) traffic between the client and servers, place demands on the servers that cannot be easily simulated by the traditional protocol based load-testing solutions like JMeter and Gatling.  If you want to do so, you will have to write a lot of code and jump through a number of hoops to get to something that is even remotely close to the experience of a user on an actual browser.

Selenium or Browser based testing

Running load tests with browsers remains the easiest way to simulate and measure user experience for digital applications.  Tests that use tools like Selenium to control the browser can realistically simulate the user journey and measure the response times to update various UI/content elements on the screen. Since the browser handles these updates (via AJAX or jQuery or similar technologies), it eliminates the need for custom coding making these tests a lot easier to build. 

The metrics gathered by tests run on browsers also provide a more accurate representation of the user experience under load conditions – just what you want from your performance testing solution.

The common complaint against using Selenium/browser based performance testing has been:

  • Browsers have a larger footprint (CPU and memory usage) compared to solutions like JMeter or Gatling, which places a greater demand on the machine simulating the load
  • The cost of provisioning large numbers of browsers can be high
  • The effort involved in coordinating tests running across these browsers. 

These may have been valid reasons years ago, but they are no longer the case today.

Headless versions of leading browsers like Chrome have a negligible footprint, and can be scaled without placing undue loads on the machines generating the loads.  Further, the ability to massively scale and control tests running on these browsers on the cloud with platforms like eureQa have made using Selenium/browser based testing a viable, easy to use and cost-effective option.

Hybrid approaches that use HTTP or protocol based tests as a primary means to load the server, while using a few Selenium or browser based tests to measure user experience are also being use.  These have the advantage of loading the server with simple protocol based traffic, while measuring the user experience on the browser based tests.  The effectiveness of this approach for rich UI based application however depends on the ratio of the two types of traffic (i.e. virtual users) used in the test.  Higher the ratio of browser based tests to protocol based tests, the better the quality of performance test results.  In reality, most such approaches limit the browser based test traffic to less than 10% of the total virtual users.  This delivers a very limited sample set to get realistic performance metrics.

The Upshot

If your application has a relatively simple user interface with updates primarily performed via page loads, HTTP or protocol based performance testing solution should work well for you.  However, if you are testing a digital application that provides a rich, content heavy user experience, you should seriously consider a Selenium or browser based performance testing approach.

This article does not cover other factors like ease of test development and the role of test data in performance testing.  These are important considerations that will be addressed in another post.





Should I build a Selenium automation framework or buy a commercial automation tool?

Build vs. Buy – This is a common question facing people considering software test automation, especially for web browser based software. The answer depends on your specific testing needs and a number of other factors.  In this article we will try to help you analyze these factors and think through the process of making a build vs. buy decision

Even though eureQa offers a complete SaaS+Cloud test automation solution, we understand that the eureQa automation platform may not be for everyone.  Based on your specific situation, building a Selenium (or Webdriver, Selendroid or Appium) automation solution or framework in-house may work better for you. Our goal is to make sure that you have all the information you need to make an informed decision. So lets get started.

The three most common criteria that a majority of our customers consider when making this choice are – cost, customization and control.



The open source and zero license cost nature of Selenium is attractive when compared to the license or subscription costs of commercial automation solutions. But there is more to it than just that when you look at the cost of implementing an automation solution.

Selenium is open source and free, however building and using an automation solution on this ‘free’ software comes with some real costs:

  • The open source tools/libraries provide a starting point but you still have to write code for test creation, management, execution, handling errors and reporting.  Note that this is over and above the test logic that you want to automate.
  • If you are looking to test integrations with back-end systems and external systems via API, you have to build in support for that into your framework.
  • If you are looking to implement a continuous testing solution, you must add to the above effort, the effort of building integration with DevOps or CI/CD tools.

You can see that building this solution is now a second development effort, complete with its own software development team and project managers.  You will need skilled developers to build and maintain this solution and project managers to manage the effort.

  • Also to be considered are the costs of building and managing an infrastructure to run these tests.  These costs are likely to go up as you scale your infrastructure in order to support increased test volumes or to increase the pace of testing.
  • Lastly, if you want to run cross browser/device tests, you are likely to incur more costs to build out that infrastructure or you can subscribe to selenium-on-the-cloud providers like Sauce Labs or BrowserStack.

As you can see there is a definite upfront and ongoing cost of building your own selenium based automation solution.  Additionally, since you will be using development resources to build this solution, you will also have to consider the opportunity costs, i.e. if these resources were not engaged in building your solution, could they be deployed to some other potentially ‘higher value’ task. This will give you a measure of the total cost of ownership of building your own solution.

Now compare this against the cost of procuring and deploying commercial automation software that have one-time license or ongoing subscription costs:

  • The licenses and subscriptions typically include costs for upgrades, which eliminate the cost of maintaining the tool itself.
  • As you scale test automation, your license and subscription costs may go up, though most vendors tend to offer volume discounts.
  • Ease of automation, along with the skills required to build and maintain automation vary based on tools selected, but in general these are lower (less expensive) than what is required with building your own selenium solution.
  • Most commercial automation solutions require you to build and manage your own test infrastructure to run the solution, which adds to the total cost of using these solutions.  Others like eureQa deploy their solution as SaaS+Cloud including all infrastructure costs into subscription costs and provide the convenience of a fixed predictable cost structure for test automation.

The Upshot

A good understanding of total costs of ownership (the true cost) of each option is important before you make a choice of a test automation solution.  Building your own Selenium based automation solution may work when you have a small set of tests that don’t change much, but as your testing needs, test complexity and your test suite grow, a commercial solution may be the more cost-effective option.



Does the automation solution work for my specific application and testing needs?

This is another common question from companies that have specific application features, workflows or integrations. 

A custom developed Selenium solution provides the flexibility to handle specific workflows or integration needs, but it comes with the added cost (see above) of the development of this capability and more importantly maintenance of it.

Most commercial test automation software tools provide some type of support for customization.  Building this customization typically requires a higher level of (programming) skill than test automation with these tools, and this cost has to be factored into your cost calculations (above).

One important factor to consider when building customizations is the upgrade path for the commercial software.  Make sure that customizations use a well-published API that will continue to be supported, and will not interfere with upgrades.  While premises installed (or single tenant) software provide a lot more flexibility with respect to customizations, SaaS solutions, like eureQa, typically do a better job of making sure that customizations are upgrade friendly, helping reduce your costs in the long run. 

The Upshot

Among companies that have this need, it is important to point out that the tests for the truly specific features and integrations are usually a smaller subset of the overall suite of tests to be automated.  Thus any decision on an approach to customization must take into account the relative weightage of the tests that are impacted.



I am worried that I will be boxed into a solution that does not evolve as my applications and testing needs change.

This is a perfectly valid concern for anyone making a technology choice in today’s fast paced world. 

A custom built Selenium framework gives you complete control over the features that are built into it. It allows you to add any features that you need as and when you need them, so long as you are willing and able to spend the time, effort and money to build these features into the framework. Conversely, it also means that the only features your framework will have are the ones that you build in it, which limits innovation to a single source – you!

With commercial automation solutions, you are dependent on the tool vendor to add new features and updates to the tool. Most vendors don’t charge for updates, but you also don’t have too much control on their release schedule.  You however have the benefit of feedback and feature requests from a large pool of user/customers, that go into determining what gets added to the tool.  You gain from the learning, experience and innovation coming out of a larger group of peers.

If you are evaluating commercial automation software ask your vendor about their practice for soliciting customer/user feedback and the frequency of upgrades. At eureQa, our customers request features and enhancements through our support portal and one third of our minor enhancement release cycle (every two weeks) is dedicated to these requests making sure that we evolve with our customers’ needs. As a SaaS platform, we do have the added benefit of being able to make sure that all our customers get all the updates in a timely manner.

The Upshot

While it is easy to evaluate the current capabilities of an automation solution at the time of purchase, the future feature set is more nebulous.  As a result, it is easy to succumb to the ‘fear of the unknown’, and ignoring all other factors lean heavily, towards a custom Selenium solution in an attempt to control the uncertain.

A preferred approach would be to look at this feature uncertainty as a ‘risk’ and identify a mitigation strategy that takes into account all the factors discussed above in the decision-making process.

The choice of an automation solution is an important component of your software quality process.  We hope this article is helpful in your evaluation process. 

Feel free to share your thoughts and feedback on this article.


5 takeaways from IRCE 2017

The eureQa team had a great time at the Internet Retailer Conference and Expo (IRCE) this year. Like many companies in our space we spend much of our time in the office, heads down working to refine our product and generate leads, so it was nice to get out and be amongst others who have a passion for eCommerce.

Going into the show we knew there would be a ton of interesting products and companies to check out. We were not disappointed.  While there were some many things to talk about, here are the top 5 takeaways for us from IRCE 2017:

The Platform Wars

eCommerce should just be called “Commerce” at this point, we’re seeing more retailers start online and then build brick and mortar locations as a sign of their viability. The big (and not so big) players in this space know this. We saw incredible displays from CloudCraze,  Shopify, Magento, WooCommerce, BigCommerce and more (shoutout to the small but mighty “” team here, they were excellent neighbors and had a great foosball table). These providers understand how important is to have a platform that is customizable, robust, easy to use and most importantly can scale. When Shopify is making it snow in Chicago in June, you know these platforms are working hard to bring eyeballs to their products.

Warehouse Automation Tools

You don’t need to sell the eureQa team on the benefits of automation, we preach it every day. However, seeing these tools in action, as shelves zipped around grabbing a specific part, keeping track of stock in what felt like nanoseconds is the type of in person experience we dream off.

Commerce Focused Agencies

Digital agencies know the importance of specialization. Just as you have agencies focusing on App development, SEO and Animation we’re seeing the rise of the Commerce agency. Some of these hitch their wagons to a specific eCommerce platform that is either open source or proprietary, while others are more platform agnostic. They all clearly have the same goal: Converting visitors to paying customers. Seeing this granular focus on site architecture, customer engagement and CX was a wonderful thing.

Curalate’s Image Recognition and Suggestion Engine:

Maybe it’s the Philly connection, maybe it’s that they had their booth 2 spots down from ours but kudos to the new image based suggestion engine from Curalate. The ability to upsell based on the image itself, not just other shoppers history, or tracking is the type of advancement we wanted to see at the show, Curalate did not disappoint.

Fidget Spinners

No list about what you saw at IRCE would be complete without a shout out to the most dominant cultural trend of 2017 (Salt Bae who?) the Fidget Spinner. Watching people go from mocking these toys to mindlessly spinning them as they walked the convention floor was just incredible. I brought home 7 different types which pleased my kids to no end.

IRCE 2017 was a great time, not to get emotional, but when the lights went down at the end of the show and the exhibitors all clapped at the same time it was hard not to get swept up in the shared purpose of us all. eCommerce isn’t the future, it is now, it represents the crossroads of what makes America great. Manufacturers and developers working to build something memorable and enjoyable.  Operations, fulfilment and shipping all working together to brighten someone’s day. All these people, and more were in one giant room and for 3 days the purpose was clear: “Put the customer first and success will follow.” We at eureQa are proud to be part of this and can’t wait to see what the future brings.

Testing the Wave - Part 2

This is a follow up to Part 1 of this series looking at the Forrester Wave report from a Tester’s point of view

In Part 1 we discussed some of the capabilities that Forrester reported customers demanding from Commerce Suite vendors - Omni-channel engagement, Full Spectrum Selling and Personalization of Customer experiences. In this post we will look at the impact of using data to drive customer engagement and revenue, and the emphasis on speed to market.

Data Driven Merchant Tools

As companies are increasing using personalization to drive customer engagement and revenue they are evaluating better ways of designing these personalized experiences. They are demanding better tools, including machine learning , reporting and decision support tools to better understand customer behavior and preferences, so that they can personalize experience to deliver a positive outcome.

But machine learning tools are only as good as the data used to help in the ‘learning’ and the algorithms that drive the learning.  Incomplete data or ‘built-in bias’ within the data or the algorithms can skew the model and the results produced may not be optimal or desirable.

Identifying unacceptable models or solutions can be tricky and require a different type of testing.

There are two questions a tester must aim to answer when testing machine learning or AI capabilities:

Are the models created, valid? 

For example if the model says that you have to sell at 90% discount to maximize units sold, it may not be an acceptable model.  You must define rules that allow you to determine what is an acceptable model and what is not. Your testing must focus on validating the model against rules to make sure that they are not broken.  You may need to run your tests with a variety of data to check against these rules, as some of these failures may not easily manifest themselves. And you must be able to do this validation at scale.

Have they been implemented correctly to deliver the desired personalization

Tests should include simulations to cover all user profiles and data combinations to make sure that the models are robust. 

Agility and Faster Time to Market

Companies are looking for a faster time to market and are looking to Cloud based solutions and service oriented architecture.  The ability to roll out changes faster on Cloud platforms is an advantage, but also places a great deal of responsibility on the testing team to make sure that things that work well don’t break when new features are released.  Regression testing is critical and has to be done regularly and consistently.  Regression testing of all features and making sure the implementation works correctly on all browsers and devices require a lot of effort and can be time consuming if done manually.  Automating these tests will bring the speed, scale and efficiency required to get to market faster.  Further, integrating these automated tests with DevOps will help implement continuous testing into agile development processes and speed up time to market.

In the final part of this series we will explore various approaches taken by companies while implementing eCommerce platforms and their implications on testing.

Stay Tuned!




Testing the Wave – A Tester’s view of The Forrester Wave reports on B2C Commerce and B2B Commerce Suites – Part I

Forrester released its 2017 version of the Forrester WaveTM analysis of the B2C and B2B platform suite last month.  The report details what customers are looking for in eCommerce platforms and how they are buying these.

In the author’s words, “This report shows how each provider measures up and helps eBusiness and channel strategy professionals make the right choice.”

I however, chose to look at it with a different point of view. Of a tester responsible for making sure that the “right choice” actually “works right”.

More than just Shopping Carts

In this first of three blog posts I will focus on some of the capabilities that customers are demanding from eCommerce Platform vendors and implications of having to test for these capabilities.

Omni-Channel Engagement

Companies want to engage their customers on mobile, social and in the store with the same ease as they do on the Web. And they want this engagement to flow seamlessly across all these channels.  As a tester, your work just got a lot more complicated.  You no longer just have to test the user’s experience on a single web site, but you have to be able to simulate (and validate) this experience across the mobile site, POS systems used in-store and social engagement platforms. 

Here are a couple of things you can do to accomplish this :

  • Avoid the temptation to keep your tests short and within a single application. Make sure your test scenarios span user experience across the applications/systems driving various channels. 
  • Make sure that your test data and test scenario flows cover all possible combinations of omni-channel use.  You never know which path your customers end up taking, and you want to make sure that their experience is perfect.

Full Spectrum Selling

As more companies provide B2B buying online, customers want the flexibility to buy online, call in or fax in (is anyone still using this) an order, or use a combination of these methods to complete their order.  [Note: While the report did not cover machine-to-machine ordering and fulfillment, these too play a large role in B2B commerce.]

As a tester you have to make sure that the flow of orders and the experience of the customer across these channels works great.  Just as in the case of omni-channel testing, you have to make sure that your test scenarios span the use of all channels and can integrate with machine-to-machine transactions with ease.  This is an area that can get complicated if you are only doing manual testing and may require that you look into automation solutions to handle these.

Personalized Digital Commerce Experiences

Companies are looking to personalize their customer’s digital experience by providing them buying options and incentives based on their purchase preferences, location or by timing these offer correctly.  These solutions combine two main aspects that place great demands on testing – Customer data that fuels the experience and the front-end experience of the customer that is driven based off this data.  Testing these personalized digital experiences requires a strong data driven testing approach using robust and representative test data to simulate all possible experiences

  • Testing the personalization options built into the eCommerce solution will require you to simulate a variety of customer data (profiles). Your test data should include a large test data set covering all combinations of customer preference/behavior/location etc.
  • You should then be able to include data on the corresponding customer experience that has to be validated. 
  • Finally, your ability to test this multitude of combinations accurately, at scale, with efficiency is critical to a successful implementation of personalized digital experiences.

In the next part of this series we will explore how machine learning and AI concepts are making their way into eCommerce and the impact of moving Commerce to the Cloud. 

Stay tuned!

Stop Revenue Bleed - Complete the Sale!

67 percent of eCommerce shopping carts are abandoned!

Think about that statistic for a minute - Two out of three shoppers on your site leave without completing the purchase. Take the time, money and effort you spent in getting those shoppers to your site, and imagine that two-thirds of that just went up in smoke.  Poof! Something happened along the way that made them change their minds. Focus on that, fix that, and you will see your revenue climb.

Let’s dive into the statistic that I opened with - Most abandoned carts are due to factors well under your control – and you can do something about them.  

Reasons range from unexpected costs (usually shipping and handling that were not stated up front) to challenges in completing the purchase. Detecting these issues and fixing them will directly impact your revenue and bottom line.  

In addition to being upfront with your customers about the costs they can expect, you have to make sure their experience is seamless and error-free no matter what path they take in your site to make the purchase.  Otherwise, given the short attention span of shoppers, it’s Hasta la Vista – they’re gone, probably for good.

These are three areas you should focus on to make sure that your customer’s experience is a rewarding one:


  •  Provide customers with a way to checkout without having to register on your site. This is extremely important for first-time and impulse buyers.
  • Make sure registered customers can log in easily. If they can log in with a single click, so much the better.
  • For returning customers, remember their preferences, like shipping details so they don’t have to reenter the information. (Don’t you hate that?)
  •  Remember your customers’ shopping history, including abandoned carts. Offer to review/include those abandoned carts in the current sale.

Pricing, Promotions and Discounts

  • Are cart totals displayed accurately, including applicable taxes? Consistency is critical – beware of discrepancies in different paths through your site.

  • Make sure product prices are correct – particularly when an off-size item is selected. State clearly if a price is different from the price on the product page and why it is different.

  • What about different currencies? Make sure you’re handling them correctly, especially when selling internationally.
  • Verify that promotions, discounts, and sales are correct.  Itemize when necessary, and make sure the transaction is easy to understand.


  • Make sure all displayed payment options work properly. There’s nothing worse than a credit card or banking transaction that goes haywire – don’t let it happen on your site.
  • If certain options require users to register, make sure you let them know upfront – preferably during the checkout process. Make registration as effortless as possible, and bring users back to where they were.
  • For registered users, provide the option to use past payment methods.
  • Be able to send payment confirmation multiple ways – on-screen, email (if you have the address), text message, etc.

These simple steps go a long way in increasing your conversion rates and your revenue.

Change Is Constant – How Can You Keep Up?

Your eCommerce site has complex workflows behind the scenes connecting to diverse backend systems and external services.  You need to make sure that all these elements work correctly and continue working correctly with the constant changes on a dynamic retail site. Thorough testing is essential – but doing it every time you adjust a price, unveil a new product, launch a promotion and make other changes is a daunting challenge for many retailers. Traditional manual testing approaches take too long and cost too much.

Continuous testing of all site features and user journeys across the myriad of browsers and devices your customers use requires an automated testing solution. It is the only way to do testing at scale, with efficiency.

Stop the bleed!. Detect problems on your site before they afflict your customers. Get them to complete the purchase, and come back for more.

Get Personalization Wrong! It's Personal!

"Kickoff the Jim Harbaugh era with an autographed Michigan Wolverines Harbaugh T-Shirt", said the pop-up ad on the eCommerce site I had just logged on to.  It was a very personalized message designed to target a college football fan like me, on a site selling, among other things, college football merchandise at the beginning of the Big Ten football season.

It was perfect except for one thing – I went to The Ohio State University!

For the uninitiated, presenting a personalized message hawking Michigan Wolverines merchandise, like the one above, to an Ohio State Buckeye is like waving a red cape in front of a bull.  It triggers responses that are fueled by one of the storied college football rivalries dating back over a hundred years. These are not pleasant responses and certainly not the kind that the person who designed this promotion was hoping for.

So what went wrong?  It could be any one of a number of things from insufficient user profile data to incorrect or conflicting personalization rules.

One thing however is certain – the personalization implemented on the site was not adequately tested.

Personalization tools and platforms providing a myriad of ways to present relevant content and promotions to users on an eCommerce site have abounded in the past few years.  They have at their disposal a variety of data – volunteered and inferred, about the user to build personalization rules and drive specific digital experiences.  This same variety of data and complexity of rules makes the task of testing personalization that much more complex, and the likelihood that the testing is shortchanged that much more real.

It does not have to be so. Here are three things you can do to make sure that the personalization you build on your site works correctly:

Simulate all attributes that drive personalization in your testing: Your personalization rules depend on a lot of attributes (data) about your user.  Make sure that your test data includes all combinations of these to verify the personalization rules.  Pay particular attention to test data that drive negative tests – if the person testing the site mentioned above had used test data corresponding to a Buckeye in addition to a Wolverine, they would have caught the problem before it made it to the public site.  Testing for large combinations of test data is easier and significantly more economical if you use test automation that supports data driven testing.

Test the integration of the personalization engine and the website:Unit testing the personalization rules are important, but do not skip the step of testing the website after the personalization has been integrated.  Testing the web site from the user interface will show you exactly what the user experiences and will help you identify details that can be easily missed in unit testing.
Enable people designing personalization rules to perform the tests themselves: It is obvious that the person designing the personalization rules knows how they should work and is best equipped to come up with test scenarios to make sure that they work correctly.   So why not equip them to run these tests themselves right after the personalization rules are built.  They can validate the results and make sure that the personalization works just the way they intend it to.  By moving this testing process closer to the experts and testing often, you can reduce the time and cost required to test your personalization.

I would like to add a final note of caution when dealing with personalization that uses inferred data like geo-location or language in addition to user profile data.  Make sure that your test data includes these variations to tease out potential conflicts in rules between the two.  I am unsure if the site above uses this in its personalization, but as I look forward to the football season this year, I have a note of caution to the person testing this personalization – I may live in the state of the Scarlet Knights, but I still bleed Scarlet and Gray!