Testing the Wave - Part 2

This is a follow up to Part 1 of this series looking at the Forrester Wave report from a Tester’s point of view

In Part 1 we discussed some of the capabilities that Forrester reported customers demanding from Commerce Suite vendors - Omni-channel engagement, Full Spectrum Selling and Personalization of Customer experiences. In this post we will look at the impact of using data to drive customer engagement and revenue, and the emphasis on speed to market.

Data Driven Merchant Tools

As companies are increasing using personalization to drive customer engagement and revenue they are evaluating better ways of designing these personalized experiences. They are demanding better tools, including machine learning , reporting and decision support tools to better understand customer behavior and preferences, so that they can personalize experience to deliver a positive outcome.

But machine learning tools are only as good as the data used to help in the ‘learning’ and the algorithms that drive the learning.  Incomplete data or ‘built-in bias’ within the data or the algorithms can skew the model and the results produced may not be optimal or desirable.

Identifying unacceptable models or solutions can be tricky and require a different type of testing.

There are two questions a tester must aim to answer when testing machine learning or AI capabilities:

Are the models created, valid? 

For example if the model says that you have to sell at 90% discount to maximize units sold, it may not be an acceptable model.  You must define rules that allow you to determine what is an acceptable model and what is not. Your testing must focus on validating the model against rules to make sure that they are not broken.  You may need to run your tests with a variety of data to check against these rules, as some of these failures may not easily manifest themselves. And you must be able to do this validation at scale.

Have they been implemented correctly to deliver the desired personalization

Tests should include simulations to cover all user profiles and data combinations to make sure that the models are robust. 

Agility and Faster Time to Market

Companies are looking for a faster time to market and are looking to Cloud based solutions and service oriented architecture.  The ability to roll out changes faster on Cloud platforms is an advantage, but also places a great deal of responsibility on the testing team to make sure that things that work well don’t break when new features are released.  Regression testing is critical and has to be done regularly and consistently.  Regression testing of all features and making sure the implementation works correctly on all browsers and devices require a lot of effort and can be time consuming if done manually.  Automating these tests will bring the speed, scale and efficiency required to get to market faster.  Further, integrating these automated tests with DevOps will help implement continuous testing into agile development processes and speed up time to market.

In the final part of this series we will explore various approaches taken by companies while implementing eCommerce platforms and their implications on testing.

Stay Tuned!