What tests should I automate for digital commerce?

We are seeing a great deal of interest amongst online merchants in using automation to increase the amount of testing on their digital commerce applications.  One question that keeps coming up – “I believe automation can help me with my testing, but I am not sure what I should automate.  How do I get started?

We will try to answer this question here.

Automation can help improve the testing process for all types of software including digital commerce applications.  Automation works best for tests that must be run frequently, in a highly scalable manner where large numbers of tests have to be completed in a short period of time. Automation also has the advantage of ensuring consistency by eliminating ‘human error’ in testing.  

Testing helps identify errors introduced when ‘something changes’ in an application. While code/feature change is the frequent driver of tests, digital commerce applications have ‘change’ introduced by many factors.  Focusing the testing on these factors will provide the best return on investment in automation.

  • Regression testing application features: When you release new features you have to make sure that you don’t break something that already works.  Automating tests that cover all features in the user journey and path to purchase workflows in your eCommerce web store or mobile app helps you quickly identify regression errors early in the development process.  Your regression test should include product search, PLP, PDP, shopping cart, checkout, payment, registration, order status, approvals (for B2B) and any other key revenue producing workflows. 
  • Content testing: Content changes, especially those relating to product information occur frequently as merchants add, remove or update their catalogs.  If these changes are being applied to multiple sites in different countries and languages, the permutations of tests to be performed can go up significantly. Verifying that content is being updated accurately on all these channels, and is done so in a timely manner requires fast testing at a scale that is best done with automation.
  • Cross browser & device testing: Your customer can get to your web storefront using a myriad of mobile or desktop devices running different operating systems and browsers.  You have to test every workflow of the user journey and path to purchase on this ever-changing list of device/OS/browser combinations to make sure that your users have a consistent experience on your applications.  Using automation for this cross browser and cross device testing is an efficient way to solve this problem.  You can avoid the hassle, effort and expense of building out the infrastructure to run these tests by using automation platforms that provide access to a large set of device/OS/browsers combinations on the Cloud.
  • Promotions, Coupons and Discounts: Promotions, coupons, discounts, daily deals and a number of other mechanisms to drive eCommerce sales are gaining popularity.  Most of these are ‘configured’ in the eCommerce applications as rules or data, which means that changing them does not require a ‘code release’.  Given the frequency of these changes and the fact that multiple promotions or discounts can be applied at the same time, these changes can easily give rise to errors.  Automated tests that leverage data driven testing approaches can help validate the large volume of scenarios required to make sure that these promotions work the way they are designed, in a very short period of time.
  • Personalization & Localization: Personalization of user experience, user loyalty programs, wishlists, favorites and many features designed to improve a customer’s experience on an eCommerce app, place great demands on testing.  B2B commerce sites should include testing custom catalogs and customer specific contracts as well.  Automated tests that use data driven testing to simulate different experiences for users as they navigate the application can ensure adequate test coverage in a cost and time efficient manner.
  • Integrations: Integrations to external and back end systems like payment gateway, shipping/logistics systems, PIM, OMS, ERP, CMS and many other are integral parts of most eCommerce applications.  Automation suites for eCommerce applications must include tests to make sure that these integrations are working correctly and reliably. These automation suites must include tests that validate both the integration points and end-to-end transaction workflows.
  • Load/Performance Testing:  The performance of your eCommerce web store has direct correlation to customer conversion on your eCommerce app.  Measuring application response times under load conditions for critical user actions like displaying results of a product search, rendering PDP, adding/removing items from cart, placing an order etc., is important. And doing so on a regular basis (as content, configurations, data and features change) is critical to identifying performance bottlenecks and eliminating problems that reduce conversion and bleed revenue.

While the above is by no means a comprehensive list, it is a great start to get the most benefit out of test automation for digital commerce applications.

Should I use JMeter or Selenium for load and performance testing of Web applications?

Making sure that your software performs well under expected (or anticipated) peak loads is critical.  Load and performance testing can provide valuable data to identify bottlenecks and problem areas in your software. 

The main options available for load and performance testing of Web applications are:

  • HTTP or Protocol based testing
  • Selenium or browser based testing
  • Hybrid option using a combination of protocol and browser based testing

These options are very different in the way they work and the results they deliver.

HTTP or Protocol based testing

Protocol based testing solutions (like the ones built with JMeter, Gatling etc.) load the servers with traffic at the protocol (HTTP) level and assess the response of the server to that traffic.  So if you are looking to see how your servers perform under load conditions this works really well.  Additionally, since you are driving protocol (HTTP/S) level traffic, it is easy to generate really large volumes of traffic with relative ease.

Protocol based testing solutions however don’t do a good job of providing a measure of the user experience under these load conditions, especially for Web applications that have a rich UI. 

Today’s digital applications place a great deal of emphasis on a rich user experience and rely heavily on technologies like AJAX and jQuery that asynchronously update the UI without requiring a complete page refresh.  These UI with their asynchronous (and sometimes continuous) traffic between the client and servers, place demands on the servers that cannot be easily simulated by the traditional protocol based load-testing solutions like JMeter and Gatling.  If you want to do so, you will have to write a lot of code and jump through a number of hoops to get to something that is even remotely close to the experience of a user on an actual browser.

Selenium or Browser based testing

Running load tests with browsers remains the easiest way to simulate and measure user experience for digital applications.  Tests that use tools like Selenium to control the browser can realistically simulate the user journey and measure the response times to update various UI/content elements on the screen. Since the browser handles these updates (via AJAX or jQuery or similar technologies), it eliminates the need for custom coding making these tests a lot easier to build. 

The metrics gathered by tests run on browsers also provide a more accurate representation of the user experience under load conditions – just what you want from your performance testing solution.

The common complaint against using Selenium/browser based performance testing has been:

  • Browsers have a larger footprint (CPU and memory usage) compared to solutions like JMeter or Gatling, which places a greater demand on the machine simulating the load
  • The cost of provisioning large numbers of browsers can be high
  • The effort involved in coordinating tests running across these browsers. 

These may have been valid reasons years ago, but they are no longer the case today.

Headless versions of leading browsers like Chrome have a negligible footprint, and can be scaled without placing undue loads on the machines generating the loads.  Further, the ability to massively scale and control tests running on these browsers on the cloud with platforms like eureQa have made using Selenium/browser based testing a viable, easy to use and cost-effective option.

Hybrid approaches that use HTTP or protocol based tests as a primary means to load the server, while using a few Selenium or browser based tests to measure user experience are also being use.  These have the advantage of loading the server with simple protocol based traffic, while measuring the user experience on the browser based tests.  The effectiveness of this approach for rich UI based application however depends on the ratio of the two types of traffic (i.e. virtual users) used in the test.  Higher the ratio of browser based tests to protocol based tests, the better the quality of performance test results.  In reality, most such approaches limit the browser based test traffic to less than 10% of the total virtual users.  This delivers a very limited sample set to get realistic performance metrics.

The Upshot

If your application has a relatively simple user interface with updates primarily performed via page loads, HTTP or protocol based performance testing solution should work well for you.  However, if you are testing a digital application that provides a rich, content heavy user experience, you should seriously consider a Selenium or browser based performance testing approach.

This article does not cover other factors like ease of test development and the role of test data in performance testing.  These are important considerations that will be addressed in another post.





Should I build a Selenium automation framework or buy a commercial automation tool?

Build vs. Buy – This is a common question facing people considering software test automation, especially for web browser based software. The answer depends on your specific testing needs and a number of other factors.  In this article we will try to help you analyze these factors and think through the process of making a build vs. buy decision

Even though eureQa offers a complete SaaS+Cloud test automation solution, we understand that the eureQa automation platform may not be for everyone.  Based on your specific situation, building a Selenium (or Webdriver, Selendroid or Appium) automation solution or framework in-house may work better for you. Our goal is to make sure that you have all the information you need to make an informed decision. So lets get started.

The three most common criteria that a majority of our customers consider when making this choice are – cost, customization and control.



The open source and zero license cost nature of Selenium is attractive when compared to the license or subscription costs of commercial automation solutions. But there is more to it than just that when you look at the cost of implementing an automation solution.

Selenium is open source and free, however building and using an automation solution on this ‘free’ software comes with some real costs:

  • The open source tools/libraries provide a starting point but you still have to write code for test creation, management, execution, handling errors and reporting.  Note that this is over and above the test logic that you want to automate.
  • If you are looking to test integrations with back-end systems and external systems via API, you have to build in support for that into your framework.
  • If you are looking to implement a continuous testing solution, you must add to the above effort, the effort of building integration with DevOps or CI/CD tools.

You can see that building this solution is now a second development effort, complete with its own software development team and project managers.  You will need skilled developers to build and maintain this solution and project managers to manage the effort.

  • Also to be considered are the costs of building and managing an infrastructure to run these tests.  These costs are likely to go up as you scale your infrastructure in order to support increased test volumes or to increase the pace of testing.
  • Lastly, if you want to run cross browser/device tests, you are likely to incur more costs to build out that infrastructure or you can subscribe to selenium-on-the-cloud providers like Sauce Labs or BrowserStack.

As you can see there is a definite upfront and ongoing cost of building your own selenium based automation solution.  Additionally, since you will be using development resources to build this solution, you will also have to consider the opportunity costs, i.e. if these resources were not engaged in building your solution, could they be deployed to some other potentially ‘higher value’ task. This will give you a measure of the total cost of ownership of building your own solution.

Now compare this against the cost of procuring and deploying commercial automation software that have one-time license or ongoing subscription costs:

  • The licenses and subscriptions typically include costs for upgrades, which eliminate the cost of maintaining the tool itself.
  • As you scale test automation, your license and subscription costs may go up, though most vendors tend to offer volume discounts.
  • Ease of automation, along with the skills required to build and maintain automation vary based on tools selected, but in general these are lower (less expensive) than what is required with building your own selenium solution.
  • Most commercial automation solutions require you to build and manage your own test infrastructure to run the solution, which adds to the total cost of using these solutions.  Others like eureQa deploy their solution as SaaS+Cloud including all infrastructure costs into subscription costs and provide the convenience of a fixed predictable cost structure for test automation.

The Upshot

A good understanding of total costs of ownership (the true cost) of each option is important before you make a choice of a test automation solution.  Building your own Selenium based automation solution may work when you have a small set of tests that don’t change much, but as your testing needs, test complexity and your test suite grow, a commercial solution may be the more cost-effective option.



Does the automation solution work for my specific application and testing needs?

This is another common question from companies that have specific application features, workflows or integrations. 

A custom developed Selenium solution provides the flexibility to handle specific workflows or integration needs, but it comes with the added cost (see above) of the development of this capability and more importantly maintenance of it.

Most commercial test automation software tools provide some type of support for customization.  Building this customization typically requires a higher level of (programming) skill than test automation with these tools, and this cost has to be factored into your cost calculations (above).

One important factor to consider when building customizations is the upgrade path for the commercial software.  Make sure that customizations use a well-published API that will continue to be supported, and will not interfere with upgrades.  While premises installed (or single tenant) software provide a lot more flexibility with respect to customizations, SaaS solutions, like eureQa, typically do a better job of making sure that customizations are upgrade friendly, helping reduce your costs in the long run. 

The Upshot

Among companies that have this need, it is important to point out that the tests for the truly specific features and integrations are usually a smaller subset of the overall suite of tests to be automated.  Thus any decision on an approach to customization must take into account the relative weightage of the tests that are impacted.



I am worried that I will be boxed into a solution that does not evolve as my applications and testing needs change.

This is a perfectly valid concern for anyone making a technology choice in today’s fast paced world. 

A custom built Selenium framework gives you complete control over the features that are built into it. It allows you to add any features that you need as and when you need them, so long as you are willing and able to spend the time, effort and money to build these features into the framework. Conversely, it also means that the only features your framework will have are the ones that you build in it, which limits innovation to a single source – you!

With commercial automation solutions, you are dependent on the tool vendor to add new features and updates to the tool. Most vendors don’t charge for updates, but you also don’t have too much control on their release schedule.  You however have the benefit of feedback and feature requests from a large pool of user/customers, that go into determining what gets added to the tool.  You gain from the learning, experience and innovation coming out of a larger group of peers.

If you are evaluating commercial automation software ask your vendor about their practice for soliciting customer/user feedback and the frequency of upgrades. At eureQa, our customers request features and enhancements through our support portal and one third of our minor enhancement release cycle (every two weeks) is dedicated to these requests making sure that we evolve with our customers’ needs. As a SaaS platform, we do have the added benefit of being able to make sure that all our customers get all the updates in a timely manner.

The Upshot

While it is easy to evaluate the current capabilities of an automation solution at the time of purchase, the future feature set is more nebulous.  As a result, it is easy to succumb to the ‘fear of the unknown’, and ignoring all other factors lean heavily, towards a custom Selenium solution in an attempt to control the uncertain.

A preferred approach would be to look at this feature uncertainty as a ‘risk’ and identify a mitigation strategy that takes into account all the factors discussed above in the decision-making process.

The choice of an automation solution is an important component of your software quality process.  We hope this article is helpful in your evaluation process. 

Feel free to share your thoughts and feedback on this article.


5 takeaways from IRCE 2017

The eureQa team had a great time at the Internet Retailer Conference and Expo (IRCE) this year. Like many companies in our space we spend much of our time in the office, heads down working to refine our product and generate leads, so it was nice to get out and be amongst others who have a passion for eCommerce.

Going into the show we knew there would be a ton of interesting products and companies to check out. We were not disappointed.  While there were some many things to talk about, here are the top 5 takeaways for us from IRCE 2017:

The Platform Wars

eCommerce should just be called “Commerce” at this point, we’re seeing more retailers start online and then build brick and mortar locations as a sign of their viability. The big (and not so big) players in this space know this. We saw incredible displays from CloudCraze,  Shopify, Magento, WooCommerce, BigCommerce and more (shoutout to the small but mighty “freewebstore.com” team here, they were excellent neighbors and had a great foosball table). These providers understand how important is to have a platform that is customizable, robust, easy to use and most importantly can scale. When Shopify is making it snow in Chicago in June, you know these platforms are working hard to bring eyeballs to their products.

Warehouse Automation Tools

You don’t need to sell the eureQa team on the benefits of automation, we preach it every day. However, seeing these tools in action, as shelves zipped around grabbing a specific part, keeping track of stock in what felt like nanoseconds is the type of in person experience we dream off.

Commerce Focused Agencies

Digital agencies know the importance of specialization. Just as you have agencies focusing on App development, SEO and Animation we’re seeing the rise of the Commerce agency. Some of these hitch their wagons to a specific eCommerce platform that is either open source or proprietary, while others are more platform agnostic. They all clearly have the same goal: Converting visitors to paying customers. Seeing this granular focus on site architecture, customer engagement and CX was a wonderful thing.

Curalate’s Image Recognition and Suggestion Engine:

Maybe it’s the Philly connection, maybe it’s that they had their booth 2 spots down from ours but kudos to the new image based suggestion engine from Curalate. The ability to upsell based on the image itself, not just other shoppers history, or tracking is the type of advancement we wanted to see at the show, Curalate did not disappoint.

Fidget Spinners

No list about what you saw at IRCE would be complete without a shout out to the most dominant cultural trend of 2017 (Salt Bae who?) the Fidget Spinner. Watching people go from mocking these toys to mindlessly spinning them as they walked the convention floor was just incredible. I brought home 7 different types which pleased my kids to no end.

IRCE 2017 was a great time, not to get emotional, but when the lights went down at the end of the show and the exhibitors all clapped at the same time it was hard not to get swept up in the shared purpose of us all. eCommerce isn’t the future, it is now, it represents the crossroads of what makes America great. Manufacturers and developers working to build something memorable and enjoyable.  Operations, fulfilment and shipping all working together to brighten someone’s day. All these people, and more were in one giant room and for 3 days the purpose was clear: “Put the customer first and success will follow.” We at eureQa are proud to be part of this and can’t wait to see what the future brings.

Testing the Wave - Part 2

This is a follow up to Part 1 of this series looking at the Forrester Wave report from a Tester’s point of view

In Part 1 we discussed some of the capabilities that Forrester reported customers demanding from Commerce Suite vendors - Omni-channel engagement, Full Spectrum Selling and Personalization of Customer experiences. In this post we will look at the impact of using data to drive customer engagement and revenue, and the emphasis on speed to market.

Data Driven Merchant Tools

As companies are increasing using personalization to drive customer engagement and revenue they are evaluating better ways of designing these personalized experiences. They are demanding better tools, including machine learning , reporting and decision support tools to better understand customer behavior and preferences, so that they can personalize experience to deliver a positive outcome.

But machine learning tools are only as good as the data used to help in the ‘learning’ and the algorithms that drive the learning.  Incomplete data or ‘built-in bias’ within the data or the algorithms can skew the model and the results produced may not be optimal or desirable.

Identifying unacceptable models or solutions can be tricky and require a different type of testing.

There are two questions a tester must aim to answer when testing machine learning or AI capabilities:

Are the models created, valid? 

For example if the model says that you have to sell at 90% discount to maximize units sold, it may not be an acceptable model.  You must define rules that allow you to determine what is an acceptable model and what is not. Your testing must focus on validating the model against rules to make sure that they are not broken.  You may need to run your tests with a variety of data to check against these rules, as some of these failures may not easily manifest themselves. And you must be able to do this validation at scale.

Have they been implemented correctly to deliver the desired personalization

Tests should include simulations to cover all user profiles and data combinations to make sure that the models are robust. 

Agility and Faster Time to Market

Companies are looking for a faster time to market and are looking to Cloud based solutions and service oriented architecture.  The ability to roll out changes faster on Cloud platforms is an advantage, but also places a great deal of responsibility on the testing team to make sure that things that work well don’t break when new features are released.  Regression testing is critical and has to be done regularly and consistently.  Regression testing of all features and making sure the implementation works correctly on all browsers and devices require a lot of effort and can be time consuming if done manually.  Automating these tests will bring the speed, scale and efficiency required to get to market faster.  Further, integrating these automated tests with DevOps will help implement continuous testing into agile development processes and speed up time to market.

In the final part of this series we will explore various approaches taken by companies while implementing eCommerce platforms and their implications on testing.

Stay Tuned!




Testing the Wave – A Tester’s view of The Forrester Wave reports on B2C Commerce and B2B Commerce Suites – Part I

Forrester released its 2017 version of the Forrester WaveTM analysis of the B2C and B2B platform suite last month.  The report details what customers are looking for in eCommerce platforms and how they are buying these.

In the author’s words, “This report shows how each provider measures up and helps eBusiness and channel strategy professionals make the right choice.”

I however, chose to look at it with a different point of view. Of a tester responsible for making sure that the “right choice” actually “works right”.

More than just Shopping Carts

In this first of three blog posts I will focus on some of the capabilities that customers are demanding from eCommerce Platform vendors and implications of having to test for these capabilities.

Omni-Channel Engagement

Companies want to engage their customers on mobile, social and in the store with the same ease as they do on the Web. And they want this engagement to flow seamlessly across all these channels.  As a tester, your work just got a lot more complicated.  You no longer just have to test the user’s experience on a single web site, but you have to be able to simulate (and validate) this experience across the mobile site, POS systems used in-store and social engagement platforms. 

Here are a couple of things you can do to accomplish this :

  • Avoid the temptation to keep your tests short and within a single application. Make sure your test scenarios span user experience across the applications/systems driving various channels. 
  • Make sure that your test data and test scenario flows cover all possible combinations of omni-channel use.  You never know which path your customers end up taking, and you want to make sure that their experience is perfect.

Full Spectrum Selling

As more companies provide B2B buying online, customers want the flexibility to buy online, call in or fax in (is anyone still using this) an order, or use a combination of these methods to complete their order.  [Note: While the report did not cover machine-to-machine ordering and fulfillment, these too play a large role in B2B commerce.]

As a tester you have to make sure that the flow of orders and the experience of the customer across these channels works great.  Just as in the case of omni-channel testing, you have to make sure that your test scenarios span the use of all channels and can integrate with machine-to-machine transactions with ease.  This is an area that can get complicated if you are only doing manual testing and may require that you look into automation solutions to handle these.

Personalized Digital Commerce Experiences

Companies are looking to personalize their customer’s digital experience by providing them buying options and incentives based on their purchase preferences, location or by timing these offer correctly.  These solutions combine two main aspects that place great demands on testing – Customer data that fuels the experience and the front-end experience of the customer that is driven based off this data.  Testing these personalized digital experiences requires a strong data driven testing approach using robust and representative test data to simulate all possible experiences

  • Testing the personalization options built into the eCommerce solution will require you to simulate a variety of customer data (profiles). Your test data should include a large test data set covering all combinations of customer preference/behavior/location etc.
  • You should then be able to include data on the corresponding customer experience that has to be validated. 
  • Finally, your ability to test this multitude of combinations accurately, at scale, with efficiency is critical to a successful implementation of personalized digital experiences.

In the next part of this series we will explore how machine learning and AI concepts are making their way into eCommerce and the impact of moving Commerce to the Cloud. 

Stay tuned!

Stop Revenue Bleed - Complete the Sale!

67 percent of eCommerce shopping carts are abandoned!

Think about that statistic for a minute - Two out of three shoppers on your site leave without completing the purchase. Take the time, money and effort you spent in getting those shoppers to your site, and imagine that two-thirds of that just went up in smoke.  Poof! Something happened along the way that made them change their minds. Focus on that, fix that, and you will see your revenue climb.

Let’s dive into the statistic that I opened with - Most abandoned carts are due to factors well under your control – and you can do something about them.  

Reasons range from unexpected costs (usually shipping and handling that were not stated up front) to challenges in completing the purchase. Detecting these issues and fixing them will directly impact your revenue and bottom line.  

In addition to being upfront with your customers about the costs they can expect, you have to make sure their experience is seamless and error-free no matter what path they take in your site to make the purchase.  Otherwise, given the short attention span of shoppers, it’s Hasta la Vista – they’re gone, probably for good.

These are three areas you should focus on to make sure that your customer’s experience is a rewarding one:


  •  Provide customers with a way to checkout without having to register on your site. This is extremely important for first-time and impulse buyers.
  • Make sure registered customers can log in easily. If they can log in with a single click, so much the better.
  • For returning customers, remember their preferences, like shipping details so they don’t have to reenter the information. (Don’t you hate that?)
  •  Remember your customers’ shopping history, including abandoned carts. Offer to review/include those abandoned carts in the current sale.

Pricing, Promotions and Discounts

  • Are cart totals displayed accurately, including applicable taxes? Consistency is critical – beware of discrepancies in different paths through your site.

  • Make sure product prices are correct – particularly when an off-size item is selected. State clearly if a price is different from the price on the product page and why it is different.

  • What about different currencies? Make sure you’re handling them correctly, especially when selling internationally.
  • Verify that promotions, discounts, and sales are correct.  Itemize when necessary, and make sure the transaction is easy to understand.


  • Make sure all displayed payment options work properly. There’s nothing worse than a credit card or banking transaction that goes haywire – don’t let it happen on your site.
  • If certain options require users to register, make sure you let them know upfront – preferably during the checkout process. Make registration as effortless as possible, and bring users back to where they were.
  • For registered users, provide the option to use past payment methods.
  • Be able to send payment confirmation multiple ways – on-screen, email (if you have the address), text message, etc.

These simple steps go a long way in increasing your conversion rates and your revenue.

Change Is Constant – How Can You Keep Up?

Your eCommerce site has complex workflows behind the scenes connecting to diverse backend systems and external services.  You need to make sure that all these elements work correctly and continue working correctly with the constant changes on a dynamic retail site. Thorough testing is essential – but doing it every time you adjust a price, unveil a new product, launch a promotion and make other changes is a daunting challenge for many retailers. Traditional manual testing approaches take too long and cost too much.

Continuous testing of all site features and user journeys across the myriad of browsers and devices your customers use requires an automated testing solution. It is the only way to do testing at scale, with efficiency.

Stop the bleed!. Detect problems on your site before they afflict your customers. Get them to complete the purchase, and come back for more.

Get Personalization Wrong! It's Personal!

"Kickoff the Jim Harbaugh era with an autographed Michigan Wolverines Harbaugh T-Shirt", said the pop-up ad on the eCommerce site I had just logged on to.  It was a very personalized message designed to target a college football fan like me, on a site selling, among other things, college football merchandise at the beginning of the Big Ten football season.

It was perfect except for one thing – I went to The Ohio State University!

For the uninitiated, presenting a personalized message hawking Michigan Wolverines merchandise, like the one above, to an Ohio State Buckeye is like waving a red cape in front of a bull.  It triggers responses that are fueled by one of the storied college football rivalries dating back over a hundred years. These are not pleasant responses and certainly not the kind that the person who designed this promotion was hoping for.

So what went wrong?  It could be any one of a number of things from insufficient user profile data to incorrect or conflicting personalization rules.

One thing however is certain – the personalization implemented on the site was not adequately tested.

Personalization tools and platforms providing a myriad of ways to present relevant content and promotions to users on an eCommerce site have abounded in the past few years.  They have at their disposal a variety of data – volunteered and inferred, about the user to build personalization rules and drive specific digital experiences.  This same variety of data and complexity of rules makes the task of testing personalization that much more complex, and the likelihood that the testing is shortchanged that much more real.

It does not have to be so. Here are three things you can do to make sure that the personalization you build on your site works correctly:

Simulate all attributes that drive personalization in your testing: Your personalization rules depend on a lot of attributes (data) about your user.  Make sure that your test data includes all combinations of these to verify the personalization rules.  Pay particular attention to test data that drive negative tests – if the person testing the site mentioned above had used test data corresponding to a Buckeye in addition to a Wolverine, they would have caught the problem before it made it to the public site.  Testing for large combinations of test data is easier and significantly more economical if you use test automation that supports data driven testing.

Test the integration of the personalization engine and the website:Unit testing the personalization rules are important, but do not skip the step of testing the website after the personalization has been integrated.  Testing the web site from the user interface will show you exactly what the user experiences and will help you identify details that can be easily missed in unit testing.
Enable people designing personalization rules to perform the tests themselves: It is obvious that the person designing the personalization rules knows how they should work and is best equipped to come up with test scenarios to make sure that they work correctly.   So why not equip them to run these tests themselves right after the personalization rules are built.  They can validate the results and make sure that the personalization works just the way they intend it to.  By moving this testing process closer to the experts and testing often, you can reduce the time and cost required to test your personalization.

I would like to add a final note of caution when dealing with personalization that uses inferred data like geo-location or language in addition to user profile data.  Make sure that your test data includes these variations to tease out potential conflicts in rules between the two.  I am unsure if the site above uses this in its personalization, but as I look forward to the football season this year, I have a note of caution to the person testing this personalization – I may live in the state of the Scarlet Knights, but I still bleed Scarlet and Gray!

Winter Is Coming! A Call to Action for Software Testing

As I look forward to the start of Season 6 of the HBO Series, Game of Thrones in April, the motto of House Stark - Winter is Coming! both a warning and a call to action to maintain constant vigilance to prepare for tough times ahead, is beginning to take a strangely familiar tone.

Falling oil prices, stock market turmoil and Unicorns crashing down from the skies could well be harbingers of tough times ahead – Winter. And we may do well to heed warnings to remain vigilant.

The need for Vigilance

Public companies facing falling stock prices are beginning to cut costs to boost balance sheets. Falling commodity prices are also going to reduce the amount of cash available for investment by sovereign funds and large investment funds. IT budget cuts and the prospect of delayed or smaller future funding rounds will force those responsible for building and managing software to look for more efficient ways to deploy their capital. This vigilance to achieve capital efficiency has to start at the top, in the C-Suite.

One of the lasting outcomes of the last winter in the tech economy in the US was a significant increase in offshoring as a way of achieving capital efficiency. Until then, offshoring was seen as something only large enterprises did, but it became a must have for even startups and mid sized companies. Investors emphasized the need for a ‘lower cost’ development center as a way to stretch their investment dollars.

But winters pass, seasons change and we tend to forget the travails of the cold as we bask in the warmth of fat balance sheets flush with cash and easy funding.

The vigilance that was the norm during winter has been relaxed and at times forgotten as we saw unicorns emerge and rise to the skies. We will however be wise to bring that vigilance back so that we can continue to thrive when the frost sets in.

With most software testing done offshore, labor costs already are at an all time low. Steps to bring greater efficiency in the deployment of capital for testing have to involve something other than just moving jobs to lower cost economies. These steps must include a level of automation that takes humans out of performing low value grunt tasks in testing and frees them to perform activities that provide a higher value. Activities like test design and planning require a level of domain expertise and subject matter knowledge, and when done right, contribute significantly more towards improving software quality.

Freeing up your testing teams to pursue the high-value tasks means you have to first get them out of doing the mundane manual testing that they have been used to.

The challenge is two-fold. Firstly, building and maintaining test automation has typically required specialized skills and expensive tools. Tools you may not have the capital budget to buy and skills you many not have in-house. Secondly the learning curve for manual testers to take up test automation has been extremely steep and one that most are unable to navigate. This dual challenge of high upfront capital costs and dealing with a workforce that cannot transition to a new future has kept inefficient testing practices going for far too long.

There is a new reality in test automation today! A reality that when adopted will not just allow us to survive the winter, but thrive as well.

A New Reality

It is possible today to build and manage test automation without using expensive and proprietary tools that require upfront capital expenditure. Legacy tools that carry the burden of per-user or node locked licenses, and tools that are primarily workstation-centric in building the running test automation are being replaced by nimble SaaS and Cloud based testing platforms. These are easy to use and support the current reality of distributed software development and testing teams. Out-of-the box and standards-based integration with DevOps tools available in these platforms ensure that product management teams, development and testing teams work in sync. Further by replacing the upfront capital expenditure required for software licenses with pay-as-you-go subscriptions these platforms help companies conserve cash and provide a true measure of productivity and ROI in testing operations.

Move spending on tools from capital to operational budgets to conserve cash

While unit tests built by developers work well to test the building blocks of software, testing real user journeys and business workflows require a great deal of understanding of the business. The new test automation platforms are making it possible to build automation to test these user journeys and business workflows using people that understand the business – subject matter experts and domain experts. By eliminating the dependence on programmers for designing and building this test automation, companies can significantly reduce the cost associated with test automation. Further by helping companies use their manual testers that understand the business to build the automation, these platforms help transition an existing testing workforce to a new reality.

Make it easy for your existing manual testing teams to adopt test automation. Your testing workforce will be more efficient and productive helping you stay competitive during tough times.

Finally the testing platforms in this new reality also help companies improve their software development and testing processes by providing them with valuable and actionable insights about their software quality. Insights powered by analytics and detailed reports on test results that are readily available to all stakeholders without having to spend long hours or precious resources crunching numbers. Test automation is critical to obtain the data that powers these insights. The greater the extent of test automation, the better the insights to improve software quality.

A Call to Action

At the current time in the Game of Thrones, it has been almost nine years since the last winter and most of the key characters in the plot were too young to remember it. They are engrossed in their current squabbles to realize the larger threat to the realm that winter brings. It is up to the older characters that have experienced previous winters to caution them to be prepared or perish.

Nine years – In our present context that would be 2007. How would those of us that have been through the last economic winter remember it and caution each other to prepare for the next one.

It is not a matter of if, but when – Winter Is Coming!

Changing Face of Retail

Tuesday, November 3rd 2015, a store opened in a sub-urban shopping mall. An everyday occurrence in cities around the country that would normally not have garnered much attention – except this was no ordinary store. Amazon.com opened its first brick-and-mortar store in open-air shopping mall at University Village outside Seattle, WA.

The fact that this was a bookstore had a lot of people scratching their heads. The company that single-handedly led to the demise of a number of independent bookstores (and some chains like Borders), the company whose founder had famously said that the bookstore was redundant, was actually opening a physical bookstore. Media reaction was mixed, while some thought Amazon was going the way of Apple and Microsoft, others were convinced it wanted to add insult to injury by muscling out the remaining few independent bookstores.

I think there is more to this move by Amazon.

Amazon built a business for itself by eliminating the inherent inefficiencies in the supply chain in some business, while using a new (at that time) medium for engaging with its customers. That model worked well and once Amazon had established its self as the leading player in the industries that (until then) were really inefficient, it moved on to others and continued to repeat the model. That was over two decades ago.

The face of retail has changed.

Amazon’s success has pushed most retailers to add an online presence to their physical stores. These two avatars of the business however were and still are largely run independent of each other with the customer’s experience significantly differing in each channel.

The inefficiencies that existed in the supply chain two decades ago may be a thing of the past, but the ones that exist in the way customers are engaged across the multiple channels that a retailer uses to interact with its customers are glaring.

Amazon has set its sights on this inefficiency. Omni-channel strategies for marketing, customer engagement and operating a business in general are in their infancy and have a long way to go. Vendors are offering tools to build omni-channel marketing programs, but building a winning strategy and executing on it is not easy.

Amazon’s brick-and-mortar store is its first step to showcase its learning and thought leadership in this area. The fact that Amazon calls its store a ‘physical extension of Amazon.com’ should be evidence enough.

Books are just a start. Be prepared to have this strategy expand to other areas as Amazon learns and continues to innovate from this initial experience.

Here is a company that changed the retail, supply chain & logistics and the technology landscape (AWS). I believe that we are looking at the next wave of innovation to come from this company.

If it gets this one right, Amazon may just not be the World’s Largest Online Retailer. It may as well drop the Online from that tag line.