Increase Test Coverage 95% and Double Speed to Market

The Vermont Country Store takes pride in being the ‘Purveyor of the Practical and Hard-to-Find’. From its founding in 1946 by Vrest and Mildred Ellen Orton, to the present day when the company is run by 4th and 5th generation of the Orton family, merchants at Vermont Country Store source goods globally that fill the needs of its customers. Over the decades, these needs have changed, and the customer base has grown, but one thing has remained constant – Vermont Country Store’s commitment to customer satisfaction.

We were able to increase our test coverage to 95% of the browsers and devices our customers use, while simultaneously speeding up our testing 2X with eureQa
— Russell Glitman, eCommerce Manager, The Vermont Country Store

This focus on the customer and the quality of customer experience is well represented in The Vermont Country Store’s Customer Bill of Rights, which is prominently displayed on their Web site. When The Vermont Country Store was looking for a solution to ensure the quality of customer experience on their online storefront, they chose eureQa.

As The Vermont Country Store adopted the Oracle Commerce Cloud (OCC) platform they wanted to make sure their customers’ online shopping experience was maintained at the same high levels that generations of in-store or catalog shoppers have been experiencing. The OCC platform made it easy for The Vermont Country Store to make changes to the online store providing a rich shopping experience to its customers, and to add new features and services through integrations to external systems.  These changes however required a great deal of testing, often requiring a ‘all-hands-on-deck’ approach, tapping into customer support staff to augment the QA team to complete the testing in a timely manner.

The Oracle Commerce Cloud platform made it easy to push new features, but testing them took a lot of time and effort.

Challenge

With the frequency and pace of changes to the eCommerce site, guaranteeing a great customer experience means that the frequency and volume of testing had to be significantly increased:

  • The web site’s OCC platform is upgraded quarterly. Testing these upgrades could take over a week and require additional staff loaned from customer service to complete.
  • Feature updates are applied to the eCommerce site every 3-4 weeks, and the effort requires running regression tests on the development and pre-production environments in a timely manner placed significant demands on the QA team and were a constraint on the pace of development.
  • Validating the customer’s experience on mobile devices and the primary browsers used by customers required a test lab of browsers and devices, which added to the cost of testing.
  • As The Vermont Country Store prepared for the holiday season, it also had to make sure that the online storefront could hold up to the increased holiday traffic without long page load times, which could drive online shoppers away.

Solution

The Vermont Country Store chose eureQa to automate its tests, and run them to ensure changes made to the online store worked correctly and did not affect user experience after updates are made to the live site. To accomplish this, eureQa helped The Vermont Country Store:

  • Automate its smoke and regression tests to quickly assess impact of changes made to the online store.
  • Run tests every time a new release is deployed to the TEST environment or a platform upgrade is applied to identify and fix issues before going live.
  • Run automated tests on the primary desktop and mobile browsers used by its customers to guarantee a consistent and seamless shopping experience no matter how a customer chooses to shop
  • Simulate 2x the user load expected during the peak holiday shopping season, and assess the eCommerce platform performance to identify and fix any performance bottlenecks. eureQa generates load from Web browsers instead of at the protocol-level to measure performance of rich UI elements on the online storefront providing a true measure of customer experience.

Benefits

eureQa helped Vermont Country Store make sure that its customers had a great shopping experience online, thus staying true to its focus on customer satisfaction through quality that has been the hallmark of its business since its founding.

  • Vermont Country Store was able to reduce the time taken to fully test its online store from more than a week to less than three days, a 2x savings in time. More importantly, it was able to reduce the number of customer support staff involved in testing by half so that they could spend more time helping their customers.
  • By testing on the most popular browsers and devices used by about 95% of its customers, Vermont Country Store was able to make sure that its customers had the best shopping experience.
  • By running performance tests on its online store prior to the holiday season, Vermont Country Store was able to start the peak shopping season with the confidence that its eCommerce store could hold up to the expected customer traffic. This confidence was borne-out as Vermont Country Store was able to get through a holiday season that saw an increase in sales, without a performance glitch.
  • By performing all these tests on a single platform, Vermont Country Store was able to reduce QA costs while continuing to ensure top-flight site quality.