Should I build a Selenium automation framework or buy a commercial automation tool?

Build vs. Buy – This is a common question facing people considering software test automation, especially for web browser based software. The answer depends on your specific testing needs and a number of other factors.  In this article we will try to help you analyze these factors and think through the process of making a build vs. buy decision

Even though eureQa offers a complete SaaS+Cloud test automation solution, we understand that the eureQa automation platform may not be for everyone.  Based on your specific situation, building a Selenium (or Webdriver, Selendroid or Appium) automation solution or framework in-house may work better for you. Our goal is to make sure that you have all the information you need to make an informed decision. So lets get started.

The three most common criteria that a majority of our customers consider when making this choice are – cost, customization and control.

 

Cost

The open source and zero license cost nature of Selenium is attractive when compared to the license or subscription costs of commercial automation solutions. But there is more to it than just that when you look at the cost of implementing an automation solution.

Selenium is open source and free, however building and using an automation solution on this ‘free’ software comes with some real costs:

  • The open source tools/libraries provide a starting point but you still have to write code for test creation, management, execution, handling errors and reporting.  Note that this is over and above the test logic that you want to automate.
  • If you are looking to test integrations with back-end systems and external systems via API, you have to build in support for that into your framework.
  • If you are looking to implement a continuous testing solution, you must add to the above effort, the effort of building integration with DevOps or CI/CD tools.

You can see that building this solution is now a second development effort, complete with its own software development team and project managers.  You will need skilled developers to build and maintain this solution and project managers to manage the effort.

  • Also to be considered are the costs of building and managing an infrastructure to run these tests.  These costs are likely to go up as you scale your infrastructure in order to support increased test volumes or to increase the pace of testing.
  • Lastly, if you want to run cross browser/device tests, you are likely to incur more costs to build out that infrastructure or you can subscribe to selenium-on-the-cloud providers like Sauce Labs or BrowserStack.

As you can see there is a definite upfront and ongoing cost of building your own selenium based automation solution.  Additionally, since you will be using development resources to build this solution, you will also have to consider the opportunity costs, i.e. if these resources were not engaged in building your solution, could they be deployed to some other potentially ‘higher value’ task. This will give you a measure of the total cost of ownership of building your own solution.

Now compare this against the cost of procuring and deploying commercial automation software that have one-time license or ongoing subscription costs:

  • The licenses and subscriptions typically include costs for upgrades, which eliminate the cost of maintaining the tool itself.
  • As you scale test automation, your license and subscription costs may go up, though most vendors tend to offer volume discounts.
  • Ease of automation, along with the skills required to build and maintain automation vary based on tools selected, but in general these are lower (less expensive) than what is required with building your own selenium solution.
  • Most commercial automation solutions require you to build and manage your own test infrastructure to run the solution, which adds to the total cost of using these solutions.  Others like eureQa deploy their solution as SaaS+Cloud including all infrastructure costs into subscription costs and provide the convenience of a fixed predictable cost structure for test automation.

The Upshot

A good understanding of total costs of ownership (the true cost) of each option is important before you make a choice of a test automation solution.  Building your own Selenium based automation solution may work when you have a small set of tests that don’t change much, but as your testing needs, test complexity and your test suite grow, a commercial solution may be the more cost-effective option.

 

Customization

Does the automation solution work for my specific application and testing needs?

This is another common question from companies that have specific application features, workflows or integrations. 

A custom developed Selenium solution provides the flexibility to handle specific workflows or integration needs, but it comes with the added cost (see above) of the development of this capability and more importantly maintenance of it.

Most commercial test automation software tools provide some type of support for customization.  Building this customization typically requires a higher level of (programming) skill than test automation with these tools, and this cost has to be factored into your cost calculations (above).

One important factor to consider when building customizations is the upgrade path for the commercial software.  Make sure that customizations use a well-published API that will continue to be supported, and will not interfere with upgrades.  While premises installed (or single tenant) software provide a lot more flexibility with respect to customizations, SaaS solutions, like eureQa, typically do a better job of making sure that customizations are upgrade friendly, helping reduce your costs in the long run. 

The Upshot

Among companies that have this need, it is important to point out that the tests for the truly specific features and integrations are usually a smaller subset of the overall suite of tests to be automated.  Thus any decision on an approach to customization must take into account the relative weightage of the tests that are impacted.

 

Control

I am worried that I will be boxed into a solution that does not evolve as my applications and testing needs change.

This is a perfectly valid concern for anyone making a technology choice in today’s fast paced world. 

A custom built Selenium framework gives you complete control over the features that are built into it. It allows you to add any features that you need as and when you need them, so long as you are willing and able to spend the time, effort and money to build these features into the framework. Conversely, it also means that the only features your framework will have are the ones that you build in it, which limits innovation to a single source – you!

With commercial automation solutions, you are dependent on the tool vendor to add new features and updates to the tool. Most vendors don’t charge for updates, but you also don’t have too much control on their release schedule.  You however have the benefit of feedback and feature requests from a large pool of user/customers, that go into determining what gets added to the tool.  You gain from the learning, experience and innovation coming out of a larger group of peers.

If you are evaluating commercial automation software ask your vendor about their practice for soliciting customer/user feedback and the frequency of upgrades. At eureQa, our customers request features and enhancements through our support portal and one third of our minor enhancement release cycle (every two weeks) is dedicated to these requests making sure that we evolve with our customers’ needs. As a SaaS platform, we do have the added benefit of being able to make sure that all our customers get all the updates in a timely manner.

The Upshot

While it is easy to evaluate the current capabilities of an automation solution at the time of purchase, the future feature set is more nebulous.  As a result, it is easy to succumb to the ‘fear of the unknown’, and ignoring all other factors lean heavily, towards a custom Selenium solution in an attempt to control the uncertain.

A preferred approach would be to look at this feature uncertainty as a ‘risk’ and identify a mitigation strategy that takes into account all the factors discussed above in the decision-making process.

The choice of an automation solution is an important component of your software quality process.  We hope this article is helpful in your evaluation process. 

Feel free to share your thoughts and feedback on this article.