Choosing a Front-End Test Automation Framework

Note: This is our latest technical engineering post written by principal engineer, Justin Zimmerman. For more posts like this, check out our technical blogroll. 

As SendGrid has evolved from the controlled chaos of a startup to a more mature software development organization, we’ve had to revisit some of our test automation infrastructure. In particular, the test automation tools we used to test Marketing Campaigns and other front-end applications have had to evolve along with those applications they test.

Marketing Campaigns v1 was developed in Rails, a Ruby-based MVC framework, so it made sense to develop our own in-house Ruby test framework (Gridium) and test harness (known internally as SiteTestUI, or STUI).

After the initial release, however, increasing customer demand for Marketing Campaigns exposed some performance limitations with Rails, and front-end developers started moving to separate the View part of MVC from Rails into JavaScript via Facebook’s React.

Finding the cause of developer frustration

As the shift on the development side was going on, developers were expected to contribute automated integration and end-to-end tests as they added features. As the test harness became a point of contention for some delivery teams, we did some internal research to determine what was causing the most developer frustration with STUI test development.

We found primarily that it required too much context-switching between languages (JavaScript to Ruby) as well as code repos.

In an effort to partner more effectively with development teams, the automation team decided to move away from the in-house STUI test harness to JavaScript tooling in order to reduce context switching. To that end, we decide to evaluate off-the-shelf JavaScript-based test automation tools.

We developed a list of evaluation criteria. The framework had to:

• Support multiple browsers (with the eventual intent of integrating with an external service such as Sauce Labs or BrowserStack)
• Report test results to a single file that could be consumed by an external test aggregation/dashboard
• Be able to run single tests and single suites
• Run tests in parallel
• Support asynchronous communication
• Be available as open source (strong community support a big plus)
• Support test case tagging
• Run in multiple environments (our internal testing and staging environments, for example)

From an initial list of 8 tools, the test automation team did a deeper dive into each one, with individual engineers evaluating tools against the requirements matrix. After a thorough review, the team chose the Node.js-based WebDriverIO as the best potential fit.

Executing sample tests

After we made the decision, the automation team worked with front-end development leads to write and execute sample tests and see whether WebDriverIO in practice lived up to its promise.

From that, we made a recommendation to SendGrid’s Front End Architecture Guild (an internal advisory group of front-end engineers from the different delivery teams across the company) that the delivery teams adopt WebDriverIO for their front-end testing.

With that, the automation team is providing advice and support to the delivery teams, but the developers are taking on the responsibility of writing integration and end-to-end tests for new front-end features as they add them. Additionally, automation engineers are developing a new way to store and display automated test results from WebDriverIO along with our other test tools.