Note: This is our latest technical engineering post written by principal engineer, Justin Zimmerman. For more posts like this, check out our technical blogroll.
As SendGrid has evolved from the controlled chaos of a startup to a more mature software development organization, we’ve had to revisit some of our test automation infrastructure. In particular, the test automation tools we used to test Marketing Campaigns and other front-end applications have had to evolve along with those applications they test.
Marketing Campaigns v1 was developed in Rails, a Ruby-based MVC framework, so it made sense to develop our own in-house Ruby test framework (Gridium) and test harness (known internally as SiteTestUI, or STUI).
Finding the cause of developer frustration
As the shift on the development side was going on, developers were expected to contribute automated integration and end-to-end tests as they added features. As the test harness became a point of contention for some delivery teams, we did some internal research to determine what was causing the most developer frustration with STUI test development.
We developed a list of evaluation criteria. The framework had to:
- Support multiple browsers (with the eventual intent of integrating with an external service such as Sauce Labs or BrowserStack)
- Report test results to a single file that could be consumed by an external test aggregation/dashboard
- Be able to run single tests and single suites
- Run tests in parallel
- Support asynchronous communication
- Be available as open source (strong community support a big plus)
- Support test case tagging
- Run in multiple environments (our internal testing and staging environments, for example)
From an initial list of 8 tools, the test automation team did a deeper dive into each one, with individual engineers evaluating tools against the requirements matrix. After a thorough review, the team chose the Node.js-based WebDriverIO as the best potential fit.
Executing sample tests
After we made the decision, the automation team worked with front-end development leads to write and execute sample tests and see whether WebDriverIO in practice lived up to its promise.
From that, we made a recommendation to SendGrid’s Front End Architecture Guild (an internal advisory group of front-end engineers from the different delivery teams across the company) that the delivery teams adopt WebDriverIO for their front-end testing.
With that, the automation team is providing advice and support to the delivery teams, but the developers are taking on the responsibility of writing integration and end-to-end tests for new front-end features as they add them. Additionally, automation engineers are developing a new way to store and display automated test results from WebDriverIO along with our other test tools.