A little over 9 months ago, Wealthfront converted 100% of our end-to-end tests from Selenium (via Capybara) to Puppeteer. The process involved converting hundreds of tests that provide coverage for just about all of Wealthfront’s product offerings on the website.

This blog post will cover why we migrated away from Selenium, the process we used to convert the tests, and the result of the completed conversion.

Why migrate

Several years ago, Wealthfront began converting from a Ruby on Rails application to a React application. This provided many benefits when developing new features, but the end-to-end test suite was still entirely in Ruby. Over time, this began to cause more issues as new engineers joined the team who primarily worked in JavaScript, which meant these engineers had to learn a language primarily to write end-to-end tests.

In addition to this, the test suite was not meeting our SLOs set for test suite flake rate (flake meaning at least one test case in the entire suite failing due to a flaky test, as opposed to each individual test failing). As you can see in the chart below, the flake rate has been a constant problem since we first started collecting data to track the flake rate. There have been multiple attempts over the years to get these things under control. Eventually the team decided it was more worthwhile to begin investigating moving off of Selenium rather than spending a few weeks every quarter fixing flaky tests, and keeping versions of Selenium and Chrome’s webdriver up to date.

But the problems don’t stop at flake rate. Since we deploy to production multiple times per day. The amount of time it takes to run the test suite is very important to us. As you can see in the below chart, the test runtimes had been steadily increasing as we added more tests for our new features. At one point the test suite was taking over 30 minutes. Add this to the time it takes to build a side branch, merge that side branch, build master, then deploy the application and we’re looking at over an hour just to merge code into production. This made it untenable to fix forward, even when solutions are trivial, instead relying on speedier but less ideal rollbacks. To us, this was unacceptable. As you can see in the graph below, some improvements were made to runtimes by increasing the power of our build machines, but throwing more resources at the problem is not a scalable solution.

Picking a framework

There are a number of frameworks available for end-to-end testing. And it’s important to come up with a criteria for evaluating the options available, and picking one that meets all of those. For us, the criteria was:

• Works well with Wealthfront’s ecosystem
• Similar code style to how we write our unit and integration tests (we use Jest and React Testing Library)
• Good community support
• Good documentation
• Customizable to meet our needs

After evaluating multiple options, it came down to Cypress and Puppeteer. Cypress is an end-to-end testing framework and provides an extensive list of features, covering everything from build machines to run the tests in parallel to UI snapshots of test failures. Whereas Puppeteer is a lightweight node library that could just be installed as an npm package and get started with our existing infrastructure right away.

We eventually picked Puppeteer due to it being lightweight, extensible, and allowed us to write our own helper methods to mimic the API of Capybara to reduce the learning curve for our web engineers as they migrate end-to-end tests from Capybara to Puppeteer. It also allowed us to use the existing jest test runner configuration which is shared by our unit and integration tests.

It’s worth noting that at the time of this investigation, Playwright had not yet been released. But it has feature parity with Puppeteer, while also allowing for cross browser testing, auto waiting functionality for elements on the page, and its own test runner. If this investigation was done today, it’s likely that Playwright would be our main choice.

The migration process

Migrating from one end-to-end technology to another is never easy. Especially so when we have tests for just about every feature at Wealthfront. In order to reduce the burden on our engineers when migrating the test suite, we created custom assertions that are wrapper functions for the Puppeteer API. These assertions are a 1:1 mapping with the Capybara assertions in terms of how they’re used in code. It’s important to note that this has the trade-off of no longer being able to reference the Puppeteer documentation when writing tests. To account for this, we wrote documentation that lives on our internal wiki. Refer to the excerpt from our internal documentation below as an example of clicking the “Get Started” button on Wealthfront’s home page.

The result

Since the migration, test runtimes have decreased from 15 minutes per run to 6 minutes, all while the number of test files have increased from 141 to 197. This gives us confidence that our test suite will scale as we wanted when we initially started this project.

On top of that, the flake rate has been consistently within the set SLO. And while we still have some flaky tests, we are well within our defined SLOs and have incorporated fixing flakes into our on-call process to make sure that they never get out of control again.

If this work interests you or if you have thoughts on how we might make it even better, check out our careers page.

Disclosures:

The information contained in this communication is provided for general informational purposes only, and should not be construed as investment or tax advice. Nothing in this communication should be construed as a solicitation or offer, or recommendation, to buy or sell any security. Any links provided to other server sites are offered as a matter of convenience and are not intended to imply that Wealthfront Advisers or its affiliates endorses, sponsors, promotes and/or is affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.

All investing involves risk, including the possible loss of money you invest, and past performance does not guarantee future performance. Please see our Full Disclosure for important details.

Wealthfront offers a free software-based financial advice engine that delivers automated financial planning tools to help users achieve better outcomes. Investment management and advisory services are provided by Wealthfront Advisers LLC, an SEC registered investment adviser, and brokerage related products are provided by Wealthfront Brokerage LLC, a member of FINRA/SIPC.   

Wealthfront, Wealthfront Advisers and Wealthfront Brokerage are wholly owned subsidiaries of Wealthfront Corporation.

© 2021 Wealthfront Corporation. All rights reserved.