Model based testing with XState + Cypress or Puppeteer or Playwright made easy

The goal of this project is to define a simple configuration "language" allowing developers to create model based end-to-end tests for their applications using state machines created using XState and test models generated by @xstate/test in combination with Cypress or Puppeteer or Playwright and Jest.

You should be somewhat familiar with XState but you don't necessarily need to be comfortable with the other dependencies. If you are using Cypress and wanting to intercept and mock network requests you wil have have to enable the experimental route2 API.

Clone or fork this repository and run:

yarn
yarn test:ci

Running yarn will install both the dependencies of this project plus those in the packages folder and the example project in ./examples/app. yarn test:ci will build the packages in the packages folder, build the example app, and run the Playwright and [Puppteer] tests for the example app in headless browser mode.

You can also start up the example ReactJS project in development mode running at localhost:3000 and just run yarn test in the root of the project to view the tests in non-headless mode. To run the example Cypress tests run yarn cypress open after you have started the development server.

Depending on which test runner you want to use:

npm install xstate-marionettist-cypress
npm install xstate-marionettist-puppteer
npm install xstate-marionettist-playwright

Cypress:

• cypress
• @xstate/test
• xstate

Playwright:

• jest
• jest-playwright-preset
• playwright
• @xstate/test
• xstate

Puppeteer:

• jest
• jest-puppeteer
• puppeteer
• @xstate/test
• xstate

The example project is a good example of implementeting testing with Cypress or Jest with Puppeteer or Playwright. Feel free to open an issue or send me questions if you are having trouble with this. You can open an issue, contact me on Twitter and I'm also active in the Statecharts chat on Spectrum.

npm install xstate-marionettist-cypress

Be sure to enable experimentalNetworkStubbing cypress.json

// cypress.json
{
  "experimentalNetworkStubbing": true
}

npm install xstate-marionettist-playwright
npm install xstate-marionettist-puppeteer

For usage with Puppeteer or Playwright you should at least implement similiar Jest configuration files:

• jest-config.puppeteer.js
• jest-puppeteer.config.js

Or if you are using Playwright

• jest-config.playwright.js
• jest-playwright.config.js

jest-e2e-setup.js is optional but helps with bailing out quickly when tests are failing. You'll need to add jasmine-fail-fast as a dependency to your project for this.

Actions are instructions and can be attached to tests or events. By default, anything using a selector expects that is was defined using a data-testid attribute. All selector actions wrap names with: [data-testid~="name"] matching any words. You can override this behaviour by importing the make function rather than the test function from xstate-marionettist-.... and create your own version of the test function:

// custom-test.ts
import { create } from "xstate-marionettist-cypress";
import { create } from "xstate-marionettist-playwright";
import { create } from "xstate-marionettist-puppeteer";


export const test = create({ selectorWrapper: (name: string) => name }); // for no wrapping at all
export const test = create({
  selectorWrapper: (name: string) => `[data-my-custom-test-id="${name}"]`,
});

// my-e2e.ts
import { test } from './custom-test'

test({ ... })

Other options that can be passed to create are:

{
  // Defaults to "localhost"
  server?: string;

  // Defaults to false
  // Adds the query paramter xstate-inspect=false|true to the visited
  // url allowing you to disable the inspector when running tests.
  // See the example project's index.tsx file.
  xstateInspect?: boolean,

  // What port the visited url is hosted at
  // Listed values are the defaults
  ports?: {
    ci?: 7777;
    prod?: 7777;
    dev?: 3000;
  };
}

Availeable actions are:

Let's review the example test's configuration availabe in the example project. The project is a ReactJS application created using create-react-app and renders a simple login form. The test is located at ./examples/xstate-marionettist-example/src/apps/auth/auth.e2e.ts.

The configuration models 4 different states:

• noop: State where users can enter their email and password and click login.
• authenticating: State reflecting the UI is blocked and busy
• failure: Failure outcome state
• authenticated: Success outcome state

The configuration for the noop state is as follows:

noop: {
    tests: [
      page => page.waitForSelector("div#root"),
      ["waitForSelector", "frm-login"],
      ["waitForFocus", "txt-email"],
      ["waitForSelector", "txt-password"],
      ["expectProperty", "btn-login", "disabled", false],
    ],
    on: {
        LOGIN: {
            target: "authenticating",
            actions: [
            ["type", "txt-email", "[email protected]"],
            ["type", "txt-password", "statechartsareawesome"],
            ["defer", "submitting"],
            ["click", "btn-login"],
            ],
        },
    },
},

This state has a couple of tests. The first one examplifies usage of plain functions allowing you to write your own tests with the page object. This is followed by a "waitForSelector" action which is just a wrapper around Puppeteer's page.waitForSelector. "waitForFocus" uses the Page object to compose document.activeElement against the selected element. In our case the email input should take focus after the first render. The test verifies if the password field is present and if the submit button is enabled.

.on.LOGIN should look familiar to you as it maps directly to a proper state machine configured using XState. The actions defined here however are translated into proper functions passed to the test model built using @xstate/test's createModel API. THE "type" and "click" actions map directly to puppeteer intructions simulating typing and clicking:

    await page.type('[data-testid="txt-email"]', "[email protected]"),
    await page.type('[data-testid="txt-password"]', "statechartsareawesome"),
    ...
    await page.click('[data-testid="btn-login"]'),

The "defer" action is executed before clicking the button. Under the hood puppeteer's API for intercepting network requests is used to mock and block API responses. The order of execution here will be:

1. "defer" puts a promise in a buffer
2. Puppeteer clicks the button in the UI
3. The UI makes a network request to login
4. Puppeteer intercepts the network request, checks the buffer for unresolved promises and blocks if any are in there.
5. The test model moves on to the authenticating state's tests.

The next state that will be tested is the authenticating state:

authenticating: {
    tests: [
        ["expectProperty", "btn-login", "disabled", true],
        ["resolve", "submitting"],
      ],
      on: {
        OK: {
          target: "authenticated",
        },
        BAD: {
          target: "failure",
        },
    },
},

The first test translates to a puppeteer instruction for retrieving an element's property and using Jest's expect API to test the value. In this case the button should be disabled while the form is busy submitting.

The next action in the list of tests in not really a test but an instruction to resolve the promise that was put in the buffer using the "defer" action. This will cause the intercepted request to continue on. Our configuration has defined a matching path and for such cases the intercepted request is mocked. Requests that have no matching path are not mocked.

Each event type that transitions the model to a state where a mocked/blocked network request is made should be added to the top-level .outcomes property. This important or otherwise mocked responses are not matched up. I'll write up an explanation of the underlying machinery in a different readme.

In this case we have an OK and a BAD outcome. The API configuration also allows you to define a response status and payload for each outcome. The default response status is 200 and an empty body. You can also define a status and body for all outcome using an asterisk:

  apis: [
    {
      path: "/greet",
      deferrals: ["greet"],
      outcomes: {
        *: {
          status: 200,
          body: "Hello World!"
        },
      },
    },
  ],

From here the test model splits up in to two different outcomes since the authenticating state has two events configured that transition to different states. Let's look at the BAD/failure combination first.

 failure: {
    tests: [
        ["expectProperty", "btn-login", "disabled", true],
        ["resolve", "submitting"],
        ["waitForSelector", "txt-email-helptext-error"],
    ],
    on: {
        NOOP: {
            target: "noop",
        }
    }
},

The tests for this state first determine if the button is disabled since this should be the case while the network request is in progress. Then it resolves the promise that was pushed into the buffer earlier causuing the network request to be unblocked and continue. Since we have defined a custom payload for outcome BAD the response should have status 400 and contain "Invalid token".

Before allowing the model to transition back to noop state we block with a test expecting an error message. The configuration and tests for the authenticated state follow the same pattern.