Performance monitoring with Lighthouse CI

How to add Lighthouse to a continuous integration system, such as GitHub Actions.

Katie Hempenius

Lighthouse CI is a suite of tools for using Lighthouse during continuous integration. Lighthouse CI can be incorporated into developer workflows in many different ways. This guide covers the following topics:

Using the Lighthouse CI CLI.
Configuring your CI provider to run Lighthouse CI.
Setting up a GitHub Action and status check for Lighthouse CI. This will automatically display Lighthouse results on GitHub pull requests.
Building a performance dashboard and data store for Lighthouse reports.

Overview

Lighthouse CI is a suite of free tools that facilitate using Lighthouse for performance monitoring. A single Lighthouse report provides a snapshot of a web page's performance at the time that it is run; Lighthouse CI shows how these findings have changed over time. This can be used to identify the impact of particular code changes or ensure that performance thresholds are met during continuous integration processes. Although performance monitoring is the most common use case for Lighthouse CI, it can be used to monitor other aspects of the Lighthouse report - for example, SEO or accessibility.

The core functionality of Lighthouse CI is provided by the Lighthouse CI command line interface. (Note: This is a separate tool than the Lighthouse CLI.) The Lighthouse CI CLI provides a set of commands for using Lighthouse CI. For example, the autorun command executes multiple Lighthouse runs, identifies the median Lighthouse report, and uploads the report for storage. This behavior can be heavily customized by passing additional flags or customizing Lighthouse CI's configuration file, lighthouserc.js.

Although the core functionality of Lighthouse CI is primarily encapsulated in the Lighthouse CI CLI, Lighthouse CI is typically used through one of the following approaches:

Running Lighthouse CI as part of continuous integration
Using a Lighthouse CI GitHub Action that runs and comments on every pull request
Tracking performance over time via the dashboard provided by Lighthouse Server.

All of these approaches are built upon the Lighthouse CI CLI.

Alternatives to Lighthouse CI include third-party performance monitoring services or writing your own script to collect performance data during the CI process. You should consider using a third-party service if you'd prefer to let someone else handle the management of your performance monitoring server and test devices, or, if you want notification capabilities (such as email or Slack integration) without having to build these features yourself.

Use Lighthouse CI locally

This section explains how to run and install the Lighthouse CI CLI locally and how to configure lighthouserc.js. Running the Lighthouse CI CLI locally is the easiest way to make sure that your lighthouserc.js is configured correctly.

Install the Lighthouse CI CLI.
```
npm install -g @lhci/cli
```
Lighthouse CI is configured by placing a lighthouserc.js file in the root of your project's repo. This file is mandatory and will contain Lighthouse CI related configuration information. Although Lighthouse CI can be configured to be used without a git repo, the instructions in this article assume that your project repo is configured to use git.
In the root of your repository, create a lighthouserc.js configuration file.
```
touch lighthouserc.js
```

Add the following code to lighthouserc.js. This code is an empty Lighthouse CI configuration. You will be adding to this configuration in later steps.

module.exports = {
  ci: {
    collect: {
      /* Add configuration here */
    },
    upload: {
      /* Add configuration here */
    },
  },
};

Every time that Lighthouse CI runs, it starts a server to serve your site. This server is what enables Lighthouse to load your site even when no other servers are running. When Lighthouse CI finishes running, it will automatically shutdown the server. To ensure that serving works correctly, you should configure either the staticDistDir or startServerCommand properties.

If your site is static, add the staticDistDir property to the ci.collect object to indicate where your static files are located. Lighthouse CI will use its own server to serve these files while testing your site. If your site is not static, add the startServerCommand property to the ci.collect object to indicate the command that starts your server. Lighthouse CI will start a new server process during testing and shut it down after.
```
// Static site example
collect: {
  staticDistDir: './public',
}
```
```
// Dynamic site example
collect: {
  startServerCommand: 'npm run start',
}
```
Add the url property to the ci.collect object to indicate URL(s) that Lighthouse CI should run Lighthouse against. The value of the url property should be provided as an array of URLs; this array can contain one or more URLs. By default, Lighthouse CI will run Lighthouse three times against each URL.
```
collect: {
  // ...
  url: ['http://localhost:8080']
}
```
Note: These URLs should be serveable by the server you configured in the previous step. Thus, if you're running Lighthouse CI locally, these URLs should probably include localhost rather than your production host.
Add the target property to the ci.upload object and set the value to 'temporary-public-storage'. The Lighthouse report(s) collected by Lighthouse CI will be uploaded to temporary public storage. The report will remain there for seven days and then be automatically deleted. This setup guide uses the "temporary public storage" upload option because it is quick to setup. For information on other ways of storing Lighthouse reports, refer to the documentation.
```
upload: {
  target: 'temporary-public-storage',
}
```
The storage location of the report will be similar to this:
```
https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1580152437799-46441.report.html
```
(This URL won't work because the report has already been deleted.)
Run the Lighthouse CI CLI from the terminal using the autorun command. This will run Lighthouse three times and upload the median Lighthouse report.
```
lhci autorun
```
If you've correctly configured Lighthouse CI, running this command should produce output similar to this:
```
✅  .lighthouseci/ directory writable
✅  Configuration file found
✅  Chrome installation found
⚠️   GitHub token not set
Healthcheck passed!

Started a web server on port 65324...
Running Lighthouse 3 time(s) on http://localhost:65324/index.html
Run #1...done.
Run #2...done.
Run #3...done.
Done running Lighthouse!

Uploading median LHR of http://localhost:65324/index.html...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1591720514021-82403.report.html
No GitHub token set, skipping GitHub status check.

Done running autorun.
```
You can ignore the GitHub token not set message in the console output. A GitHub token is only necessary if you want to use Lighthouse CI with a GitHub Action. How to setup a GitHub Action is explained later in this article.

Clicking on the link in the output that begins with https://storage.googleapis.com... will take you to the Lighthouse report corresponding to the median Lighthouse run.

The defaults used by autorun can be overridden via the command line or lighthouserc.js. For example, the lighthouserc.js configuration below indicates that five Lighthouse runs should be collected every time autorun executes.

Update lighthouserc.js to use the numberOfRuns property:

module.exports = {
    // ...
    collect: {
      numberOfRuns: 5
    },
  // ...
  },
};

Re-run the autorun command:

lhci autorun

The terminal output should show that Lighthouse has been run five times rather than the default three:

✅  .lighthouseci/ directory writable
✅  Configuration file found
✅  Chrome installation found
⚠️   GitHub token not set
Healthcheck passed!

Automatically determined ./dist as `staticDistDir`.
Set it explicitly in lighthouserc.json if incorrect.

Started a web server on port 64444...
Running Lighthouse 5 time(s) on http://localhost:64444/index.html
Run #1...done.
Run #2...done.
Run #3...done.
Run #4...done.
Run #5...done.
Done running Lighthouse!

Uploading median LHR of http://localhost:64444/index.html...success!
Open the report at https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1591716944028-6048.report.html
No GitHub token set, skipping GitHub status check.

Done running autorun.

To learn about other configuration options, refer to the Lighthouse CI configuration documentation.

Setup your CI process to run Lighthouse CI

Lighthouse CI can be used with your favorite CI tool. The Configure Your CI Provider section of the Lighthouse CI documentation contains code samples showing how to incorporate Lighthouse CI into the configuration files of common CI tools. Specifically, these code samples show how to run Lighthouse CI to collect performance measurements during the CI process.

Using Lighthouse CI to collect performance measurements is a good place to start with performance monitoring. However, advanced users may want to go a step further and use Lighthouse CI to fail builds if they don't meet pre-defined criteria such as passing particular Lighthouse audits or meeting all performance budgets. This behavior is configured through the assert property of the lighthouserc.js file.

Lighthouse CI supports three levels of assertions:

off: ignore assertions
warn: print failures to stderr
error: print failures to stderr and exit Lighthouse CI with a non-zero exit code

Below is an example of a lighthouserc.js configuration that includes assertions. It sets assertions for the scores of Lighthouse's performance and accessibility categories. To try this out, add the assertions shown below to your lighthouserc.js file, then rerun Lighthouse CI.

module.exports = {
  ci: {
    collect: {
      // ...
    },
    assert: {
      assertions: {
        'categories:performance': ['warn', {minScore: 1}],
        'categories:accessibility': ['error', {minScore: 1}]
      }
    },
    upload: {
      // ...
    },
  },
};

The console output that it generates looks like this:

Screenshot of a warning message generated by Lighthouse CI

For more information on Lighthouse CI assertions, refer to the documentation.

Set up a GitHub Action to run Lighthouse CI

A GitHub Action can be used to run Lighthouse CI. This will generate a new Lighthouse report every time that a code change is pushed to any branch of a GitHub repository. Use this in conjunction with a status check to display these results on each pull request.

In the root of your repository, create a directory named .github/workflows. The workflows for your project will go in this directory. A workflow is a process that runs at a predetermined time (for example, when code is pushed) and is composed of one or more actions.
```
mkdir .github
mkdir .github/workflows
```
In .github/workflows create a file named lighthouse-ci.yaml. This file will hold the configuration for a new workflow.
```
touch lighthouse-ci.yaml
```

Add the following text to lighthouse-ci.yaml.

name: Build project and run Lighthouse CI
on: [push]
jobs:
  lhci:
    name: Lighthouse CI
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - name: Use Node.js 10.x
        uses: actions/setup-node@v1
        with:
          node-version: 10.x
      - name: npm install
        run: |
          npm install
      - name: run Lighthouse CI
        run: |
          npm install -g @lhci/[email protected]
          lhci autorun --upload.target=temporary-public-storage || echo "LHCI failed!"

This configuration sets up a workflow consisting of a single job that will run whenever new code is pushed to the repository. This job has four steps:

Check out the repository that Lighthouse CI will be run against
Install and configure Node
Install required npm packages
Run Lighthouse CI and upload the results to temporary public storage.

Commit these changes and push them to GitHub. If you've correctly followed the steps above, pushing code to GitHub will trigger running the workflow you just added.
To confirm that Lighthouse CI has triggered and to view the report it generated, go to the Actions tab of your project. You should see the Build project and Run Lighthouse CI workflow listed under your most recent commit.

You can navigate to the Lighthouse report corresponding to a particular commit from the Actions tab. Click on the commit, click on the Lighthouse CI workflow step, then expand the results of the run Lighthouse CI step.

You've just set up a GitHub Action to run Lighthouse CI. This will be most useful when used in conjunction with a GitHub status check.

Set up a GitHub status check

A status check, if configured, is a message that appears on every PR and typically includes information such as the results of a test or the success of a build.

The steps below explain how to set up a status check for Lighthouse CI.

Go to the Lighthouse CI GitHub App page and click Configure.
(Optional) If you're part of multiple organizations on GitHub, choose the organization that owns the repository for which you want to use Lighthouse CI.
Select All repositories if you want to enable Lighthouse CI in all repositories or select Only select repositories if you only want to use it in specific repositories, and then select the repositories. Then click Install & Authorize.
Copy the token that is displayed. You'll use it in the next step.
To add the token, navigate to the Settings page of your GitHub repository, click Secrets, then click Add a new secret.
Set the Name field to LHCI_GITHUB_APP_TOKEN and set the Value field to the token that you copied in the last step and then click the Add secret button.
Go to the lighthouse-ci.yaml file and add the new environment secret to the "run Lighthouse CI" command.

-           name: run Lighthouse CI
            run: |
              npm install -g @lhci/[email protected]
              lhci autorun --upload.target=temporary-public-storage || echo "LHCI failed!"
+            env:
+              LHCI_GITHUB_APP_TOKEN: $

The status check is ready for use. To test it, create a new pull request or push a commit to an existing pull request.

Set up the Lighthouse CI Server

The Lighthouse CI server provides a dashboard for exploring historical Lighthouse reporting. It can also act as a private, long-term datastore for Lighthouse reports.

Screenshot of the Lighthouse CI Server dashboard

Screenshot of comparing two Lighthouse reports in Lighthouse CI Server

Choose which commits to compare.
The amount that the Lighthouse score has changed between the two commits.
This section only shows metrics that have changed between the two commits.
Regressions are highlighted in pink.
Improvements are highlighted in blue.

Lighthouse CI Server is best-suited to users who are comfortable deploying and managing their own infrastructure.

For information on setting up the Lighthouse CI server, including recipes for using Heroku and Docker for deployment, refer to these instructions.

Find out more

Lighthouse CI GitHub repo