This post was updated on 8 August 2023 to include changes in Artillery v2 (from v1).

Artillery is an open-source command-line tool purpose-built for load testing and smoke testing web applications. It is written in JavaScript and it supports testing HTTP, Socket.io, and WebSockets APIs.

This article will get you started with load testing your Node.js APIs using Artillery. You'll be able to detect and fix critical performance issues before you deploy code to production.

Before we dive in and set up Artillery for a Node.js app, though, let's first answer the question: what is load testing and why is it important?

Why Should You Do Load Tests in Node.js?

Load testing is essential to quantify system performance and identify breaking points at which an application starts to fail. A load test generally involves simulating user queries to a remote server.

Load tests reproduce real-world workloads to measure how a system responds to a specified load volume over time. You can determine if a system behaves correctly under loads it is designed to handle and how adaptable it is to spikes in traffic. It is closely related to stress testing, which assesses how a system behaves under extreme loads and if it can recover once traffic returns to normal levels.

Load testing can help validate if an application can withstand realistic load scenarios without a degradation in performance. It can also help uncover issues like:

• Increased response times
• Memory leaks
• Poor performance of various system components under load

As well as other design issues that contribute to a suboptimal user experience.

In this article, we'll focus on the free and open-source version of Artillery to explore load testing. However, bear in mind that a pro version of Artillery is also available for those whose needs exceed what can be achieved through the free version. It provides added features for testing at scale and is designed to be usable even if you don't have prior DevOps experience.

Note: Currently, Artillery Pro is in maintenance mode and will be officially sunset in June 2024. The distributed load testing functionality of Artillery Pro is now available in the main open source distribution of Artillery.

Installing Artillery for Node.js

Artillery is an npm package so you can install it through npm or yarn:

$ yarn global add artillery

If this is successful, the artillery program should be accessible from the command line:

$ artillery -V
 
        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/
 
 
VERSION INFO:
 
Artillery: 2.0.0-35
Node.js:   v18.16.1
OS:        linux
 

Basic Artillery Usage

Once you've installed the Artillery CLI, you can start using it to send traffic to a web server. It provides a quick subcommand that lets you run a test without writing a test script first.

You'll need to specify:

• an endpoint
• the rate of virtual users per second or a fixed amount of virtual users
• how many requests should be made per user

$ artillery quick --count 20 --num 10 http://localhost:4000/example

The --count parameter above specifies the total number of virtual users, while --num indicates the number of requests that should be made per user. Therefore, 200 (20*10) GET requests are sent to the specified endpoint. On successful completion of the test, a report is printed out to the console.

All VUs finished. Total time: 3 seconds
 
--------------------------------
Summary report @ 05:14:34(+0300)
--------------------------------
 
http.codes.200: ................................................................ 200
http.downloaded_bytes: ......................................................... 1400
http.request_rate: ............................................................. 200/sec
http.requests: ................................................................. 200
http.response_time:
  min: ......................................................................... 0
  max: ......................................................................... 3
  median: ...................................................................... 1
  p95: ......................................................................... 1
  p99: ......................................................................... 2
http.responses: ................................................................ 200
vusers.completed: .............................................................. 20
vusers.created: ................................................................ 20
vusers.created_by_name.0: ...................................................... 20
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 12.1
  max: ......................................................................... 53.4
  median: ...................................................................... 14.2
  p95: ......................................................................... 24.8
  p99: ......................................................................... 24.8

This shows several details about the test run, such as the requests completed, response times, time taken for the test, and more. It also displays the response codes received on each request so that you can determine if your API handles failures gracefully in cases of overload.

While the quick subcommand is handy for performing one-off tests from the command line, it's quite limited in what it can achieve. That's why Artillery provides a way to configure different load testing scenarios through test definition files in YAML or JSON formats. This allows great flexibility to simulate the expected flows at one or more of your application's endpoints.

Writing Your First Artillery Test Script

In this section, I'll demonstrate a basic test configuration that you can apply to any application. If you want to follow along, you can set up a test environment for your project, or run the tests locally so that your production environment is not affected. Ensure you install Artillery as a development dependency so that the version you use is consistent across all deployments.

$ yarn add -D artillery

An Artillery test script consists of two main sections: config and scenarios. config includes the general configuration settings for the test such as the target, response timeouts, default HTTP headers, etc. scenarios consist of the various requests that virtual users should make during a test. Here's a script that tests an endpoint by sending 10 virtual users every second for 30 seconds:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 30
      arrivalRate: 10
 
scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"

In the above script, the config section defines the base URL for the application that's being tested in the target property. All the endpoints defined later in the script will run against this base URL.

The phases property is then used to set up the number of virtual users generated in a period of time and how frequently these users are sent to specified endpoints.

In this test, duration determines that virtual users will be generated for 30 seconds and arrivalRate determines the number of virtual users sent to the endpoints per second (10 users).

On the other hand, the scenarios section defines the various operations that a virtual user should perform. This is controlled through the flow property, which specifies the exact steps that should be executed in order. In this case, we have a single step: a GET request to the /example endpoint on the base URL. Every virtual user that Artillery generates will make this request.

Now that we've written our first script, let's dive into how to run a load test.

Running a Load Test in Artillery

Save your test script to a file (such as load-test.yml) and execute it through the command below:

$ artillery run path/to/script.yml

This command will start sending virtual users to the specified endpoints at a rate of 10 requests per second. A report will be printed to the console every 10 seconds, informing you of the number of test scenarios launched and completed within the time period, and other statistics such as mean response time, HTTP response codes, and errors (if any).

Once the test concludes, a summary report (identical to the one we examined earlier) is printed out before the command exits.

All VUs finished. Total time: 31 seconds
 
--------------------------------
Summary report @ 05:30:46(+0300)
--------------------------------
 
http.codes.200: ................................................................ 300
http.downloaded_bytes: ......................................................... 2100
http.request_rate: ............................................................. 10/sec
http.requests: ................................................................. 300
http.response_time:
  min: ......................................................................... 0
  max: ......................................................................... 28
  median: ...................................................................... 1
  p95: ......................................................................... 3
  p99: ......................................................................... 4
http.responses: ................................................................ 300
vusers.completed: .............................................................. 300
vusers.created: ................................................................ 300
vusers.created_by_name.Retrieve data: .......................................... 300
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 1.8
  max: ......................................................................... 44.4
  median: ...................................................................... 4
  p95: ......................................................................... 13.6
  p99: ......................................................................... 21.5

How to Create Realistic User Flows

The test script we executed in the previous section is not very different from the quick example in that it makes requests to only a single endpoint. However, you can use Artillery to test more complex user flows in an application.

In a SaaS product, for example, a user flow could be: someone lands on your homepage, checks out the pricing page, and then signs up for a free trial. You'll definitely want to find out how this flow will perform under stress if hundreds or thousands of users are trying to perform these actions at the same time.

Here's how you can define such a user flow in an Artillery test script:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
      name: "Warming up"
    - duration: 240
      arrivalRate: 20
      rampTo: 100
      name: "Ramping up"
    - duration: 500
      arrivalRate: 100
      name: "Sustained load"
  processor: "./processor.js"
 
scenarios:
  - name: "Sign up flow"
    flow:
      - get:
          url: "/"
      - think: 1
      - get:
          url: "/pricing"
      - think: 2
      - get:
          url: "/signup"
      - think: 3
      - post:
          url: "/signup"
          beforeRequest: generateSignupData
          json:
            email: "{{ email }}"
            password: "{{ password }}"

In the above script, we define three test phases in config.phases:

• The first phase sends 20 virtual users per second to the application for 60 seconds.
• In the second phase, the load will start at 20 users per second and gradually increase to 100 users per second over 240 seconds.
• The third and final phase simulates a sustained load of 100 users per second for 500 seconds.

By providing several phases, you can accurately simulate real-world traffic patterns and test how adaptable your system is to a sudden barrage of requests.

The steps that each virtual user takes in the application are under scenarios.flow. The first request is GET / which leads to the homepage. Afterward, there is a pause for 1 second (configured with think) to simulate user scrolling or reading before making the next GET request to /pricing. After a further delay of 2 seconds, the virtual user makes a GET request to /signup. The last request is POST /signup, which sends a JSON payload in the request body.

The {{ email }} and {{ password }} placeholders are populated through the generateSignupData function, which executes before the request is made. This function is defined in the processor.js file referenced in config.processor. In this way, Artillery lets you specify custom hooks to execute at specific points during a test run. Here are the contents of processor.js:

const Faker = require("faker");
 
function generateSignupData(requestParams, ctx, ee, next) {
  ctx.vars["email"] = Faker.internet.exampleEmail();
  ctx.vars["password"] = Faker.internet.password(10);
 
  return next();
}
 
module.exports = {
  generateSignupData,
};

The generateSignupData function uses methods provided by Faker.js to generate a random email address and password each time it is called. The results are then set on the virtual user's context, and next() is called so that the scenario can continue to execute. You can use this approach to inject dynamic random content into your tests so they're as close as possible to real-world requests.

Note that other hooks are available aside from beforeRequest, including the following:

• afterResponse - Executes one or more functions after a response has been received from the endpoint:

- post:
    url: "/login"
    afterResponse:
      - "logHeaders"
      - "logBody"

• beforeScenario and afterScenario - Used to execute one or more functions before or after each request in a scenario:

scenarios:
  - beforeScenario: "setData"
    afterScenario: "logResults"
    flow:
      - get:
          url: "/auth"

• function - Can execute functions at any point in a scenario:

- post:
    url: "/login"
    function: "doSomething"

Injecting Data from a Payload File

Artillery also lets you inject custom data through a payload file in CSV format. For example, instead of generating fake email addresses and passwords on the fly as we did in the previous section, you can have a predefined list of such data in a CSV file:

[email protected],rwkWspKUKy
[email protected],7BaFHbaWga
[email protected],CWvc6Bznnh
[email protected],1vlT_02i6h
[email protected],1N0PRraQU7

To access the data in this file, you need to reference it in the test script through the config.payload.path property. Secondly, you need to specify the names of the fields you'd like to access through config.payload.fields. The config.payload property provides several other options to configure its behavior, and it's also possible to specify multiple payload files in a single script.

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./auth.csv"
    fields:
      - "email"
      - "password"
 
scenarios:
  - name: "Authenticating users"
    flow:
      - post:
          url: "/login"
          json:
            email: "{{ email }}"
            password: "{{ password }}"

Capturing Response Data From an Endpoint

Artillery makes it easy to capture the response of a request and reuse certain fields in a subsequent request. This is helpful if you're simulating flows with requests that depend on an earlier action's execution.

Let's assume you're providing a geocoding API that accepts the name of a place and returns its longitude and latitude in the following format:

{
  "longitude": -73.935242,
  "latitude": 40.73061
}

You can populate a CSV file with a list of cities:

Seattle
London
Paris
Monaco
Milan

Here's how you can configure Artillery to use each city's longitude and latitude values in another request. For example, you can use the values to retrieve the current weather through another endpoint:

config:
  target: "http://localhost:4000"
  phases:
    - duration: 60
      arrivalRate: 20
  payload:
    path: "./cities.csv"
    fields:
      - "city"
 
scenarios:
  - flow:
      - get:
          url: "/geocode?city={{ city }}"
          capture:
            - json: "$.longitude"
              as: "lon"
            - json: "$.latitude"
              as: "lat"
      - get:
          url: "/weather?lon={{ lon }}&lat={{ lat }}"

The capture property above is where all the magic happens. It's where you can access the JSON response of a request and store it in a variable to reuse in subsequent requests. The longitude and latitude properties from the /geocode response body (with the aliases lon and lat, respectively) are then passed on as query parameters to the /weather endpoint.

Using Artillery in a CI/CD Environment

An obvious place to run your load testing scripts is in a CI/CD pipeline so that your application is put through its paces before being deployed to production.

When using Artillery in such environments, it's necessary to set failure conditions that cause the program to exit with a non-zero code. Your deployment should abort if performance objectives are not met. Artillery provides support for this use case through its config.ensure property.

ensure is available as a [Plugin]. Artillery Plugins are distributed as npm packages named with an artillery-plugin- prefix, e.g., artillery-plugin-ensure.

First install the plugin:

$ yarn global add artillery-plugin-ensure

If you installed Artillery as a project dependency, then install the plugin in the same way:

$ yarn add -D artillery-plugin-ensure

To use a plugin, you first have to enable it in config.plugins:

config:
  plugins:
    ensure: {}

Then you can set some checks with config.ensure.

You can set two types of checks:

• thresholds - check that a metric's value is less than the defined integer value
• conditions - can be used to create advanced checks combining multiple metrics and conditions

Below, we set a threshold check to ensure that 95% of all requests have an aggregate response time of less than 100.

config:
  target: "http://localhost:4000"
  plugins:
    ensure: {}
  phases:
    - duration: 60
      arrivalRate: 20
  ensure:
    thresholds:
      # p95 of response time must be <100:
      - "http.response_time.p95": 100
 
scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"

Once you run the test, it will continue as before, except that assertions are verified at the end of the test and cause the program to exit with a non-zero exit code if requirements are not met. The reason for a test failure is printed at the bottom of the summary report.

All VUs finished. Total time: 1 minute, 1 second
 
--------------------------------
Summary report @ 07:49:53(+0300)
--------------------------------
 
http.codes.200: ................................................................ 1200
http.downloaded_bytes: ......................................................... 8400
http.request_rate: ............................................................. 20/sec
http.requests: ................................................................. 1200
http.response_time:
  min: ......................................................................... 449
  max: ......................................................................... 488
  median: ...................................................................... 450.4
  p95: ......................................................................... 450.4
  p99: ......................................................................... 459.5
http.responses: ................................................................ 1200
vusers.completed: .............................................................. 1200
vusers.created: ................................................................ 1200
vusers.created_by_name.Retrieve data: .......................................... 1200
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 451.2
  max: ......................................................................... 559.2
  median: ...................................................................... 450.4
  p95: ......................................................................... 459.5
  p99: ......................................................................... 468.8
 
Checks:
fail: http.response_time.p95 < 100

Artillery still supports the v1 basic checks as shown below, but it is recommended that you use the previously mentioned thresholds and conditions. Below we use the old format to ensure that 99% of all requests have an aggregate response time of 150 milliseconds or less and that 1% or less of all requests are allowed to fail.

config:
  target: "http://localhost:4000"
  plugins:
    ensure: {}
  phases:
    - duration: 60
      arrivalRate: 20
  ensure:
    p99: 150
    maxErrorRate: 1
 
scenarios:
  - name: "Retrieve data"
    flow:
      - get:
          url: "/example"

Below is the summary report showing one failed and one okay check:

All VUs finished. Total time: 1 minute, 1 second
 
--------------------------------
Summary report @ 07:52:21(+0300)
--------------------------------
 
http.codes.200: ................................................................ 1200
http.downloaded_bytes: ......................................................... 8400
http.request_rate: ............................................................. 18/sec
http.requests: ................................................................. 1200
http.response_time:
  min: ......................................................................... 446
  max: ......................................................................... 465
  median: ...................................................................... 450.4
  p95: ......................................................................... 450.4
  p99: ......................................................................... 459.5
http.responses: ................................................................ 1200
vusers.completed: .............................................................. 1200
vusers.created: ................................................................ 1200
vusers.created_by_name.Retrieve data: .......................................... 1200
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 451.3
  max: ......................................................................... 558.3
  median: ...................................................................... 450.4
  p95: ......................................................................... 459.5
  p99: ......................................................................... 468.8
 
Checks:
fail: p99 < 150
ok: maxErrorRate < 1

Generating Status Reports in Artillery

Artillery prints a summary report for each test run to the standard output, but it's also possible to output detailed statistics for a test run into a JSON file by utilizing the --output flag:

$ artillery run config.yml --output test.json

Once the test completes, its report is placed in a test.json file in the current working directory. This JSON file can be visualized by converting it into an HTML report through the report subcommand:

$ artillery report --output report.html test.json
Report generated: report.html

You can open the report.html file in your browser to view a full report of the test run. It includes tables and several charts that should give you a good idea of how your application performed under load:

Note: This will be deprecated in the next major release of Artillery. In the future, Artillery reports will be visualized in the Artillery Dashboard which will be part of Artillery Cloud.

Extending Artillery With Plugins

Artillery's built-in tools for testing HTTP, Socket.io, and Websocket APIs can take you quite far in your load testing process. However, if you have additional requirements, you can search for plugins on NPM to extend Artillery's functionality.

Here are some official Artillery plugins that you might want to check out:

• artillery-plugin-expect: Helps with adding expectations to HTTP requests for functional or acceptance testing.
• artillery-plugin-publish-metrics: Used to send statistics from test runs to some external monitoring and observability systems.
• artillery-plugin-fuzzer: Helps you fuzz test your APIs with random and unexpected payloads to your API endpoints so you can catch errors. It is based on the Big List Of Naughty Strings.
• artillery-plugin-metrics-by-endpoint: Breaks down response time metrics by endpoint rather than displaying aggregate values across all endpoints.

You can also extend Artillery by creating your own plugins.

Use Artillery for Node.js Apps to Avoid Downtime

In this article, we've described how you can set up a load testing workflow for your Node.js applications with Artillery. This setup will ensure that your application performance stays predictable under various traffic conditions. You'll be able to account well for traffic-heavy periods and avoid downtime, even when faced with a sudden influx of users.

We've covered a sizeable chunk of what Artillery can do for you, but there's still lots more to discover. Ensure you read Artillery's official documentation to learn about the other features on offer.

Thanks for reading, and happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.