victoriametrics.com

About #

VictoriaMetrics Anomaly Detection (or shortly, vmanomaly) is a service that continuously scans VictoriaMetrics time series and detects unexpected changes within data patterns in real-time. It does so by utilizing user-configurable machine learning models.

It periodically queries user-specified metrics, computes an “anomaly score” for them, based on how well they fit a predicted distribution, taking into account periodical data patterns with trends, and pushes back the computed “anomaly score” to VictoriaMetrics. Then, users can enable alerting rules based on the “anomaly score”.

Compared to classical alerting rules, anomaly detection is more “hands-off” i.e. it allows users to avoid setting up manual alerting rules set up and catching anomalies that were not expected to happen. In other words, by setting up alerting rules, a user must know what to look for, ahead of time, while anomaly detection looks for any deviations from past behavior.

In addition to that, setting up alerting rules manually has been proven to be tedious and error-prone, while anomaly detection can be easier to set up, and use the same model for different metrics.

vmanomaly can be used as a helper to set up your own alerting. You can rely on the spikes you see in anomaly scores to form the metric queries for alerting rules.

Note: vmanomaly is a part of enterprise package. You need to get a free trial license for evaluation.

How? #

VictoriaMetrics Anomaly Detection service (vmanomaly) allows you to apply several built-in anomaly detection algorithms. You can also plug in your own detection models, code doesn’t make any distinction between built-in models or external ones.

All the service parameters (model, schedule, input-output) are defined in a config file.

Single config file supports only one model, but it’s totally OK to run multiple vmanomaly processes in parallel, each using its own config.

Models #

Currently, vmanomaly ships with a set of built-in models:

For a detailed overview, see model section

  1. ZScore

    (useful for testing)

    Simplistic model, that detects outliers as all the points that lie farther than a certain amount from time-series mean (straight line). Keeps only two model parameters internally: mean and std (standard deviation).

  2. Prophet

    (simplest in configuration, recommended for getting started)

    Uses Facebook Prophet for forecasting. The anomaly score is computed of how close the actual time series values follow the forecasted values (yhat), and whether it’s within forecasted bounds (yhat_lower, yhat_upper). The anomaly score reaches 1.0 if the actual data values are equal to yhat_lower or yhat_upper. The anomaly score is above 1.0 if the actual data values are outside the yhat_lower/yhat_upper bounds.

    See Prophet documentation

  3. Holt-Winters

    Very popular forecasting algorithm. See statsmodels.org documentation for Holt-Winters exponential smoothing.

  4. Seasonal-Trend Decomposition

    Extracts three components: season, trend, and residual, that can be plotted individually for easier debugging. Uses LOESS (locally estimated scatterplot smoothing). See statsmodels.org documentation for LOESS STD.

  5. Rolling Quantile

    A simple moving window of quantiles. Easy to use, easy to understand, but not as powerful as other models.

  6. Isolation Forest

    Detects anomalies using binary trees. It works for both univariate and multivariate data. Be aware of the curse of dimensionality in the case of multivariate data - we advise against using a single model when handling multiple time series if the number of these series significantly exceeds their average length (# of data points).

    The algorithm has a linear time complexity and a low memory requirement, which works well with high-volume data. See scikit-learn.org documentation for Isolation Forest.

  7. MAD (Median Absolute Deviation)

    A robust method for anomaly detection that is less sensitive to outliers in data compared to standard deviation-based models. It considers a point as an anomaly if the absolute deviation from the median is significantly large.

  8. AutoTuned

    Designed to take the cognitive load off the user, allowing any of built-in models below to be re-tuned for best params on data seen during each fit phase of the algorithm. Tradeoff is between increased computational time and optimized results / simpler maintenance.

  9. Online Z-score

    Online alternative to Z-score model with exact same behavior and use cases.

  10. Online MAD

    Approximate online alternative to MAD model, appropriate for the same use cases.

  11. Online Seasonal Quantile

    Best used on de-trended data with strong (possibly multiple) seasonalities. Can act as a (slightly less powerful) online replacement to ProphetModel.

Note: The full list of models can be found here

Examples #

For example, here’s how Prophet predictions could look like on a real-data example
(Prophet auto-detected seasonality interval):

propher-example

And here’s what Holt-Winters predictions real-world data could look like (seasonality manually set to 1 week). Notice that it predicts anomalies in different places than Prophet because the model noticed there are usually spikes on Friday morning, so it accounted for that:

holtwinters-example

Process #

Upon starting, vmanomaly queries the initial range of data, and trains its model (“fit” by convention).

Then, reads new data from VictoriaMetrics, according to schedule, and invokes its model to compute “anomaly score” for each data point. The anomaly score ranges from 0 to positive infinity. Values less than 1.0 are considered “not an anomaly”, values greater or equal than 1.0 are considered “anomalous”, with greater values corresponding to larger anomaly. Then, vmanomaly pushes the metric to vminsert (under the user-configured metric name, optionally preserving labels).

Usage #

Starting from v1.5.0, vmanomaly requires a license key to run. You can obtain a trial license key here.

See Quickstart.

See Integration guide: vmanomaly and vmalert.

Config file #

There are 4 required sections in config file:

  • schedulers - defines how often to run and make inferences, as well as what timerange to use to train the model.
  • models - specific model parameters and configurations.
  • reader - how to read data and where it is located
  • writer - where and how to write the generated output.

monitoring - defines how to monitor work of vmanomaly service. This config section is optional.

For a detailed description, see config sections

Config example #

Here is an example of config file that will run Facebook’s Prophet model, that will be retrained every 2 hours on 14 days of previous data. It will generate inference results (including anomaly_score metric) every 1 minute.

You need to specify your datasource urls to use it:

schedulers:
  periodic:
    infer_every: "1m"
    fit_every: "2h"
    fit_window: "14d"

models:
  prophet:  # or use a model alias of your choice here
    class: "prophet"  # or "model.prophet.ProphetModel" until v1.13.0
    args:
      interval_width: 0.98

reader:
  datasource_url: [YOUR_DATASOURCE_URL] #Example: "http://victoriametrics:8428/"
  queries:
    cache: "sum(rate(vm_cache_entries))"

writer:
  datasource_url: [YOUR_DATASOURCE_URL] # Example: "http://victoriametrics:8428/"

Monitoring #

vmanomaly can be monitored by using push or pull approach. It can push metrics to VictoriaMetrics or expose metrics in Prometheus exposition format.

For a detailed description, see monitoring section

Push approach #

vmanomaly can push metrics to VictoriaMetrics single-node or cluster version. In order to enable push approach, specify push section in config file:

monitoring:
   push:
      url: [YOUR_DATASOURCE_URL] #Example: "http://victoriametrics:8428/"
      extra_labels:
         job: "vmanomaly-push"

Pull approach #

vmanomaly can export internal metrics in Prometheus exposition format at /metrics page. These metrics can be scraped via vmagent or Prometheus.

In order to enable pull approach, specify pull section in config file:

monitoring:
   pull:
      enable: true
      port: 8080

This will expose metrics at http://0.0.0.0:8080/metrics page.

Run vmanomaly Docker Container #

To use vmanomaly you need to pull docker image:

docker pull victoriametrics/vmanomaly:v1.18.8

Note: please check what is latest release in CHANGELOG

Note: us-docker.pkg.dev/victoriametrics-test/public/vmanomaly-trial is deprecated since v1.6.0. Use DockerHub repo instead

You can put a tag on it for your convenience:

docker image tag victoriametrics/vmanomaly:v1.18.8 vmanomaly

Here is an example of how to run vmanomaly docker container with license file:

export YOUR_LICENSE_FILE_PATH=path/to/license/file
export YOUR_CONFIG_FILE_PATH=path/to/config/file
docker run -it --net [YOUR_NETWORK] \
               -v YOUR_LICENSE_FILE_PATH:/license \
               -v YOUR_CONFIG_FILE_PATH:/config.yml \
               vmanomaly /config.yml \
               --licenseFile=/license

Licensing #

The license key can be passed via the following command-line flags:

  --license     STRING  License key for VictoriaMetrics Enterprise.
                        See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.
  --licenseFile STRING  Path to file with license key for VictoriaMetrics Enterprise.
                        See https://victoriametrics.com/products/enterprise/trial/ to obtain a trial license.
  --license.forceOffline 
                        Whether to force offline verification for VictoriaMetrics Enterprise license key, 
                        which has been passed either via -license or via -licenseFile command-line flag. 
                        The issued license key must support offline verification feature. 
                        Contact [email protected] if you need offline license verification.

In order to make it easier to monitor the license expiration date, the following metrics are exposed(see Monitoring section for details on how to scrape them):

# HELP vm_license_expires_at When the license expires as a Unix timestamp in seconds
# TYPE vm_license_expires_at gauge
vm_license_expires_at 1.6963776e+09
# HELP vm_license_expires_in_seconds Amount of seconds until the license expires
# TYPE vm_license_expires_in_seconds gauge
vm_license_expires_in_seconds 4.886608e+06

Example alerts for vmalert:

groups:
  - name: vm-license
    # note the `job` label and update accordingly to your setup
    rules:
      - alert: LicenseExpiresInLessThan30Days
        expr: vm_license_expires_in_seconds < 30 * 24 * 3600
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} instance {{ $labels.instance }} license expires in less than 30 days"
          description: "{{ $labels.instance }} of job {{ $labels.job }} license expires in {{ $value | humanizeDuration }}. 
            Please make sure to update the license before it expires."

      - alert: LicenseExpiresInLessThan7Days
        expr: vm_license_expires_in_seconds < 7 * 24 * 3600
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.job }} instance {{ $labels.instance }} license expires in less than 7 days"
          description: "{{ $labels.instance }} of job {{ $labels.job }} license expires in {{ $value | humanizeDuration }}. 
            Please make sure to update the license before it expires."

Content

Back to Page