The first bug fix release for Prometheus 3.

• [BUGFIX] Promql: Make subqueries left open. #15431
• [BUGFIX] Fix memory leak when query log is enabled. #15434
• [BUGFIX] Support utf8 names on /v1/label/:name/values endpoint. #15399

This release includes new features such as a brand new UI and UTF-8 support enabled by default. As this marks the first new major version in seven years, several breaking changes are introduced. The breaking changes are mainly around the removal of deprecated feature flags and CLI arguments, and the full list can be found below. For users that want to upgrade we recommend to read through our migration guide.

• [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376
• [CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373
• [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136
• [CHANGE] Remote-write: default enable_http2 to false. #15219
• [CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #15164
• [CHANGE] Scraping: config scrape_classic_histograms was renamed to always_scrape_classic_histograms. #15178
• [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657
• [CHANGE] Disallow configuring AM with the v1 api. #13883
• [CHANGE] regexp . now matches all characters (performance improvement). #14505
• [CHANGE] holt_winters is now called double_exponential_smoothing and moves behind the experimental-promql-functions feature flag. #14930
• [CHANGE] API: The OTLP receiver endpoint can now be enabled using --web.enable-otlp-receiver instead of --enable-feature=otlp-write-receiver. #14894
• [CHANGE] Prometheus will not add or remove port numbers from the target address. no-default-scrape-port feature flag removed. #14160
• [CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #14906
• [CHANGE] Don't create extra _created timeseries if feature-flag created-timestamp-zero-ingestion is enabled. #14738
• [CHANGE] Float literals and time durations being the same is now a stable fetaure. #15111
• [CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag --enable-feature=old-ui for the time being. #14872
• [CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #13904
• [CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #14365
• [CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #14365
• [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #14705
• [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #14807
• [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed. #14770
• [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #14747
• [CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #13456, #14526
• [CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #14640, #14643
• [FEATURE] OTLP receiver: Ability to skip UTF-8 normalization using otlp.translation_strategy = NoUTF8EscapingWithSuffixes configuration option. #15384
• [FEATURE] Support config reload automatically - feature flag auto-reload-config. #14769
• [ENHANCEMENT] Scraping, rules: handle targets reappearing, or rules moving group, when out-of-order is enabled. #14710
• [ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #15196
• [ENHANCEMENT] Scraping: support Created-Timestamp feature on native histograms. #14694
• [ENHANCEMENT] UI: Many fixes and improvements. #14898, #14899, #14907, #14908, #14912, #14913, #14914, #14931, #14940, #14945, #14946, #14972, #14981, #14982, #14994, #15096
• [ENHANCEMENT] UI: Web UI now displays notifications, e.g. when starting up and shutting down. #15082
• [ENHANCEMENT] PromQL: Introduce exponential interpolation for native histograms. #14677
• [ENHANCEMENT] TSDB: Add support for ingestion of out-of-order native histogram samples. #14850, #14546
• [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909
• [ENHANCEMENT] Kubernetes SD: Support sidecar containers in endpoint discovery. #14929
• [ENHANCEMENT] Consul SD: Support catalog filters. #11224
• [ENHANCEMENT] Move AM discovery page from "Monitoring status" to "Server status". #14875
• [PERF] TSDB: Parallelize deletion of postings after head compaction. #14975
• [PERF] TSDB: Chunk encoding: shorten some write sequences. #14932
• [PERF] TSDB: Grow postings by doubling. #14721
• [PERF] Relabeling: Optimize adding a constant label pair. #12180
• [BUGFIX] UI: fix selector / series formatting for empty metric names. #15341
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
• [BUGFIX] OTLP receiver: Preserve colons when generating metric names in suffix adding mode (this mode is always enabled, unless one uses Prometheus as a library). #15251
• [BUGFIX] Scraping: Unit was missing when using protobuf format. #15095
• [BUGFIX] PromQL: Only return "possible non-counter" annotation when rate returns points. #14910
• [BUGFIX] TSDB: Chunks could have one unnecessary zero byte at the end. #14854
• [BUGFIX] "superfluous response.WriteHeader call" messages in log. #14884
• [BUGFIX] PromQL: Unary negation of native histograms. #14821
• [BUGFIX] PromQL: Handle stale marker in native histogram series (e.g. if series goes away and comes back). #15025
• [BUGFIX] Autoreload: Reload invalid yaml files. #14947
• [BUGFIX] Scrape: Do not override target parameter labels with config params. #11029

Full Changelog: v2.55.0...v3.0.0

This is the next Release Candidate of Prometheus 3.0.0.

Please test it out on non-critical environments; nothing further will be changed in 3.0.0 except to fix bugs.

Prometheus 3.0.0 includes new features such as a brand new UI and UTF-8 support enabled by default.

Migration guide is here.

New changes since v3.0.0-rc.0:

• [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #15376
• [CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #15373
• [FEATURE] OTLP receiver: Ability to skip UTF-8 normalization using otlp.translation_strategy = NoUTF8EscapingWithSuffixes configuration option. #15384
• [BUGFIX] Scraping: Don't log errors on empty scrapes. #15357
• [BUGFIX] UI: fix selector / series formatting for empty metric names. #15341

Full Changelog: v2.55.0...v3.0.0-rc.1

• [BUGFIX] round() function did not remove __name__ label. #15250

• [BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685, #14740

This is the first Release Candidate of Prometheus 3.0.0.
Please test it out on non-critical environments; nothing further will be changed in 3.0.0 except to fix bugs.

Prometheus 3.0.0 includes new features such as a brand new UI and UTF-8 support enabled by default.

New changes since v3.0.0-beta.1:

• [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #15136
• [CHANGE] Remote-write: default enable_http2 to false. #15219
• [CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #15164
• [CHANGE] Scraping: config scrape_classic_histograms was renamed to always_scrape_classic_histograms. #15178
• [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #14657
• [CHANGE] Disallow configuring AM with the v1 api. #13883
• [ENHANCEMENT] Scraping, rules: handle targets reappearing, or rules moving group, when out-of-order is enabled. #14710
• [ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #15196
• [ENHANCEMENT] Scraping: support Created-Timestamp feature on native histograms. #14694
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #14941
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #14941
• [BUGFIX] OTLP receiver: Preserve colons when generating metric names in suffix adding mode (this mode is always enabled, unless one uses Prometheus as a library). #15251

Changes from v3.0.0-beta.1 and v3.0.0-beta.0 are also included in this version.

Full Changelog: v2.55.0...v3.0.0-rc.0

What's Changed

• [FEATURE] PromQL: Add experimental info function. #14495
• [FEATURE] Support UTF-8 characters in label names - feature flag utf8-names. #14482, #14880, #14736, #14727
• [FEATURE] Scraping: Add the ability to set custom http_headers in config. #14817
• [FEATURE] Scraping: Support feature flag created-timestamp-zero-ingestion in OpenMetrics. #14356, #14815
• [FEATURE] Scraping: scrape_failure_log_file option to log failures to a file. #14734
• [FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
• [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
• [FEATURE] Promtool: tsdb create-blocks new option to add labels. #14403
• [FEATURE] Promtool: promtool test adds --junit flag to format results. #14506
• [FEATURE] TSDB: Add delayed-compaction feature flag, for people running many Prometheus to randomize timing. #12532
• [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
• [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
• [ENHANCEMENT] Remote Read client: Enable streaming remote read if the server supports it. #11379
• [ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450
• [ENHANCEMENT] PromQL: Delay deletion of __name__ label to the end of the query evaluation. This is experimental and enabled under the feature-flag promql-delayed-name-removal. #14477
• [ENHANCEMENT] PromQL: Experimental sort_by_label and sort_by_label_desc sort by all labels when label is equal. #14655, #14985
• [ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
• [ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in avg and avg_over_time. #14413
• [ENHANCEMENT] Tracing: Improve PromQL tracing, including showing the operation performed for aggregates, operators, and calls. #14816
• [ENHANCEMENT] API: Support multiple listening addresses. #14665
• [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
• [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948, #15120
• [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
• [BUGFIX] PromQL: make sort_by_label stable. #14985
• [BUGFIX] SD: Fix dropping targets (with feature flag new-service-discovery-manager). #13147
• [BUGFIX] SD: Stop storing stale targets (with feature flag new-service-discovery-manager). #13622
• [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
• [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
• [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
• [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14513, #14575, #14598, #14609, #14611, #14771, #14821
• [BUGFIX] TSDB: Only count unknown record types in record_decode_failures_total metric. #14042

New Contributors

• @maxamins made their first contribution in #14346
• @cuiweiyuan made their first contribution in #14626
• @harshitasao made their first contribution in #14690
• @patilsuraj767 made their first contribution in #14403
• @riskrole made their first contribution in #14751
• @jcreixell made their first contribution in #14477
• @kevinrawal made their first contribution in #14765
• @electron0zero made their first contribution in #14650
• @shandongzhejiang made their first contribution in #14700

Full Changelog: v2.54.1...v2.55.0

The next release candidate of Prometheus v2.55 has a few bug-fixes to queries, plus the experimental info function which we are including to facilitate feedback.
The auto-reload-config feature was removed from v2.55 as it needed more fixes; it is available in v3.0 beta 1.

• [FEATURE] PromQL: Add experimental info function. #14495
• [BUGFIX] PromQL: make sort_by_label stable. #14985
• [BUGFIX] TSDB: Don't read in-order chunks from before head MinTime #15162

This is yet another beta release for Prometheus 3.0 and is not ready for production usage.

The changes mentioned below are all the new changes since v3.0.0-beta.0. For a complete changelog for 3.0 please read the notes from both beta.0 and beta.1.

• [CHANGE] regexp . now matches all characters (performance improvement). #14505
• [CHANGE] holt_winters is now called double_exponential_smoothing and moves behind the experimental-promql-functions feature flag. #14930
• [CHANGE] API: The OTLP receiver endpoint can now be enabled using --web.enable-otlp-receiver instead of --enable-feature=otlp-write-receiver. #14894
• [CHANGE] Prometheus will not add or remove port numbers from the target address. no-default-scrape-port feature flag removed. #14160
• [CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #14906
• [CHANGE] Don't create extra _created timeseries if feature-flag `created-timestamp-zero-ingestion' is enabled. #14738
• [CHANGE] Float literals and time durations being the same is now a stable fetaure. #15111
• [ENHANCEMENT] UI: Many fixes and improvements. #14898, #14899, #14907, #14908, #14912, #14913, #14914, #14931, #14940, #14945, #14946, #14972, #14981, #14982, #14994, #15096
• [ENHANCEMENT] UI: Web UI now displays notifications, e.g. when starting up and shutting down. #15082
• [ENHANCEMENT] PromQL: Introduce exponential interpolation for native histograms. #14677
• [ENHANCEMENT] TSDB: Add support for ingestion of out-of-order native histogram samples. #14850, #14546
• [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #13909
• [ENHANCEMENT] Scraping: support Created-Timestamp feature on native histograms. #14694
• [ENHANCEMENT] Kubernetes SD: Support sidecar containers in endpoint discovery. #14929
• [ENHANCEMENT] Consul SD: Support catalog filters. #11224
• [PERF] TSDB: Parallelize deletion of postings after head compaction. #14975
• [PERF] TSDB: Chunk encoding: shorten some write sequences. #14932
• [PERF] TSDB: Grow postings by doubling. #14721
• [PERF] Relabeling: Optimize adding a constant label pair. #12180
• [BUGFIX] Scraping: Unit was missing when using protobuf format. #15095
• [BUGFIX] PromQL: Only return "possible non-counter" annotation when rate returns points. #14910
• [BUGFIX] TSDB: Chunks could have one unnecessary zero byte at the end. #14854
• [BUGFIX] "superfluous response.WriteHeader call" messages in log. #14884
• [BUGFIX] PromQL: Unary negation of native histograms. #14821
• [BUGFIX] PromQL: Handle stale marker in native histogram series (e.g. if series goes away and comes back). #15025
• [BUGFIX] Autoreload: Reload invalid yaml files. #14947

While all the excitement is focused on Prometheus 3.0, we are preparing release 2.55.

This release has quite a few new features, some of which require a feature-flag to enable, a major optimisation of queries when out-of-order data is present, and a number of bug-fixes.

• [FEATURE] Support UTF-8 characters in label names - feature flag utf8-names. #14482, #14880, #14736, #14727
• [FEATURE] Support config reload automatically - feature flag auto-reload-config. #14769
• [FEATURE] Scraping: Add the ability to set custom http_headers in config. #14817
• [FEATURE] Scraping: Support feature flag created-timestamp-zero-ingestion in OpenMetrics. #14356, #14815
• [FEATURE] Scraping: scrape_failure_log_file option to log failures to a file. #14734
• [FEATURE] PromQL: Delay removal of __name__ label - feature flag promql-delayed-name-removal. #14477
• [FEATURE] OTLP receiver: Optional promotion of resource attributes to series labels. #14200
• [FEATURE] API: Support multiple --web.listen-address. #14665
• [FEATURE] Remote-Write: Support Google Cloud Monitoring authorization. #14346
• [FEATURE] Promtool: tsdb create-blocks new option to add labels. #14403
• [FEATURE] Promtool: promtool test adds --junit flag to format results. #14506
• [ENHANCEMENT] OTLP receiver: Warn on exponential histograms with zero count and non-zero sum. #14706
• [ENHANCEMENT] OTLP receiver: Interrupt translation on context cancellation/timeout. #14612
• [ENHANCEMENT] PromQL: Improve detail in distributed tracing. #14816
• [ENHANCEMENT] PromQL: Experimental sort_by_label and sort_by_label_desc sort by all labels when label is equal. #14655
• [ENHANCEMENT] PromQL: Clarify error message logged when Go runtime panic occurs during query evaluation. #14621
• [ENHANCEMENT] PromQL: Use Kahan summation for better accuracy in avg and avg_over_time. #14413
• [ENHANCEMENT] TSDB: Backward compatibility with upcoming index v3. #14934
• [PERF] Remote-Read: Support streaming mode. #11379
• [PERF] TSDB: Query in-order and out-of-order series together. #14354, #14693, #14714, #14831, #14874, #14948
• [PERF] TSDB: Streamline reading of overlapping out-of-order head chunks. #14729
• [BUGFIX] SD: Fix dropping targets (with feature flag new-service-discovery-manager). #13147
• [BUGFIX] SD: Stop storing stale targets (with feature flag new-service-discovery-manager). #13622
• [BUGFIX] fix(utf8): propagate validationScheme config to scraping options. #14880
• [BUGFIX] PromQL: Experimental Native Histograms: Do not re-use spans between histograms. #14771
• [BUGFIX] Scraping: exemplars could be dropped in protobuf scraping. #14810
• [BUGFIX] Remote-Write: fix metadata sending for experimental Remote-Write V2. #14766
• [BUGFIX] Remote-Write: Return 4xx not 5xx when timeseries has duplicate label. #14716
• [BUGFIX] Experimental Native Histograms: many fixes for incorrect results, panics, warnings. #14598, #14611, #14609, #14575, #14513, #14821
• [BUGFIX] TSDB: Only count unknown record types in record_decode_failures_total metric. #14042
• [ENHANCEMENT] Remote-Write: Don't reshard if we haven't successfully sent a sample since last update. #14450