Note: In version 3.9, Native Histograms is no longer experimental, and the feature flag native-histogram has no effect. You must now turn on the config setting scrape_native_histogram to collect Native Histogram samples from exporters.

• [CHANGE] Native Histograms are no longer experimental! Make the native-histogram feature flag a no-op. Use scrape_native_histograms config option instead. #17528
• [CHANGE] API: Add maximum limit of 10,000 sets of statistics to TSDB status endpoint. #17647
• [FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
• [FEATURE] Promtool: Add start_timestamp field for unit tests. #17636
• [FEATURE] Promtool: Add --format seriesjson option to tsdb dump to output just series labels in JSON format. #13409
• [FEATURE] Add --storage.tsdb.delay-compact-file.path flag for better interoperability with Thanos. #17435
• [FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
• [ENHANCEMENT]: TSDB: add flag --storage.tsdb.block-reload-interval to configure TSDB Block Reload Interval. #16728
• [ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
• [ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
• [ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
• [ENHANCEMENT] Notifications: Add a histogram prometheus_notifications_latency_histogram_seconds to complement the existing summary. #16637
• [ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
• [ENHANCEMENT] SD: add a config label with job name for most prometheus_sd_refresh metrics. #17138
• [ENHANCEMENT] TSDB: New histogram prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
• [ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
• [PERF] TSDB: Small optimizations to postings index. #17439
• [PERF] Scraping: Speed up relabelling of series. #17530
• [PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
• [BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in _total. #17682
• [BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
• [BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
• [BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
• [BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty ignoring() and non-empty grouping. #17643
• [BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
• [BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
• [BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
• [BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
• [BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
• [BUGFIX] PromQL: Ensure that rate/increase/delta of histograms results in a gauge histogram. #17608
• [BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
• [BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
• [BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
• [BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
• [BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472

• [BUGFIX] remote: Fix Remote Write receiver, so it does not send wrong response headers for v1 flow and cause Prometheus senders to emit false partial error log and metrics. #17683

Note for users of Native Histograms

This is the first release with Native Histograms as a stable feature. However, scraping Native Histograms has to be activated explicitly via the scrape_native_histograms config setting (newly introduced in this release). To ease the transition, the --enable-feature=native-histograms flag is not a complete no-op in this release, but changes the default value of scrape_native_histograms to true. In the next release (v3.9), the feature flag will be a complete no-op, and the default value of scrape_native_histograms will always be false. If you have been using the feature flag so far, the recommended course of action is the following:

1. Upgrade to v3.8 and keep the feature flag. Everything should work as before.
2. At your own pace, set scrape_native_histograms to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histograms, allowing granular control if needed. It is a good idea to also set scrape_native_histograms explicitly to false where you do not want to scrape Native Histograms. In this way, you do not depend on the default value of the setting anymore.)
3. Remove the feature flag and make sure that everything still works as intended.
4. Now you are ready for an upgrade to the next release (v3.9).

Changelog

• [CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
• [CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
• [FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592
• [FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
• [FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
• [FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histograms config setting. #17232 #17315
• [FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
• [FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
• [FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
• [FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
• [ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
• [ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
• [ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
• [ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
• [ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
• [ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
• [ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
• [ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
• [ENHANCEMENT] Document Prometheus Agent. #14519
• [PERF] PromQL: Speed up parsing of variadic functions. #17316
• [PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
• [PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
• [PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
• [BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
• [BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
• [BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
• [BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
• [BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
• [BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
• [BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
• [BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
• [BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
• [BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424

• [CHANGE] Remote-write 2 (receiving): Update to 2.0-rc.4 spec. "created timestamp" (CT) is now called "start timestamp" (ST). #17411
• [FEATURE] OAuth2: support jwt-bearer grant-type (RFC7523 3.1). #17592

Note for users of Native Histograms

This is the first release with Native Histograms as a stable feature. However, scraping Native Histograms has to be activated explicitly via the scrape_native_histogram config setting (newly introduced in this release). To ease the transition, the --enable-feature=native-histograms flag is not a complete no-op in this release, but changes the default value of scrape_native_histogram to true. In the next release (v3.9), the feature flag will be a complete no-op, and the default value of scrape_native_histogram will always be false. If you have been using the feature flag so far, the recommended course of action is the following:

1. Upgrade to v3.8 and keep the feature flag. Everything should work as before.
2. At your own pace, set scrape_native_histogram to true in all relevant scrape configs. (There is a global and a per-scrape-config version of scrape_native_histogram, allowing granular control if needed. It is a good idea to also set scrape_native_histogram explicitly to false where you do not want to scrape Native Histograms. In this way, you do not depend on the default value of the setting anymore.)
3. Remove the feature flag and make sure that everything still works as intended.
4. Now you are ready for an upgrade to the next release (v3.9).

Changelog

• [CHANGE] TSDB: Native Histogram Custom Bounds with a NaN threshold are now rejected. #17287
• [FEATURE] Dockerfile: Add OpenContainers spec labels to Dockerfile. #16483
• [FEATURE] SD: Add unified AWS service discovery for ec2, lightsail and ecs services. #17046
• [FEATURE] Native histograms are now a stable, but optional feature, use the scrape_native_histogram config setting. #17232 #17315
• [FEATURE] UI: Support anchored and smoothed keyword in promql editor. #17239
• [FEATURE] UI: Show detailed relabeling steps for each discovered target. #17337
• [FEATURE] Alerting: Add urlQueryEscape to template functions. #17403
• [FEATURE] Promtool: Add Remote-Write 2.0 support to promtool push metrics via the --protobuf_message flag. #17417
• [ENHANCEMENT] Clarify the docs about handling negative native histograms. #17249
• [ENHANCEMENT] Mixin: Add static UID to the remote-write dashboard. #17256
• [ENHANCEMENT] PromQL: Reconcile mismatched NHCB bounds in Add and Sub. #17278
• [ENHANCEMENT] Alerting: Add "unknown" state for alerting rules that haven't been evaluated yet. #17282
• [ENHANCEMENT] Scrape: Allow simultaneous use of classic histogram → NHCB conversion and zero-timestamp ingestion. #17305
• [ENHANCEMENT] UI: Add smoothed/anchored in explain. #17334
• [ENHANCEMENT] OTLP: De-duplicate any target_info samples with the same timestamp for the same series. #17400
• [ENHANCEMENT] Document use_fips_sts_endpoint in sigv4 config sections. #17304
• [ENHANCEMENT] Document Prometheus Agent. #14519
• [PERF] PromQL: Speed up parsing of variadic functions. #17316
• [PERF] UI: Speed up alerts/rules/... pages by not rendering collapsed content. #17485
• [PERF] UI: Performance improvement when getting label name and values in promql editor. #17194
• [PERF] UI: Speed up /alerts for many firing alerts via virtual scrolling. #17254
• [BUGFIX] PromQL: Fix slice indexing bug in info function on churning series. #17199
• [BUGFIX] API: Reduce lock contention on /api/v1/targets. #17306
• [BUGFIX] PromQL: Consistent handling of gauge vs. counter histograms in aggregations. #17312
• [BUGFIX] TSDB: Allow NHCB with -Inf as the first custom value. #17320
• [BUGFIX] UI: Fix duplicate loading of data from the API speed up rendering of some pages. #17357
• [BUGFIX] Old UI: Fix createExpressionLink to correctly build /graph URLs so links from Alerts/Rules work again. #17365
• [BUGFIX] PromQL: Avoid panic when parsing malformed info call. #17379
• [BUGFIX] PromQL: Include histograms when enforcing sample_limit. #17390
• [BUGFIX] Config: Fix panic if TLS CA file is absent. #17418
• [BUGFIX] PromQL: Fix histogram_fraction for classic histograms and NHCB if lower bound is in the first bucket. #17424

• [BUGFIX] UI: Revert changed (and breaking) redirect behavior for -web.external-url if -web.route-prefix is configured, which was introduced in #17240. #17389
• [BUGFIX] Fix federation of some native histograms. #17299 #17409
• [BUGFIX] promtool: check config would fail when --lint=none flag was set. #17399 #17414
• [BUGFIX] Remote-write: fix a deadlock in the queue resharding logic that can lead to suboptimal queue behavior. #17412

• [BUGFIX] AWS SD: Fix AWS SDK v2 credentials handling for EC2 and Lightsail discovery. #17355
• [BUGFIX] AWS SD: Load AWS region from IMDS when not set. #17376
• [BUGFIX] Relabeling: Fix labelmap action validation with the legacy metric name validation scheme. #17372
• [BUGFIX] PromQL: Fix parsing failure when anchored and smoothed are used as metric names and label names. #17353
• [BUGFIX] PromQL: Fix formatting of range vector selectors with smoothed/anchored modifier. #17354

• [BUGFIX] OTLP: Prefix key_ to label name when translating an OTel attribute name starting with a single underscore, and keep multiple consecutive underscores in label name when translating an OTel attribute name. This reverts the breaking changes introduced in 3.7.0. #17344

Warning

We discovered a breaking change in the OTLP endpoint: OpenTelemetry attribute names starting with underscore, for example _attrib, are no longer automatically translated to key_attrib. Please use 3.7.1 instead for OTLP ingestion.

Changelog

• [CHANGE] Remote-write: the following metrics are deprecated:
  • prometheus_remote_storage_samples_in_total, use prometheus_wal_watcher_records_read_total{type="samples"} and prometheus_remote_storage_samples_dropped_total instead,
  • prometheus_remote_storage_histograms_in_total, use prometheus_wal_watcher_records_read_total{type=~".*histogram_samples"} and prometheus_remote_storage_histograms_dropped_total instead,
  • prometheus_remote_storage_exemplars_in_total, use prometheus_wal_watcher_records_read_total{type="exemplars"} and prometheus_remote_storage_exemplars_dropped_total instead,
  • prometheus_remote_storage_highest_timestamp_in_seconds, use the more accurate prometheus_remote_storage_queue_highest_timestamp_seconds instead in dashboards and alerts to properly account for relabeling and for more accuracy. #17065
• [FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
• [FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
• [FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
• [FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
• [FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
• [FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
• [ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
• [ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
• [ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
• [ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
• [ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
• [ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
• [PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
• [PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
• [PERF] PromQL: Move more work to preprocessing step. #16896
• [PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
• [PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
• [PERF] TSDB: Improve speed of querying a series with multiple matchers. #13971
• [BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
• [BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
• [BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
• [BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
• [BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping. #17156
• [BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
• [BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
• [BUGFIX] PromQL: Correct inaccurate character positions in errors for some aggregate expressions. #16996 #17031
• [BUGFIX] PromQL: Fix info() function on churning series. #17135
• [BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
• [BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
• [BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
• [BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
• [BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
• [BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
• [BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
• [BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
• [BUGIFX] Remote-write: Do not panic on invalid symbol table in remote-write 2.0. #17160

• [CHANGE] Remote-write: Deprecate prometheus_remote_storage_{samples,exemplars,histograms}_in_total and prometheus_remote_storage_highest_timestamp_in_seconds metrics, see their respective descriptions for alternatives. #17065
• [FEATURE] PromQL: Add support for experimental anchored and smoothed rate behind feature flag promql-extended-range-selectors. #16457
• [FEATURE] Federation: Add support for native histograms with custom buckets (NHCB). #17215
• [FEATURE] PromQL: Add first_over_time(...) and ts_of_first_over_time(...) behind feature flag experimental-promql-functions. #16963 #17021
• [FEATURE] Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver. #16788
• [FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. #17033
• [FEATURE] OTLP: Write start time of metrics as created time zero samples into TSDB when created-timestamp-zero-ingestion feature flag is enabled. #16951
• [ENHANCEMENT] PromQL: Add warn-level annotations for counter reset conflicts in certain histogram operations. #17051 #17094
• [ENHANCEMENT] UI: Add scrape interval and scrape timeout to targets page. #17158
• [ENHANCEMENT] TSDB: Reduce the resolution of native histograms read from chunks or remote read if the schema is exponential. #17213
• [ENHANCEMENT] Remote write: Add logging for unexpected metadata in sample batches, when metadata entries are found in samples-only batches. #17034 #17082
• [ENHANCEMENT] Remote-write: Add prometheus_remote_storage_queue_highest_timestamp_seconds metric, tracking the highest timestamp actually enqueued, taking write_relabel_configs into account. #17065
• [ENHANCEMENT] Mixin: Replace prometheus_remote_storage_highest_timestamp_in_seconds metric with the new prometheus_remote_storage_queue_highest_timestamp_seconds metric in dashboards and alerts to properly account for relabeling, for better accuracy. #17065
• [ENHANCEMENT] Rules: Support concurrent evaluation for rules querying ALERTS and ALERTS_FOR_STATE. #17064
• [ENHANCEMENT] TSDB: Add logs to improve visibility into internal operations. #17074
• [PERF] OTLP: Write directly to TSDB instead of passing through a Remote-Write adapter when receiving OTLP metrics. #16951
• [PERF] OTLP: Reduce number of logs emitted from OTLP endpoint. No need to log duplicate sample errors. #17201
• [PERF] PromQL: Move more work to preprocessing step. #16896
• [PERF] PromQL: Reduce allocations when walking the syntax tree. #16593
• [PERF] TSDB: Optimize appender creation, slightly speeding up startup. #16922
• [PERF] TSDB: Improve reverse index lookup performance. #13971
• [BUGFIX] Alerting: Mutating alerts relabeling (using replace actions, etc.) within a alertmanager_config.alert_relabel_configs block is now scoped correctly and no longer yields altered alerts to subsequent blocks. #17063
• [BUGFIX] Config: Infer valid escaping scheme when scrape config validation scheme is set. #16923
• [BUGFIX] TSDB: Correctly handle appending mixed-typed samples to the same series. #17071 #17241 #17290 #17295 #17296
• [BUGFIX] Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning. #17146
• [BUGFIX] TSDB: Fix metadata entries handling on metadata-wal-records feature for Native Histograms with custom buckets in protobuf scraping. #17156
• [BUGFIX] TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay. #17214
• [BUGFIX] PromQL: Avoid empty metric names in annotations for histogram_quantile(). #16794
• [BUGFIX] PromQL: Fix the character position of errors in some aggregate expressions. #16996 #17031
• [BUGFIX] PromQL: Fix info() function on churning series. #17135
• [BUGFIX] PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors. #17004
• [BUGFIX] TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52. #17189
• [BUGFIX] Remote-write: Fix HTTP handler to return after writing error response for invalid compression. #17050
• [BUGFIX] Remote-write: Return HTTP error 400 instead of 5xx for wrongly formatted Native Histograms. #17210
• [BUGFIX] Scrape: Prevent staleness markers from generating unnecessary series. #16429
• [BUGFIX] TSDB: Avoid misleading Failed to calculate size of \"wal\" dir error logs during WAL clean-up. #17006
• [BUGFIX] TSDB: Prevent erroneously dropping series records during WAL checkpoints. #17029
• [BUGFIX] UI: Fix redirect to path of -web.external-url if -web.route-prefix is configured. #17240
• [BUGIFX] Remote-write: Do not panic on invalid symbol table in remote-write 2.0. #17160