Rule: log warning spam from PromQL info "metric might not be a counter" #8229
Description
Thanos, Prometheus and Golang version used:
$ thanos --version
thanos, version 0.38.0 (branch: non-git, revision: non-git)
build user: pbuilder@build2001
build date: 20250414-09:27:40
go version: go1.24.2
platform: linux/amd64
tags: netgo
What happened:
After 0.38 upgrade thanos-rule spams logs at each evaluation when encountering promql warning/info messages (PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket) for example:
ts=2025-04-28T08:17:36.893830677Z caller=rule.go:1080 level=warn
component=rules warning="PromQL info: metric might not be a counter, name does
not end in _total/_sum/_count/_bucket: \"varnish_sli_all\""
query="sum(rate(varnish_sli_bad{cluster=\"cache_text\",site=\"magru\"}[15m])) /
sum(rate(varnish_sli_all{cluster=\"cache_text\",site=\"magru\"}[15m]))"
ts=2025-04-28T08:17:37.01919047Z caller=rule.go:1080 level=warn component=rules
warning="PromQL info: metric might not be a counter, name does not end in
_total/_sum/_count/_bucket: \"varnish_sli_bad\""
query="sum(rate(varnish_sli_bad{cluster=\"cache_text\",site=\"magru\"}[1h30m]))
/ sum(rate(varnish_sli_all{cluster=\"cache_text\",site=\"magru\"}[1h30m]))"
ts=2025-04-28T08:17:37.019256315Z caller=rule.go:1080 level=warn
component=rules warning="PromQL info: metric might not be a counter, name does
not end in _total/_sum/_count/_bucket: \"varnish_sli_all\""
query="sum(rate(varnish_sli_bad{cluster=\"cache_text\",site=\"magru\"}[1h30m]))
/ sum(rate(varnish_sli_all{cluster=\"cache_text\",site=\"magru\"}[1h30m]))"
What you expected to happen:
Not get logs spammed, maybe being able to tune the log level for filterOutPromQLWarnings ? Or since this is an info extannotation, maybe match log level for warning vs info extannotations ?
How to reproduce it (as minimally and precisely as possible):
Give thanos-rule a non-counter-named metric for rate()