Thank you for the feedback @dcaro, appreciate it! We did resolve the gaps issue by extending the rate() window in T382396 and that indeed fixed the issue.

see also this re: statuspage and ipv6 support https://community.atlassian.com/t5/Statuspage-questions/Enabling-IPv6-IPv4-dualstack-for-a-status-page/qaq-p/2847737

In T368953#10416707, @herron wrote:

Re: reverting liftwing to previous recording rules, the trouble is it'd be a revert to a broken state in Pyrra where the SLOs would be recording bad values and would activate several burn alerts. If we do need to revert the safest thing to do would be offboard of the liftwing SLOs, but it'd be nice to avoid that if we can. Would it be fair to prep an offboard patch and keep it in our back pocket in case of issue?

Done in T370506: Replace check_ripe_atlas with prometheus alert

No longer valid I think, also MXes now use postfix

Has been done at some point in host overview dashboard, sample query: node_cpu_frequency_hertz{instance=~"titan1001:.*"} or node_cpu_scaling_frequency_hertz{instance=~"titan1001:.*"}

I believe this is done for all intents and purposes, what do you think @Clement_Goubert ?

I'll boldly resolve this task, we haven't seen a reoccurrence

The check has moved to Prometheus in T370506: Replace check_ripe_atlas with prometheus alert

In T368953#10413075, @herron wrote:

Initial results with Thanos store caching bucket enabled look promising. I'm seeing reductions in duration, errors, network/socket utilization and slight decrease in cpu util

{F58028229} {F58028264}{F58028221} {F58028224}

Along with some increases in disk read/write (see above) and GC time, should be able to tune these out if they become a problem

{F58028233}

I added new panels to track Thanos store caching bucket metrics here https://grafana-rw.wikimedia.org/d/c2b5ccc9-0c16-45ae-99bc-a244e3f73808/thanos-cache-overview

Since we're staring to see some positive effects I think its worth trying to tune this further. Theres still room for improvement in iter and get hits, and in bucket cache evictions overall. Also you can see Thanos store quickly consumed all the memory available for cache

Thank you all for looking into this -- let's indeed see how 3m (or larger) goes and if that is satisfactory!

Indeed the underlying data/samples are there as expected: I tested this theory by removing all functions and look at the raw data, which has indeed no gaps. I noticed the interval used for (i)rate calculation is 2m, which I believe is the culprit. In the sense that we're scraping data every 1m and the raw data points (i.e. two) might fall outside the window looked by irate(...[2m]) for a given interval, therefore returning no points for that interval. Switching the rate calculation from 2m to 5m widens the window to look for data and eliminates the gaps, note the values are unchanged since rate/irate always return per-second values. Let me know what you think! cc @CDanis

We can certainly try the caching bucket in thanos store, since it is easy to do. I'd be happy to be wrong although I don't think that's going to help anything, since the problem in my mind is the quantity of data that thanos components (thanos-store, thanos-query, thanos-query-frontend) have to process.

No worries at all @cmooney, I've opened T382396: Investigate gnmic metric gaps and counters going to zero to investigate/followup on the two issues you mentioned

Thank you for the extensive explanation @cmooney ! Yes definitely let's go over the issues you outlined in the gnmi task and I'm happy to assist! Also thank you for the dashboards, I'll take a look and no cheekiness has been detected

I took a look as well at the general performance degradation when switching away from recording rules (i.e. increased CPU and network bandwidth on titan[12]001). Despite the attempts at optimizing what we have with @herron (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1103365?usp=search https://gerrit.wikimedia.org/r/c/operations/puppet/+/1104690?usp=search https://gerrit.wikimedia.org/r/c/operations/puppet/+/1104678?usp=search https://gerrit.wikimedia.org/r/c/operations/puppet/+/1103352?usp=search) I'm not seeing a significant change.

Thank you @Scott_French! I'm happy to help with ops-maint-gcal.js changes, feel free to send reviews my way

I took another look at the dashboards and it looks like to me we now have all interesting switch port metrics in Prometheus via gnmi for cloudsw devices. Specifically port in/out bytes and discards seem the most/only graphed metrics. Does that track @cmooney @dcaro ? If so I'll start a test conversion of https://grafana.wikimedia.org/d/613dNf3Gz/wmcs-ceph-eqiad-performance to swap out graphite for prometheus for said metrics.

FWIW as stated now the task is not actionable, I recommend setting alerts up as instructed by https://wikitech.wikimedia.org/wiki/Alertmanager#Grafana_alerts

I'm optimistically resolving this since I don't think we've seen it recently

On the thanos-compact side the signal for backlogged operations are the following metrics:

In T367065#10250012, @SLyngshede-WMF wrote:

To some extend I question the value of these checks. It's more or less just an "isalive" check for Apache and does not reveal anything about the underlying service.

For the purposes of routing instance alerts to their owner team we have been using role_owner variable, joining said variable on instance will attach the correct team label and thus alertmanager can apply the correct routing. Taking NodeTextfileStale as inspiration, the critical expression would be: ( node_nf_conntrack_entries / node_nf_conntrack_entries_limit * 100 >= 90 ) * on (instance) group_left(team) role_owner . Since team label will be in the expression, it can be removed from the alert itself. Hope that helps!

Indeed, grafana alerts are to be sent to alertmanager (i.e. centrally) and from there alerts can trigger email notifications. Moving to cxserver folks, I'm assuming an alert misconfiguration (?)

In T371426#10174127, @ecarg wrote:

Wow, looks great 🤩! Yes, that sounds like a plan, TY!
How do we route these alerts to trigger our Slack?

Nice! Thank you @hnowlan, resolving as we're done

In T375066#10158541, @Dzahn wrote:

Would it make sense to have clusters named after a team to group all machines owned by a specific subteam?

Or would that go against the purpose of clusters and they should stricly group servers by service / puppet role?

In T374842#10157255, @fnegri wrote:

Reading through the comments I found:

For centrallog we'll do the in place upgrade to Bookworm as part of T353912, I'd say sooner rather than later. Not sure about cloudlb* hosts, thus cc cloud-services-team to see if they do have indeed an ETA for cloudlb on Bookworm or not.

I've created a subtask for the cloudlb bookworm upgrade, it's not in our immediate plans but shouldn't be too hard: T375082: Upgrade cloudlb hosts to bookworm.

In T374842#10153975, @ssingh wrote:

On perhaps a related note, while it is true that many of the things the script is doing are taken care by the self-healing nature of anycast-hc in 0.9 and beyond, I do wonder if there is value in keeping this script around. anycast-hc is a critical part of our infrastructure especially on the DNS boxes because we announce all ns[0-2] routes from there. In a hypothetical case where it is not running, this would mean that all our nameservers are down. This leads me to wonder that in the absence of this script, what kind of an alert will be generated if anycast-hc is not running if it failed to start or is manually stopped for whatever reason?

One such alert can be the BGP one but there are many of them and the SNR is quite terrible. The other alert can be the systemd service not running but that again is shadowed by other such alerts. Does my concern make sense? In which case I am thinking perhaps we can restrict the scope of the script but still keep it around because if we get rid of this, there isn't a single dedicated script to check if anycast-hc is running or and in case it failed or didn't start, we won't probably catch it.

In T374916#10153020, @dcausse wrote:

We could perhaps adapt modules/query_service/files/monitor/prometheus-blazegraph-exporter.py to take care of running this query by possibly re-using the same gauge blazegraph_lastupdated but adapting the query depending on the namespace it's running.

In T374842#10149670, @ssingh wrote:

In T374842#10149337, @fgiunchedi wrote:

Thank you @ssingh, there's also centrallog hosts running bullseye + 0.8 in addition to cloudlb hosts, at any rate I tried installing the bookworm anycast-healthchecker 0.9 package on bullseye on a test host and it seems to work as expected (i.e. a rebuild for bullseye doesn't seem to be needed)

Ah, that's interesting. I mean it's not surprising it worked given the code but I am not sure if we want to extend it like that. Or maybe, it's fine and we do this all the time! (I know of some cases in which we do it but I am not sure if it is frowned upon.)

Doh! You are quite right, the relevant alert is MaxConntrack

Thank you @ssingh, there's also centrallog hosts running bullseye + 0.8 in addition to cloudlb hosts, at any rate I tried installing the bookworm anycast-healthchecker 0.9 package on bullseye on a test host and it seems to work as expected (i.e. a rebuild for bullseye doesn't seem to be needed)

@ssingh from T370068: Upgrade anycast-healthchecker to 0.9.8 (from 0.9.1-1+wmf12u1) I couldn't find any obvious blocker to have 0.9 on bullseye. Do you remember if there were there any obvious blockers for that to happen ? Thank you

Also note that prometheus-postgres-exporter since 0.12.0 has gained support for replication monitoring. This is IMHO the proper solution, though it will require >= trixie unless we chose to backport the package instead: https://github.com/prometheus-community/postgres_exporter/blob/master/CHANGELOG.md#0120--2023-03-21
It is an option that can/should be considered I think since it would mean future-proofing the postgresql monitoring infrastructure.

Resolving since we haven't run into further problems with envoy metrics ingestion

From my investigation so far on IRC:

script is deployed!

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

This is done, I've set the service as non-paging since we're using it for the prometheus web interface (i.e. humans) whereas the http service is paging since that's for automated access

This is done! The only other use of graphite_threshold is 'zuul_gearman_wait_queue' which will be addressed as part of T233089: Export zuul metrics to Prometheus. There are of course graphite-itself alerts left, which will be removed together with graphite.

I have merged the ported mw alerts (thank you @Clement_Goubert !) and changed the referenced dashboard at https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts to show the prometheus metrics.