Had a closer look into the threshold tunables and I don't actually see a way to change this natively within Pyrra. As-is the "for" duration of SLOMetricAbsent alerts is 6m. Pyrra has options to enable/disable the absent alert, but maybe we can configure something to put this alert in a silence/inhibit waiting room for 30-60m before it alerts (or recovers on its own)

SLOMetricAbsent for dead_letters_hits, varnish_sli_bad, trafficserver_backend_sli_bad and haproxy_sli_bad occurred today, at two different times.

Reviewing thanos-rule logs I'm seeing related discards with err="out of order sample"

It occurred to me that deploying absent metric alerts for the metrics where we're seeing gaps would be a reasonable next step. That'd let us troubleshoot gaps in/closer to their broken state which should help toward better understanding the issue and steps to resolve manually. Plus, it'd of course help us respond faster and shrink the gaps. I'll work on a patch.

Proposal SGTM for the near-term. Thank you for organizing this!

Buster is looking fine with this deb as well. So I've gone ahead and uploaded 1.8.0 to bookworm-wikimedia, bullseye-wikimedia, and buster-wikimedia. Up next is a small canary upgrade before fully rolling out.

Nice, thanks for the pointer! It looks like export CGO_ENABLED=0 does the right thing. At least, with this set the package builds and installs successfully on my bullseye test host.

In T368088#9917836, @MoritzMuehlenhoff wrote:

Given that this is a Go static ELF we can also simply build on bookworm and copy over the deb to bullseye-wikimedia, we're doing this for other exporters as well. buster might be tricky due to it's old libc6, but we can also ignore it, there's less than 150 hosts left and they can simply live the old IPMI monitoring.

Quick update: prometheus-ipmi-exporter-1.8.0 was a straightforward backport for bookworm https://gitlab.wikimedia.org/repos/sre/prometheus-ipmi-exporter/-/jobs/291532

In T352756#9913669, @elukey wrote:

On the 6th in the thanos-rule logs I see a ton of errors while connecting to Prometheus nodes, and around 14 UTC a reload was issued on both titan active nodes.

In T253810#9485013, @fgiunchedi wrote:

ipmi_exporter now has support to collect generic SEL entries and export metrics from those: https://github.com/prometheus-community/ipmi_exporter/pull/179

Sounds like a plan!

In T365265#9902980, @Clement_Goubert wrote:

I think we may want to enable it deployment per deployment so SRE Observability can monitor the load on prometheus. @colewhite or @herron can we coordinate on this?

Looking longer-term I think it'd be generally worthwhile to support more than baseline blackbox checks on the mgmt interfaces and I'm personally open to exploring something like the redfish exporter. But I think this would be a medium to large sized project since AIUI it will involve a decent sized chunk of setup effort on the hardware itself, and offhand I'm also not sure the percentage of hw in the fleet that will support the approach today. I'm also assuming we would run into some hw vendor oddities/bugs along the way.

In T359879#9875881, @elukey wrote:

@herron let's double check, maybe we can drop the secondary rules and keep going with the "regular" ones?

Example of the fix: https://grafana.wikimedia.org/d/slo-Lift_Wing_Revert_Risk_LA/lift-wing-revert-risk-la-slo-s?orgId=1&from=2024-03-01%2000:00:00&to=2024-05-31%2023:59:59

done

Group membership has been provisioned, thanks!

Hi @odimitrijevic @Milimetric @WDoranWMF @Ahoelzl could one of you please approve this request for analytics-privatedata-users? Thanks in advance!

This was completed yesterday (during stashbot outage, this task unfortunately missed the !log)

The patch to provision this access has been merged, and will be fully propagated within the next 30 minutes. I'll transition this to resolved now, please reopen if any followup is needed. Thanks!

The patch to provision this access has been merged and will fully propagate within the next 30 minutes. I'll transition this to resolved now, please re-open if any followup is needed. Thanks!

Hi @Soda could you please coordinate obtaining a comment of support on this task from a sponsor as outlined in https://wikitech.wikimedia.org/wiki/Volunteer_NDA? Thanks!

Closing as this has been stalled for weeks. Please re-open if/when ready to proceed. Thanks!

The patch to provision this access has now been merged and will be fully deployed within the next 30 minutes. I'll transition this task to resolved now, please re-open if any followup is needed. Thanks!

The patch to provision this access has been merged, and will fully propagate within 30 minutes. I'll go ahead and transition this task to resolved now, please re-open if any followup is needed. Thanks!

In T365832#9830059, @elappen-WMF wrote:

Approving access from my end.

Hi @JayCano, assigning to you for approval. Thanks!

(SSH key verification email sent)

Resolving as the access looks to have been provisioned, please reopen if any followup is needed. Thanks!

The patch to provision this access has been merged and will be propagated fully within the next 30 minutes. I'll transition this to resolved now, please reopen if any followup is needed. Thanks!

Hi @Ifrahkhanyaree_WMDE I see the SSH key in the description is in use already. Could you please generate a fresh ssh key for production use and update the task description with it? Details/rationale for this are outlined in https://wikitech.wikimedia.org/wiki/SRE/Production_access#Generating_your_SSH_key

In T360895#9875773, @Jhancock.wm wrote:

I got the ram out of the servers. I can do the addition tomorrow morning at 8am CDT (1600 UTC) or another time that works for you.

(from the task description)
pyrra (includes slo/slos)
I believe we could point these to thanos-query.discovery.wmnet right away; what do you think @herron ?

Deployed an SLO change to Pyrra just now and looking much better -- No alerts, and thanos rule was automatically reloaded shortly after pyrra filesystem 😎

In T364645#9793355, @fgiunchedi wrote:

Thank you, indeed that's probably that issue I was vaguely remembering.

To clarify I'm referring to --prometheus-url http://localhost:17902/rule/ to pyrra filesystem, not pyrra api; AFAICS the prometheus client in filesystem is used only for reload so it should work as expected. The reload will work on titan hosts that do currently run thanos-rule, and will fail on hosts that don't run it which is benign

In T364645#9789322, @fgiunchedi wrote:

A bit of a different route, though I can't remember if we have tried telling pyrra filesystem about "prometheus" being thanos-rule on localhost? i.e. --prometheus-url http://localhost:17902/rule/ ? In other words let pyrra filesystem effectively do the reload

This started happening because I added a grouping workaround in the parent task where essentially the grouping is done by puppet instead of pyrra itself. It results in pyrra generating more output config files, e.g. now slo-$site.yaml vs what used to be slo.yaml

Reviewed fs utilization on codfw/eqiad prom hosts and grew their k8s and ops filesystems targeting ~85% free space each

Still seeing two spaces after the status e.g. FIRING: although not seeing a clear cause for that

Overall this dashboard is meant to show graphite utilization for the whole installation, so I think the thing to do is add filters to drill down as needed.

Added two panels at the bottom of the dashboard to display count over time details using the time picker

FWIW we recently added disk capacity to these hosts with about ~1T free in the VG. I've made a note also to discuss/plan next week how best to allocate the additional space for the long-term with the team. In the mean time should it fire again it is safe to grow the LV again, though hopefully it wont be necessary.

fwiw this look quite similar to the diff from the experimental patch where the global cert name was changed to a discovery.wmnet domain:

Thanks! Looks good!

Prometheus1005 is down and depooled, any time works!

Reopening -- today we experienced a memory issue on prometheus1005 which presumably relates to this maintenance. Could we arrange to swap the faulty DIMM outlined in T362990? Thanks in advance!

FWIW I think the current alert text makes sense based on the premise that all alert recipients will/should know about how alerting system internals are structured.

While considering this I'd also like to propose moving the (alert name) to the end of message at the same time. For example:

Hey @VRiley-WMF, I'll help out with this one for the o11y side.

With T352756 T359879 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1017873 in mind I think It'd be worth spending some time here to work out a strategy for bringing backfilled metrics into production.

I think we're in good shape here, please reopen if anything else is needed

In T359879#9669840, @elukey wrote:

@herron something really strange: https://w.wiki/9bMW

I compared the recording rule with the actual metric, trying to aggregate with the same labels, and the results of the recording rule are strange. I see both eqiad and codfw traffic reported, while the original metric shows mostly eqiad traffic. Anything that comes up to mind on Thanos that could cause this?

SSD and RAM upgrades have been installed thanks @Jhancock.wm!