Filing from IRC
09:04 <godog> herron: looks like thanos-rule on titan2001 never got reloaded 09:10 <elukey> sigh 09:10 <elukey> Maybe we could add a notify in puppet so when pyrra reloads it will signal thanos-rule as well? 09:23 <elukey> reloaded thanos-rule on 2001 09:26 <godog> yes that's what needs to happen, for some reason I thought that was the case already
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T290924 Tooling for end-of-quarter SLO reporting | |||
| Open | None | T302995 Transition to Pyrra for SLO Visualization and Management | |||
| Resolved | herron | T353691 Reload thanos-rule on new pyrra rules deployed | |||
| Open | None | T353716 Pyrra: cleanup output-rules when config is removed |
In theory pyrra should do the right thing, but we're running into a couple issues in our current deployment:
Having puppet issue reloads of pyrra-filesystem and thanos-rule when puppet deploys a config change should help. Only concern would be the thanos-rule reload racing against pyrra generation of the output rules, but lets try it and see how it goes
Change 984220 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] pyrra: reload pyrra-filesystem and thanos-rule on cfg change
Change 984220 merged by Herron:
[operations/puppet@production] pyrra: reload pyrra-filesystem and thanos-rule on cfg change
Change 990126 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] thanos::rule: set reload service to stopped
Change 990126 merged by Herron:
[operations/puppet@production] thanos::rule: set reload service to stopped
Can confirm we're good in terms of puppet runs not changing state at every run
titan1001:~$ pat Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for titan1001.eqiad.wmnet Info: Applying configuration version '(996c73dc41) Taavi Väänänen - P:toolforge::mailrelay: reject mail not using Toolforge domains' Notice: Applied catalog in 24.79 seconds