In T351927#9994610, @MatthewVernon wrote:

Hi @fgiunchedi and sorry to bug you again (again) about this, but we have a bunch of alerts about thanos backend disk space again.

[relatedly, we're trying to expedite the two new backend servers to get them in this quarter]

In T369825#9994586, @Papaul wrote:

No problem just give me a heads up on when y'all want to do the changes.

I captured otelcoll traffic on mw2310 (looks like it is always and only otel-coll running on this host that causes problems) in the hopes to understand what's the problematic trace. I've left /root/otelcoll-T370043.cap in place in case it is of interest; I could decode the http2 traffic with wireshark (select traffic with source port 4317 then "decode as" http2) though I couldn't get wireshark to decode the grpc traffic.

I'm investigating this again today, and yes there's a single otel-coll pod in codfw that is problematic: https://w.wiki/AhDr

I'm tentatively resolving this since even though the topic isn't perfectly balanced I'm calling it good enough™. We can revisit as needed

Thank you all for the clarification, I'm glad we're able to do the move without re-addressing after all!

In T369825#9990558, @VRiley-WMF wrote:

Hey @fgiunchedi Just wanted to verify, since we would have to physically move this server into another rack (and in turn, have to change the IP) this activity is no longer needed, correct? If so, I will close this ticket. Thanks!

So I looked where the probes come from, and they are part of the generic "probe mgmt network hosts for ssh" and data comes from netbox, specifically these bits in modules/profile/manifests/prometheus/ops.pp:

I'm removing grafana since turns out this issue is unrelated

I have reached out to upstream at https://github.com/redpanda-data/connect/issues/2705 to make them aware, in case they have seen this before. I'm not overly concerned just yet in the sense that IIRC this is the first time we've seen such a failure and it could be a unlucky coincidence

In T369256#9989598, @cmooney wrote:

Though there didn't seem to be a problem afterwards, the timing makes me think of T365997: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad

Yeah the times in those logs for connection failures to kafka-jumbo1014 exactly match when it was offline due to the switch upgrade. So probably I'm complicating things for everyone as usual :)

There's similar logs in benthos@webrequest_live on centrallog1002:

Also this in logs around the time consuming stopped:

In T369445#9989260, @fgiunchedi wrote:

In T369445#9989244, @Jelto wrote:

In T369445#9987520, @Arnoldokoth wrote:

I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.

I think this option solved the mail issue. As far as I can tell the mails no longer go to root@. The latest mails from VRTS Notifications (with the same error) were sent to our team mail address sre-service-ops-collab@ instead of root@.

root@ got a bunch of these emails e.g. just now

In T369445#9989244, @Jelto wrote:

In T369445#9987520, @Arnoldokoth wrote:

I think I found the setting that affects this. It's configured on the VRTS dashboard and was initially set to root@localhost. Updated it to our team's mailing list. Let's see where the next one goes.

I think this option solved the mail issue. As far as I can tell the mails no longer go to root@. The latest mails from VRTS Notifications (with the same error) were sent to our team mail address sre-service-ops-collab@ instead of root@.

This is done! We went with the procedure I suggested above, namely I took the host side configuration by logging back in via console to adjust the network configuration with the new interface name. @Jhancock.wm did the host physical move by connecting everything to its final location, and @Papaul did the netbox + network part

Perhaps unsurprisingly, the problem is back only on a different pod. Which in my mind excludes a otel-coll problem and points to a client sending troublesome spans

A very low hanging fruit to make progress on this task is the following prometheus-based checks:

Thank you @Papaul that is quite helpful!

In T315866#9984611, @fnegri wrote:

Adding a note to mention that currently Icinga alerts related to clouddb* hosts are getting tagged with team=wmcs when they are forwarded from icinga to alertmanager. I tried to figure out where that tagging happens but I haven't found it. We should aim to maintain that tagging when the alerts are migrated to alertmanager.

In T352756#9965404, @gerritbot wrote:

Change #1052784 merged by jenkins-bot:

[operations/alerts@master] istio_sli_avail: alert if metric goes absent

https://gerrit.wikimedia.org/r/1052784

I'm tentatively resolving this since the increase in partitions in T369256 has helped with increased concurrency

I'm happy to report that increasing partition count and restarting benthos mw sampler did the trick, lag was reduced even before the restart and completely gone after restart:

This is ongoing, to exclude a otel-coll problem I've deleted main-opentelemetry-collector-agent-msvrc pod in wikikube codfw. Let's see if it happens again

In T369826#9982422, @Jhancock.wm wrote:

We won't need to move racks. But because of the way the switches are, we can't reuse the same port on the switch. we'll be moving to a different set of 4. Are you going to reimage the server?

In T369826#9982167, @Jhancock.wm wrote:

Got the card back @fgiunchedi. I'm free to swap it anytime on Tuesday or Thursday between 8am and 4pm CDT

Thank you @LSobanski ! I'll be reaching out to the individual service owners

Today I've done extensive tests and tweaking of benthos@webrequest_live settings, namely:

Thank you @Jhancock.wm that's great! Please LMK a day and time of next week that would work for you

In T369825#9974444, @VRiley-WMF wrote:

@wiki_willy Yes, I was able to locate one. @fgiunchedi is there an estimated time and date for us to bring the server down and install the it?

A bit of context: benthos@webrequest_live normally runs on centrallog1002 and centrallog2002 consuming the webrequest firehose from jumbo-eqiad. While at steady state this is more or less fine, on sudden spikes of messages codfw consumer struggles with clearing its backlog / lag.

I've taken a look at this as well for the benthos bits, and there was significant kafka lag for the benthos consumer group at the time, which would explain the delay in data

Thank you @EUwandu-WMF ! Everything checks out, you are now part of wmf group and therefore have access. I'm tentatively resolving, though please reopen as needed

Patch is merged and I've added soda to nda ldap group, tentatively resolving though please reopen if needed

Calling this done, Pontoon now supports Puppet 7 (puppetserver) and I've updated the wikitech documentation

This is done (the monitoring project is no longer a thing)

No worries at all and totally fair @Aklapper !

Definitely +1 to use recording rules for histograms, since they are broken anyways already. I think also switching the istio dashboard to thanos might help, as there is caching within thanos (the stateless part, thanos-query-frontend and thanos-query).

In T369348#9964576, @Aklapper wrote:

I'll bite per https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#WMF_Group

@Lferreira you are now part of the wmf ldap group, I'm optimistically resolving the task, though please reopen if sth is amiss!

You should be able to access the dashboards in ~30 min from now, please confirm that is the case.

In T369500#9964364, @fgiunchedi wrote:

In T369500#9963066, @EUwandu-WMF wrote:

Hello @fgiunchedi , Please can you check again to see if it works now? Here is the screenshot of my sign-in with Uniquemia on Wikitech as well if it is helpful{F56297576}

Thank you, that helped realize I made a mistake in my search! I'll do the next steps to get you access to superset

In T369500#9963066, @EUwandu-WMF wrote:

Hello @fgiunchedi , Please can you check again to see if it works now? Here is the screenshot of my sign-in with Uniquemia on Wikitech as well if it is helpful{F56297576}

Hello @XiaoXiao-WMF; I've added you to wmf ldap group. I'm tentatively resolving the task though please reopen if sth is amiss

Hello @EUwandu-WMF, I couldn't find the uniquemia account on wikitech, or at least one with euwandu-ctr@wikimedia.org as its email, what wikitech account should we be using? thank you!

There is actually a mwlog02 bullseye instance which is already active, I've proceeded to delete mwlog01

Got it now, thank you @Marostegui. My recommendation in this case is to use mtail to get metrics out of logs; where such parsing should happen we can decide. My recommendation is to have mtail on db hosts that tails log files, this way you don't depend on: rsyslog delivering to centrallog, centrallog working, etc. There are some examples in puppet already for roles that use mtail (e.g. hieradata/role/common/mail/mx.yaml) which should be a good starting point, we (o11y) are of course happy to help too.

I think I'm missing context here, would you mind expanding on what are you trying to achieve?

I suspect this is related to T369256: Kafka lag for benthos-mw-accesslog-sampler and mediawiki.httpd.accesslog topic; we'll know more once that task is done