Example invocation of the WIP implementation:

klaxon/victorops.py -v incidents_to_annotations -a 2024-07-01  -b 2024-08-01 \
  | xargs -d$'\n' -I@ curl -H "Authorization: Bearer $GRAFANA_SVC_ACCT_TOKEN" -H 'content-type: application/json' --data '@' https://grafana.wikimedia.org/api/annotations -s

Just FTR -- we didn't actually saturate anything at all on the Kafka hosts. The hottest that any NIC was was about 70% of line rate, which is warm but not hot. No drops or errors on the switch ports, nic-saturation-exporter metrics show that we weren't micro-bursting above 800mbit/s except for a one-minute interval (and nic-sat says were below 900mbit/s during then). Disk i/o queues were fine, all the other NICs were fine, and it wasn't some goofy thing where we were saturating core 0 or anything on kafka-main2006 either (link).

In T372507#10088733, @Scott_French wrote:

In puppet, this comment [2] would suggest it's only needed for fundraising use cases. That's a little surprising, as we wouldn't expect that to be used in a fundraising context (although I'm told there are cases where "fundraising" colloquially can refer to specific functionality in, e.g., CentralNotice).

In T372943#10083764, @Marostegui wrote:

In T372943#10081783, @CDanis wrote:

Concrete proposal: monitor client connections as a load indicator for masters

I really like this idea, we'd need help from IF though but I am happy to help as much as I can. I don't think it will have a huge impact on the performance, but we can start trying them on misc masters. Ideally we should have this tooling/metric everywhere, not just masters. However, the amount of writes compare with reads is way lower, so we'd need to measure the impact on reads as well.
If we prefer to have this only on masters for performance reasons, we should make this controlled by puppet and the master role, which we assign to the hiera file of each master, so it gets enabled/disable when we promote/demote a master.
We can of course explore the possibility to enable it for reads too.

@Jelto and @fgiunchedi handled a repeat Jio set-top box hotlink issue between 0600-0730 UTC today. The new requestctl support for cache hits in Varnish (T317794) was successfully exercised by Jelto to stop answering the requests, which is awesome.

@Ottomata wrote:

If eventgate cannot produce to Kafka, it is broken. In this case, Kafka overloaded was maybe the problem, but in other cases, it could be misconfigured pods. In that case, it would be better to fail a deployment and rollback.

ES has a hardcoded maximum of 1000 top-level fields per index, after which point it will refuse writes that add any more fields, so that's one argument to not enable the flag that puts all tags as fields .... but I could still be convinced. I guess you could just drop the index for that day and start over and it would be fine ... yeah. Yeah actually let's just do all tags as fields? I don't think there's any other downside.

In T372411#10077892, @fgiunchedi wrote:

Very interesting, yes definitely we should be experimenting with tags-as-fields. I skimmed the upstream post and at this time it isn't clear to me how the transition looks like. Or said otherwise whether jaeger UI transparently supports reading from both "formats"

In T370304#10063330, @jeremyb-phone wrote:

In T370304#10063127, @CDanis wrote:

can anyone besides you see this file?

@Ladsgroup @Marostegui I have a mix of concrete proposals and naive questions for you re: MariaDB observability :)

In T344171#10080323, @JMeybohm wrote:

For the record: The CoreDNS Pods are, like all other Pods, reachable by their Pod IP. So going through a debug Pod to resolve the PTRs is not really necessary.

I think in light of T370304 we need to prioritize this -- and a proper version, not just creating static placeholder records.

Nice! This is very cool.

In T370304#10073845, @Ladsgroup wrote:

Delete is much much more expensive than edits (at least currently due to T20493) it has to remove all the rows from revision table and insert them in archive table, we will eventually fix this and it would be much easier to allow batching and so on but for now we even had to move in the opposite direction (Nuke extension switched to actually queue jobs that would delete each page).

With a lot of encouragement I was able to get a long-term graph of that row locks metric in s4:

In T320556#10063763, @pmiazga wrote:

There is a quick way already - Telemetry::getRequestHeaders().

Oh, and here's the summarized perf report output from one of those perfs:

Artifacts I collected:

Internal incident doc: https://docs.google.com/document/d/1lscZB565H5z610ECTpit0lzke-rS3au0Cokn-MAy9xw/edit

I have a very quick first draft at https://wikitech.wikimedia.org/wiki/Distributed_tracing/Propagating_tracing_context

Thanks to @JMeybohm for giving us a good head start on this:

I pushed a branch (wmf-v1.6.5) to our gitlab cfssl repo with the 1.6.5 version (current upstream) plus our patches. I've also updated the debian patches with gbp

BTW I opened a new task T371842: Come up with a roadmap for supporting tracing for batch- or message-driven systems with a bunch of pointers for what I think is the current state of the OTel/Jaeger world on this.

In T368389#10038133, @Pablo wrote:

• I also think that silent-drop occurring during attack time windows might be a good signal... @CDanis, could you share with me an expanded version of the silentdrop dataset covering the entire month of July to see if these bots triggered any silentdrop action?

I ran otelcol with 1Gi RAM in codfw for half an hour.

Oh, and, silent-drop occurring during attack time windows should still make for a good signal, I think.

No, this is not expected behavior. I'm guessing this is external Mediawiki instances using our "Mathoid as a service" . I also see some requests there for "Instant Commons".

There's not a finished plan, no, but I'd be very happy to try some experiments with you next Q perhaps?

Thanks so much!

That's really odd -- I see the collector starting itself back up before the container gets restarted ... maybe externally?

In T371144#10026447, @Joe wrote:

In T370739#10024506, @akosiaris wrote:

Rate limiting is broken in service-runner for a long time now. See T200374#9558666. As far as the worker pool goes, we 've conducted some experiments in that past. It does have some benefits reliability wise (faster restarts in case of failures), but otherwise, in a Kubernetes environment, it's more of a nice to have than a requirement.

Yep, Logstash presently, although it would be nice if we had them in Hive some day as well :)

A way to limit concurrency from specific IPs, only for requests corresponding to a specific pattern? So limit concurrency for a user over the threshold only if they're hitting url X with user-agent Y?

In T370739#10019839, @Catrope wrote:

@akosiaris I'm trying to figure out how we should proceed based on your comment. Should we develop a service based on (an up-to-date version of) service-runner, and maybe look at service-template-node loosely for inspiration but not copy all of it verbatim, and then have that reviewed by serviceops + security prior to deployment? We probably don't need most of service-template-node's dependencies anyway, I think we probably just need service-runner, express and maybe body-parser. The other dependencies are either outdated (e.g. bluebird), irrelevant (e.g. domino, we don't need to parse HTML) or overkill for our use case (e.g. swagger-router, we expect there to only be one endpoint).

In T320556#9216907, @Ottomata wrote:

Would also like some guidance for how to pass on this info in emitted data events. We already produce the request_id, but perhaps we can make a more standardized tracing schema fragment for this.

@Jelto @Dzahn FYI ^ about nft and requestctl support without Ferm.

If you make the traces service also export to the basic logging service, then (post-processing) all traces will get dumped out pretty verbosely into the logs.