User Details
- User Since
- Nov 5 2018, 2:54 PM (303 w, 6 d)
- Availability
- Available
- IRC Nick
- cdanis
- LDAP User
- CDanis
- MediaWiki User
- CDanis (WMF) [ Global Accounts ]
Fri, Aug 23
Example invocation of the WIP implementation:
klaxon/victorops.py -v incidents_to_annotations -a 2024-07-01 -b 2024-08-01 \ | xargs -d$'\n' -I@ curl -H "Authorization: Bearer $GRAFANA_SVC_ACCT_TOKEN" -H 'content-type: application/json' --data '@' https://grafana.wikimedia.org/api/annotations -s
Just FTR -- we didn't actually saturate anything at all on the Kafka hosts. The hottest that any NIC was was about 70% of line rate, which is warm but not hot. No drops or errors on the switch ports, nic-saturation-exporter metrics show that we weren't micro-bursting above 800mbit/s except for a one-minute interval (and nic-sat says were below 900mbit/s during then). Disk i/o queues were fine, all the other NICs were fine, and it wasn't some goofy thing where we were saturating core 0 or anything on kafka-main2006 either (link).
11:47:53 <vgutierrez> hmmm if those are megabytes my intention was to set it to 300 / 8 = 37.5 11:48:00 <cdanis> ahahaha 11:48:03 <cdanis> that ALSO sounds good to me 11:48:25 <vgutierrez> traffic shaping in the cp servers doesn't allow a single stream to go over 300mbps IIRC
@Jelto and @fgiunchedi handled a repeat Jio set-top box hotlink issue between 0600-0730 UTC today. The new requestctl support for cache hits in Varnish (T317794) was successfully exercised by Jelto to stop answering the requests, which is awesome.
@Ottomata wrote:
If eventgate cannot produce to Kafka, it is broken. In this case, Kafka overloaded was maybe the problem, but in other cases, it could be misconfigured pods. In that case, it would be better to fail a deployment and rollback.
Thu, Aug 22
ES has a hardcoded maximum of 1000 top-level fields per index, after which point it will refuse writes that add any more fields, so that's one argument to not enable the flag that puts all tags as fields .... but I could still be convinced. I guess you could just drop the index for that day and start over and it would be fine ... yeah. Yeah actually let's just do all tags as fields? I don't think there's any other downside.
Wed, Aug 21
@Ladsgroup @Marostegui I have a mix of concrete proposals and naive questions for you re: MariaDB observability :)
I think in light of T370304 we need to prioritize this -- and a proper version, not just creating static placeholder records.
Tue, Aug 20
Nice! This is very cool.
Mon, Aug 19
With a lot of encouragement I was able to get a long-term graph of that row locks metric in s4:
Thu, Aug 15
Wed, Aug 14
Tue, Aug 13
Oh, and here's the summarized perf report output from one of those perfs:
Artifacts I collected:
Internal incident doc: https://docs.google.com/document/d/1lscZB565H5z610ECTpit0lzke-rS3au0Cokn-MAy9xw/edit
I have a very quick first draft at https://wikitech.wikimedia.org/wiki/Distributed_tracing/Propagating_tracing_context
Thu, Aug 8
Thanks to @JMeybohm for giving us a good head start on this:
I pushed a branch (wmf-v1.6.5) to our gitlab cfssl repo with the 1.6.5 version (current upstream) plus our patches. I've also updated the debian patches with gbp
Tue, Aug 6
Mon, Aug 5
BTW I opened a new task T371842: Come up with a roadmap for supporting tracing for batch- or message-driven systems with a bunch of pointers for what I think is the current state of the OTel/Jaeger world on this.
Aug 2 2024
Aug 1 2024
I ran otelcol with 1Gi RAM in codfw for half an hour.
Oh, and, silent-drop occurring during attack time windows should still make for a good signal, I think.
No, this is not expected behavior. I'm guessing this is external Mediawiki instances using our "Mathoid as a service" . I also see some requests there for "Instant Commons".
Jul 31 2024
There's not a finished plan, no, but I'd be very happy to try some experiments with you next Q perhaps?
Thanks so much!
Jul 30 2024
That's really odd -- I see the collector starting itself back up before the container gets restarted ... maybe externally?
Jul 29 2024
Yep, Logstash presently, although it would be nice if we had them in Hive some day as well :)
A way to limit concurrency from specific IPs, only for requests corresponding to a specific pattern? So limit concurrency for a user over the threshold only if they're hitting url X with user-agent Y?
Jul 27 2024
Jul 26 2024
Jul 25 2024
If you make the traces service also export to the basic logging service, then (post-processing) all traces will get dumped out pretty verbosely into the logs.