We've started getting kafka lag alerts for benthos-mw-accesslog-sampler consumer group (this is the benthos instance that runs on logstash hosts, consumes mediawiki.httpd.accesslog and writes a sample to mediawiki.httpd.accesslog-sampled.
There's indeed a background lag of ~4k messages for a bunch of partitions
Given the following:
- the problems we've had with the same topic in T367076 (cc @kamila)
- benthos webrequest live (webrequest firehose) doesn't have lag for a different topic
- benthos webrequest live uses ~24 cores at peak (on centrallog hosts) to consume ~160k msg/s and sample them
- logstash hosts (where benthos accesslog sampler runs) are not starved for CPU nor memory
I've taken a look at mediawiki.httpd.accesslog topic, and indeed the topic has 6 partitions, though kafka-logging1001 is leader for 3 of them:
Topic:mediawiki.httpd.accesslog PartitionCount:6 ReplicationFactor:3 Configs:retention.ms=172800000
Topic: mediawiki.httpd.accesslog Partition: 0 Leader: 1001 Replicas: 1001,1002,1005 Isr: 1001,1005,1002
Topic: mediawiki.httpd.accesslog Partition: 1 Leader: 1004 Replicas: 1004,1001,1002 Isr: 1001,1002,1004
Topic: mediawiki.httpd.accesslog Partition: 2 Leader: 1001 Replicas: 1001,1002,1004 Isr: 1001,1002,1004
Topic: mediawiki.httpd.accesslog Partition: 3 Leader: 1006 Replicas: 1006,1002,1001 Isr: 1001,1006,1002
Topic: mediawiki.httpd.accesslog Partition: 4 Leader: 1001 Replicas: 1001,1006,1002 Isr: 1001,1006,1002
Topic: mediawiki.httpd.accesslog Partition: 5 Leader: 1005 Replicas: 1005,1002,1001 Isr: 1001,1002,1005Unsurprisingly 1001 gets 3x the traffic in msg/s of others (green line)
This topic is also by far the one with the most traffic on kafka-logging by far:
Therefore I think we should be increasing the number of partitions (to 12 just to get an even number), therefore increasing concurrency for consumers and thus performance. We should also be rebalancing the topic to ensure even broker distribution.