As part of T369256 I'm rebalancing mediawiki.httpd.accesslog topic on kafka-logging cluster (both eqiad and codfw, though focusing on eqiad since that's where logs are written all the time, codfw is idle)
I wanted to test first topicmappr to equalize throughput, following the instructions here https://wikitech.wikimedia.org/wiki/Kafka/Administration#Rebalance_topic_partitions_to_equalize_broker_storage
kafka-logging1001:~$ prometheus-metricsfetcher time="2024-07-16T09:13:21.242Z" level=info msg="Getting broker storage stats from Prometheus" time="2024-07-16T09:13:21.243Z" level=info msg="Connected to [2620:0:861:101:10:64:0:207]:2181" logger=zk time="2024-07-16T09:13:21.245Z" level=info msg="authenticated: id=5983579338673291539, timeout=20000" logger=zk time="2024-07-16T09:13:21.245Z" level=info msg="re-submitting `0` credentials after reconnect" logger=zk time="2024-07-16T09:13:21.253Z" level=info msg="Broker ID Map: map[kafka-logging1001:9100:1001 kafka-logging1002:9100:1002 kafka-logging1003:9100:1006 kafka-logging1004:9100:1004 kafka-logging1005:9100:1005]" time="2024-07-16T09:13:21.253Z" level=warning msg="Returned metrics not equal to broker ID override map" time="2024-07-16T09:13:21.253Z" level=info msg="Getting partition sizes from Prometheus" time="2024-07-16T09:13:21.299Z" level=info msg="writing data to /kafka/logging-eqiad/topicmappr/partitionmeta" time="2024-07-16T09:13:21.305Z" level=info msg="writing data to /kafka/logging-eqiad/topicmappr/brokermetrics" time="2024-07-16T09:13:21.309Z" level=info msg="recv loop terminated: err=EOF" logger=zk kafka-logging1001:~$ topicmappr rebalance --brokers -2 --topics mediawiki.httpd.accesslog --out-file cluster-rebalance Error unmarshalling broker metrics: json: cannot unmarshal number logging-hd1001:9100 into Go value of type int
And I'm not quite sure if we've seen this before or where to go next. I'm also confused by logging-hd1001:9100 popping up since that is a opensearch storage host. Maybe @brouberol you have seen this before and/or can assist? thank you !