IMHO zookeeper is probably the eadiest place to get this. There are many
zookeeper libraries out there for the most popular languages.

If you perform a GetChildren query on /consumers you'll get the list of
consumer groups. A GetChildren in /consumers/[group]/ids will give you the
consumer instances.

On Tue, 1 Dec 2015 at 20:54, Ivan Shumkov notifications@github.com wrote:

—
Reply to this email directly or view it on GitHub
#131.

Ah, sorry… ZK won't give you that. 😊

Yep :(. Endpoint GET /consumers would be great.

+1 for this endpoint

Yeah, makes sense to me. Maybe we can add:

GET /consumers
GET /consumers/{groupId}/instances

Might also be helpful to have a way to query current offsets:

GET /consumers/{groupId}/instances/{instanceId}/offsets

@hachikuji Do we need to be careful about how we group these too under the resources? The current set of APIs under /consumers only support operations for consumers on the current instances, but that's because they are all mutating operations (e.g. create, delete, and reading from a topic is as well despite being a GET).

In contrast, I might expect GET /consumers to list all consumers in the cluster, i.e. be able to list them similar to the ConsumerGroupCommand.

I'm not sure if mixing both under /consumers is going to be confusing, if we should add another top-level resource, or maybe we can find a completely different home for the API endpoint for a cluster-wide listing of consumer groups.

@ewencp I'm not sure I see any problem with having GET /consumers return the full list of groups in the cluster, but it's definitely trickier if we wanted to implement GET /consumers/{groupId}/instances. A user might expect it to return the full list of instances in the group or just the ones hosted on the proxy receiving the request. Either option would probably be confusing for some.

Now that I'm thinking about it, there seems to be two use cases here. In one case, the user wants to see the full list of instances in the group (e.g. to view the group's assignments and lag). In the other, the user wants to see only the instances hosted on the proxy in order to monitor the proxy itself (this seems to be @shumkov's use case).

Given these cases, I would probably favor having GET /consumers only return groups which have active instances on the proxy. Similarly for the instances call (if we actually want to support it). Then maybe there could be a separate path (e.g. under /cluster) to query the full list of groups.

@hachikuji Yeah, that's exactly what I was getting at. In retrospect, I probably would have preferred the cluster-wide version to be /consumers and the instances on the proxy to be under something like /proxy/consumers since that would be more consistent with the other top-level resources that are cluster-wide. But I guess that's what I get for lack of foresight...

Was this ever implemented? I assume not as it's not in the docs. I think it would be useful, especially when used with generated consumer names.

@joewood If you're referring to the ability to list all the consumers in the cluster, it hasn't been implemented yet. It'd probably fall under the admin API. There's a new Java AdminClient API that is coming soon in Apache Kafka which would probably be the best way to get this info, although I'm not sure when info about consumer groups will be exposed in it (it is part of KIP-4 which is a large, multi-stage effort).

Hi, folks.
I've made pull request PR.
It is not return all consumer infos from cluster, but show consumers are restracted by consumer group and topic wich the group reads. Any comments are welcome.

you can check the number of consumers and some information about consumers.
like this : kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-group

Information :
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
my-topic 2 0 0 0 consumer-3-75a0020f-a76f-492a-bb6e-cbd70faa8593 /172.30.1.58 consumer-3
my-topic 3 0 0 0 consumer-4-b7992669-a2b6-4355-ab8d-f869183e6b8e /172.30.1.58 consumer-4
my-topic 1 0 0 0 consumer-2-2292686a-a001-4c09-9224-ee9697b4932d /172.30.1.58 consumer-2

Does anybody know how to use those kafka sh scripts with the remote bootstrap server?