Author Message
RajatVerma
Joined: Mar 5, 2014
Messages: 101
Offline
Hello Experts,
I am developing a solution as a Kafka consumer for getting evernts from POM kafka server for any call attempt. I have developed the consumer application and successfully get the events.

Now requirement is to deploy 3 instances of Kafka cosumer clients with load balancing and failover between them. I need to understand where how can make changes in Kafka producer (POM component) to support this.

FYI, this is standard Kafka functionality. I have developed a kafka producer and client of my own and deployed in my local machine. Here in producer I define multiple partistions (> 1) for the topic which I want consumers to subscriber. In consumers I define the topic and keep all consumers in same gorupID. Kafka events are equally distributed between these consumers and if one goes down, new events are distributed to different consumers.

Thanks,
Rajat Verma
MikeMalcorINI [Avatar]

Joined: Jan 14, 2015
Messages: 72
Offline
I'm not the Kafka SME but just wanted to share the Kafka HA guide in case there is something in there that is useful.
https://downloads.avaya.com/css/P8/documents/101063367
RajatVerma
Joined: Mar 5, 2014
Messages: 101
Offline
Hello Mike,
Thanks for this as this was also one of detail which I was looking for.

However, my current challenge is how I can make my consumer (client) deployed in multiple instances supporting failover, and Kafka events should be distributed to each of them equally. Currently shared document mostly specifies details for Kafka Server HA, but configurations related to supporting client side multiple instances.

Just to add as I understand there should be multiple partitions for a topic only then multiple clients will work to get events. But read somewhere that Kafka topic supports only one partition. There should be some way of doing this.
Go to:   
Mobile view