Setting up Kafka topics with Docker Compose: spotify/kafka vs wurstmeister/kafka

Other Questions Spotify
1

I’m trying to set up Kafka topics for my Java Spring app using Docker Compose. I’ve tried two different Kafka images but I’m running into some issues.

First, I used wurstmeister/kafka. I got the topics running, but I can’t connect them to my app. Here’s what my docker-compose.yml looks like for that:

kafka:
  image: wurstmeister/kafka:0.10.2.0
  ports:
    - "9092:9092"
  environment:
    KAFKA_ADVERTISED_HOST_NAME: localhost
    KAFKA_CREATE_TOPICS: "example-topic1:1:1, example-topic2:1:1"
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - zookeeper

Then I tried spotify/kafka, but I can’t get it to create topics. I’ve set the TOPICS environment variable like this:

kafka:
  image: spotify/kafka
  ports:
    - "9092:9092"
    - "2181:2181"
  environment:
    TOPICS: example-topic

But it’s not working. I’ve also tried putting quotes around the topic name. No luck.

Any ideas on how to get either of these setups working? I need to create Kafka topics and connect them to my Java Spring app. Thanks!

2

I’ve encountered similar challenges with Kafka setup in Docker. For the wurstmeister/kafka image, ensure your Spring application.properties includes bootstrap-servers=localhost:9092. Also, verify that your app’s network can reach the Kafka container.

Regarding spotify/kafka, it’s worth noting that this image is no longer maintained. I’d recommend sticking with wurstmeister/kafka or considering confluentinc/cp-kafka, which offers robust features and documentation.

If you’re still facing issues, try exposing Kafka on your host network:

kafka:
  network_mode: host
  environment:
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://your_host_ip:9092

Replace your_host_ip with your machine’s actual IP. This approach often resolves connectivity problems between your app and Kafka.

3

hey mate, i’ve used spotify/kafka before and had similar issues. try setting KAFKA_CREATE_TOPICS instead of TOPICS. also, make sure you’re using the right port in your spring app config (9092). if that doesn’t work, wurstmeister/kafka is usually more reliable in my experience. good luck!