How to create Kafka topics in Docker Compose using different images?

Other Questions Spotify
1

I’m trying to set up Kafka topics for my Java Spring app using Docker Compose. I’ve tried two different Kafka images but I’m having trouble.

First, I used wurstmeister/kafka. The topics start up, but I can’t connect them to my app. Here’s what my docker-compose.yml looks like for that:

kafka:
  image: wurstmeister/kafka:0.10.2.0
  ports:
    - '9092:9092'
  environment:
    KAFKA_ADVERTISED_HOST_NAME: localhost
    KAFKA_CREATE_TOPICS: 'topic-a:1:1, topic-b:1:1'
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - zookeeper

Then I tried spotify/kafka, but I can’t get the topics to create. I’ve set the TOPICS environment variable like this:

kafka:
  image: spotify/kafka
  ports:
    - '9092:9092'
    - '2181:2181'
  hostname: kafka
  environment:
    TOPICS: sample-topic

But it’s not working. Am I missing something? How can I get Kafka topics running and connected to my Spring app using Docker Compose?

2

I’ve been down this road before, and I can tell you from experience that the Confluent Kafka image suggested by Dave_17Sketch is indeed a solid choice. However, if you’re set on using wurstmeister/kafka, there’s a trick I’ve found that might help.

Instead of using KAFKA_CREATE_TOPICS, try creating a separate service for topic creation. Here’s what I mean:

kafka-topics:
  image: wurstmeister/kafka:0.10.2.0
  command: >
    bash -c
      "sleep 10 &&
      kafka-topics --create --topic topic-a --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181 &&
      kafka-topics --create --topic topic-b --partitions 1 --replication-factor 1 --if-not-exists --zookeeper zookeeper:2181"
  depends_on:
    - kafka

This approach ensures that Kafka is fully up before attempting to create topics. As for connecting to your Spring app, double-check your application.properties. Make sure you’re using the correct bootstrap servers address (localhost:9092 if running on the same machine).

3

hey there, i’ve had some luck with the bitnami/kafka image. it’s pretty straightforward to use. here’s a quick example:

kafka:
  image: bitnami/kafka:latest
  ports:
    - '9092:9092'
  environment:
    - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
    - ALLOW_PLAINTEXT_LISTENER=yes
    - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT
    - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
    - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092

for topics, you can use the KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true environment variable. hope this helps!

4

I’ve been wrestling with Kafka setups in Docker Compose too, and I found a solution that’s been working well for me. Instead of using pre-built images, I’ve had success with a custom Dockerfile approach. Here’s what I do:

Create a Dockerfile for Kafka:

FROM openjdk:8-jre-alpine
RUN apk add --no-cache bash
ADD kafka_2.13-2.8.1.tgz /opt
ENV KAFKA_HOME /opt/kafka_2.13-2.8.1
COPY start-kafka.sh /usr/bin
CMD ["start-kafka.sh"]

Then in your docker-compose.yml:

kafka:
  build: .
  ports:
    - '9092:9092'
  environment:
    KAFKA_ADVERTISED_HOST_NAME: localhost
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

This gives you more control over the Kafka setup. For topic creation, I use a separate init container that runs kafka-topics.sh after Kafka is up. It’s a bit more work upfront, but it’s been rock solid for my Spring projects.

5

I’ve had success using the Confluent Platform’s cp-kafka image. It’s well-maintained and integrates smoothly with Spring Boot apps. Here’s a streamlined setup:

kafka:
  image: confluentinc/cp-kafka:latest
  ports:
    - '9092:9092'
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

For topic creation, I recommend using Kafka’s built-in auto-create feature. Set KAFKA_AUTO_CREATE_TOPICS_ENABLE=true in the environment. This approach has proven reliable in my projects, eliminating the need for separate topic creation steps.

6

I’ve encountered similar issues when setting up Kafka with Docker Compose. From my experience, the Confluent Kafka image tends to be more reliable and easier to configure. Here’s a snippet that’s worked well for me:

kafka:
  image: confluentinc/cp-kafka:latest
  ports:
    - '9092:9092'
  environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  depends_on:
    - zookeeper

For topic creation, I’d recommend using a separate service or init container to run the kafka-topics command after Kafka is up. This approach has been more consistent in my projects.