How to ensure Kafka message schema validation using Schema Registry?

Other Questions API
1

I’m trying to set up Kafka message schema validation with Schema Registry. I’ve got Kafka and Schema Registry running in Docker containers. My schemas are registered using the Schema Registry UI.

However, it seems like the schemas aren’t being validated properly. I keep getting a ‘Schema not found’ error most of the time. When this error doesn’t show up, the messages just get sent to another topic without any validation.

Here’s a snippet of my KafkaEventProducer class:

public class KafkaEventProducer {
    private KafkaProducer<String, GenericRecord> producer;

    public KafkaEventProducer(String kafkaServers, String schemaRegistryUrl) {
        Properties config = new Properties();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServers);
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
        config.put(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        config.put(AbstractKafkaAvroSerDeConfig.AUTO_REGISTER_SCHEMAS, false);

        producer = new KafkaProducer<>(config);
    }

    public void sendMessage(String topic, GenericRecord event) {
        String key = UUID.randomUUID().toString();
        ProducerRecord<String, GenericRecord> record = new ProducerRecord<>(topic, key, event);
        producer.send(record);
    }
}

Am I missing any crucial configuration? How can I make sure the schema validation is working correctly?

2

I’ve encountered similar issues with Kafka schema validation. From your code, it appears you’ve disabled auto-registration of schemas, which is correct if you’re aiming for controlled schema management. However, ensure that your Schema Registry URL is accurate and that the registry is accessible from your Kafka broker. It might also help to verify that the GenericRecord fully aligns with the registered schema. Enabling detailed logging can reveal any subtle mismatches or network issues that might be causing the ‘Schema not found’ error.

3

I’ve dealt with Schema Registry validation issues before, and it can be tricky. One thing that helped me was adding error handling and retries to my producer. Sometimes network hiccups can cause temporary ‘Schema not found’ errors.

Also, make sure your GenericRecord is using the latest schema version. If you’ve updated the schema recently, there might be a mismatch. I found it helpful to explicitly specify the schema version when creating the GenericRecord.

Another thing to check is your topic configuration. Ensure that the topic you’re producing to is set up to use Schema Registry validation. Sometimes, messages can be sent to topics without validation if this isn’t configured correctly.

Lastly, if you’re still having issues, try temporarily enabling schema auto-registration (set AUTO_REGISTER_SCHEMAS to true) just to test if the producer can successfully communicate with the Schema Registry. This helped me isolate whether the problem was with schema matching or connectivity.

4

hey dude, i had same prob. check ur schema registry url, make sure its reachable from kafka. also, double check ur genericrecord matches the schema u registered. if that dont work, try enabling auto register schemas (set it to true) and see if that helps. good luck!