detect and fix kafka issues

src: Détectez et corrigez automatiquement les problèmes les plus fréquents avec Apache Kafka - 2023-04-26

Abstract

Common issues:

• topic name: how to make a mess
• nb partition: how to waste a precious resource
• retention: how to waste storage
• ack != all: how to loose data
• min.isr: how to loose data again
• linger.ms: how to be slow and penalize kafka
• compression.codec: how to waste storage and network
• consumer group names: how make bad magic appear
• poison pill: how to fail miserably
• invalid schema: how to fail miserably again
• latency: how to fail at kstream or ksqlDB at the worst time
• fetch.min.bytes: how to be a smart ass and breaking kafka
• duplicates: how to discover you’re not idempotent
• rebalance storm: how to discover you’re half-alive
• old version: how to play with fire
• over transacting: how to no understand transactions

Protocol to extend: https://kafka.apache.org/protocol. They present a way to enforce the way to communicate with Kafka, e.g. enforce the topic syntax, by using a proxy in front of Kafka.