Kafka concepts

  Kafka concepts every Data Engineer should know:

✅ What is Kafka?
✅ Producer & Consumer
✅ Topics
✅ Partitions
✅ Replication
✅ Consumer Groups
✅ Offsets
✅ Offset Commit
✅ Message Retention
✅ Delivery Semantics
✅ High Watermark
✅ Log Compaction
✅ Idempotent Producer
✅ Transactions
✅ Consumer Rebalancing
✅ ZooKeeper vs KRaft
✅ ACL (Access Control List)
✅ Kafka Connect
✅ Kafka Streams
✅ Topic Configuration
✅ Log Segments
✅ Mirror Maker
✅ Dead Letter Queue (DLQ)
✅ Idempotent Consumer
These are the concepts interviewers use to test whether you've worked with Kafka in real-world systems.
Here are a few questions you should be able to answer:
• Why do we need partitions?
• What happens if a broker crashes?
• How does Kafka prevent duplicate writes?
• What is the difference between offset and offset commit?
• How does log compaction differ from retention?
• When should you use DLQ?
• What triggers consumer rebalance?
• What is the High Watermark?
• How does Mirror Maker replicate data?
• What is the difference between Kafka Connect and Kafka Streams?
• How does Exactly-Once processing actually work?
Knowing the definitions is easy.
Understanding why these features exist is what separates beginners from experienced Data Engineers.
Bookmark this guide it covers the Kafka concepts you'll revisit throughout your Data Engineering journey.