Apache Kafka: Data Streaming
Topics, partitions, producers/consumers, Kafka Connect and Streams. Build resilient real-time data pipelines.
- Understand Kafka's architecture
- Produce and consume messages
- Design resilient topics
- Integrate with Connect and Streams
- Decouple two systems that called each other directly, with no more fragile synchronous dependency
- Absorb load spikes without losing data: Kafka keeps messages for as long as needed
- Build a real-time data pipeline (ingestion → transformation → delivery) from end to end
- Replay the event history to debug or to feed a new consumer
The programme
Day 1 - Architecture & fundamentals
- Brokers, topics, partitions, offsets and replication
- Producers: acks, batching, compression and idempotence
- Consumers and consumer groups: assignment and rebalancing
- Kafka CLI and basic administration
- Workshop: multi-broker Kafka cluster, producing and consuming messages
Day 2 - Reliability & performance
- Delivery guarantees: at-most-once, at-least-once, exactly-once
- Retention, compaction and topic management
- Tuning producers and consumers (throughput vs latency)
- Security: TLS, SASL, ACLs
- Workshop: setting up high availability and failure testing
Day 3 - Kafka Connect, Streams & ksqlDB
- Kafka Connect: source and sink connectors (JDBC, S3, Elasticsearch)
- Schema Registry and Avro/Protobuf serialisation
- Kafka Streams: topologies, KTable, time windows
- ksqlDB: SQL queries on real-time streams
- Final workshop: complete pipeline - ingestion → transformation → sink with ksqlDB
Who is this for
Data engineers, back-end developers and architects.
Prerequisites
- Programming basics
- Notions of distributed systems (an asset)
Format
Frequently asked questions
Is the course eligible for OPCO or CPF funding?
No. Sphinx-Digital is not Qualiopi certified and is not listed on Mon Compte Formation. The course is invoiced directly to the company or the individual, with no external funding.
What is the difference between the public (inter-company) and in-company formats?
In the public format, you join a session with participants from other companies (3 to 10 people). In-company, we run a session dedicated to your team, on your site or remotely, with content adapted to your context. In-company pricing is available on quote.
Remote or on-site: what are the practical differences?
Content and labs are identical. Remotely, labs run on cloud environments accessible from a browser, with nothing to install. On-site, you provide the workstations; labs can run in the cloud or locally depending on your infrastructure.
How many participants per session?
3 to 10 participants at most. This deliberate limit ensures everyone can ask questions, adapt the exercises to their context and get a detailed answer.
What is the evening format?
The evening format spreads the same course over 4 sessions of 3.5 hours (14 hours in total), in the evening from 6:30 pm to 10:00 pm. Same content, same labs, same price, so you can build skills without taking days off.
How does payment work?
Courses are invoiced on quote, by bank transfer. Payment in instalments is possible. We issue a training agreement for companies that want their training department to cover the cost.
Interested in this course?
Join the next session or request a tailored in-company session.
What participants say
Not sure this course fits your context?
A 20-minute call to take stock before you enrol, with no sales pitch.