Apache Flink

Failure-first · PyFlink · Kafka

Stateful Stream Processing with Apache Flink

Build GatewayMeter, an AI usage meter. Kill TaskManagers, stall watermarks, inject late data, and prove the invoice survives.

Start module 1 — free Explore the curriculum

12 modules · about 16 hoursFreeHands-on labs

Python · Kafka · Docker · no prior Flink. The labs run Flink and Kafka locally in containers.

A two-day, failure-first Apache Flink course for experienced engineers. You build GatewayMeter, a public and self-contained reference pipeline for AI usage metering, then defend its invoice through bad keys, late data, stalled partitions, TaskManager deaths, slow dependencies, upgrades, and hot tenants. The goal is not merely to make a job run; it is to prove where each correctness guarantee begins and ends.

Try the signature failure

One Kafka partition goes quiet. Every window stops.

Module 5 shows why a watermark is the minimum across its inputs, so one silent partition holds every event-time window hostage — and how idleness releases it.

One partition goes silent
Its watermark stops
The minimum holds every window
Idleness releases it

Stall a watermark yourself

Your Learning Path

Each module builds on the last. Open any published lesson or lab and continue at your own pace.

The Machine — JobManager, TaskManagers, slots, operators, parallelism — and what a job is when nothing is checkpointed

What runs where. A Flink job is a dataflow graph deployed as parallel tasks across TaskManagers, coordinated by a JobManager. GatewayMeter reads the gateway's event stream and counts events per model. You kill a TaskManager and learn the first hard fact: without checkpoints there is nothing to recover from.

Free60 minReading materialHands-on lab (local)

Open lab

Streams, Keys, and Operators — Partitioning, keyBy, key groups, operator chains — the same request must always meet the same subtask

How records find their operator. keyBy is a routing decision that also decides where state can live. GatewayMeter assembles each request's lifecycle from its events; drop the keyBy and one request's start and completion land on different subtasks and no record is ever emitted.

Free60 minReading materialHands-on lab (local)

Open lab

Stateful Operators — Keyed state, state categories, TTL — the running total that outlives the record that changed it

State may be keyed, operator-local, or broadcast; this lab exercises keyed state directly. GatewayMeter keeps an open-request record per request. Requests that never complete leave state behind, so you measure the growth and defend a TTL backstop without confusing expiry with durability.

Free75 minReading materialHands-on lab (local)

Open lab

Event Time and Processing Time — Which clock a minute belongs to — and why the gateway's timestamp, not the wall clock, decides the bill

Two notions of time, one of them wrong for billing. GatewayMeter computes cost per tenant per minute; a 30-second retry delay makes processing-time windows bill the wrong minute. Event time bills the right one, at the price of deciding when a minute is over.

Free75 minReading materialHands-on lab (local)

Open lab

Watermarks — Bounded out-of-orderness, the minimum rule, idleness, alignment — who decides that a minute is over

A watermark says the system is willing to treat earlier timestamps as late; it does not make their arrival impossible. One silent Kafka partition can hold every window hostage. The lab diagnoses that case and repairs it with idleness; the reading contrasts the separate fast-versus-slow-input problem that watermark alignment addresses.

Free90 minReading materialHands-on lab (local)

Open lab

Windows and Timers — Tumbling windows, allowed lateness, side outputs, event-time timers — a late event can change an answer already sent

Windows are state plus a trigger plus a timer. GatewayMeter exercises a one-minute tenant-cost window, a late correction, a too-late side output, and abandoned-request detection with an event-time timer. The second value for one natural key is a correction, not automatically a recovery duplicate.

Free90 minReading materialHands-on lab (local)

Open lab

Checkpoints — Barriers, alignment, snapshots, recovery — the distributed snapshot problem, and Flink's answer to it

The module the course is built around. A checkpoint is a consistent cut across every operator's state and every source's position, drawn by barriers that flow with the records. You kill a TaskManager between checkpoints and watch state restore to checkpoint k and Kafka rewind to S_k: per-tenant totals neither lost nor double-counted inside Flink.

Free105 minReading materialHands-on lab (local)

Open lab

Exactly Once — Replayable source, checkpointed state, transactional or idempotent sink — what the phrase means and where it stops

Exactly-once state semantics are Flink's; end-to-end exactly-once needs the source and the sink to cooperate. The sink writes a cost record, the TaskManager dies before the checkpoint completes, the record is written again after recovery, and the billing consumer double-bills. Two fixes: a transactional Kafka sink with read_committed consumers, or an idempotent upsert keyed by tenant and window.

Free90 minReading materialHands-on lab (local)

Open lab

Backpressure — Network buffers, credit-based flow control, chaining, async I/O — what a slow downstream call does to everything upstream

Flink propagates slowness backward by design. GatewayMeter enriches records with a negotiated price; when each lookup takes two seconds, capacity one collapses throughput. You measure the propagation, estimate required concurrency, compare bounded async capacities, and inspect the checkpoint trade-off under load.

Free75 minReading materialHands-on lab (local)

Open lab

Scale and Upgrade — Savepoints, rescaling, max parallelism, operator UIDs, state schema evolution — changing the job without losing the state

A savepoint is a snapshot you own. On the lab cluster GatewayMeter restores from parallelism 4 to 8; 4 to 32 remains a production thought experiment. You meet the max-parallelism ceiling and missing-UID trap, then reason explicitly about the serializer and persisted bytes that make a state upgrade compatible.

Free75 minReading materialHands-on lab (local)

Open lab

Operating GatewayMeter in Production — Skew, hot keys, metrics, capacity — one tenant becomes 40% of traffic

Production is the job you cannot restart on a whim. One tenant becomes a hot key; adding parallelism cannot split that key. You diagnose the skew by subtask, use deterministic salting and replacement merge, and prove the invoice against the unsalted oracle. JobManager HA and disaggregated state remain advanced reading topics, not lab claims.

Free60 minReading materialHands-on lab (local)

Open lab

Capstone: GatewayMeter Under Fire — The whole job, a failure contract per operator, and a reproducible incident review

Derive GatewayMeter from its correctness requirements: write the failure contract for every operator, retain one seeded trace, inject a staged sequence of silence, TaskManager loss, slow pricing, late wake-up, and skew, then diff the materialized invoice against the offline oracle. Every guarantee must name an owner and an artifact.

Free120 minReading materialHands-on lab (local)

Open lab

Built for

Who this course helps

Engineers who have run a service that consumed Kafka, SQS or Kinesis and had to think about duplicates
Backend and data engineers who need to know where a streaming correctness guarantee begins and ends
Python practitioners new to Flink who would rather break a real job than read another API tour

What you leave with

A metering pipeline whose failure contract you can defend operator by operator

Explain what survives a TaskManager death, with and without checkpoints, and trace recovery from checkpoint k
Diagnose a stalled watermark and predict when a late event corrects an emitted window
Qualify every use of 'exactly-once' with its scope and build an end-to-end path with a transactional or idempotent sink

One reference pipeline runs through all twelve modules. Each lab breaks it on purpose, and every claim about state, time or exactly-once is checked against the running job.

About This Course

Flink's central idea is small: a computation over an unbounded stream is a set of parallel operators, each with local state partitioned by key, whose progress through time is measured by watermarks and whose state is made recoverable by consistent distributed snapshots. Everything else — windows, timers, backpressure, savepoints, and transactional sinks — follows from those mechanisms.

The one evolving system is GatewayMeter, an open reference pipeline for metering AI usage. It is simplified teaching code, inspired by the metering problems Meterry works on, and needs no Meterry account or service.

Every module adds one capability and injects one failure. The lab method is predict, break, observe, explain, repair, and prove. Day 1 builds the machine — tasks and slots, keys, state, time, watermarks, windows. Day 2 establishes the correctness boundary — checkpoints, end-to-end delivery, backpressure, upgrades, skew, and a capstone invoice review. Each lab page states whether its walkthrough has been verified end to end on the supported Docker Compose stack; a job returning to RUNNING is never accepted as the business oracle.

You do not finish this course by getting a Flink job to run. You finish by defending why its materialized invoice remains correct when the system fails.

Course design original to SciMigo, developed from a two-day learning path for stateful stream processing. Reading pages adapt the Apache Flink documentation, the Flink Kafka connector documentation and the Flink Agents documentation (Apache License 2.0, © The Apache Software Foundation); see THIRD_PARTY_NOTICES.md. Apache Flink, Flink and the Flink logo are trademarks of The Apache Software Foundation; this course is independent and not endorsed by the ASF or the Flink project.

Prerequisites

Comfortable Python; some familiarity with async/await
Have run a service that consumed a queue or a log (Kafka, SQS, Kinesis) and had to think about duplicates
Docker installed locally (labs run Flink and Kafka in containers); no prior Flink

What You Will Learn

Explain what a Flink job is at runtime — JobManager, TaskManagers, slots, tasks, chains — and what survives a TaskManager death with and without checkpoints
Choose keys deliberately and explain why keyed state can only live behind a keyBy
Distinguish keyed, operator, and broadcast state; use keyed state, TTL, and timers to keep per-entity state bounded
State the difference between event time and processing time and choose the right one for a billing computation
Explain that a watermark makes earlier timestamps late rather than impossible; diagnose a stalled input and reason about the idleness/alignment trade-off
Build a tumbling event-time cost window with allowed lateness and a side output; predict when a late event corrects an emitted result
Explain checkpoint barriers, alignment and unaligned checkpoints, and trace a recovery from checkpoint k
Qualify every use of 'exactly-once' with its scope and build an end-to-end exactly-once path with a transactional or idempotent sink
Read backpressure off the metrics and relieve a slow dependency with bounded asynchronous I/O
Rescale and upgrade a stateful job through savepoints without losing state — and know the two ways to lose it
Detect and mitigate a hot key; read the six metrics that predict trouble
Design and defend a production metering pipeline on Flink: failure contract per operator, key topology, time, checkpoints, sinks, observability

Terminology Mapping

How classic concepts map to the terminology used in this course.

Classic	This Course (Python)
Job / JobGraph / ExecutionGraph	StreamExecutionEnvironment.get_execution_environment(); env.execute('GatewayMeter') submits the graph
Operator / Task / Subtask	each .map / .key_by / .process / .window call is an operator; parallel instances are subtasks; chained operators form one task
Keyed State	ValueState / ListState / MapState obtained from runtime_context.get_state(...) inside a KeyedProcessFunction after key_by
Event Time Watermark	WatermarkStrategy.for_bounded_out_of_orderness(Duration.of_seconds(20)).with_timestamp_assigner(...).with_idleness(...)
Window / Trigger / Allowed Lateness	.window(TumblingEventTimeWindows.of(Time.minutes(1))).allowed_lateness(...).side_output_late_data(tag).aggregate(...)
Timer	ctx.timer_service().register_event_time_timer(ts) in process_element; on_timer(ts, ctx) fires it
Checkpoint	env.enable_checkpointing(10_000, CheckpointingMode.EXACTLY_ONCE); env.get_checkpoint_config() for timeout, min pause, unaligned
Savepoint	flink stop --savepointPath ...; flink run -s <path> (CLI); job.uid('tenant-cost') on every stateful operator
Exactly-once sink	KafkaSink.builder().set_delivery_guarantee(DeliveryGuarantee.EXACTLY_ONCE).set_transactional_id_prefix(...)
Async I/O	AsyncDataStream.unordered_wait(stream, AsyncFunction, timeout, capacity, output_type)

Appendices

Reading that belongs to the course rather than to one module. None of it is required to finish the learning path above; each page says who it is for.

Optional module 13 — Temporal vs Flink vs Flink Agents: which runtime does an agent belong in?Reading material

Start now

Module 1: The Machine

JobManager, TaskManagers, slots, operators, parallelism — and what a job is when nothing is checkpointed. About 60 minutes.

Start module 1 — free

Course interest

Help shape what we build next

Tell us what you want to learn. This records your interest; it does not enroll you or promise a launch email.