Part XXI — Streaming State: Watermarks, State Stores, and the 146GB Checkpoint
Every Structured Streaming query is a stateful program wearing a batch costume. The watermark is the only thing standing between your state store and unbounded growth — and when it is misconfigured, the checkpoint doesn’t save you. It becomes the incident.
Towards AI11 min read