Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks

Databricks Runtime 18 introduces on-demand state repartitioning for Apache Spark Structured Streaming, allowing developers to resize partitions without losing checkpoint state. This feature addresses scaling bottlenecks in stateful queries like fraud detection. Early adopter Coveo reports a 40% reduction in S3 API costs by scaling infrastructure dynamically without rebuilding checkpoints.