Optimising Databricks Costs Without Sacrificing Performance

The cost challenge

Databricks is powerful, flexible and — without careful management — expensive. The same elasticity that makes it valuable (spin up clusters on demand, scale to any workload) can lead to runaway costs if teams aren't disciplined about how they use compute resources.

We've seen organisations spend three to five times more than necessary on Databricks, not because the platform is overpriced but because the usage patterns are inefficient. The good news is that most cost optimisation doesn't require sacrificing performance — it requires better configuration, smarter scheduling and a culture of cost awareness.

Right-sizing clusters

The single biggest cost driver in Databricks is cluster configuration. Over-provisioned clusters — too many workers, too much memory, instance types that exceed workload requirements — burn through credits rapidly.

Key practices:

• Match instance types to workloads — memory-optimised instances for Spark-heavy transformations, compute-optimised for SQL queries, GPU instances only for ML workloads
• Use auto-scaling wisely — set minimum and maximum worker counts that reflect actual workload patterns, with scaling steps that respond to demand without over-provisioning
• Enable auto-termination — set aggressive idle termination (15–30 minutes) so clusters shut down automatically when work is complete
• Use spot instances for non-critical workloads — development and staging environments can leverage spot instances at a fraction of the cost

Job cluster vs all-purpose clusters

A common mistake is using all-purpose clusters for scheduled jobs. All-purpose clusters have a higher DBU rate than job clusters — for the same compute. Always use job clusters for automated, scheduled workloads. The savings are immediate and significant.

Photon engine

Photon is Databricks' native vectorised execution engine. It accelerates SQL and DataFrame operations significantly, often reducing runtime by 30–50%. While Photon-enabled instances have a higher DBU rate, the reduced runtime frequently results in lower overall cost. Benchmark your workloads — the speedup often more than compensates for the higher rate.

Delta caching and Z-Ordering

• Delta caching — enables clusters to cache data locally, reducing I/O for repeated queries. This is particularly effective for interactive workloads where the same data is queried multiple times
• Z-Ordering — reorganises data on disk to co-locate related records, dramatically reducing the amount of data scanned for filtered queries. Z-Order on columns that are frequently filtered on, not on every column
• Liquid clustering — Databricks' newer clustering approach that dynamically adapts to query patterns without manual Z-Order maintenance

Monitoring and cost allocation

You can't optimise what you can't see. Implement cost tagging and cluster-level monitoring to understand where spend is going:

• Tag clusters by project, team and environment — enables cost allocation and chargeback
• Use Databricks cost analysis — review the cluster usage dashboard weekly to identify underutilised or misconfigured clusters
• Set budget alerts — configure notifications when spend exceeds thresholds so issues are caught early
• Review job runtimes — jobs that used to take 10 minutes but now take 30 are a signal that something has changed — data volume, cluster configuration or query performance

A culture of cost awareness

Technical optimisation is necessary but not sufficient. The most effective cost management comes from a culture where engineers understand the cost implications of their choices. Share cost dashboards with teams, include cost review in sprint retrospectives and celebrate optimisation wins.

Cost optimisation is an ongoing discipline, not a one-time project. Start with the highest-impact changes — job clusters, auto-termination, right-sizing — and iterate from there.