Cost Control and Cluster Policies

Databricks bills DBUs per second; your cloud provider bills the VMs underneath. Both scale with choices people make in a UI, so cost control here is mostly about constraining those choices and then measuring what is left.

Where the money goes

select
    u.sku_name,
    round(sum(u.usage_quantity), 1)                          as dbus,
    round(sum(u.usage_quantity * p.pricing.default), 2)      as usd
from system.billing.usage u
join system.billing.list_prices p
  on u.sku_name = p.sku_name
 and u.usage_end_time between p.price_start_time and coalesce(p.price_end_time, current_timestamp())
where u.usage_date >= current_date() - interval 30 days
group by all
order by usd desc;

sku_name                        dbus      usd
------------------------------  --------  ---------
PREMIUM_ALL_PURPOSE_COMPUTE     12488.4   6868.62
PREMIUM_JOBS_COMPUTE             5602.1   1568.59
PREMIUM_SQL_COMPUTE              2884.0   1902.44
PREMIUM_SERVERLESS_SQL_COMPUTE   1204.8    841.36
PREMIUM_DLT_ADVANCED              882.0    308.70

All-purpose compute is 62% of the bill. On almost every account that means one of two things: interactive clusters left running, or scheduled jobs pointed at all-purpose compute — which costs roughly 2.5× the jobs rate for identical work.

select
    case when j.job_id is not null then 'scheduled work' else 'interactive' end as kind,
    round(sum(u.usage_quantity), 1) as dbus
from system.billing.usage u
left join system.lakeflow.jobs j on u.usage_metadata.job_id = j.job_id
where u.sku_name = 'PREMIUM_ALL_PURPOSE_COMPUTE'
  and u.usage_date >= current_date() - interval 30 days
group by all;

kind             dbus
---------------  -------
scheduled work    8204.2
interactive       4284.2

8,204 DBUs of scheduled work on interactive compute — about $2,700 a month, recoverable by changing a cluster setting on each job.

Attributing it

select
    u.custom_tags['team']        as team,
    u.custom_tags['cost_centre'] as cost_centre,
    round(sum(u.usage_quantity), 1) as dbus
from system.billing.usage u
where u.usage_date >= current_date() - interval 30 days
group by all
order by dbus desc
limit 5;

team        cost_centre  dbus
----------  -----------  -------
data-eng    CC-4021       8412.2
NULL        NULL          6204.8
analytics   CC-4088       3102.1
ml-platform CC-4155       2841.0

The NULL row is 6,204 DBUs nobody owns — the second largest line. Untagged resources are always the problem, and asking people to remember tags does not fix it. Enforce them:

Cluster policies

{
  "spark_version": {
    "type": "regex",
    "pattern": "1[5-9]\\..*",
    "defaultValue": "16.4.x-scala2.13"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["m6gd.large", "m6gd.xlarge", "m6gd.2xlarge"],
    "defaultValue": "m6gd.xlarge"
  },
  "autotermination_minutes": {
    "type": "range",
    "minValue": 10,
    "maxValue": 60,
    "defaultValue": 30
  },
  "autoscale.max_workers": {
    "type": "range",
    "maxValue": 8,
    "defaultValue": 4
  },
  "custom_tags.team": {
    "type": "fixed",
    "value": "analytics"
  },
  "custom_tags.cost_centre": {
    "type": "fixed",
    "value": "CC-4088"
  },
  "aws_attributes.availability": {
    "type": "fixed",
    "value": "SPOT_WITH_FALLBACK"
  },
  "cluster_type": {
    "type": "fixed",
    "value": "all-purpose"
  }
}

Attempting to exceed it:

Cluster validation failed:
  autotermination_minutes: value 0 is not within the allowed range [10, 60]
  autoscale.max_workers: value 32 exceeds the maximum allowed value of 8

The cluster is not created. Four things this policy guarantees: every cluster terminates, no cluster exceeds 8 workers, every cluster carries team and cost-centre tags, and workers run on spot with on-demand fallback.

Grant policies per group so each team gets appropriate limits, and make Unrestricted available to nobody outside the platform team.

Spot instances

"aws_attributes": {
  "availability": "SPOT_WITH_FALLBACK",
  "first_on_demand": 1,
  "spot_bid_price_percent": 100
}

Spot capacity runs at a substantial discount and can be reclaimed at any time. first_on_demand: 1 keeps the driver on-demand — losing a worker costs a retried task, while losing the driver kills the whole job.

Suitable for: batch ETL with retries, development clusters, anything idempotent. Unsuitable for: long streaming jobs where restarts are expensive, and time-critical SLA work.

Photon

-- without Photon, Large cluster
select country_code, date_trunc('month', ordered_at) as month, sum(amount) as revenue
from bookshop.silver.orders join bookshop.silver.customers using (customer_id)
group by all;

-- 244.1 seconds, 8 DBU/hour → 0.542 DBUs

-- with Photon, same cluster: 88.4 seconds, ~1.9× DBU rate → 0.373 DBUs

Faster and cheaper here, because the speedup exceeds the rate increase. But:

@udf("double")
def apply_fx(amount, rate):
    return amount * rate

df.withColumn("gbp", apply_fx("amount", "rate")).groupBy("country").sum("gbp")

-- without Photon: 188.2s → 0.418 DBUs
-- with Photon:    182.1s → 0.769 DBUs

A Python UDF falls back to the JVM path, so Photon gives 3% and costs 84%. Photon accelerates SQL and DataFrame operations, not Python UDFs, RDD code or most ML libraries. Enable it for SQL warehouses and scan-heavy pipelines; measure before turning it on across everything.

SQL warehouses

select
    warehouse_id,
    count(*)                                        as queries,
    round(avg(total_duration_ms) / 1000, 2)         as avg_seconds,
    round(sum(case when total_duration_ms > 60000 then 1 else 0 end) * 100.0 / count(*), 1) as pct_over_1m
from system.query.history
where start_time >= current_date() - interval 7 days
group by all
order by queries desc;

warehouse_id      queries  avg_seconds  pct_over_1m
----------------  -------  -----------  -----------
a1b2c3d4e5f60718   182044         1.42          0.8
3d9c4b21771a4e02     4120        48.11         38.2

The second warehouse runs 4,120 queries averaging 48 seconds, 38% over a minute — a sizing problem. The first runs 182,000 fast queries, which is a dashboard polling far more often than anyone reads it.

Serverless removes the idle question entirely, and for bursty BI traffic it is usually cheaper despite the higher rate — a Pro warehouse with a 10-minute auto-stop, hit every 8 minutes, never stops.

Budgets and alerts

create or replace view ops.daily_spend as
select
    usage_date,
    custom_tags['team'] as team,
    round(sum(usage_quantity * p.pricing.default), 2) as usd
from system.billing.usage u
join system.billing.list_prices p on u.sku_name = p.sku_name
where usage_date >= current_date() - interval 90 days
group by all;

select team, usd,
       round(avg(usd) over (partition by team order by usage_date rows between 7 preceding and 1 preceding), 2) as trailing_avg
from ops.daily_spend
where usage_date = current_date() - 1
order by usd desc;

team         usd     trailing_avg
-----------  ------  ------------
data-eng     412.60        188.44
analytics    102.11        108.20
ml-platform   88.02         91.44

data-eng is at 2.2× its trailing average — worth a look this morning rather than at month end. An alert on a multiple of the trailing average beats a fixed threshold, which is either noisy or useless as the platform grows.

A checklist that pays for itself

1. Move scheduled work to job compute. Usually the single largest saving.
2. Auto-termination on every interactive cluster, enforced by policy, not by asking.
3. Enforce tags through policies so nothing is unattributable.
4. Cap max_workers per team — an autoscaling cluster scales to whatever it is allowed.
5. Spot for batch, on-demand driver.
6. Serverless SQL for BI, so idle warehouses stop being a line item.
7. Measure Photon per workload rather than enabling it globally.
8. Alert on a multiple of the trailing average, per team.

Practice

kind             dbus
---------------  -------
scheduled work    8204.2
interactive       4284.2

Cluster validation failed:
  autoscale.max_workers: value 32 exceeds the maximum allowed value of 8

-- SQL aggregation:  244.1s / 0.542 DBUs  →  88.4s / 0.373 DBUs
-- Python UDF job:   188.2s / 0.418 DBUs  → 182.1s / 0.769 DBUs

team        dbus
----------  -------
data-eng     8412.2
NULL         6204.8

That closes the Databricks track. The thread through all ten lessons: the platform is Delta tables in your own storage plus compute you rent by the second — governance, pipelines, performance and cost all follow from keeping those two things straight.