The Databricks Workspace and Compute

Databricks is a workspace over object storage: your data stays in your cloud account as Delta files, and the platform provides the compute, the catalog, and the orchestration. The first thing worth understanding is which kind of compute you are using, because it decides both how fast you start and how much you pay.

The compute options

The rule that saves the most money: never schedule production work on an all-purpose cluster. It is billed at roughly double the job rate, and it usually stays up between runs.

# a notebook cell
print(spark.version)
print(spark.conf.get("spark.databricks.clusterUsageTags.clusterName"))

4.0.1
bookshop-dev

spark already exists — Databricks creates the session before your first cell runs. So does dbutils, the utility object for filesystem, secrets and notebook control.

Where data lives

Tables are Delta files in object storage, registered in Unity Catalog under a three-level name: catalog.schema.table.

-- a %sql cell
create catalog if not exists bookshop;
create schema if not exists bookshop.raw;
use catalog bookshop;
use schema raw;

OK

create or replace table customers (
    customer_id  bigint,
    full_name    string,
    country_code string,
    created_at   timestamp
);

insert into customers values
    (1, 'Ada Lovelace',      'GB', '2025-11-02 09:14:00'),
    (2, 'Grace Hopper',      'US', '2025-11-04 16:02:00'),
    (3, 'Alan Turing',       'GB', '2026-01-03 11:41:00'),
    (4, 'Katherine Johnson', 'US', '2026-01-08 08:20:00');

num_affected_rows  num_inserted_rows
-----------------  -----------------
                4                  4

create or replace table orders (
    order_id    bigint,
    customer_id bigint,
    ordered_at  date,
    status      string,
    amount      decimal(10,2)
);

insert into orders values
    (1001, 1, '2026-01-04', 'completed', 25.50),
    (1002, 2, '2026-01-05', 'completed', 12.00),
    (1003, 1, '2026-01-07', 'returned',  40.00),
    (1004, 3, '2026-01-09', 'completed',  8.75),
    (1005, 2, '2026-01-11', 'pending',   63.20);

num_affected_rows  num_inserted_rows
-----------------  -----------------
                5                  5

Every table created this way is Delta by default — there is no USING DELTA needed, and choosing anything else is almost always a mistake on this platform.

Mixing languages

A notebook has a default language, and any cell can override it with a magic:

%sql
select c.country_code, count(*) as orders, sum(o.amount) as revenue
from orders o join customers c using (customer_id)
where o.status = 'completed'
group by 1
order by revenue desc;

country_code  orders  revenue
------------  ------  -------
GB                 2    34.25
US                 1    12.00

df = spark.table("bookshop.raw.orders").filter("status = 'completed'")
df.show()

+--------+-----------+----------+---------+------+
|order_id|customer_id|ordered_at|   status|amount|
+--------+-----------+----------+---------+------+
|    1001|          1|2026-01-04|completed| 25.50|
|    1002|          2|2026-01-05|completed| 12.00|
|    1004|          3|2026-01-09|completed|  8.75|
+--------+-----------+----------+---------+------+

%scala
spark.table("bookshop.raw.orders").count()

res0: Long = 5

The same tables from SQL, Python and Scala, in one notebook. This is the practical argument for the platform: the analyst writing SQL and the engineer writing PySpark are working on one copy of the data, not on an extract.

Passing values between languages goes through a temporary view rather than variables:

spark.sql("select * from orders where amount > 20").createOrReplaceTempView("big_orders")

%sql
select count(*) from big_orders;

count(1)
--------
       3

dbutils

dbutils.fs.ls("/Volumes/bookshop/raw/landing/")

[FileInfo(path='dbfs:/Volumes/bookshop/raw/landing/orders_2026_01.csv', name='orders_2026_01.csv', size=124500, modificationTime=1789459200000),
 FileInfo(path='dbfs:/Volumes/bookshop/raw/landing/orders_2025_12.csv', name='orders_2025_12.csv', size=118204, modificationTime=1786867200000)]

dbutils.help("fs")

fs: DbfsUtils -> Manipulates the Databricks filesystem (DBFS)
    cp(from: String, to: String, recurse: boolean = false): boolean -> Copies a file or directory
    ls(dir: String): Seq -> Lists the contents of a directory
    mkdirs(dir: String): boolean -> Creates the given directory
    mv(from: String, to: String, recurse: boolean = false): boolean -> Moves a file or directory
    put(file: String, contents: String, overwrite: boolean = false): boolean -> Writes the string
    rm(dir: String, recurse: boolean = false): boolean -> Removes a file or directory

Use volumes (/Volumes/catalog/schema/name/) for new work rather than raw DBFS paths. Volumes are governed by Unity Catalog, so file access follows the same grants as tables — lesson 3 covers them properly.

Parameters make a notebook reusable, and they are how a job passes values in:

dbutils.widgets.text("run_date", "2026-01-04")
run_date = dbutils.widgets.get("run_date")
print(f"processing {run_date}")

processing 2026-01-04

What a cluster costs while you read this

%sql
select
    sku_name,
    usage_date,
    sum(usage_quantity) as dbus
from system.billing.usage
where usage_date >= current_date() - interval 7 days
group by all
order by dbus desc
limit 5;

sku_name                              usage_date  dbus
------------------------------------  ----------  ------
PREMIUM_ALL_PURPOSE_COMPUTE           2026-09-08  412.60
PREMIUM_JOBS_COMPUTE                  2026-09-08  188.44
PREMIUM_SQL_COMPUTE                   2026-09-08   88.02
PREMIUM_ALL_PURPOSE_COMPUTE           2026-09-07  402.11
PREMIUM_JOBS_COMPUTE                  2026-09-07  190.28

All-purpose compute is the largest line, which on most accounts means interactive clusters left running, or jobs pointed at the wrong compute. The system.billing.usage table is available in every Unity Catalog workspace and is the honest answer to “what are we spending this on” — lesson 10 builds on it.

Set an auto-termination window on every interactive cluster:

Auto termination: 30 minutes of inactivity

Practice

describe detail bookshop.raw.orders;

format  id                                    numFiles  sizeInBytes  location
------  ------------------------------------  --------  -----------  ------------------------------------------
delta   9f2c1a44-8e21-4b0e-9a3c-1d84f0b27a51         1         2048  s3://bookshop-lake/raw/orders

-- SQL
count(1)
--------
       5

# PySpark
5

dbutils.widgets.dropdown("country", "GB", ["GB", "US", "NL"])
spark.sql(f"select count(*) from customers where country_code = '{dbutils.widgets.get('country')}'").show()

+--------+
|count(1)|
+--------+
|       2|
+--------+

select sku_name, round(sum(usage_quantity), 1) as dbus
from system.billing.usage
where usage_date >= current_date() - interval 30 days
group by all order by dbus desc;

sku_name                     dbus
---------------------------  -------
PREMIUM_ALL_PURPOSE_COMPUTE  12488.4
PREMIUM_JOBS_COMPUTE          5602.1
PREMIUM_SQL_COMPUTE           2884.0

Next: Delta Lake — the transaction log that makes all of this behave like a database.