PySpark.
Distributed data engineering with Apache Spark, PySpark DataFrames, Databricks pipelines, and Structured Streaming.
Beginner
Start here — no prior experience needed
PySpark in Ten Minutes: Your First DataFrame
Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.
Reading and Writing Data in PySpark
Read CSV, JSON, and Parquet with an explicit schema, handle malformed rows deliberately, and partition output so downstream reads skip what they do not need.
Transforming Columns in PySpark
Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.
Grouping and Aggregation in PySpark
Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.
Intermediate
For developers with core concepts down
Joins and Broadcast Joins in PySpark
Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.
Window Functions in PySpark
Rank rows within groups, compute running totals, and compare each row to its neighbours — without collapsing the DataFrame the way groupBy does.
UDFs and Pandas UDFs in PySpark
Measure what a Python UDF costs against a built-in function, then rewrite it as a pandas UDF and watch most of that cost disappear.
Advanced
Production-grade patterns for experienced engineers
The Spark Execution Model: Jobs, Stages, and Caching
Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.
Tuning PySpark Jobs
Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.
Structured Streaming in PySpark
Run the same DataFrame code over an unbounded stream — with windows, watermarks, checkpoints, and the output modes that decide what you are allowed to compute.