Apache Spark Tutorial with Examples.
Apache Spark tutorial covering the core architecture: RDDs, DAG execution, shuffles, Spark SQL and cluster tuning, with real job output.
Architecture at a glance
Click any box to open the lesson that teaches it
Spark architecture at a glance
spark-submit → driver (DAG + Catalyst) → cluster manager → executors
How it fits together (6 steps)
- spark-submit launches the driver, which hosts your SparkSession and builds the plan — no cluster work happens in the driver itself.
- The driver's DAG scheduler turns one action into a job, splits it into stages wherever a shuffle must happen, and splits each stage into one task per partition.
- Catalyst rewrites that plan before it runs — pushing filters down to the data source, pruning columns, and choosing a join strategy — the same optimizer whether the query was written as SQL or as DataFrame calls.
- The driver asks the cluster manager (YARN, Kubernetes, or Spark's own standalone mode) for resources, which launches the executors that actually run tasks and hold cached data in a unified memory pool shared between caching and execution.
- A shuffle between stages is the expensive part — reduceByKey combines values on each partition before the shuffle, while groupByKey moves every value across the network, a five-times difference for the same result.
- When a job is slow, the Spark UI's Jobs, Stages, and SQL tabs show what actually happened, and most failures are one of the memory problems in driver OOM, executor OOM, or disk spill.
Beginner
Start here — no prior experience needed
Spark Architecture: Driver, Executors, and Cluster Managers
Submit a job and watch the driver plan it, executors run it, and the cluster manager allocate resources — reading each component's output live.
RDDs: The API Underneath DataFrames
Work directly with RDDs, measure how much slower they are than the equivalent DataFrame, and identify the few cases where you still need them.
Spark SQL and the Catalyst Optimizer
Register views, query them with SQL, then read Catalyst's four plan stages to see exactly which rewrites it applied to your query.
Intermediate
For developers with core concepts down
Deploying Spark Jobs with spark-submit
Package an application, submit it to a cluster, and work through failures that only appear outside the shell — missing jars and wrong Python.
Reading the Spark UI to Diagnose a Slow Job
Work through the Jobs, Stages, SQL, and Executors tabs on a deliberately broken job, and learn which number to look at first for each kind of slowness.
Advanced
Production-grade patterns for experienced engineers
Also part of Apache Spark
Taught in other topics. Each link opens the lesson in its home topic.
Structured Streaming in PySpark
Run the same DataFrame code over an unbounded stream — with windows, watermarks, checkpoints, and output modes that decide what you compute.
MLlib Basics: A Real Pipeline, Fit and Evaluated
Assemble feature vectors, chain them into a Pipeline with a classifier, and evaluate the fitted model with AUC and accuracy on held-out data.
PySpark in Ten Minutes: Your First DataFrame
Install PySpark, start a SparkSession, build a DataFrame, and see why nothing runs until you call an action.
Transforming Columns in PySpark
Add, rename, and reshape columns with withColumn, when/otherwise, and the functions module — and see why chaining withColumn in a loop is a trap.
Grouping and Aggregation in PySpark
Group, aggregate, and pivot — then read the query plan to see the shuffle that groupBy introduces and why it dominates the job's runtime.
Joins and Broadcast Joins in PySpark
Work through every join type, then compare a shuffle join against a broadcast join on the same data and read the difference in the query plan.
The Spark Execution Model: Jobs, Stages, and Caching
Trace one query from DataFrame to tasks, see where stage boundaries come from, and prove that caching changes the plan rather than just the timing.
Tuning PySpark Jobs
Diagnose and fix the four things that actually make Spark jobs slow — skew, small files, spill, and wasted shuffles — measuring each one.