GPU Fundamentals: SIMT & Memory Hierarchy

How GPUs Execute (Theory)

GPUs run many threads in parallel. The key execution model is:

• Warp (32 threads): the unit of scheduling.
• SIMT: all threads in a warp follow the same instruction stream.
• Divergence: if threads take different branches, the warp executes multiple paths serially.

Practical implication

When writing kernels:

• Prefer straight-line code
• Reduce branch divergence
• Ensure memory access patterns are friendly (coalesced)

Code Example 1 — Visualizing Warp-Level Control Flow (Branch Divergence)

// divergence.cu
#include <cuda_runtime.h>

__global__ void divergence_example(int* out, int N) {
  int idx = blockIdx.x * blockDim.x + threadIdx.x;
  if (idx >= N) return;

  // Example divergence: half threads take one path, half take another
  if (threadIdx.x % 2 == 0) {
    out[idx] = 1;   // path A
  } else {
    out[idx] = 2;   // path B
  }
}

What to learn

Even though every thread does different work, the warp still executes both paths if they differ.

Code Example 2 — Mapping Indices to Threads (Global Index)

A foundational skill: compute a global thread index and guard against out-of-range work.

#include <cuda_runtime.h>

__global__ void scale(const float* x, float* y, int N, float a) {
  int i = blockIdx.x * blockDim.x + threadIdx.x;
  if (i < N) {
    y[i] = a * x[i];
  }
}

GPU Memory Hierarchy (Theory)

Typical memory layers:

• Registers: fastest, per-thread, limited count.
• Shared memory: fast, per-block, programmer-managed.
• L2 cache: device-wide cache for reuse.
• Global/DRAM: large but slow.

Optimization priorities (beginner rule)

1. Reuse data: keep it in registers or shared when possible.
2. Make accesses coalesced: consecutive threads access nearby addresses.
3. Avoid unnecessary global loads/stores.

Common Gotchas

• Ignoring bounds checks: threads may run beyond array size.
• Assuming all threads run independently: warps execute in lockstep (SIMT).
• Poor data layout: strided access ruins bandwidth.

Quick Checklist

• Compute global_id = blockIdx.x * blockDim.x + threadIdx.x
• Add if (global_id < N) guards
• Reduce divergence
• Favor coalesced reads/writes
• Reuse data via shared/registers