File I/O, pathlib, and JSON in Python

Working with files is a fundamental skill in Python. The standard library provides three complementary tools: the built-in open() function for reading and writing, pathlib for path manipulation, and json/csv modules for structured data.

Reading Files with open()

Almost every Python program eventually needs to read data from disk — configuration, logs, user input saved from a previous run. The built-in open() function handles this, but raw file handles must be closed manually or they leak operating system resources. The with statement solves this by wrapping the file in a context manager that guarantees the handle is closed the moment the block exits, even if an exception is raised mid-read.

# Read entire file as a single string — good for small config files
with open("report.txt", "r", encoding="utf-8") as f:
    content = f.read()

# Read line by line — memory-efficient for large log files
# The file object is itself an iterator; no extra list is created
with open("log.txt", "r", encoding="utf-8") as f:
    for line in f:
        process(line.rstrip("\n"))   # strip the trailing newline each line carries

# Read all lines into a list when you need random access by index
with open("data.txt", "r", encoding="utf-8") as f:
    lines = f.readlines()   # each element includes its "\n" character

Writing Files

Writing to disk lets your program persist results, produce reports, and communicate with other processes. Python gives you three distinct modes: "w" creates or overwrites, "a" appends without destroying existing content, and "x" creates exclusively (fails if the file already exists, which prevents accidental overwrites). Always specify encoding="utf-8" explicitly — the default encoding varies by platform and can cause silent data corruption.

# Write a new file — creates it if absent, overwrites if present
with open("output.txt", "w", encoding="utf-8") as f:
    f.write("First line\n")
    f.write("Second line\n")

# Append a log entry without touching existing content
with open("events.log", "a", encoding="utf-8") as f:
    f.write("[2024-01-15] User logged in\n")

# Write many lines at once — writelines() is slightly faster than repeated write()
lines = ["apple\n", "banana\n", "cherry\n"]
with open("fruits.txt", "w", encoding="utf-8") as f:
    f.writelines(lines)   # note: writelines does NOT add newlines automatically

File Modes Reference

Choosing the wrong mode is a common source of bugs — "w" silently destroys an existing file, while "r+" requires the file to already exist. This table is worth bookmarking.

pathlib — Object-Oriented Filesystem Paths

The older os.path module treats paths as plain strings, which leads to fragile string concatenation and platform-specific separators. pathlib.Path (introduced in Python 3.4) represents paths as objects. You join segments with the / operator, call methods to read or write directly, and never worry about whether you’re on Windows or Linux. It is the recommended approach for all new code.

from pathlib import Path

# Build paths with the / operator — works on Windows and Unix alike
project = Path("/projects/myapp")
config_file = project / "config" / "settings.toml"

# Inspect path components without string slicing
print(config_file.parent)   # /projects/myapp/config
print(config_file.name)     # settings.toml
print(config_file.stem)     # settings  (filename without extension)
print(config_file.suffix)   # .toml

# Check existence before reading to avoid exceptions
if config_file.exists():
    content = config_file.read_text(encoding="utf-8")

# Create a nested directory structure in one call
output_dir = project / "output"
output_dir.mkdir(parents=True, exist_ok=True)   # no error if it already exists

# Path objects can read and write directly — no open() needed for simple cases
log_file = output_dir / "run.log"
log_file.write_text("Started successfully\n", encoding="utf-8")

# Glob — find all Python files recursively without os.walk()
py_files = list(project.rglob("*.py"))

# Rename / move a file
old = project / "legacy.py"
new = project / "modern.py"
old.rename(new)

# Inspect file metadata
stat = config_file.stat()
print(f"Size: {stat.st_size} bytes")

Working with JSON

JSON is the lingua franca of data exchange — REST APIs, configuration files, and inter-service communication all rely on it. Python’s built-in json module converts between Python dicts/lists and JSON strings with two pairs of functions: dumps/loads for strings, and dump/load for file handles. The indent parameter produces human-readable output, which is invaluable when debugging.

import json
from pathlib import Path

# Python dict → formatted JSON string
data = {
    "name": "Alice",
    "age": 30,
    "skills": ["Python", "SQL"],
    "active": True,
}

json_string = json.dumps(data, indent=2)
print(json_string)
# {
#   "name": "Alice",
#   "age": 30,
#   ...
# }

# JSON string → Python dict — json.loads handles the reverse conversion
parsed = json.loads(json_string)
print(parsed["skills"])   # ['Python', 'SQL']

# Write JSON to a file using pathlib — concise and readable
path = Path("user.json")
path.write_text(json.dumps(data, indent=2), encoding="utf-8")

# Read JSON from a file
loaded = json.loads(path.read_text(encoding="utf-8"))

# Alternatively, use file handles — useful when streaming large files
with open("user.json", "w") as f:
    json.dump(data, f, indent=2)   # writes directly to file handle

with open("user.json", "r") as f:
    loaded = json.load(f)          # reads directly from file handle

Working with CSV

CSV (comma-separated values) is the standard format for tabular data — spreadsheets, database exports, and data pipelines all use it. Python’s csv module handles quoting, escaping, and dialect differences that make naive string splitting unreliable. DictWriter and DictReader let you work with rows as dictionaries keyed by column name, which is far less error-prone than positional indexing.

import csv
from pathlib import Path

# Define rows as dictionaries — column order is controlled by fieldnames
rows = [
    {"name": "Alice", "score": 95, "grade": "A"},
    {"name": "Bob",   "score": 82, "grade": "B"},
    {"name": "Carol", "score": 78, "grade": "C"},
]

csv_path = Path("results.csv")
# newline="" is required on Windows to prevent double line endings
with csv_path.open("w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "score", "grade"])
    writer.writeheader()   # writes the column names as the first row
    writer.writerows(rows)

# Read CSV — each row comes back as a dict with column names as keys
with csv_path.open("r", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(f"{row['name']}: {row['score']}")

Handling File Errors Gracefully

File operations fail for many reasons outside your control: the file may not exist, the process may lack read permission, or the encoding may not match. Catching specific exceptions rather than a bare except Exception lets you report the right problem to the user and handle each case appropriately. Returning None on failure is a common pattern that lets callers decide what to do instead of crashing unexpectedly.

from pathlib import Path

def safe_read(path: str) -> str | None:
    """Read a file and return its content, or None on error."""
    try:
        return Path(path).read_text(encoding="utf-8")
    except FileNotFoundError:
        # The path doesn't exist — common when config is optional
        print(f"File not found: {path}")
    except PermissionError:
        # Process lacks read access — worth logging at a higher level
        print(f"Permission denied: {path}")
    except UnicodeDecodeError:
        # File exists but isn't valid UTF-8 — might be binary or Latin-1
        print(f"Cannot decode file as UTF-8: {path}")
    return None