Bash Text Processing

grep — Pattern Matching

grep filters lines by pattern. It is typically the first stage in a pipeline — narrowing a large file down to the lines you care about before passing them to other tools. Knowing grep’s flags well means you rarely need to reach for anything else for simple filtering tasks.

# Basic search — print lines that contain "error"
grep "error" /var/log/app.log

# Case-insensitive match
grep -i "error" /var/log/app.log

# Invert match — print lines that do NOT match
grep -v "^#" /etc/hosts          # strip comment lines

# Extended regex — match any of several patterns
grep -E "error|warning|critical" /var/log/app.log

# Count matching lines
grep -c "Failed" /var/log/auth.log

# Show line numbers alongside matches
grep -n "TODO" *.sh

# Show only the matching part of each line (not the whole line)
grep -o "192\.168\.[0-9]+\.[0-9]+" access.log

# Recursive search across a directory tree
grep -r "database_url" /etc/myapp/

# Print only the filenames that contain a match
grep -rl "API_KEY" ./src/

# Context lines around each match
grep -A 3 "ERROR" app.log     # 3 lines After the match
grep -B 2 "ERROR" app.log     # 2 lines Before the match
grep -C 2 "ERROR" app.log     # 2 lines Context (before and after)

# Fixed string — no regex interpretation (faster for literal strings)
grep -F "192.168.1.1" access.log

# Quiet mode — just check if a match exists, use the exit code
if grep -q "FAILED" test_results.txt; then
  echo "Tests failed!"
fi

sed — Stream Editor

sed applies transformations to text line by line. Its most common use is substitution — replacing one string with another across a file. It is the right tool when you know exactly what text to change and what to change it to.

# Substitute first occurrence per line
sed 's/old/new/' file.txt

# Substitute all occurrences per line (g flag)
sed 's/old/new/g' file.txt

# Case-insensitive substitution
sed 's/error/ERROR/gi' file.txt

# In-place edit — modifies the file directly (creates no output)
sed -i 's/localhost/db.internal/g' config.ini

# In-place edit with a backup file
sed -i.bak 's/localhost/db.internal/g' config.ini

# Delete lines matching a pattern
sed '/^#/d' config.txt          # remove comment lines
sed '/^$/d' file.txt            # remove blank lines

# Print only lines matching a pattern (like grep)
sed -n '/ERROR/p' app.log

# Print specific line numbers
sed -n '5,10p' file.txt         # lines 5 through 10
sed -n '5p' file.txt            # line 5 only

# Insert a line before a match
sed '/^server/i # Server configuration' nginx.conf

# Append a line after a match
sed '/^server/a listen 443 ssl;' nginx.conf

# Multiple expressions in one command
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

# Replace within a range between two patterns
sed '/START/,/END/s/old/new/g' file.txt

awk — Field Processing

awk is a complete programming language built around processing columns of text. While grep filters rows and sed transforms text, awk is the tool for working with structured data — CSV files, log lines with fixed fields, command output with columns. It can filter, transform, aggregate, and format in a single pass.

# Print a specific field (default delimiter: whitespace)
awk '{print $1}' /etc/passwd          # first field of every line
awk '{print $1, $3}' file.txt         # fields 1 and 3

# Custom field separator with -F
awk -F: '{print $1, $6}' /etc/passwd  # username and home directory
awk -F, '{print $2}' data.csv         # second column of a CSV

# Arithmetic on fields
df -h | awk 'NR>1 {print $1, $5}'    # skip header (NR>1), print name and usage%

# Filter rows by condition
awk '$3 > 1000' /etc/passwd           # only users with UID > 1000
awk '/error/ {print NR, $0}' app.log  # lines matching "error", with line number

# Sum a column
awk '{sum += $1} END {print "Total:", sum}' numbers.txt

# Count occurrences of a pattern
awk '/ERROR/{count++} END {print count}' app.log

# Formatted output
awk -F: '{printf "%-15s %s\n", $1, $7}' /etc/passwd

# BEGIN and END blocks run before and after all lines are processed
awk '
  BEGIN { print "=== Report ===" }
  /ERROR/ { errors++ }
  /WARN/  { warns++ }
  END {
    printf "Errors: %d\nWarnings: %d\n", errors, warns
  }
' app.log

# Process CSV and calculate an average
awk -F, 'NR>1 { sum+=$3; count++ } END { printf "Avg: %.2f\n", sum/count }' sales.csv

cut — Extract Columns

cut extracts specific fields or character positions from each line. It is simpler than awk for straightforward column extraction and works well when the delimiter is consistent.

# Cut by delimiter — extract specific fields
cut -d: -f1 /etc/passwd          # first field (username)
cut -d: -f1,6 /etc/passwd        # fields 1 and 6
cut -d, -f2-4 data.csv           # fields 2 through 4

# Cut by character position
cut -c1-10 file.txt              # first 10 characters of each line
cut -c5- file.txt                # from character 5 to end of line

# Practical: extract IP addresses from a log and count them
cut -d' ' -f1 /var/log/nginx/access.log | sort | uniq -c | sort -rn

sort — Sorting Lines

sort orders lines of text. It becomes essential as soon as you need to find the top N of something, deduplicate with uniq, or present data in a predictable order.

# Alphabetical sort
sort names.txt

# Reverse sort
sort -r names.txt

# Numeric sort (treats values as numbers, not strings)
sort -n numbers.txt

# Sort by a specific field (delimiter defaults to whitespace)
sort -k2 data.txt                # sort by second field
sort -t: -k3 -n /etc/passwd      # sort by UID (field 3, numeric)

# Remove duplicates while sorting
sort -u names.txt

# Sort by multiple keys — primary key first, then tiebreaker
sort -t, -k2,2 -k1,1 data.csv   # primary: field 2, secondary: field 1

# Sort human-readable sizes (10K, 5M, 2G) — requires GNU sort
du -sh /var/log/* | sort -h

uniq — Deduplicate Adjacent Lines

uniq removes or counts duplicate adjacent lines. Because it only compares adjacent lines, it almost always needs to be preceded by sort. The most common use is counting how many times each value appears.

# Remove duplicates (input must be sorted first)
sort names.txt | uniq

# Count occurrences of each unique line
sort names.txt | uniq -c

# Show only lines that appear more than once
sort names.txt | uniq -d

# Show only lines that appear exactly once
sort names.txt | uniq -u

# Case-insensitive deduplication
sort names.txt | uniq -i

# Practical: top IP addresses in an nginx access log
awk '{print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

wc — Word and Line Count

wc counts lines, words, and bytes. Its most common use in scripts is counting lines of output to validate that a command produced the expected number of results.

wc -l file.txt      # line count
wc -w file.txt      # word count
wc -c file.txt      # byte count
wc -m file.txt      # character count (differs from bytes for multi-byte chars)

# Count the output of a command
ls /etc | wc -l

# In a script — count errors and take action
error_count=$(grep -c "ERROR" app.log)
echo "Found $error_count errors"
(( error_count > 100 )) && echo "Too many errors — alerting" >&2

tr — Translate Characters

tr translates or deletes characters from its input. It operates on individual characters (not strings or patterns), making it the right tool for case conversion, removing unwanted characters, and squeezing repeated characters.

# Convert lowercase to uppercase
echo "hello world" | tr 'a-z' 'A-Z'    # HELLO WORLD

# Delete characters
echo "hello 123 world" | tr -d '0-9'   # hello  world

# Squeeze repeated characters — collapse multiple spaces into one
echo "hello   world" | tr -s ' '       # hello world

# Replace newlines with spaces (join lines)
cat file.txt | tr '\n' ' '

# Remove non-printable characters from a file
tr -cd '[:print:]\n' < input.txt > clean.txt

# Display PATH entries one per line
echo "$PATH" | tr ':' '\n'

column — Tabular Formatting

column aligns output into readable columns. It is useful for displaying the results of scripts in a human-readable table without writing custom formatting code.

# Auto-align whitespace-delimited columns
column -t data.txt

# Custom delimiter
column -t -s, data.csv

# Format /etc/passwd as a table
cut -d: -f1,3,6 /etc/passwd | column -t -s:

# Format mount output
mount | column -t

# Display key=value pairs as a table
env | sort | column -t -s=

Combining Tools — Real Pipeline Examples

The real power of these tools comes from combining them. Each tool does one thing; pipelines combine them into workflows that would take significant code in any other language.

Find the top 10 slowest API endpoints

# Log format: timestamp method path response_time_ms
awk '{print $3, $4}' api.log \
  | sort -k2 -rn \
  | head -10 \
  | column -t

Count HTTP status codes

awk '{print $9}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn \
  | awk '{printf "HTTP %s: %d requests\n", $2, $1}'

Find recently modified files with suspicious content

find /var/www -name "*.php" -mmin -60 \
  | xargs grep -l "eval\|base64_decode" 2>/dev/null \
  | while IFS= read -r f; do
      echo "Suspicious: $f"
    done

Parse a CSV and generate a report

#!/usr/bin/env bash
# Summarize sales totals by region from a CSV file

awk -F, '
  NR == 1 { next }   # skip header row
  {
    region=$2
    amount=$4
    totals[region] += amount   # accumulate per region
    counts[region]++
  }
  END {
    printf "%-15s %10s %10s\n", "Region", "Total", "Count"
    printf "%-15s %10s %10s\n", "------", "-----", "-----"
    for (r in totals) {
      printf "%-15s %10.2f %10d\n", r, totals[r], counts[r]
    }
  }
' sales.csv | sort -k2 -rn

What’s Next

The next tutorial covers process management — background jobs, signals, trap, and subshells.