VPCs and the Connectivity Debug Ladder
Subnets, route tables, security groups and NACLs, and the ordered checklist that finds why two things cannot talk — the most common hands-on cloud interview question.
The scenario question — “instance A cannot reach database B, walk me through your debugging” — is asked in almost every cloud round. It has an ordered answer.
The topology, and what makes a subnet public
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-0abc \
--query 'Subnets[].{id:SubnetId,cidr:CidrBlock,az:AvailabilityZone,public:MapPublicIpOnLaunch}' \
--output table-------------------------------------------------------------------
| DescribeSubnets |
+------------+-------------------+--------------+-----------------+
| az | cidr | id | public |
+------------+-------------------+--------------+-----------------+
| us-east-1a| 10.0.1.0/24 | subnet-0a1 | True |
| us-east-1b| 10.0.2.0/24 | subnet-0b2 | True |
| us-east-1a| 10.0.11.0/24 | subnet-0c3 | False |
| us-east-1b| 10.0.12.0/24 | subnet-0d4 | False |
+------------+-------------------+--------------+-----------------+MapPublicIpOnLaunch is not what makes a subnet public. It only decides whether instances
get a public IP automatically. The route table decides:
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-0a1 \
--query 'RouteTables[].Routes[].{dest:DestinationCidrBlock,gw:GatewayId,nat:NatGatewayId}' \
--output table-----------------------------------------------
| DescribeRouteTables |
+-------------+---------------+---------------+
| dest | gw | nat |
+-------------+---------------+---------------+
| 10.0.0.0/16| local | None |
| 0.0.0.0/0 | igw-0f1e2d | None |
+-------------+---------------+---------------+PUBLIC SUBNET 0.0.0.0/0 → internet gateway
PRIVATE SUBNET 0.0.0.0/0 → NAT gateway (outbound only)
ISOLATED SUBNET no 0.0.0.0/0 route at all (nothing in or out)“A subnet is public if and only if its route table sends 0.0.0.0/0 to an internet gateway. The name is a convention, the auto-assign flag only affects public IPs, and I’ve debugged a ‘public’ subnet that had no internet gateway route.”
The standard three-tier layout, and why:
┌─────────────┐
internet ────────▶│ IGW │
└──────┬──────┘
┌─────────────┴─────────────┐
│ PUBLIC 10.0.1.0/24 │ load balancer, NAT gateway
│ 10.0.2.0/24 │ (one per AZ, or lose the AZ)
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PRIVATE 10.0.11.0/24 │ application instances
│ 10.0.12.0/24 │ outbound via NAT only
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ ISOLATED 10.0.21.0/24 │ database — no internet route
│ 10.0.22.0/24 │ at all, in either direction
└───────────────────────────┘Two things worth volunteering about this diagram:
- One NAT gateway per AZ, not one for the VPC. A single NAT gateway is a single point of failure and sends cross-AZ traffic that you pay for in both directions.
- The database tier has no internet route. Not a restrictive one — none. That is a real security boundary rather than a rule someone can loosen.
Security groups versus NACLs
aws ec2 describe-security-groups --group-ids sg-0app \
--query 'SecurityGroups[0].{in:IpPermissions,out:IpPermissionsEgress}'{
"in": [
{
"IpProtocol": "tcp", "FromPort": 8080, "ToPort": 8080,
"UserIdGroupPairs": [{"GroupId": "sg-0alb"}] ← source is another SG
}
],
"out": [
{"IpProtocol": "-1", "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}
]
}Referencing sg-0alb as the source rather than a CIDR block is the idiom worth knowing: the
rule follows the load balancer’s instances wherever they are, and it survives IP changes and
scaling events. Hardcoding CIDRs is what makes security groups rot.
SECURITY GROUP NETWORK ACL
attaches to instance / ENI subnet
stateful? YES — return traffic NO — allow both directions
is automatic explicitly
rules allow only allow AND deny
evaluation all rules, any match numbered, first match wins
permits
default deny all in, allow all in and out
allow all out
use for everything coarse subnet blocks, IP
denylistsStateful versus stateless is the distinction being tested. The failure it produces:
Security group: allow inbound 443 from 0.0.0.0/0
→ response goes out automatically. Works.
NACL: allow inbound 443 from 0.0.0.0/0
→ response tries to leave on an ephemeral port (1024-65535)
→ outbound rule does not allow it
→ connection hangs, then times out. No error, no log line.The hang rather than a refusal is the signature. A security group blocking traffic and a NACL blocking the return path look identical from the client — both time out — which is why the debug ladder checks them separately.
The debug ladder
Work these in order. Checking randomly is what turns five minutes into three hours.
1. DNS Does the name resolve, and to the address you expect?
dig +short db.internal → 10.0.21.15
2. ROUTE Is there a route from source subnet to destination subnet?
Same VPC → the "local" route covers it.
Different VPC → peering or Transit Gateway, and BOTH route
tables need the entry.
3. SG (out) Does the SOURCE security group allow outbound to the target?
Default is allow-all out, so usually yes — but not if hardened.
4. SG (in) Does the TARGET security group allow inbound on that port from
the source SG or CIDR? This is the answer about 60% of the time.
5. NACL Does the subnet NACL allow inbound on the port AND outbound on
ephemeral ports 1024-65535? Stateless — both are required.
6. LISTENING Is anything actually bound to that port, and on which address?
ss -tlnp → 127.0.0.1:5432 binds loopback only, not the network.
7. HOST FW iptables / nftables / Windows Firewall on the instance itself.The two that account for most real failures are 4 and 6. Step 6 in particular:
ss -tlnp | grep 5432LISTEN 0 244 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=812,fd=6))Bound to 127.0.0.1, so nothing outside the host can connect regardless of how the security
groups are set. Every network-layer check passes and the connection still fails.
The managed tool
aws ec2 create-network-insights-path --source i-0app --destination i-0db \
--destination-port 5432 --protocol tcp --query 'NetworkInsightsPath.NetworkInsightsPathId'
aws ec2 start-network-insights-analysis --network-insights-path-id nip-0123 \
--query 'NetworkInsightsAnalysis.NetworkInsightsAnalysisId'
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids nia-0456 \
--query 'NetworkInsightsAnalyses[0].{path:NetworkPathFound,explanations:Explanations}'{
"path": false,
"explanations": [
{
"Direction": "ingress",
"ExplanationCode": "ENI_SG_RULES_MISMATCH",
"SecurityGroups": [{"Id": "sg-0db"}],
"Port": 5432
}
]
}ENI_SG_RULES_MISMATCH — step 4 on the ladder, found in one call. Naming this tool is a good
answer to “how would you debug this in production”, because it beats reading rules by hand and
it checks the whole path rather than the part you suspected.
VPC Flow Logs are the other one, and they distinguish the two cases the client cannot:
aws logs filter-log-events --log-group-name /vpc/flowlogs \
--filter-pattern '[version, account, eni, source, destination, srcport, destport="5432", ...]' \
--query 'events[0].message' --output text2 123456789012 eni-0abc 10.0.11.20 10.0.21.15 45122 5432 6 3 180 1725966000 1725966060 REJECT OKREJECT means a security group or NACL dropped it. ACCEPT with no application response means
the packet arrived and the process did not answer — steps 6 and 7. That single field splits
the problem in half, and knowing it is a strong signal.
Connecting two VPCs
PEERING TRANSIT GATEWAY PRIVATELINK
topology 1:1 hub and spoke service endpoint
scales to ~125 peers, N² thousands per service
routing manual, both sides centralised none needed
transitive? NO yes n/a
overlapping CIDRs not allowed not allowed allowed
cost data transfer only $0.05/attach-hr $0.01/hr + data
+ $0.02/GB per endpointPeering is not transitive — that is the fact being probed. A peered to B and B peered to C does not let A reach C, and the fix is either a third peering or a Transit Gateway.
The CIDR planning consequence: if two VPCs might ever need to talk, their address ranges must not
overlap. Two VPCs both using 10.0.0.0/16 can never be peered, and re-addressing a live VPC is
a migration, not a change. Allocating non-overlapping ranges up front costs nothing and is the
kind of forethought interviewers listen for.
PrivateLink is the exception worth knowing: it exposes a single service through an endpoint rather than joining networks, so overlapping CIDRs are fine and only the one service is exposed. It is the right answer for “expose our API to a customer’s VPC”.
Subnet sizing
# cidr.py
import ipaddress
vpc = ipaddress.ip_network("10.0.0.0/16")
print(f"VPC {vpc} usable addresses: {vpc.num_addresses:,}\n")
print(f"{'prefix':>8} {'total':>8} {'usable':>8} how many fit in the /16")
for prefix in (20, 22, 24, 26, 28):
net = ipaddress.ip_network(f"10.0.0.0/{prefix}")
fits = vpc.num_addresses // net.num_addresses
print(f" /{prefix} {net.num_addresses:>8,} {net.num_addresses - 5:>8,} {fits:>6,}")$ python cidr.py
VPC 10.0.0.0/16 usable addresses: 65,536
prefix total usable how many fit in the /16
/20 4,096 4,091 16
/22 1,024 1,019 64
/24 256 251 256
/26 64 59 1,024
/28 16 11 4,096Five addresses per subnet are reserved, not two: network, broadcast, and three for the
provider’s DNS, router, and future use. A /28 gives you eleven usable addresses, which sounds
like sixteen until an autoscaling group fails to launch.
The sizing mistake that hurts: a /24 per subnet across six subnets uses 1,536 of 65,536
addresses and looks generous — until a container platform assigns an IP per pod and a single
node consumes 30. Container networking is the reason to size subnets larger than feels
necessary, and saying so unprompted is a good signal.
Recognising it
SYMPTOM CHECK
connection times out (hangs) SG inbound, or NACL return path
connection refused (immediate) nothing listening on that port
works from one subnet, not another route table, or NACL on that subnet
works by IP, fails by name DNS — check the resolver and the zone
outbound works, inbound does not private subnet — that is the design
worked yesterday, fails today an IP changed; SG references a CIDR
two VPCs cannot reach each other peering is not transitive; check both routes
"we need to peer these" first check whether the CIDRs overlap
intermittent failures one AZ misconfigured; check per-AZ resourcesPractice
1. Determine whether a subnet is public.
0.0.0.0/0 → igw-0f1e2d public
0.0.0.0/0 → nat-0a1b private
no 0.0.0.0/0 route isolatedThe name and the auto-assign-public-IP flag are irrelevant. Only the route table decides.
2. Allow inbound 443 in a NACL and leave outbound closed.
Connection hangs, then times out. No error, no log line.NACLs are stateless — the response leaves on an ephemeral port and needs its own rule. The hang is the signature.
3. Check what a service is bound to.
LISTEN 127.0.0.1:5432 — loopback only, nothing external can connectEvery network-layer check passes and the connection still fails. Step 6 on the ladder, and one of the two most common real causes.
4. Read ACCEPT versus REJECT in a VPC flow log.
REJECT → a security group or NACL dropped it
ACCEPT with no response → the packet arrived; the process did not answerThat field splits the problem in half in one query.
Next: IAM and least privilege — how a policy decision is actually evaluated, and the questions that follow.