Primers › Networking

AWS VPCs, subnets, and routing

Most of the subnets I deal with live in AWS, inside a VPC (Virtual Private Cloud), which is your own private network in an AWS region. AWS follows the same CIDR rules as everywhere else and adds a few of its own on top. Inside the VPC, route tables decide where traffic can go and two kinds of firewall decide what’s allowed to get there.

This builds on IP addresses and CIDR, so if /20 or “starts on a multiple of 16” doesn’t mean anything yet, start there.

The VPC and its subnets

• A VPC gets an IPv4 CIDR block between /16 (65,536 addresses) and /28 (16). 10.0.0.0/16 is the common choice, and it’s the largest a single block can be [1].
• A subnet is a slice of the VPC’s range (what a subnet is), and it’s where your resources actually get their addresses. Every instance, database, or load balancer is launched into a subnet and gets an address from that subnet’s range. Subnets also have to be between /16 and /28 [2]. They can’t overlap each other, and each one lives in exactly one availability zone (roughly, one data center in the region) [3].
• You can’t change a subnet’s CIDR after creating it [4]. If a subnet fills up, you make a new one. If the whole VPC fills up, you can add more CIDR blocks to it (up to five by default) and put new subnets in those [1], [5].

The five reserved addresses

AWS takes five addresses out of every subnet, the first four and the last one, instead of the usual two (network and broadcast). In 10.0.1.0/24:

That leaves 2^{(32 − n)} − 5 usable addresses [2]: 251 in a /24, 59 in a /26, and only 11 in a /28, the smallest subnet AWS allows. The first address you can hand out is always the fifth one (10.0.1.4 here).

Route tables

Every subnet is associated with one route table [6]. The packet’s source IP picks the route table, and its destination IP picks the row in it. Whenever something in a subnet sends a packet, the VPC uses that subnet’s route table, looks up the packet’s destination address in it, and the matching row says where to send it next. A route table doesn’t allow or block anything (that’s the firewalls’ job, further down). It only picks the next step.

Source picks the table, destination picks the row

Every packet carries two addresses: its source, where it came from, and its destination, where it’s going. When an instance at 10.0.1.25 sends something to a server at 10.20.5.9, the source is 10.0.1.25 and the destination is 10.20.5.9, and both stay written on the packet the whole way.

The route table uses them for two different jobs:

1. The source IP picks the route table. 10.0.1.25 belongs to one subnet, so the packet uses that subnet’s route table. (Strictly, AWS uses the route table of the subnet the packet leaves from. For an instance sending its own traffic, that’s the subnet its source IP is in, so it comes to the same thing.) The source address never appears inside the table. It’s already settled by which table you’re looking at.
2. The destination IP picks the row. The VPC compares 10.20.5.9 against each row’s Destination column and uses the row that matches (the most specific one, if several do).

So a route table never asks “where did this come from?” It only asks “where is this going, and which way out gets it closer?”

Destination and target

Each route is one row with two columns, and AWS’s names for them are confusing, because in everyday English “destination” and “target” mean the same thing. In a route table they don’t:

• Destination is a range of addresses the packet might be going to, a CIDR block, and it’s compared against the packet’s destination address (not its source). Read it as “if the packet is headed for an address in this range…”
• Target is the next hop: which way to send the packet next, not where it ends up. Read it as “…hand it to this.” It’s usually a gateway or connection out of the VPC, or local, which means “it’s somewhere inside this VPC, deliver it directly.” Most networking equipment outside AWS calls this column “next hop” or “gateway,” which describes it much better.

Driving directions work the same way. If you’re going to Salt Lake City, that’s your destination, and the sign that says “Salt Lake City: take I-15 North” doesn’t take you there. It tells you which road to get on next. When you reach the next junction, another sign tells you the next road. A route table is a set of those signs for one subnet: for addresses in this range, take this way out. The packet keeps its real destination the whole time, and each place it passes through looks it up in its own route table to pick the next hop. A peering connection hands the packet to the other VPC, and that VPC’s route tables take it from there.

Here’s a route table for a subnet in a VPC that uses 10.0.0.0/16, is peered with another VPC that uses 10.20.0.0/16, and has an internet gateway:

0.0.0.0/0 matches every IPv4 address there is, because a /0 locks none of the bits. It’s called the default route, the place traffic goes when nothing more specific applies.

When more than one route matches

Since 0.0.0.0/0 matches everything, almost every packet matches at least two routes: the default route and something more specific. The route table needs a rule for picking one, and the rule is that the route with the longest prefix wins, meaning the biggest number after the slash. AWS calls this longest prefix match [7].

A bigger number after the slash means a smaller, more specific range. /0 is every address, /16 is 65,536 of them, /24 is 256, and /32 is exactly one. So the rule amounts to this: the route that describes the destination most precisely wins.

Sorting mail works the same way. Say there’s one bin for anything going to the US, one for anything going to Utah, and one for anything going to Salt Lake City. A letter for Salt Lake City fits in all three bins, but it goes in the Salt Lake City one, because that’s the most specific. A letter for Denver only fits the US bin, so that’s where it goes. Routes are the same, with CIDR ranges in place of places, and 0.0.0.0/0 as the “anywhere” bin.

Here’s the table above deciding where a packet headed for 10.20.5.9 goes:

Two routes match, and 16 is longer than 0, so the packet goes over the peering connection.

The order of the rows doesn’t matter, only how specific each route is. (Network ACLs work the other way, checking rules in number order, which is one of the reasons the two get mixed up.) This makes it easy to have a broad default and carve out exceptions. If the table also had 10.20.8.0/24 → tgw-…, then 10.20.8.7 would go to the transit gateway (/24 beats /16), while 10.20.5.9 would still go over the peering connection, because it isn’t in 10.20.8.0/24.

Following a packet

Here are three packets leaving an instance at 10.0.1.25 (the source, which is why this subnet’s route table is the one used), each headed for a different destination:

Try any destination address against the same table:

(This version includes the 10.20.8.0/24 route from above, so 10.20.8.7 and 10.20.5.9 end up going different ways.)

Every subnet has its own

• Every route table has the local route for the VPC’s range, and it can’t be deleted. That’s why two subnets in the same VPC can reach each other without you adding anything, as long as the firewalls allow it.
• A subnet you don’t explicitly associate with a route table uses the VPC’s main route table, so changing the main one quietly changes every subnet still relying on it [3].
• A route table only decides where traffic leaving its subnet goes. When the database at 10.0.2.40 replies, the reply leaves the database’s subnet, so the database subnet’s route table decides where the reply goes. Inside one VPC the local route covers that. For traffic from outside the VPC, like a VPN or a peered VPC, the destination’s subnet needs its own route back to wherever the request came from. A missing return route is behind a lot of “the request gets there but nothing comes back” problems, and it’s step 2 of the checklist below.

What the targets are

These are the next hops a route can point at. Each one is a way out of the subnet, and the ID prefix tells you which kind it is when you’re reading a route table in the console.

AWS’s routing options page shows an example route for each [8].

Public and private subnets

AWS has no “public” setting on a subnet. A subnet is public or private because of its route table [3]:

• Public subnet: has a route to an internet gateway, usually 0.0.0.0/0 → igw-…. A resource in it also needs a public IPv4 address (or an Elastic IP) to use that route. With only a private address, it can’t reach the internet even from a public subnet.
• Private subnet: no route to an internet gateway. To reach out (downloading packages, calling an outside API), it sends 0.0.0.0/0 to a NAT gateway. The NAT gateway sends the traffic out from its own public address and passes the replies back, but nothing on the internet can start a connection into the private subnet through it [9].
• VPN-only subnet: has a route to a VPN connection and none to the internet.
• Isolated subnet: no routes outside the VPC at all, only local.

NAT gateways come in two kinds. The standard kind, which AWS now calls a zonal NAT gateway, lives in one availability zone and sits in a public subnet, so the usual setup is one per zone, with each zone’s private subnets routing to the NAT gateway in the same zone. Then losing a zone only takes out that zone’s internet access [10]. A regional NAT gateway is newer: one NAT gateway that spreads across availability zones on its own as your workloads appear in them, and doesn’t need a public subnet at all [11]. Either way, NAT gateways charge for the data they process, which is why traffic to S3 usually gets its own more specific route to a gateway VPC endpoint instead of going through the NAT [8].

Security groups and network ACLs

A VPC has two layers of firewall, and they behave differently enough that mixing them up causes a lot of confusion.

Security groups attach to a resource’s network interface: an EC2 instance, a load balancer, a database, a Lambda function running in the VPC.

• They’re stateful. If a request is allowed in, the reply is allowed back out automatically, so you only write rules for whoever starts the connection [12].
• They only have allow rules. Anything no rule allows is dropped.
• A new security group allows nothing in and everything out.
• A rule’s source can be a CIDR range or another security group. “Allow port 5432 from the app servers’ security group” keeps working as app servers come and go, with no addresses to keep up to date.

Network ACLs attach to a subnet, and every subnet has exactly one. NACL is short for network access control list (people usually say it like “nackle”). An access control list, or ACL, is just what it sounds like: a list of rules, each one saying a kind of traffic (a protocol, a port range, and an address range) is allowed or denied, checked in order from the top. The term comes from older routers and firewalls, and the “network” in front distinguishes AWS’s subnet-level version from other ACLs in AWS, like the ones on S3 buckets.

• They’re stateless. Every packet is checked on its own, so replies need their own rules [13].
• Rules are numbered and checked from the lowest number up. The first match decides, and a final * rule denies anything nothing else matched [14].
• Rules can deny as well as allow, which makes NACLs useful for blocking a specific range.
• The VPC’s default NACL allows everything in both directions, which is why many VPCs effectively run on security groups alone. A NACL you create yourself denies everything until you add rules [15], [16].
• They only check traffic entering or leaving the subnet, not traffic between two resources inside it.

Because NACLs are stateless, the reply side catches people out. A reply goes back to whatever port the client picked for its end of the connection, called an ephemeral port, and different clients pick from different ranges. Linux usually uses 32768–61000, newer Windows uses 49152–65535, and NAT gateways, load balancers, and Lambda use 1024–65535. So a web server’s subnet needs an inbound NACL rule for port 443 and an outbound rule for 1024–65535 (AWS’s suggested catch-all) to let the replies out [14].

Traffic coming into a subnet passes the NACL first and then the resource’s security group, and it has to get through both [17]. Neither one filters traffic to the VPC’s own DNS server (the base address plus two) or the instance metadata service, so you can’t block those with either [13], [12].

Which one to use

Security groups, nearly always. AWS says so directly: “in most cases, security groups can meet your needs,” and NACLs are there “if you want an additional layer of security” [3]. AWS’s comparison of the two says the same: you can secure instances with security groups alone and add NACLs as an extra layer [17]. Plenty of VPCs leave the default NACL, which allows everything, alone for good. In practice, NACLs get used for a few specific jobs a security group can’t do:

1. Denying something. Security groups can only allow. If one address range is causing trouble and everything else should still get in, that takes a NACL deny rule. (For web traffic, a WAF is usually a better place to block, since it can see the requests.)
2. A rule for a whole subnet that no security group can override. “The database subnets only accept traffic from the app subnets’ range” is a good NACL. Even if someone attaches a wide-open security group to a database by mistake, the NACL still blocks everything else. It’s a backstop, and it’s often owned by a different team than the security groups are.
3. A requirement to separate subnets at the network level, which some compliance frameworks ask for.

NACLs make a poor main firewall. They’re stateless, so you manage reply ports yourself. They only understand address ranges, so you can’t say “from the app servers’ security group.” They allow only 20 inbound and 20 outbound rules by default, and 40 each at most, compared with 60 each per security group [5]. And they apply to everything in the subnet at once, so one mistake breaks every resource in it.

An apartment building is a decent comparison. The security group is the lock on each apartment’s door, set up for whoever lives there. The NACL is the guard at the building’s front gate with one list for everybody, which is useful for keeping certain people out of the whole building and no good for deciding who gets into apartment 4B.

When traffic doesn’t get through

Most “can’t connect” problems in a VPC come down to one of these, and the return path is the one that gets missed:

1. A route there. The source’s subnet has a route that covers the destination.
2. A route back. The destination’s subnet has a route back to the source. For anything coming from outside the VPC (a VPN, a peered VPC), this is a separate route you have to add, and without it the request arrives but the reply has nowhere to go.
3. The NACLs, both ways. On both subnets, including the outbound rule for ephemeral ports.
4. The security group on the destination. It allows the source’s range (or security group) on the right port.
5. Something is actually listening. The service is running and listening on that address and port, not only on localhost.

Planning a layout

A common pattern is a /16 VPC with a public subnet and a private subnet in each of three availability zones. Public subnets hold the few things that face the internet, mostly load balancers and zonal NAT gateways, so they can stay small. Private subnets hold everything else and should be much bigger:

Each /20 starts on a multiple of 16 in the third octet, which is the boundary rule from the CIDR primer. 10.0.3.0 through 10.0.15.255 and everything from 10.0.64.0 up are left empty on purpose, so new subnets have room later without any renumbering.

Private subnets need room because more things use VPC addresses than you’d expect. Every EC2 instance, load balancer node, database, and Lambda network interface takes at least one. On EKS with the default networking setup, every Kubernetes pod takes its own VPC address too [18], so a cluster can run through a /24 surprisingly quickly.

Don’t overlap with networks you’ll connect to

If a VPC will ever be connected to another VPC (peering, a Transit Gateway) or to an office network over a VPN, their ranges can’t overlap. A route table has no way to tell which 10.0.1.0/24 you mean, and VPC peering refuses overlapping ranges outright [19]. Giving every VPC 10.0.0.0/16 works fine until the day two of them need to talk, so give each VPC its own range from the start, like 10.0.0.0/16, 10.1.0.0/16, 10.2.0.0/16, and so on.

(The default VPC AWS creates in every region uses 172.31.0.0/16, with a /20 subnet in each availability zone [20]. It’s handy for experiments. Leave 172.31.0.0/16 out of the VPCs you plan yourself, so they never overlap with a default VPC you might want to connect to later.)

References

Every source cited across the primers is collected on one page.