Primers › Networking

Reaching private resources

Databases, internal tools, and Kubernetes nodes live in private subnets on purpose, so nothing on the internet can reach them. People still need to, though, and so do other networks. The options run from permanent links between whole networks down to a tunnel to one port that lasts as long as a terminal window.

This leans on AWS VPCs, subnets, and routing, especially route tables and the checklist for when traffic doesn’t get through.

The options at a glance

Each one gets a section below, in that order (PrivateLink is covered under VPCs in different AWS accounts), and the last section covers what to check when you’re connected and the database still doesn’t answer.

Connecting whole networks

These connect one network to another, so everything on one side can reach (whatever the firewalls allow on) the other side. Peering and Transit Gateway also work between VPCs that belong to different AWS accounts.

Site-to-site VPN

A site-to-site VPN is an encrypted tunnel between two whole networks, usually an office or data center and AWS. A tunnel is a packet wrapped inside another packet. The original packet, still addressed from one private address to another, becomes the payload of a new packet addressed between the two VPN devices, and the far device unwraps it and sends the original on. It’s the same encapsulation every layer of the network already does, with a second IP header on the outside [7].

AWS’s site-to-site VPN uses IPsec for the tunnel [8]. IPsec works at the IP layer, so it encrypts every packet between the two networks whatever program sent it [7], and any protocol can ride inside: HTTP, database connections, anything. TLS, the encryption behind HTTPS (see the TLS handshake), works differently: it protects one application’s connection, set up by that application [9]. A site-to-site VPN protects all the traffic between two places, and TLS protects one conversation wherever it goes.

In AWS, the far end is described by a customer gateway (the office’s VPN device), and the AWS end is a virtual private gateway on one VPC or a Transit Gateway. Each VPN connection comes with two tunnels that end in different availability zones, and the office device should have both up, because AWS takes one down from time to time for maintenance [8], [10]. The routes for the office’s ranges are either typed in as static routes or learned over BGP (Border Gateway Protocol, which lets the office device announce its ranges itself), and AWS recommends BGP when the device supports it because its checks help traffic fail over to the second tunnel [11].

# in account 111111111111: ask to peer VPC A with VPC B in the other account
aws ec2 create-vpc-peering-connection --vpc-id vpc-aaaa1111 \
  --peer-vpc-id vpc-bbbb2222 --peer-owner-id 222222222222

# in account 222222222222: accept it, using the pcx- ID from the request
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id pcx-0123456789abcdef0

# in account 111111111111: ask to peer VPC A with VPC B in the other account
aws ec2 create-vpc-peering-connection --vpc-id vpc-aaaa1111 `
  --peer-vpc-id vpc-bbbb2222 --peer-owner-id 222222222222

# in account 222222222222: accept it, using the pcx- ID from the request
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id pcx-0123456789abcdef0

Connecting people: client VPN

A client VPN connects one person’s laptop to a network instead of connecting two networks. AWS has its own (AWS Client VPN), and plenty of teams run their own VPN server (OpenVPN, WireGuard, and so on) in a VPC instead.

Either way, the laptop gets an address from the VPN’s own client range, a block set aside for VPN users, like 10.250.0.0/22. It can’t overlap with the networks it connects to, and AWS Client VPN requires it to be between a /22 and a /12 [26]. In AWS Client VPN, the endpoint is the resource you create that every VPN session ends at [27]. Associating it with a subnet puts network interfaces for it in that subnet, with addresses from the subnet’s range [28].

Which source address the destination sees

A reply goes to whatever source address the packet carried when it arrived, and VPNs differ on what that address is:

The database answers whatever source address it sees. With AWS Client VPN that's an address inside the VPC, and with a VPN that passes the client address through it's an address from the client range, which the VPC knows nothing about until you add a route for it.

• AWS Client VPN translates the address. It applies source NAT (network address translation, rewriting the packet’s source address) as traffic leaves the endpoint’s network interface, so the database sees 10.0.1.25, not the laptop’s 10.250.0.7 [28]. Inside that VPC no extra return route is needed, because the VPC’s local route covers the interface’s address. The database’s security group allows the endpoint’s subnet (or the endpoint’s security group), not the client range. AWS Client VPN also has its own authorization rules (which client groups may reach which ranges) and its own route table, and both have to allow a destination before traffic even leaves the VPN [29], [28].
• Many self-run VPNs pass the client address through. The database sees 10.250.0.7 itself. Then every subnet the client needs to reach has to have a route for 10.250.0.0/22 pointing at the VPN instance, and every security group has to allow 10.250.0.0/22. The VPN instance also needs its source/destination check turned off. By default AWS checks that an instance is the source or the destination of all the traffic it handles, and a VPN server forwarding the laptop’s packets is neither [4].

That difference explains a lot of “I’m connected but can’t reach anything.” The right return route and security group rule depend on which address the destination actually sees, and the last section walks through a full example.

One port, for now: port forwarding

Sometimes you only need to reach one database for an afternoon, and a VPN is more than the job needs. Port forwarding makes a port on your laptop connect through something that’s already inside the network.

Through a Kubernetes cluster

If you can already reach a Kubernetes cluster inside the VPC with kubectl, the cluster can act as the way in. kubectl port-forward can only forward to a pod, or to a pod picked by a service or deployment, not to an arbitrary host like a database [30]. So the first step runs a tiny relay pod that forwards to the database, and the second forwards a local port to that pod:

# terminal 1: a throwaway pod that relays port 5432 to the database (leave it running)
kubectl run pg-proxy --rm -i --restart=Never --image=alpine/socat -- \
  TCP-LISTEN:5432,fork,reuseaddr \
  TCP:my-db.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432

# terminal 2, once the pod is running: connect local port 5433 to the pod's port 5432
kubectl port-forward pod/pg-proxy 5433:5432

# window 1: a throwaway pod that relays port 5432 to the database (leave it running)
kubectl run pg-proxy --rm -i --restart=Never --image=alpine/socat -- `
  TCP-LISTEN:5432,fork,reuseaddr `
  TCP:my-db.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432

# window 2, once the pod is running: connect local port 5433 to the pod's port 5432
kubectl port-forward pod/pg-proxy 5433:5432

Then the database client connects to localhost on port 5433. The connection takes four hops, and only the last two happen inside the VPC:

The teal hop is the encrypted connection to the API server that kubectl already uses for everything else. The database itself is only ever reached from the pod, inside the VPC.

The pieces of the first command:

• kubectl run pg-proxy creates a pod named pg-proxy from the alpine/socat image [31].
• --rm deletes the pod after it exits. It only works when kubectl is attached to the pod, and -i attaches it [31].
• -i (--stdin) keeps the pod’s input open and attaches kubectl to it, so the command keeps running for as long as the pod does [31].
• --restart=Never sets the pod’s restart policy. The default is Always, which would start socat again if it stopped. Never leaves it stopped, which suits a throwaway pod [31].
• -- ends kubectl’s own options. Everything after it goes to the container as arguments to its default command [31], and this image’s default command is socat [32].
• socat (SOcket CAT) connects two streams of bytes, and takes two addresses [33]:
  • TCP-LISTEN:5432 listens on port 5432 inside the pod.
  • fork handles each new connection in its own child process and keeps listening, so more than one connection (and a reconnect) works [33].
  • reuseaddr is harmless to include, and recent socat versions set it on listening addresses already [33].
  • TCP:my-db…:5432 connects each one to the database on port 5432. The xxxxxxxxxxxx and us-east-1 are placeholders for your own database’s endpoint name and region, which the RDS console shows.
• Two terminals, because the first command stays attached to the pod and keeps running. Start the second once the pod is up (kubectl get pod pg-proxy shows Running).

And the second:

• kubectl port-forward pod/pg-proxy 5433:5432 listens on local port 5433 and forwards each connection to port 5432 in the pod. The order is always local first, then the pod’s [30]. The traffic travels through the encrypted connection to the Kubernetes API that kubectl already has, and the API server passes it to the pod through the node’s kubelet, so the database never has to be reachable from outside the VPC [34], [35].
• Local port 5433 instead of 5432, so it doesn’t clash with a Postgres you might have running locally.

It only lasts as long as both commands keep running. Stop the port forward with Ctrl+C, then stop the first command, and --rm deletes the pod. If a pod outlives it anyway, kubectl delete pod pg-proxy removes it. The database’s security group still has to allow traffic from the pod, which usually means allowing the cluster nodes’ security group.

Through Session Manager or SSH

The same idea works with other ways in:

• AWS Systems Manager Session Manager can forward a local port through an EC2 instance to another host in the VPC, with no SSH keys and no inbound ports open on the instance [36], [37]:

  macOS / Linux

  aws ssm start-session --target i-xxxxxxxxxxxxxxxxx \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["my-db.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["5433"]}'

  Windows (PowerShell)

  aws ssm start-session --target i-xxxxxxxxxxxxxxxxx `
    --document-name AWS-StartPortForwardingSessionToRemoteHost `
    --parameters 'host=my-db.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com,portNumber=5432,localPortNumber=5433'

  The pieces:

  • --target is the instance the session goes through, by its instance ID. i-xxxxxxxxxxxxxxxxx is a placeholder for yours [38].
  • --document-name picks what kind of session to start. In Systems Manager a document isn’t a file you pass in. It’s a named, predefined set of actions that Systems Manager runs, and the ones starting with AWS- are written and maintained by AWS [39]. AWS-StartPortForwardingSessionToRemoteHost forwards a port to some other host [37].
  • host is the database’s name (or IP address), which the instance has to be able to resolve and reach. portNumber is the port on that host, and localPortNumber is the port on your laptop, so the database client connects to localhost:5433 [37].

  A few things have to be in place first. Your laptop needs the AWS CLI and its Session Manager plugin installed [37]. The instance needs the SSM Agent running, recent enough to support port forwarding to a remote host [37], plus outbound HTTPS to the Systems Manager endpoints and permission to talk to Systems Manager, usually through an instance profile (the IAM role attached to the instance) with the AmazonSSMManagedInstanceCore policy [40], [41]. The database’s security group has to allow the instance, because the instance opens the connection to it.

  The PowerShell version uses the AWS CLI’s shorthand instead of JSON, because Windows PowerShell 5.1 drops the JSON’s inner double quotes [42].

• An SSH bastion host does the same with ssh -L, at the cost of keeping an SSH port open and managing keys. Forwarding a port through a bastion walks through the command.

Connected but can’t reach it

Work through the path in order, from the laptop to the resource and back. The steps marked general are the same checks as for any traffic in a VPC, covered in when traffic doesn’t get through.

1. The VPN allows it. For AWS Client VPN, an authorization rule covers the destination range, and the endpoint’s route table has a route for it. For a self-run VPN, the laptop sends that range into the tunnel at all.
2. A route toward it (general). From the VPN’s network to the destination, through the peering connection or Transit Gateway if it’s in another VPC.
3. The Transit Gateway path, if there is one. The destination VPC’s attachment has a subnet in the destination’s availability zone, and the Transit Gateway’s route tables route both directions.
4. A route back (general). Every subnet the destination lives in has a route for the source range (the client range or the VPN’s own range, depending on the VPN) pointing back the way it came. This one is easy to miss, especially when an environment has more subnets than the ones you checked.
5. The NACLs, both ways (general). Including ephemeral ports on the way back: the port the client picked for its end of the connection, which the reply is addressed to.
6. The security group on the destination (general). It allows the source the destination actually sees, on the right port.
7. The name resolves to the private address. If the hostname only resolves inside the VPC (a private hosted zone), the laptop’s queries have to reach a resolver that can see the zone, and the zone has to be associated with the VPC that resolver answers for. The VPC’s own .2 resolver is only for queries from inside the VPC, so when the VPN’s clients reach it from outside (a self-run VPN that passes the client address through, say), the VPN should hand out a Route 53 Resolver inbound endpoint, an address in the VPC that accepts queries from outside it, or a DNS forwarder running inside the VPC [48], [49], [50].
8. Something is listening (general). On that address and port, not just on localhost.

A worked example

A self-run VPN server runs on an instance at 10.0.1.30 in VPC A (10.0.0.0/16) and hands out addresses from 10.250.0.0/22 without translating them. A laptop on the VPN at 10.250.0.7 tries to reach a Postgres database at 10.1.20.10:5432 in VPC B (10.1.0.0/16), and both VPCs are attached to one Transit Gateway. The connection hangs. Going down the list:

Four fixes. Three are on the way back to an address from the client range, and the fourth is DNS, for the same reason: the laptop’s queries arrive from outside the VPC. With AWS Client VPN in the same spot, the database would see the endpoint’s address in VPC A, and the existing 10.0.0.0/16 routes and security group rule would already cover the reply. The endpoint would only need a route and an authorization rule for 10.1.0.0/16, and its DNS setting could be 10.0.0.2, as long as the zone is associated with VPC A [51].

References

Every source cited across the primers is collected on one page.