What is IPsec?
IPsec protects IP packets themselves. Two gateways, or a laptop and a gateway, first run IKEv2 to prove who they are and agree on keys. Then every packet between the chosen networks is wrapped in ESP: encrypted, checked for changes and numbered against replay.
It is built into routers, firewalls, phones and operating systems, which is why companies use it to join offices together and to connect remote staff. Because it works at the IP layer, every app benefits without any change.
Think of it like this. IPsec is like an armoured van service between two bank branches. First the two branch managers meet, check each other's badges and agree on a lock code (IKE). After that, every bag that goes between the branches rides in a sealed, numbered box (ESP), and the outside of the van shows only the two branch addresses.
Step by step
Normal
4 stepsTwo office gateways build an IPsec tunnel with IKEv2, then carry traffic as ESP.
Normal
4 stepsTwo office gateways build an IPsec tunnel with IKEv2, then carry traffic as ESP.
- 1
Traffic needs the tunnel
A packet for the other office
A branch PC sends a packet to 10.2.0.20. The branch gateway's policy says traffic from 10.1.0.0/24 to 10.2.0.0/24 must be protected, and no tunnel exists yet.
- 2
IKE_SA_INIT
Agree on crypto and swap keys
One request and one response on UDP 500. They choose algorithms, do a Diffie-Hellman exchange, swap nonces and check for NAT. The result is an encrypted IKE session, but neither side has proved who it is yet.
- 3
IKE_AUTH
Prove identity, make the data tunnel
Inside encryption, each side sends its identity and an AUTH value made with the pre-shared key or a certificate. They also agree on the ESP settings and which networks go through the tunnel (traffic selectors). This creates the first child SA.
- 4
ESP data
Packets travel sealed
The gateway wraps the PC's whole packet in ESP, encrypts it and sends it to the other gateway, which unwraps it and forwards it to the server. Replies come back the same way.
NAT traversal
4 stepsA laptop behind a home router: IKE spots the NAT and moves everything to UDP 4500.
NAT traversal
4 stepsA laptop behind a home router: IKE spots the NAT and moves everything to UDP 4500.
- 1
IKE_SA_INIT on 500
NAT detection hashes
The laptop hashes its own address 192.168.1.20 and port 500 into the NAT detection payload. The home router rewrites the outside to 203.0.113.50 and another port on the way.
- 2
NAT found
Hashes do not match
The gateway hashes the address it actually sees and gets a different value, so it knows NAT is in the way. Plain ESP (protocol 50) has no ports, so many NAT routers cannot pass it.
- 3
Move to UDP 4500
IKE_AUTH and ESP in UDP
The laptop switches to port 4500 for IKE_AUTH. IKE messages there start with four zero bytes (the non-ESP marker). ESP packets go inside UDP 4500 too, and start with a non-zero SPI, so the receiver can tell them apart.
- 4
Keepalive
One byte, 0xFF
When idle, the laptop sends a 1-byte NAT keepalive (0xFF) to UDP 4500, often every 20 seconds, so the home router does not forget the mapping.
Wrong key
3 stepsOne gateway has a mistyped pre-shared key, so IKE_AUTH fails and no tunnel is built.
Wrong key
3 stepsOne gateway has a mistyped pre-shared key, so IKE_AUTH fails and no tunnel is built.
- 1
IKE_SA_INIT works
Encryption is agreed
The first exchange does not use the pre-shared key, so it succeeds as usual.
- 2
AUTH does not match
AUTHENTICATION_FAILED
The branch's AUTH value was made with a different key. The head office computes its own and they differ, so it answers with the AUTHENTICATION_FAILED notify (24) and deletes the session.
- 3
Nothing gets through
Traffic is dropped, not sent in the clear
Without a tunnel, the branch gateway drops packets for 10.2.0.0/24 instead of sending them unprotected. The logs on both sides name the failure, so an admin can fix the key.
Myths and mix-ups
- IPsec is not one protocol; it is IKE for keys plus ESP (or the rarely used AH) for data.
- IPsec does not break with NAT anymore; NAT traversal moves it into UDP 4500.
- The pre-shared key is never sent across the network; only a value made from it is.
- IPsec is not only for site to site links; phones and laptops have IKEv2 VPN clients built in.
Try it yourself
Run these on your own computer to see IPsec at work.
See active tunnels (Linux)
sudo ip xfrm stateLists each security association with its SPI, addresses and algorithms.
See tunnel policies (Linux)
sudo ip xfrm policyShows which source and destination networks must use IPsec.
Watch IKE and ESP
sudo tcpdump -n 'udp port 500 or udp port 4500 or ip proto 50'Start a VPN connection and watch IKE_SA_INIT, IKE_AUTH and then ESP packets.
IPsec at a glance
- Layer
- Network (3)
- Ports
- UDP 500 (IKE), UDP 4500 (behind NAT)
- ESP
- IP protocol 50, no ports
- Modes
- Tunnel (whole packet) and transport
- Key exchange
- IKEv2: IKE_SA_INIT then IKE_AUTH
- Login
- Pre-shared key, certificates or EAP
- Ciphers
- AES-GCM, ChaCha20-Poly1305
- IKEv2 standard
- RFC 7296 (2014)
Defined inRFC 7296 (IKEv2)RFC 4301 (IPsec architecture)RFC 4303 (ESP)RFC 3948 (ESP in UDP for NAT)RFC 4106 (AES-GCM in ESP)RFC 8031 (Curve25519 in IKEv2)
Header fields
What IPsec puts in its messages, with example values from the stories above.
IPv4
- Protocol
6 (TCP)- Source
10.1.0.10- Destination
10.2.0.20
IPv4 (outer)
- Protocol
50 (ESP)50 means ESP. It is not TCP or UDP, so it has no ports, which is why NAT routers struggle with it.- Source
203.0.113.10- Destination
198.51.100.1
ESP
- SPI
0xc1a2b3d4Points to the child SA, which says which keys decrypt this packet.- Sequence number
1Rises by one each packet. The receiver rejects repeats, so old packets cannot be replayed.- IV
8 bytes- Encrypted payload
10.1.0.10 > 10.2.0.20, TCP SYN to 443In tunnel mode, the whole original IP packet, addresses included, is hidden.- Next header
4 (IPv4), inside the encryption- ICV
16 bytes (AES-GCM tag)Integrity check. Any change and the packet is dropped.
Test yourself
Which IKEv2 exchange proves the identity of each side?
Questions people ask
What ports does IPsec use?
UDP 500 for IKE and UDP 4500 when NAT is in the way. ESP itself is IP protocol 50, not a port. A firewall must allow all three.
What is the difference between IKE and ESP?
IKE (Internet Key Exchange) does the handshake: it checks identities and makes keys. ESP carries the actual data, encrypted with those keys.
Is IPsec better than WireGuard?
IPsec is older, very widely supported in business gear and highly configurable. WireGuard is smaller and simpler. Both are secure when set up with modern algorithms.
What does NO_PROPOSAL_CHOSEN mean?
The two sides could not agree on algorithms, for example one wants AES-GCM and the other only offers AES-CBC. Make the IKE and ESP proposals match on both ends.
Does an IPsec VPN encrypt everything?
It encrypts the traffic its policy covers, called traffic selectors. A split tunnel sends only office traffic through IPsec and the rest straight to the internet.