IPSec VPN Toolkit
Updated Apr 13, 2026
A comprehensive IPSec VPN reference and troubleshooting tool for network engineers. Covers IKE Phase 1 and Phase 2 mechanics, a rule-based troubleshooting engine (tunnel down, phase 2 only, no traffic, flapping), vendor command references for Cisco and StrongSwan, common failure scenario cards, a definitions glossary, and example site-to-site configs. No AI — pure deterministic logic trees.
What is IPSec?
IPSec (Internet Protocol Security) is a suite of protocols that encrypts and authenticates IP traffic at the network layer. Unlike TLS which operates at the application layer, IPSec protects all IP packets flowing between two endpoints — making it transparent to applications.
It is primarily used in site-to-site VPNs (connecting two offices or data centers) and remote access VPNs (connecting individual users to a corporate network). The protocol supports both IKEv1 and the newer, more efficient IKEv2.
The core principle: two peers negotiate a Security Association (SA) — an agreement on which algorithms to use — and then encrypt all traffic between them using ESP (Encapsulating Security Payload). The negotiation itself happens over IKE (Internet Key Exchange) in two phases.
Primary Use
Site-to-site & remote access VPNs
Transport Protocol
UDP 500 (IKE), UDP 4500 (NAT-T), ESP (IP proto 50)
Versions
IKEv1 (legacy), IKEv2 (modern default)
Phase Breakdown
Establish a secure control channel
Before encrypting real traffic, both peers must authenticate each other and agree on how to protect further negotiation. This creates the ISAKMP SA.
Encrypt actual user traffic
Using the secure channel from Phase 1, peers negotiate IPSec SAs that define exactly how to encrypt and authenticate data traffic. Two SAs are created — one per direction.
Troubleshooting Engine
Tunnel completely down — Phase 1 failing
Confirm peer IP is reachable
Run ping and traceroute to the remote peer IP. A timeout here means a routing or firewall issue upstream of IKE.
Verify UDP 500 and 4500 are open
IKE uses UDP 500 for negotiation. NAT-T uses UDP 4500. Both must be permitted inbound and outbound on all firewalls between peers.
Check IKE version mismatch
If one side runs IKEv1 and the other IKEv2 only, the tunnel will never come up. Confirm both peers are configured for the same IKE version.
Verify Pre-Shared Key matches exactly
PSKs are case-sensitive and must match character-for-character on both sides. A single space or capitalization difference causes auth failure.
Check ISAKMP policy mismatches
Encryption algorithm, hash/PRF algorithm, DH group, and SA lifetime must all match the remote peer's policy. Even a lifetime mismatch can cause failure on strict implementations.
Confirm clock sync (NTP)
Certificate-based auth requires synchronized clocks. A skew greater than 5 minutes will cause cert validation to fail. Check NTP status on both peers.
Verify NAT-T configuration
If either peer is behind NAT, NAT-T (UDP 4500) must be enabled and permitted. NAT breaks ESP packets without it.
Command Reference
Common Failure Scenarios
Common Causes
- PSK mismatch — check case sensitivity
- ISAKMP policy mismatch (encryption, hash, or DH group)
- UDP 500 blocked by firewall between peers
- Wrong peer IP configured
Key Test
Check `show crypto isakmp sa` — if no SA exists, Phase 1 never started (routing/firewall issue). If MM_NO_STATE, negotiation started but failed (policy/PSK mismatch).
Common Causes
- SA lifetime mismatch causing rekey timing conflicts
- DPD misconfigured with too-short interval
- ISP packet loss causing IKE keepalive timeouts
- NAT session table timing out UDP 500/4500
Key Test
Note the interval precisely — if it matches the SA lifetime exactly, it's a rekey issue. If irregular, look at DPD or ISP packet loss.
Common Causes
- Asymmetric routing — return traffic takes a different path
- NAT exemption missing on one side
- Firewall ACL blocking return traffic post-decryption
- Crypto map not applied to correct interface on one peer
Key Test
Ping from both sides — identify which direction works. Check packet counters: `show crypto ipsec sa` — one direction should show encrypt/decrypt counts while the other shows zeros.
Common Causes
- MTU fragmentation causing retransmits
- CPU-bound encryption on VPN device (check process CPU)
- ISP congestion on the path to peer
- PFS rekey overhead at regular intervals
Key Test
Ping with large packets (`ping -s 1400 <host>`) — if large pings are significantly slower, MTU/fragmentation is the culprit. Compare base latency to peer IP vs. latency through the tunnel.
Common Causes
- Crypto ACL too narrow — missing subnets or ports for the app
- MTU/fragmentation only affecting large payloads (HTTP works, large file transfers fail)
- DNS not resolving through VPN — DNS servers not reachable via tunnel
- App uses source IP not covered by the tunnel's proxy IDs
Key Test
Capture the failing app's traffic and identify source/destination IPs and ports. Verify those are covered by the crypto ACL. Run `show crypto ipsec sa` — look for ACL hits on the interesting traffic list.
Common Causes
- Certificate expired or not yet valid
- Clock skew >5 minutes between peers
- Wrong CA chain — peer cert not signed by a trusted CA
- CRL/OCSP server unreachable — cert revocation check failing
- Distinguished Name (DN) mismatch in the IKE identity
Key Test
Check certificate expiry dates first. Verify system clocks are synchronized (NTP). Test CRL endpoint reachability from both peers. Enable debug to see the exact cert validation error message.
Definitions Glossary
The protocol used to set up IPSec Security Associations. IKEv1 uses Main Mode + Quick Mode; IKEv2 is simpler, more efficient, and the modern default. IKE runs over UDP port 500.
Internet Security Association and Key Management Protocol — the framework that defines how SAs are negotiated. Often used interchangeably with IKE, though technically IKE is the protocol that runs on top of ISAKMP.
A one-way agreement between two peers on how to encrypt and authenticate traffic. IPSec requires two SAs per tunnel — one for each direction. SAs are identified by their SPI value.
A 32-bit number included in ESP/AH packet headers that identifies which SA the receiver should use for decryption. Each SA has a unique SPI.
When enabled, each Phase 2 SA generates fresh Diffie-Hellman keys independent of Phase 1. Compromise of one session key cannot be used to decrypt other sessions. Recommended for sensitive tunnels.
Wraps ESP packets in UDP (port 4500) so IPSec traffic can pass through NAT devices. Required when either peer is behind NAT, since NAT breaks raw ESP packets (IP protocol 50).
A keepalive mechanism that sends R-U-THERE messages to detect when the remote peer has gone away. When a peer is detected dead, DPD tears down the tunnel. dpdaction controls what happens next (clear, hold, or restart).
The IPSec protocol that provides encryption and authentication of packet payloads. Used in virtually all modern IPSec tunnels. ESP is IP protocol number 50.
Authenticates packets but does NOT encrypt them. Rarely used today because it is incompatible with NAT and provides no confidentiality. ESP-only is the standard approach.
Defines the mathematical algorithm used for key exchange. Higher group numbers = stronger security + more CPU. Group 14 (2048-bit) is the minimum recommended; Groups 19/20 (ECDH) are preferred. Avoid Groups 1, 2, and 5.
Example Configurations
crypto isakmp policy 10 encr aes 256 hash sha256 authentication pre-share group 14 lifetime 86400 ! crypto isakmp key MYSHAREDKEY address 203.0.113.1 ! crypto ipsec transform-set MYSET esp-aes 256 esp-sha256-hmac mode tunnel ! crypto map MYMAP 10 ipsec-isakmp set peer 203.0.113.1 set transform-set MYSET match address VPN-ACL ! interface GigabitEthernet0/0 crypto map MYMAP ! ip access-list extended VPN-ACL permit ip 10.1.0.0 0.0.255.255 10.2.0.0 0.0.255.255
conn site-to-site keyexchange=ikev2 left=198.51.100.1 leftsubnet=10.1.0.0/16 leftauth=psk right=203.0.113.1 rightsubnet=10.2.0.0/16 rightauth=psk ike=aes256-sha256-modp2048! esp=aes256-sha256! auto=start dpdaction=restart dpddelay=30s
1. Define interesting traffic (what to encrypt) - ACL or traffic selector matching source/destination subnets 2. Configure Phase 1 policy (IKE SA) - Encryption: AES-256 | Hash: SHA-256 | Auth: PSK or cert - DH Group: 14+ | Lifetime: 86400s (24h) 3. Configure Phase 2 transform set (IPSec SA) - Protocol: ESP | Encryption: AES-256 | Hash: SHA-256 - PFS: enabled (DH Group 14+) 4. Bind Phase 1+2 into a crypto map / connection profile - Reference the peer IP, transform set, and ACL 5. Apply crypto map to WAN interface - Must be the interface with the public IP facing the peer 6. Add NAT exemption - Traffic matching the VPN ACL must bypass NAT/PAT 7. Verify routing - Route to remote subnet must exist (static or dynamic) 8. Monitor - Check SA counts and packet encrypt/decrypt counters - Alert on zero-counter SAs (tunnel up, no traffic)