In my previous post about using netcat to test ISAKMP, I validated that UDP/500 was open to the destination, and that semi-valid ISAKMP headers were being received by the peer (a Cisco ASAv in this case). However, this validation required me to be knowledgeable in troubleshooting the Cisco ASA platform so that I could easily determine if my pseudo-ISAKMP packets arrived (and also required that I control or have access to the ASA). What if the peer is outside of your control or is a platform you are not at all familiar with? In that case, what I want to do now is send a complete ISAKMP initial communication and receive a reply from the peer. This test serves the dual purpose of removing the need for debugs or logging on the peer and validates the return path (and any stateful/L7 inspection) as well.

What You’ll Learn#

This tutorial shows you how to:

  • Use a pre-crafted ISAKMP Phase 1 packet to test VPN connectivity
  • Test VPN connectivity without needing access to the remote device
  • Interpret ISAKMP responses to diagnose connectivity issues
  • Troubleshoot VPN problems using only command-line tools

Prerequisites#

Before starting, you’ll need:

  • Basic understanding of VPN/ISAKMP concepts (see Part 1)
  • netcat (nc) installed on your system
  • hexdump for analyzing responses (optional but recommended)
  • Target IP address of an ISAKMP/IKE peer to test

Important Security Considerations#

Authorization Required: Only test ISAKMP endpoints you own or have explicit written permission to test. Unauthorized VPN probing may:

  • Trigger security alerts and incident response
  • Violate computer fraud laws (CFAA in US, similar laws globally)
  • Result in IP blocking or legal action

Operational Impact: Each test attempt will:

  • Generate syslog entries on the target device
  • Potentially trigger IDS/IPS signatures
  • Consume VPN gateway resources
  • May count against rate-limiting thresholds

In Part 1, I showed how to send a minimal ISAKMP header to validate UDP/500 connectivity. That approach required debug access to the peer to confirm packet receipt. For this follow-up, I wanted a solution that works without peer access - which means sending a complete, valid ISAKMP packet that generates a response.

Rather than hand-crafting every byte again, I based this packet on an actual captured ISAKMP conversation. If you don’t have a capture handy, you can find ISAKMP protocol details and examples at the Wireshark ISAKMP Protocol page.

Want to understand how this packet was built? Check out Part 3 for a detailed byte-by-byte breakdown of ISAKMP packet construction.

The Test Packet#

I’ve constructed a complete 88-byte ISAKMP Phase 1 packet with the following transform set:

  • Encryption: AES-128-CBC
  • Hash: SHA-1
  • DH Group: Group 5 (1536-bit MODP)
  • Authentication: Pre-Shared Key
  • Lifetime: 86400 seconds (24 hours)

⚠️ Security Notice: This packet uses legacy cryptographic parameters for maximum compatibility with older devices. For production VPNs, use:

  • Hash: SHA-256 or SHA-384
  • Encryption: AES-256-CBC or AES-256-GCM
  • DH Group: Group 14 (2048-bit) or higher
  • Consider IKEv2 instead of IKEv1 for modern deployments

Quick Reference#

For those who want to skip the detailed explanation and just test connectivity, here’s everything you need:

Complete ISAKMP Test Packet:

ISAKMP_PACKET="\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x10\x02\x00\x00\x00\x00\x00\x00\x00\x00\x58\x00\x00\x00\x3c\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x30\x01\x01\x00\x01\x00\x00\x00\x28\x01\x01\x00\x00\x80\x01\x00\x07\x80\x0e\x00\x80\x80\x02\x00\x02\x80\x04\x00\x05\x80\x03\x00\x01\x80\x0b\x00\x01\x00\x0c\x00\x04\x00\x01\x51\x80"

Basic Test:

echo -ne "$ISAKMP_PACKET" | nc -u TARGET_IP 500

With Timeout and Response Capture:

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u TARGET_IP 500 | hexdump -C

What This Packet Offers:

  • Encryption: AES-128-CBC
  • Hash: SHA-1
  • DH Group: Group 5 (1536-bit MODP)
  • Authentication: Pre-Shared Key
  • Lifetime: 86400 seconds (24 hours)

Testing the Complete Datagram#

Now that we have our complete ISAKMP packet, let’s test it with netcat. The beauty of this approach is that we’ll get a proper ISAKMP response back if the peer is functioning correctly, eliminating the need for debug access.

Basic Test#

echo -ne "$ISAKMP_PACKET" | nc -u 192.168.1.1 500

Enhanced Test with Timeout#

For better control and to avoid hanging connections, use netcat with a timeout:

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u 192.168.1.1 500

Capturing the Response#

To see the hex response for analysis:

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u 192.168.1.1 500 | hexdump -C

Expected Results#

Analyzing the Response#

When you receive a response, here’s how to decode it:

Bytes 0-7 (Initiator Cookie):

00 00 00 00 00 00 00 01

Should match YOUR initiator cookie - confirms this is a reply to your packet.

Bytes 8-15 (Responder Cookie):

a1 b2 c3 d4 e5 f6 07 08

Non-zero random value - the peer’s SPI. Save this for multi-packet exchanges.

Byte 16 (Next Payload):

  • 01 = SA payload (peer accepted and is proposing parameters)
  • 0d = Notification payload (usually an error)

Byte 18 (Exchange Type):

  • 02 = Identity Protection (Main Mode) - normal
  • 05 = Informational - usually error notification

Bytes 24-27 (Length):

00 00 00 84

Total packet length (132 bytes). Longer responses typically mean successful negotiation.

Common Notification Payloads:

  • 0e 00 (14) = NO-PROPOSAL-CHOSEN - transform set mismatch
  • 18 00 (24) = AUTHENTICATION-FAILED - PSK/cert issue
  • 0d 00 (13) = INVALID-PAYLOAD-TYPE - malformed packet

Successful Response#

If the ISAKMP peer is functioning correctly, you should receive a response that:

  • Has the same initiator cookie (00 00 00 00 00 00 00 01) in the first 8 bytes
  • Contains a responder cookie (non-zero values in bytes 9-16)
  • Shows exchange type 02 (Identity Protection) in byte 18
  • Has a reasonable length (typically 100+ bytes for a full SA response)

Example successful response header:

00000000  00 00 00 00 00 00 00 01  a1 b2 c3 d4 e5 f6 07 08  |................|
00000010  01 10 02 00 00 00 00 00  00 00 00 84              |............|

Error Responses#

Different error conditions will produce different responses:

No Matching Transform Sets:

  • You’ll get a response with a Notification payload
  • The notification will indicate “NO-PROPOSAL-CHOSEN” (error 14)

Authentication Issues:

  • May receive “AUTHENTICATION-FAILED” notification
  • Or connection may be silently dropped

Firewall/NAT Issues:

  • No response at all (timeout)
  • Or ICMP unreachable messages

Troubleshooting Tips#

No Response#

  1. Check basic connectivity first with a simple UDP test
  2. Verify the target IP is actually running ISAKMP/IKE
  3. Check for NAT/firewall blocking UDP/500
  4. Try different source ports - some firewalls are picky
  5. Wait between attempts - rate limiting may block rapid tests (30-60 seconds)

Malformed Packet Errors#

  1. Double-check the hex string - any typos will break the packet
  2. Verify packet length matches the length field in the header
  3. Test with Wireshark to validate packet structure

Transform Set Mismatches#

  1. Try different encryption algorithms (3DES, AES-256)
  2. Adjust DH groups (Group 2, Group 14)
  3. Change hash algorithms (MD5, SHA-256)

NAT Considerations#

NAT Detection: If testing through NAT, be aware:

  • Source port randomization may break return traffic
  • Some firewalls require NAT-T (UDP 4500) for NATed ISAKMP
  • ESP (protocol 50) will be blocked by NAT without NAT-T

Testing through NAT:

# Bind to specific source port
echo -ne "$ISAKMP_PACKET" | nc -u -p 500 TARGET_IP 500

NAT-T Detection: Modern VPN gateways may respond on UDP 4500 instead of 500 if NAT is detected.

Aggressive Mode vs Main Mode#

This packet uses Main Mode (Identity Protection). If you get no response, the peer might only support Aggressive Mode:

  • Change Exchange Type from \x02 to \x04 (byte 18)
  • Note: Aggressive Mode is deprecated due to security weaknesses

IKEv2 Confusion#

If you get INVALID_MAJOR_VERSION notification:

  • Target only supports IKEv2 (RFC 7296)
  • This technique only works for IKEv1
  • Use ike-scan tool for IKEv2 testing

Validating Your Packet#

Before testing against production systems, validate your packet structure:

# Capture your test packet
echo -ne "$ISAKMP_PACKET" | nc -u 127.0.0.1 500 &
sudo tcpdump -i lo -w test.pcap udp port 500

Open in Wireshark and verify:

  • ISAKMP protocol is correctly decoded (not “Malformed Packet”)
  • All payload lengths match actual payload sizes
  • Transform attributes are properly formatted
  • No “Expert Info” warnings about malformed fields

Alternative Tools#

While this manual method teaches protocol internals, consider these tools for production use:

ike-scan:

ike-scan -M TARGET_IP
  • Supports IKEv1 and IKEv2
  • Tests multiple transform sets automatically
  • Better for production troubleshooting

nmap:

nmap -sU -p 500 --script ike-version TARGET_IP
  • Detects IKE version support
  • Identifies vendor implementations

When to use manual method:

  • Learning protocol internals
  • Highly restricted environments (no tools available)
  • Testing specific edge cases
  • Forensic analysis of VPN behavior

Practical Applications#

This technique is invaluable for:

  • Remote troubleshooting without device access
  • Validating VPN connectivity end-to-end
  • Testing firewall rules for ISAKMP traffic
  • Confirming peer responsiveness before complex configurations
  • Network path validation including return traffic

Conclusion#

By using a pre-crafted complete ISAKMP datagram, we’ve eliminated the dependency on having debug access to the peer device. This approach provides a more comprehensive test that validates:

  • Outbound connectivity (UDP/500 reachability)
  • Inbound connectivity (return path functionality)
  • Peer responsiveness (ISAKMP service is running)
  • Basic compatibility (peer can parse our proposal)

The hex string method, while complex to construct, gives you precise control over the ISAKMP negotiation and works regardless of the peer platform or your access level to it.

For production troubleshooting, consider creating a small script with common transform sets to quickly test different scenarios without reconstructing the hex each time.

Next Steps:

  • Want to understand how this packet was built? Read Part 3
  • Explore automating this with Python and the Scapy library
  • Check out ike-scan for more advanced VPN testing scenarios