In my previous post about using netcat to test ISAKMP, I validated that UDP/500 was open to the destination, and that semi-valid ISAKMP headers were being received by the peer (a Cisco ASAv in this case). However, this validation required me to be knowledgeable in troubleshooting the Cisco ASA platform so that I could easily determine if my pseudo-ISAKMP packets arrived (and also required that I control or have access to the ASA). What if the peer is outside of your control or is a platform you are not at all familiar with? In that case, what I want to do now is send a complete ISAKMP initial communication and receive a reply from the peer. This test serves the dual purpose of removing the need for debugs or logging on the peer and validates the return path (and any stateful/L7 inspection) as well.

Rather than hand-crafting every byte again, I based this packet on an actual captured ISAKMP conversation. If you don’t have a capture handy, you can find ISAKMP protocol details and examples at the Wireshark ISAKMP Protocol page.

Want to understand how this packet was built? Check out Part 3 for a detailed byte-by-byte breakdown of ISAKMP packet construction.

The Test Packet

I’ve constructed a complete 88-byte ISAKMP Phase 1 packet with the following transform set:

  • Encryption: AES-128-CBC
  • Hash: SHA-1
  • DH Group: Group 5 (1536-bit MODP)
  • Authentication: Pre-Shared Key
  • Lifetime: 86400 seconds (24 hours)

⚠️ Security Notice: This packet uses legacy cryptographic parameters for maximum compatibility with older devices. For production VPNs, use:

  • Hash: SHA-256 or SHA-384
  • Encryption: AES-256-CBC or AES-256-GCM
  • DH Group: Group 14 (2048-bit) or higher
  • Consider IKEv2 instead of IKEv1 for modern deployments

Quick Reference

For those who want to skip the detailed explanation and just test connectivity, here’s everything you need:

Complete ISAKMP Test Packet:

1

ISAKMP_PACKET="\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x10\x02\x00\x00\x00\x00\x00\x00\x00\x00\x58\x00\x00\x00\x3c\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x30\x01\x01\x00\x01\x00\x00\x00\x28\x01\x01\x00\x00\x80\x01\x00\x07\x80\x0e\x00\x80\x80\x02\x00\x02\x80\x04\x00\x05\x80\x03\x00\x01\x80\x0b\x00\x01\x00\x0c\x00\x04\x00\x01\x51\x80"

Basic Test:

1

echo -ne "$ISAKMP_PACKET" | nc -u TARGET_IP 500

With Timeout and Response Capture:

1

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u TARGET_IP 500 | hexdump -C

What This Packet Offers:

  • Encryption: AES-128-CBC
  • Hash: SHA-1
  • DH Group: Group 5 (1536-bit MODP)
  • Authentication: Pre-Shared Key
  • Lifetime: 86400 seconds (24 hours)

Testing the Complete Datagram

Now that we have our complete ISAKMP packet, let’s test it with netcat. The beauty of this approach is that we’ll get a proper ISAKMP response back if the peer is functioning correctly, eliminating the need for debug access.

Basic Test

1

echo -ne "$ISAKMP_PACKET" | nc -u 192.168.1.1 500

Enhanced Test with Timeout

For better control and to avoid hanging connections, use netcat with a timeout:

1

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u 192.168.1.1 500

Capturing the Response

To see the hex response for analysis:

1

echo -ne "$ISAKMP_PACKET" | timeout 5 nc -u 192.168.1.1 500 | hexdump -C

Expected Results

Analyzing the Response

When you receive a response, here’s how to decode it:

Bytes 0-7 (Initiator Cookie):

1

00 00 00 00 00 00 00 01

Should match YOUR initiator cookie - confirms this is a reply to your packet.

Bytes 8-15 (Responder Cookie):

1

a1 b2 c3 d4 e5 f6 07 08

Non-zero random value - the peer’s SPI. Save this for multi-packet exchanges.

Byte 16 (Next Payload):

  • 01 = SA payload (peer accepted and is proposing parameters)
  • 0d = Notification payload (usually an error)

Byte 18 (Exchange Type):

  • 02 = Identity Protection (Main Mode) - normal
  • 05 = Informational - usually error notification

Bytes 24-27 (Length):

1

00 00 00 84

Total packet length (132 bytes). Longer responses typically mean successful negotiation.

Common Notification Payloads:

  • 0e 00 (14) = NO-PROPOSAL-CHOSEN - transform set mismatch
  • 18 00 (24) = AUTHENTICATION-FAILED - PSK/cert issue
  • 0d 00 (13) = INVALID-PAYLOAD-TYPE - malformed packet

Successful Response

If the ISAKMP peer is functioning correctly, you should receive a response that:

  • Has the same initiator cookie (00 00 00 00 00 00 00 01) in the first 8 bytes
  • Contains a responder cookie (non-zero values in bytes 9-16)
  • Shows exchange type 02 (Identity Protection) in byte 18
  • Has a reasonable length (typically 100+ bytes for a full SA response)

Example successful response header:

1
2

00000000  00 00 00 00 00 00 00 01  a1 b2 c3 d4 e5 f6 07 08  |................|
00000010  01 10 02 00 00 00 00 00  00 00 00 84              |............|

Error Responses

Different error conditions will produce different responses:

No Matching Transform Sets:

  • You’ll get a response with a Notification payload
  • The notification will indicate “NO-PROPOSAL-CHOSEN” (error 14)

Authentication Issues:

  • May receive “AUTHENTICATION-FAILED” notification
  • Or connection may be silently dropped

Firewall/NAT Issues:

  • No response at all (timeout)
  • Or ICMP unreachable messages

Troubleshooting Tips

No Response

  1. Check basic connectivity first with a simple UDP test
  2. Verify the target IP is actually running ISAKMP/IKE
  3. Check for NAT/firewall blocking UDP/500
  4. Try different source ports - some firewalls are picky
  5. Wait between attempts - rate limiting may block rapid tests (30-60 seconds)

Malformed Packet Errors

  1. Double-check the hex string - any typos will break the packet
  2. Verify packet length matches the length field in the header
  3. Test with Wireshark to validate packet structure

Transform Set Mismatches

  1. Try different encryption algorithms (3DES, AES-256)
  2. Adjust DH groups (Group 2, Group 14)
  3. Change hash algorithms (MD5, SHA-256)

NAT Considerations

NAT Detection: If testing through NAT, be aware:

  • Source port randomization may break return traffic
  • Some firewalls require NAT-T (UDP 4500) for NATed ISAKMP
  • ESP (protocol 50) will be blocked by NAT without NAT-T

Testing through NAT:

1
2

# Bind to specific source port
echo -ne "$ISAKMP_PACKET" | nc -u -p 500 TARGET_IP 500

NAT-T Detection: Modern VPN gateways may respond on UDP 4500 instead of 500 if NAT is detected.

Aggressive Mode vs Main Mode

This packet uses Main Mode (Identity Protection). If you get no response, the peer might only support Aggressive Mode:

  • Change Exchange Type from \x02 to \x04 (byte 18)
  • Note: Aggressive Mode is deprecated due to security weaknesses

IKEv2 Confusion

If you get INVALID_MAJOR_VERSION notification:

  • Target only supports IKEv2 (RFC 7296)
  • This technique only works for IKEv1
  • Use ike-scan tool for IKEv2 testing

Validating Your Packet

Before testing against production systems, validate your packet structure:

1
2
3

# Capture your test packet
echo -ne "$ISAKMP_PACKET" | nc -u 127.0.0.1 500 &
sudo tcpdump -i lo -w test.pcap udp port 500

Open in Wireshark and verify:

  • ISAKMP protocol is correctly decoded (not “Malformed Packet”)
  • All payload lengths match actual payload sizes
  • Transform attributes are properly formatted
  • No “Expert Info” warnings about malformed fields

Alternative Tools

While this manual method teaches protocol internals, consider these tools for production use:

ike-scan:

1

ike-scan -M TARGET_IP
  • Supports IKEv1 and IKEv2
  • Tests multiple transform sets automatically
  • Better for production troubleshooting

nmap:

1

nmap -sU -p 500 --script ike-version TARGET_IP
  • Detects IKE version support
  • Identifies vendor implementations

When to use manual method:

  • Learning protocol internals
  • Highly restricted environments (no tools available)
  • Testing specific edge cases
  • Forensic analysis of VPN behavior

Conclusion

By using a pre-crafted complete ISAKMP datagram, we’ve eliminated the dependency on having debug access to the peer device. This approach validates outbound connectivity (UDP/500 reachability), peer responsiveness, and return path functionality - all without needing access to the remote device.

For a detailed breakdown of how this packet was constructed, see Part 3. For automated testing with Python and Scapy, check out Part 4.

References

Next Steps:

  • Want to understand how this packet was built? Read Part 3
  • Explore automating this with Python and the Scapy library
  • Check out ike-scan for more advanced VPN testing scenarios