Testing ISAKMP Part 1: Basic Connectivity
Table of Contents
The Scenario
In the course of my day-to-day job, I interact with VPNs on many devices (primarily IPSec VPNs on the Cisco ASA). Oftentimes the simplest way to test an IPSec VPN is to fire up vpnc in a VM, change the config file as needed, and validate the connection.
Sometimes though, you need to be more granular with your testing. Perhaps the person controlling the other endpoint/peer is not in charge of the intermediate network, and you need to validate that ISAKMP traffic is allowed through to the peer. Perhaps you want to validate an ACL that should be blocking ISAKMP. Or perhaps you’re just curious to gain a deeper understanding of what is happening at the protocol level. Either way, there’s no easy way to validate that ISAKMP is permitted end to end.
Because ISAKMP utilizes UDP port 500 for transport, you can’t simply Telnet to the port and validate that it is permitted. We’re talking about UDP, not TCP, which means there is no Layer 4 validation of an open socket (the Three-Way-Handshake of TCP). This means that you can’t do something like nc -uvv $host 500 because while netcat can certainly open and utilize UDP sockets, netcat alone can’t tell you if a UDP port is open and listening. To test this below, I attempt to use netcat to check whether my local box is listening on UDP port 500 (which I know it is not):
| |
Succeeded indeed… Although we need to do something more to validate whether UDP/500 is open and listening for ISAKMP datagrams, as mentioned above we can still actually utilize netcat We can use it to open the UDP socket and then pipe in semi-valid ISAKMP data for netcat to pass to our destination. This way, we can at least get some sort of response or validation via either an ISAKMP reply message or debugs/counters on the far side peer.
RFC2048 – Internet Security Association and Key Management Protocol
ISAKMP is defined in RFC2048, which describes in great detail the underlying structure of an ISAKMP UDP datagram. In section 3.1, it lays out the header format which is all we need for testing:
| |
Essentially, if you haven’t seen this style of packet diagram, each number across the top represents a bit, and each row represents 32 bits or 4 octets/bytes whichever you prefer. In this case, the ISAKMP header is 7 rows tall, which means that the ISAKMP header is 224 bits (7*32), or more commonly, 28 bytes long.
Now we know how long the data has to be, but what are we going to put into it? The header breaks down into the following chunks:
- Initiator and Responder Cookies
- 8 bytes per Cookie
- These are tokens that enable each endpoint to identify Security Associations (RFC2048, Sec 2.4), and to act as a method to assist in securing the communication (RFC2048, Sec 2.5.3).
- Also known as the Security Parameter Index (SPI).
- Next Payload
- 1 byte
- Indicates what type of message is following this header.
- Major Version and Minor Version
- 4 bits each
- Set to 1 and 0 respectively in RFC2048.
- Exchange Type
- 1 byte
- Tells the other system what type of messages and payloads to expect.
- Flags
- 1 byte
- Set specific ISAKMP options; only the first three bits are specified in RFC2048 rest are to be zeros.
- Message ID
- 4 bytes
- A unique ID is used in Phase 2 negotiations; as we’re simulating a Phase 1 datagram these are set to all zeros.
- Length
- 4 bytes
- The combined length of the header and payloads is represented in bytes/octets.
Taking this information, we are now able to determine what fields we need to fill out and with what values to create our testing datagram. Once we are complete, we can simply feed the appropriate arrangement of binary into netcat, and it should be able to simulate the beginning of a valid ISAKMP exchange with the peer.
The only hitch with this plan is the need for binary. For mere humans, visually tracking long strings of binary is not easy; at a glance can you tell if this is 7 or 8 digits 00000001? In addition, writing out each byte would get tedious and increase the likelihood of human error.
Hexadecimal to the Rescue
Luckily we can shortcut this process by representing the binary values with Hexadecimal in \x notation. This allows us to simply represent each byte or octet as a four-character string. For example:
| |
Working with that format, and referencing the above breakdown of the header format gives us the following Hexadecimal:
- Initiator and Responder Cookies
- 8 bytes per Cookie
- Initiator Cookie
- We’ll make this a value of 1 since it has to be something and we don’t care what it is.
\x00\x00\x00\x00\x00\x00\x00\x01
- Responder Cookie
- All zeros because this is an initial communication.
\x00\x00\x00\x00\x00\x00\x00\x00
- Next Payload
- 1 byte
- No Next Payload, so a value of 0.
\x00
- Major Version and Minor Version
- 4 bits each
- Mandatory values of 1 and 0, because we’re using ISAKMP version 1.0
- That would be 0001 and 0000 in binary.
- Together, that would look like 00010000, which equals 10 in Hex.
\x10
- Exchange Type
- 1 byte
- We’re making a fake datagram with no payload, so we’ll try none (0) which appears to be permitted by RFC2048.
\x00
- Flags
- 1 byte
- No specific flags are needed, so we’re also setting this to 0.
\x00
- Message ID
- 4 bytes
- Must be zeros per RFC2048 for an initial Phase 1 datagram like we’re simulating.
\x00\x00\x00\x00
- Length
- 4 bytes
- This will be equal to 28 since we have a header with no payload.
- 28 in Hex is 1C.
\x1C
Finally, our complete string of Hex for the simulated initial ISAKMP datagram would be:
| |
Sending Our Data
Now that we have the complete Hex string for our test datagram, how do we go about sending it? Well as described earlier, we’re going to simply pass it into netcat by printing the string and piping it into netcat. But first, we need to get a destination setup that is listening for ISAKMP traffic. In my case, I’ve spun up a virtual Cisco ASA in my lab to act as our peer, and have it listening for ISAKMP traffic on the INSIDE interface (172.16.16.1).
| |
Taking the above Hex string and using printf to send it into netcat gives us the following output (all of the Hex should be on one line, however for readability I have broken it up here):
| |
As you can see, netcat says “Connection succeeded” again, and the output on our end is not so different from when we tested to a non-ISAKMP speaking endpoint. However, on the ASA IKEv1 statistics, we can see something very interesting:
| |
So the ASA saw the packet and marked it as invalid in some way then dropped it (which makes sense because we didn’t send any payload at all).
This validates that the path for UDP/500 is open from my test box to the peer!
Bonus Validations
I’m a curious soul though, so I wanted to dig deeper still. For more detail we can turn on the IKEv1 debugs on the ASA and see as it is receiving and discarding each of the packets:
| |
And for the coup, I did a packet capture on the ASA for all UDP/500 traffic and decoded it to see if the protocol decoder could interpret the frames as ISAKMP:
| |
And there you have it, a means to quickly and easily validate a UDP/500 path!
Series Update (2026)
This post has been expanded into a comprehensive series: