Testing ISAKMP with netcat

Reading Time: 6 minutes

The Scenario

In the course of my day to day job, I interact with VPNs on many devices (primarily IPSec VPNs on the Cisco ASA).  Often times the the simplest way to test an IPSec VPN is to fire up vpnc in a VM, change the config file as needed, and validate the connection.

Sometimes though, you need to be more granular with your testing.  Perhaps the person controlling the other endpoint/peer is not in charge of the intermediate network, and you need to validate that ISAKMP traffic is allowed all the way through to the peer.  Perhaps you want to validate an ACL that should be blocking ISAKMP.  Or perhaps you’re just curious to gain a deeper understanding of what is happening at the protocol level.  Either way, there’s not a really simple way to validate that ISAKMP is permitted end to end.

Because ISAKMP utilizes UDP port 500 for transport, you can’t simply Telnet to the port and validate that it is permitted.  We’re talking about UDP not TCP, which means there is no Layer 4 validation of an open socket (the Three-Way-Handshake of TCP).  This means that you can’t do something like `nc -uvv $host 500`, because while netcat can certainly open and utilize UDP sockets, netcat alone can’t tell you if a UDP port is open and listening.  To test this below, I attempt to use netcat to check whether my local box is listening on UDP port 500 (which I know it is not):

$ nc -uvv 500
found 0 associations
found 1 connections:
1: flags=82
outif (null)
src port 59894
dst port 500
rank info not available

Connection to port 500 [udp/isakmp] succeeded!

Succeeded indeed…  Although we obviously need to do something more to validate whether UDP/500 is open and listening for ISAKMP datagrams, as mentioned above we can still actually utilize netcat.  We can use it to open the UDP socket and then pipe in semi-valid ISAKMP data for netcat to pass to our destination.  This way, we can at least get some sort of response or validation via either an ISAKMP reply message or debugs/counters on the far side peer.

RFC2048 – Internet Security Association and Key Management Protocol

ISAKMP is defined in RFC2048, which describes in great detail the underlying structure of an ISAKMP UDP datagram.  In section 3.1, it lays out the header format which is all we really need for testing:

                 1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
! Initiator !
! Cookie !
! Responder !
! Cookie !
! Next Payload ! MjVer ! MnVer ! Exchange Type ! Flags !
! Message ID !
! Length !

Essentially, if you haven’t seen this style of packet diagram, each number across the top represents a bit, and each row represents 32 bits or 4 octets/bytes whichever you prefer.  In this case the ISAKMP header is 7 rows tall , which means that the ISAKMP header is 224 bits (7*32), or more commonly, 28 bytes long.

Now we know how long the data has to be, but what are we going to put into it?  The header breaks down into the following chunks:

  • Initiator and Responder Cookies – 8 bytes per Cookie, these are tokens that enable each endpoint to identify Security Associations (RFC2048, Sec 2.4), and to act as a method to assist in securing the communication (RFC2048, Sec 2.5.3).
  • Next Payload – 1 byte, indicates what type of message is following this header.
  • Major Version and Minor Version – 4 bits each, set to 1 and 0 respectively in RFC2048.
  • Exchange Type – 1 byte, tells the other system what type of messages and payloads to expect.
  • Flags – 1 byte, set specific ISAKMP options; only the first three bits are specified in RFC2048 rest are to be zeros.
  • Message ID – 4 bytes, unique ID used in Phase 2 negotiations; as we’re simulating a Phase 1 datagram these are set to all zeros.
  • Length – 4 bytes, the combined length of the header and payloads represented in bytes/octets.

Taking this information, we are now able to determine what fields we need to fill out and with what values to create our testing datagram.  Once we are complete, we can simply feed the appropriate arrangement of binary into netcat, and it should be able to simulate the beginning of a valid ISAKMP exchange with the peer.

The only hitch with this plan is the need for binary.  For mere humans, visually tracking long strings of binary is not easy; at a glance can you tell if this is 7 or 8 digits: 00000001.  In addition, writing out for each byte would get tedious and increase the likelihood of human error.

Hexadecimal to the Rescue

Luckily we can shortcut this  process by representing the binary values with Hexadecimal in `\x` notation.  This allows us to simply represent each byte or octet as a four character string.  For example:

Binary: 000000000000000100000010000000011
Hexadecimal: \x00\x01\x02\x03
Validation using xxd:
printf '\x01\x02\x03\x04' | xxd -b 0000000: 00000001 00000010 00000011 00000100 ....

Working with that format, and referencing the above breakdown of the header format gives us the following Hexadecimal:

Initiator Cookie: 8 bytes We’ll make this a value of 1 since it has to be something and we don’t care what it is. \x00\x00\x00\x00\x00\x00\x00\x01

Responder Cookie: 8 bytes All zeros because this is an initial communication.

Next Payload: 1 byte No next payload, so a value of 0.

Major and Minor Version: 4 bits each, mandatory value of 1 and 0 In binary, that would look like 00010000, which equals 10 in Hex.

Exchange Type: 1 byte We’re making a fake datagram with no payload, so we’ll try none (0) which appears to be permitted by RFC2048.

Flags: 1 byte No specific flags needed, so we’re also setting this to 0.

Message ID: 4 bytes Must be zeros per RFC2048 for an initial Phase 1 datagram like we’re simulating.

Length: 4 bytes This will be equal to 28 since we have a header with no payload. 28 in Hex is 1C.

Finally, our complete string of Hex for the simulated initial ISAKMP datagram would be:


Sending Our Data

Now that we have the complete Hex string for our test datagram, how do we go about sending it?  Well as described earlier, we’re going to simply pass it into netcat by printing the string and piping it into netcat. But first we need to get a destination setup that is listening for ISAKMP traffic. In my case, I’ve spun up a virtual Cisco ASA in my lab to act as our peer, and have it listening for ISAKMP traffic on the INSIDE (

ASAv-1A/pri/act# show run int G0/4
interface GigabitEthernet0/4
nameif INSIDE
security-level 100
ip address
ASAv-1A/pri/act# show run crypto ikev1
crypto ikev1 enable INSIDE

Taking the above Hex string and using `printf` to send it into netcat gives us the following output (all of the Hex should be on one line, however for readability I have broken it up here):

$ printf \x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00
       \x00\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x1C | nc -uvv 500
Connection to 500 port [udp/isakmp] succeeded!

As you can see, netcat says “Connection succeeded” again, and the output on our end is not so different from when we tested to a non-ISAKMP speaking endpoint. However, on the ASA IKEv1 statistics, we can see something very interesting:

ASAv-1A/pri/act# show crypto ikev1 stats | inc Drop
In Drop Packets: 6 Out Drop Packets: 0

So the ASA saw the packet, and marked it as invalid in some way and dropped it (which makes sense because we didn’t send any payload at all).

This validates that the path for UDP/500 is open from my test box to the peer!

Bonus Validations

I’m a curious soul though, so I wanted to dig deeper still.  For more detail we can turn on the IKEv1 debugs on the ASA and see as it is receiving and discarding each of the packets:

ASAv-1A/pri/act# debug crypto ikev1 255
May 21 03:25:04 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:04 [IKEv1]IKE Receiver: Runt ISAKMP packet discarded on Port 500 from
May 21 03:25:04 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:04 [IKEv1]IKE Receiver: Runt ISAKMP packet discarded on Port 500 from
May 21 03:25:05 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:05 [IKEv1]IKE Receiver: Runt ISAKMP packet discarded on Port 500 from
May 21 03:25:06 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:06 [IKEv1]IKE Receiver: Runt ISAKMP packet discarded on Port 500 from
May 21 03:25:07 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:07 [IKEv1]IKE Receiver: Runt ISAKMP packet discarded on Port 500 from
May 21 03:25:07 [IKEv1]IKE Receiver: Packet received on from
May 21 03:25:07 [IKEv1]IKE Receiver: Discarding packet, invalid IKE version

And for the coup, I did a packet capture on the ASA for all UDP/500 traffic and decoded it to see if the protocol decoder could interpret the frames as ISAKMP:

ASAv-1A/pri/act# cap cap1 int INSIDE match udp any any eq 500
ASAv-1A/pri/act# show cap cap1 decode 6 packets captured
1: 03:34:01.443916 > udp 1 [ISAKMP header incomplete]
2: 03:34:01.444007 > udp 1 [ISAKMP header incomplete]
3: 03:34:02.444892 > udp 1 [ISAKMP header incomplete]
4: 03:34:03.445609 > udp 1 [ISAKMP header incomplete]
5: 03:34:04.447059 > udp 1 [ISAKMP header incomplete]
6: 03:34:04.450126 > udp 75 ISAKMP Header Initiator COOKIE: 78 30 30 78 30 30 78 30 Responder COOKIE: 30 78 30 30 78 30 30 78 Next Payload: IKEV2 LEAP PAYLOAD Version: 3.0 Exchange Type: DOI Specific Use Flags: MessageID: 30783031 Length: 2016424056 [ISAKMP payload corrupted or truncated]

And there you have it, a means to quickly and easily validate a UDP/500 path!