In Part 2, I provided a ready-to-use ISAKMP packet for testing VPN connectivity. But where did that hex string come from? This post is for those who want to understand the internals of ISAKMP packet construction.

Why Build Packets Manually?#

Understanding packet structure at the byte level helps you:

  • Debug protocol issues more effectively
  • Customize packets for specific testing scenarios
  • Understand what tools like ike-scan are doing under the hood
  • Develop your own VPN testing tools

The Approach#

After spending time reading RFC2408 for the basic header in Part 1, I decided to base this complete packet on an actual captured ISAKMP conversation. If you don’t have an ISAKMP capture handy, you can find protocol details and examples at the Wireshark ISAKMP Protocol page.

I still recommend reading RFC2408 to understand ISAKMP message structure, otherwise this sea of hexadecimal will be confusing.

Hand-Crafted Datagram#

The primary ingredient needed to make a complete ISAKMP datagram is an attempt to set up the SA (Security Association) including nested inner payloads and an offer of a Transform set.

As discussed previously, the header format for ISAKMP (IKE v1) is defined in RFC2408 Section 3.1.

  • Initiator and Responder Cookies
    • 8 bytes per Cookie
    • Initiator Cookie
      • AKA the SPI or Security Parameter Index
      • We’ll make this a value of 1 since it has to be something and we don’t care what it is.
      • Production note: RFC 2408 recommends pseudo-random generation to prevent DoS attacks
      • \x00\x00\x00\x00\x00\x00\x00\x01
    • Responder Cookie
      • All zeros because this is an initial communication, and as the Initiator, we don’t know what the responder will send yet.
      • \x00\x00\x00\x00\x00\x00\x00\x00
  • Next Payload
    • 1 byte
    • The Next Payload will be of a Security Association type, so per RFC2408 Section 3.1 this is a value of 1.
    • \x01
  • Major Version and Minor Version
    • 4 bits each
    • Mandatory values of 1 and 0, because we’re using ISAKMP version 1.0
    • That would be 0001 and 0000 in binary.
    • Together, that would look like 00010000, which equals 10 in Hex.
    • \x10
  • Exchange Type
    • 1 byte
    • The Exchange Types are laid out in RFC2408 Section 4.4 to Section 4.8.
    • From our sample packet capture, I see we want an Identity Protection type (02).
    • \x02
  • Flags
    • 1 byte
    • We still need no flags, so leaving this at 0.
    • \x00
  • Message ID
    • 4 bytes
    • Must be zeros per RFC2408 for an initial Phase 1 datagram like we’re simulating.
    • \x00\x00\x00\x00
  • Length
    • 4 bytes
    • I have doubled back from the bottom to fill in this value, as you are unable to calculate it until you have built all the inner payloads.
    • This totals up to 88 bytes or 58 in hex.
    • \x00\x00\x00\x58

Generic Payload Header#

There is a simple header on each ISAKMP payload as shown below, it is defined in RFC2408 Section 3.2. We’ll need one on our outer SA Payload, and one will be present on any interior/nested payloads as well (such as additional Proposals or Vendor Specific Attributes).

  • Next Payload
    • 1 byte
    • This is the last outer payload, so 0.
    • We’re not simulating Vendor Specific Attributes or anything like that, just trying to make a minimum viable datagram.
    • \x00
  • RESERVED
    • 1 byte
    • Unused, set to Zero.
    • \x00
  • Payload Length
    • 2 bytes
    • I am cheating somewhat, and know this value as I have gone forward to finish the payload, and then calculated backward to get this value.
    • It includes the Generic Payload header elements (including the length field we’re calculating now) and the length of the inner SA Payload from below.
    • That totals 60 bytes or 3c in hex.
    • \x00\x3c

Security Association Payload#

SA Payload is defined in RFC2408 Section 3.4 and laid out as shown below.

  • Domain of Interpretation (DOI)
    • 4 bytes
    • See RFC2408 Section 2.1 for more information on what the DOI is. For our purposes, this is a value of 1, for an IPSEC DOI (RFC2407).
    • \x00\x00\x00\x01
  • Situation
    • Variable length (4 bytes in this case)
    • based on the DOI For IPSEC DOI, per RFC2407 section 4.2, this will be 4 bytes.
    • Our sample capture indicates to say that this is Identity Only (value of 00000001).
    • \x00\x00\x00\x01

Proposal Payload#

Inside the SA Payload, there is the payload containing the ISAKMP Proposal on how to go about setting up the Security Association.

  • Next Payload
    • 1 byte
    • This will be zero as there is no other Proposal being sent.
    • \x00
  • RESERVED
    • 1 byte
    • Unused, set to zero.
    • \x00
  • Payload length
    • 2 bytes
    • Again, I only know this value as I am working backward, and have calculated it out knowing the length of this payload.
    • It is 48 bytes long (including the two bytes above, these two bytes, and everything in the payload below), which is 30 in hex.
    • \x00\x30
  • Proposal Number
    • 1 byte
    • If there is more than one Proposal in the payload, which Proposal is this?
    • In our case, this is the first and only Proposal, so set this to 00000001.
    • \x01
  • Protocol ID
    • 1 byte
    • This took some tracking down, but as defined in RFC2408 Section 3.6, the specific values for these are defined by the DOI in use (IPSec DOI in our case).
    • The IDs are defined in RFC2407 Section 4.4.1, and we’re using ISAKMP (value of 1).
    • \x01
  • SPI Size
    • 1 byte
    • Set to zero for ISAKMP (as the Initiator and Responder cookies will be used for the SPIs).
    • \x00
  • Number of Transforms
    • 1 byte
    • We’re just sending one Transform, so set it to a value of 1.
    • \x01

Transform Payload#

And finally, inside of THAT payload are one or more Transforms. (I told you this would get confusing…) For our purposes, I’m just going to offer a single, simple, and fairly common transform set consisting of:

  • Encryption: AES-128
  • Hash: SHA
  • Diffie-Hellman: Group 5
  • Authentication: Pre-Shared Key
  • Lifetime 86400 seconds

⚠️ Security Notice: This packet uses legacy cryptographic parameters for maximum compatibility with older devices. For production VPNs, use:

  • Hash: SHA-256 or SHA-384
  • Encryption: AES-256-CBC or AES-256-GCM
  • DH Group: Group 14 (2048-bit) or higher
  • Consider IKEv2 instead of IKEv1 for modern deployments

It is worth noting that the Transform Attributes are expressed as TLV (Type-Length-Value) or TV (Type-Value). If you’re not familiar with what a TLV is, I recommend becoming familiar with it as it crops up again and again in network protocols. TLVs allow a protocol to be extended in ways beyond the original scope, for example, TLVs are what make a routing protocol like ISIS able to be extended to support Shortest Path Bridging.

The TLV/TV information for our purposes is laid out across all three original IPSEC/ISAKMP/IKE RFCs (RFC2407, RFC2408, and RFC2409 respectively.)

  • For getting the bulk of actual Types, Values, and whether they are fixed or variable length, you need RFC2409 Appendix A which lists the IKE Attribute Assigned Numbers.
  • RFC2408 Section 3.3 however, specifies the underlying Data Attribute format.
  • But to figure out what Domain of Interpretation (DOI) to use we have to consult RFC2407 Section 4.4.2.

It is also important to call out that RFC2408 Section 3.3 indicates that the most significant (leftmost) bit of the Attribute field, will determine if these bits represent a TLV (0) or TV (1).

  • If it is a TLV (0), then there is a length field sent.
  • If it is a TV (1), then it is assumed to be two bytes, and no length field is sent.

For example, this means that a TV (1) for an Encryption Algorithm (1) of AES-CBC (7) in Binary would be:

1000 0000 0000 0001 0000 0000 0000 0111

Translated to Hexadecimal, that would come out as:

\x80\x01\x00\x07

We will similarly construct each value below (and we’ll use this value as well).

Transform Set#

  • Next Payload
    • 1 byte
    • This will be zero as no other Transforms are being sent.
    • \x00
  • RESERVED
    • 1 byte
    • Unused, set to zero.
    • \x00
  • Payload length
    • 2 bytes
    • This is a value of 40 bytes that I only know because I finished building the payload and came back to calculate this value.
    • 40 in hex is 28.
    • \x00\x28
  • Transform Number
    • 1 byte
    • This is the only transform we’re sending, and it’s the first, so 1.
    • \x01
  • Transform ID
  • RESERVED2
    • 2 bytes
    • Two more bytes of mandatory zeros.
    • \x00\x00

Transform IKE Attributes#

  • Encryption Algorithm
    • 4 bytes
    • As we worked out above, this should be four bytes for a TV (starting with 80).
    • Next, comes the encryption algorithm attribute class of 1.
    • The attribute class value portion of this TV was a little tricky because AES came out well after RFC2409 was published.
    • That information is laid out in RFC3602 (The AES-CBC Cipher Algorithm and Its Use with IPsec) Section 5.1, and indicates that the AES ID is “7”.
    • \x80\x01\x00\x07
  • Key Length
    • 4 bytes
    • We’re using a Key Length (class attribute of 14, or 0e in hex) of 128 bits (80 in hex), and this is a TV as well, so we get:
    • \x80\x0e\x00\x80
  • Hash Algorithm
    • 4 bytes
    • We’re using a hash algorithm (class attribute of 2) of SHA (which is also a value of 2), and again we’re getting these values from RFC2409 Appendix A.
    • Also, this field is a TV, which means it again starts with \x80 and is 4 bytes, so we have:
    • \x80\x02\x00\x02
  • Diffie-Hellman Group
    • 4 bytes
    • For the Group Description (class attribute of 4) we’re going to use DH Group 5, which scouring RFC2409 Appendix A again tells us is a value of 5.
    • This is again a TV (starts with \x80) and is 4 bytes:
    • \x80\x04\x00\x05
  • Authentication Method
    • 4 bytes
    • For the Authentication Method (class attribute of 3) we are using a pre-shared key (value of 1):
    • \x80\x03\x00\x01
  • Lifetime
    • 4 + variable bytes (8 in this case)
    • We’re going to set a lifetime of one day (or 86400 seconds).
    • This breaks into two fields.
      • First a TV of Life Type (what are we measuring, seconds or kilobytes) with:
        • A class attribute of 11 (0b in hex) and a value of 1 for seconds.
      • Next is a TLV (which starts with 00 instead of 80) of Life Duration (how many of that unit) with:
        • A class attribute of 12 (0c in hex), a two-byte length field with a value of 4 bytes, and a value of 86400 decimal = 0x15180 hex (written as \x00\x01\x51\x80 in big-endian).
    • \x80\x0b\x00\x01\x00\x0c\x00\x04\x00\x01\x51\x80

The Complete Hex String#

Now to add all of this into one giant string of hexadecimal:

\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x10\x02\x00\x00\x00\x00\x00\x00\x00\x00\x58\x00\x00\x00\x3c\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x30\x01\x01\x00\x01\x00\x00\x00\x28\x01\x01\x00\x00\x80\x01\x00\x07\x80\x0e\x00\x80\x80\x02\x00\x02\x80\x04\x00\x05\x80\x03\x00\x01\x80\x0b\x00\x01\x00\x0c\x00\x04\x00\x01\x51\x80

This 88-byte packet represents a complete ISAKMP Phase 1 Identity Protection exchange initiation with the transform set we defined above.

Conclusion#

Building ISAKMP packets byte-by-byte is tedious but educational. It gives you deep insight into how VPN protocols work at the wire level. For practical testing, use the ready-made packet from Part 2 or tools like ike-scan.

Key Takeaways:

  • ISAKMP packets have nested payload structures (SA → Proposal → Transform)
  • Each payload has its own header with length fields
  • Transform attributes use TLV/TV encoding
  • Length fields must be calculated backward from inner payloads
  • Understanding this structure helps debug VPN issues

For those interested in automating this process, consider exploring Python with the Scapy library for dynamic packet construction.