IPsec

Template:TOC-right IPsec (also written as IPSEC) is Internet Protocol Security; it is optional for IPv4, but all IPv6 implementations are required to include IPsec. Of course this does not mean all IPv6 systems will actually use it.

The basic idea is to provide security functions, source authentication, content integrity and content confidentiality, at the IP (Internet Protocol) level. This requires a higher-level protocol (IKE) to set things up for the IP-level services (ESP and AH).

IPsec offers two services, authentication and encryption . These can be used separately but are often used together.

Packet-level authentication allows you to be confident that a packet came from a particular machine and that its contents were not altered en route to you. No attempt is made to conceal or protect the contents, only to assure their integrity. Packet authentication can be provided separately using an Authentication Header , described just below, or it can be included as part of the ESP (Encapsulated Security Payload) service, described in the following section. That service offers encryption as well as authentication. In either case, the HMAC construct is used as the authentication mechanism.

There is a separate authentication operation at the IKE level, in which each gateway authenticates the other. This can be done in a variety of ways.

Encryption allows you to conceal the contents of a message from eavesdroppers. In IPsec this is done using a block cipher (normally Triple DES for Linux). In the most used setup, keys are automatically negotiated, and periodically re-negotiated, using the IKE (Internet Key Exchange) protocol.

The IPsec protocol offering encryption is ESP, Encapsulated Security Payload. It can also include a packet authentication service. Note that encryption should always be used with some packet authentication service. Unauthenticated encryption is vulnerable to man-in-the-middle attacks . Also note that encryption does not prevent traffic analysis.

There have been three versions of the overall specification, all titled "Security Architecture for the Internet Protocol"; the current version is RFC 4301 . Another document with broad scope is RFC 4835 "Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH)" .

Three protocols may be used in an IPsec implementation:

• ESP, Encapsulating Security Payload, encrypts and/or authenticates data
• AH, Authentication Header, provides a packet authentication service.
• IKE, Internet Key Exchange, negotiates connection parameters, including keys, for the other two using a Diffie-Hellman technique.

The term "IPsec" is slightly ambiguous. In some contexts, it includes all three of the above but in other contexts it refers only to AH and ESP. An overview of the sequence of operation contains:

• Phase one IKE (main mode exchange) — sets up a keying channel, called an ISAKMP Security Association (SA), between the two gateways
• Phase two IKE (quick mode exchange) — sets up data channels (IPsec SAs)
• IPsec proper — exchanges data using AH and/or ESP

Both phases of IKE are repeated periodically to automate re-keying. The SA contains the rules for how often a new session key is exchanged; there are options for doing it after a certain timer expiration, or when a specified number of bytes of data have been exchanged.

Implementations

IPsec is very widely deployed.

Many router or firewall devices include IPsec. For example, see these Cisco and Dlink products.

It has been built into all Microsoft operating systems since Windows 2000 [1]. BSD-derived systems — FreeBSD [2], NetBSD [3], OpenBSD [4], Dragonfly BSD [5] and Apple's MacOS — all include IPsec, based on an implementation from the Japanese KAME project [6] which built IPsec and IPv6 for BSD. Commercial Unix offerings, such as HP HP-UX and Sun Microsystems Solaris, also provide IPsec.

The original Linux implementation was the FreeS/WAN [7] project. This was very much a politically-motivated cypherpunk project; the overall goal was to deploy IPsec-based Opportunistic encryption, as defined in RFC 4322 "Opportunistic Encryption using the Internet Key Exchange (IKE)", very widely in order to frustrate widespread Internet monitoring by organisations such as the NSA, the Chinese government, and others. For details, see their goals and politics sections. This goal was not achieved. Project leader was activist John Gilmore [8] and technical lead was Henry Spencer [9]. All work was done in Canada by Canadians to avoid US export laws. To ensure that it remained free of those laws, the project would not accept even a one-line bug fix from an American resident or citizen.

Later the Linux kernel team included a different IPsec implementation — one that Americans could contribute to — in the kernel, and added KAME-based tools at the user level. Most current Linux distributions include those. After that, the FreeS/WAN project shut down but two descendants exist, StrongSWAN and OpenSWAN.

FreeS/WAN's main technical writer is now a Citizendium author, and has permission to reuse FreeS/WAN material here. Much of this article is based on that material.

The role of IPsec

Authentication and encryption functions for network data can, of course, be provided at other levels. Many security protocols work at levels above IP.

• Pretty Good Privacy (PGP) encrypts and authenticates mail messages
• Secure shell (SSH) authenticates remote logins and then encrypts the session.
• Secure sockets layer (SSL) or transport layer security (TLS) provides security at the sockets layer, e.g. for secure web browsing

and so on. Other techniques work at levels below IP. For example, data on a communications circuit or an entire network can be encrypted by specialised hardware. This is common practice in high-security applications.

There are, however, advantages to doing it at the IP level instead of, or as well as, at other levels.

IPsec is the most general way to provide these services for the Internet. Higher-level services protect a single protocol; for example PGP protects mail. Lower level services protect a single medium; for example a pair of encryption boxes on the ends of a line make wiretaps on that line useless unless the attacker is capable of breaking the encryption. IPsec can protect any protocol running above IP and any medium which IP runs over. More to the point, it can protect a mixture of application protocols running over a complex combination of media. This is the normal situation for Internet communication; IPsec is the only general solution.

While IPsec does not provide all the functions, or all the security, of a mail encryption package, it can encrypt mail. In particular, it can ensure that all mail passing between a pair or a group of sites is encrypted. An attacker looking only at external traffic, without access to anything on or behind the IPsec gateway, cannot read the mail. He or she is stymied by IPsec just as he or she would be by PGP.

The advantage is that IPsec can provide the same protection for anything transmitted over IP. In a corporate network example, PGP lets the branch offices exchange secure mail with head office. SSL and SSH allow them to securely view web pages, connect as terminals to machines, and so on. IPsec can support all those applications, plus database queries, file sharing (NFS or Windows), other protocols encapsulated in IP (Netware, Appletalk, ...), phone-over-IP, video-over-IP, ... anything-over-IP. The only limitation is that IP Multicast is not yet supported, though there are Internet Draft documents for that.

IPsec creates secure tunnels through untrusted networks. Sites connected by these tunnels form VPNs, Virtual private networks. The idea is that the VPN can be a secure trustworthy network even though some components are not trusted.

IPsec can provide some security services "in the background", with no visible impact on users. To use PGP encryption and signatures on mail, for example, the user must at least remember his or her passphrase, keep the passphrase secure, and follow procedures to validate correspondents' keys. These systems can be designed so that the burden on users is not onerous, but any system will place some requirements on users. No such system can hope to be secure if users are sloppy about meeting those requirements. The author has seen username and password stuck on terminals with post-it notes in an allegedly secure environment, for example.

No end user action is required for IPsec security to be used; they don't even have to know about it. The site administrators, of course, do have to know about it and to put some effort into making it work. Poor administration can compromise IPsec as badly as the post-it notes mentioned above. It seems reasonable, though, for organisations to hope their system administrators are generally both more security-conscious than end users and more able to follow computer security procedures. If not, at least there are fewer of them to educate or replace.

IPsec gateways can be installed wherever they are required. One organisation might choose to install IPsec only on firewalls between their LANs and the Internet; this would allow them to create a VPN linking several offices and would provide protection against anyone outside their sites. Another might install IPsec on departmental servers so everything on the corporate backbone net was encrypted; this would protect messages on that net from everyone except the sending and receiving department. Another organisation might be less concerned with information secrecy and more with controlling access to certain resources; they might use IPsec packet authentication as part of an access control mechanism, with or without also using the IPsec encryption service. It is even possible (assuming adequate processing power and an IPsec implementation in each node) to make every machine its own IPsec gateway so that everything on a LAN is encrypted; this protects information from everyone outside the sending and receiving machine.

These techniques can be combined in various ways. One might, for example, require authentication everywhere on a network while using encryption only for a few links. In a complex network there may be several layers of IPsec in play, with different security requirements and IPsec policies at each layer. Which of these, or of the many other possible variants, to use is up to the administrator setting up IPsec. IPsec provides mechanisms; you provide the policy.

Limitations of IPsec

IPsec is designed to secure IP links between machines. It does that well, but it is important to remember that there are many things it does not do. Some of the important limitations are:

IPsec cannot be secure if the system isn't. System security on IPsec gateway machines is an essential requirement if IPsec is to function as designed. No system can be trusted if the underlying machine has been subverted.

Of course, there is another side to this. IPsec can be a powerful tool for improving system and network security. For example, requiring packet authentication makes various spoofing attacks harder and IPsec tunnels can be extremely useful for secure remote administration of various things.

IPsec is not end-to-end: IPsec cannot provide the same end-to-end security as systems working at higher levels; nor can it provide all the functions of such systems. Note, however, that IPsec authentication of the underlying communication can make various attacks on higher-level protocols more difficult. In particular, authentication prevents man-in-the-middle attacks.

IPsec encrypts an IP connection between two machines, which is quite a different thing than encrypting messages between users or between applications.

For example, if you need a document electronically signed by a particular person, then you need his or her digital signature and a public key cryptosystem to verify it with. and if you need mail encrypted from the sender's desktop to the recipient's desktop and decryptable only by the recipient, then you need a mail encryption system. PGP (RFC 2440) is standard solution for both these requirements.

IPsec cannot meet either of those requirements, but it has a contribution to make. It can encrypt any or all of the links involved — between the two mail servers, or between either server and its clients. It could even be used to secure a direct IP link from the sender's desktop machine to the recipient's, cutting out any sort of network snoop. What it cannot ensure is end-to-end user-to-user security. If only IPsec is used to secure mail, then anyone with appropriate privileges on any machine where that mail is stored (at either end or on any store-and-forward servers in the path) can read it.

In another common setup, IPsec encrypts packets at a security gateway machine as they leave the sender's site and decrypts them on arrival at the gateway to the recipient's site. This does provide a useful security service — only encrypted data is passed over the Internet — but it does not even come close to providing an end-to-end service. In particular, anyone with appropriate privileges on either site's LAN can intercept the message in unencrypted form.

IPsec authenticates machines, not users. IPsec uses strong authentication mechanisms to control which messages go to which machines, but it does not have the concept of user ID, which is vital to many other security mechanisms and policies. This means some care must be taken in fitting the various security mechanisms on a network together.

For example, to control which users access a database server, some non-IPsec mechanism is required. IPsec can control which machines connect to the server, and can ensure that data transfer to those machines is done securely, but that is all. If the client machines all have strong controls on user access, then that plus IPsec controls on which machines can link to the server may be enough. The server or the database software could also require some form of user authentication, independent of IPsec. In some cases, all three — local access controls on client machines, IPsec controls on connections, and database controls on access — may be required.

IPsec does not stop denial of service attacks. Denial of service attacks aim at causing a system to crash, overload, or become confused so that legitimate users cannot get whatever services the system is supposed to provide. These are quite different from attacks in which the attacker seeks either to use the service himself or to subvert the service into delivering incorrect results.

IPsec shifts the ground for DoS attacks; the attacks possible against systems using IPsec are different than those that might be used against other systems. In particular, some DoS attacks are blocked by IPsec authentication mechanisms. IPsec does not, however, eliminate the possibility of such attacks.

IPsec does not stop traffic analysis Traffic analysis is the attempt to derive intelligence from messages without regard for their contents. In the case of IPsec, it would mean analysis based on things visible in the unencrypted headers of encrypted packets -- source and destination gateway addresses, packet size, et cetera. Given the resources to acquire such data and some skill in analysing it (both of which any national intelligence agency should have), this can be a very powerful technique.

IPsec is not designed to defend against this. Partial defenses are certainly possible, but it is not clear that any complete defense can be provided.

IPsec usage

IPsec can be, and often should be, used with along with security protocols at other levels. If two sites communicate with each other via the Internet, then IPsec is the obvious way to protect that communication. If two others have a direct link between them, either link encryption or IPsec would make sense. Choose one or use both. Whatever you use at and below the IP level, use other things as required above that level. Whatever you use above the IP level, consider what can be done with IPsec to make attacks on the higher levels harder. For example, man-in-the-middle attacks on various protocols become difficult if authentication at packet level is in use on the potential victims' communication channel.

Normal use: ESP encryption + authentication

Today, typical usage is ESP for encryption and authentication.

Authentication without encryption

IPsec can provide authentication without encryption, via an AH-only connection or ESP using null encryption. AH authenticates parts of the IP header, which ESP-null does not do.

Encryption without authentication is dangerous

Originally, the IPsec encryption protocol ESP didn't do integrity checking. It only did encryption. Steve Bellovin found many ways to attack ESP used without authentication. See his paper "Problem areas for the IP Security Protocols" [10]. To make a secure connection, you had to add an AH Authentication Header as well as ESP. Rather than incur the overhead of several layers (and rather than provide an ESP layer that didn't actually protect the traffic), the IPsec working group built integrity and replay checking directly into ESP.

Other combinations

Other variants are allowed by the standard, but not much used:

ESP can be used for encryption with AH providing authentication: This has higher overheads than using the authentication in ESP, and no obvious benefit in most cases. However, AH does protect more than ESP, specifically parts of the IP header, so policy in some organisations may require it. The combination might also appear on a network where AH authentication is widely or universally used. If AH is already used to conform with network policy, there may be no need to authenticate again in the ESP layer.

It is also possible to authenticate twice, once with AH and again with ESP. This might be done to satisfy different parts of a security policy. For example, an organisation might require AH authentication everywhere but two users within the organisation might use ESP as well. Also, some people consider "belt and suspenders" the sensible approach to security; they might use both protocols here "just to be sure".

Cryptographic components

IPsec combines a number of cryptographic techniques, all of them well-known and well-analyzed. The overall design approach was conservative; no new or poorly-understood components were included.

This section gives a brief overview of each technique. It is intended only as an introduction.

Diffie-Hellman key agreement

The Diffie-Hellman key agreement protocol allows two parties (A and B or Alice and Bob) to agree on a key in such a way that an eavesdropper who intercepts the entire conversation cannot learn the key. It is based on the discrete logarithm problem and is therefore thought to be secure. Mathematicians have been working on that problem for years and seem no closer to a solution, though there is no proof that an efficient solution is impossible.

Two variants of discrete log are supported, in a field modulo a prime or in a field defined by an elliptic curve. This is partly a safety feature; if someone should solve one variant of the discrete log problem, we could switch to the other.

The protocol involved is called called Oakley and was defined in RFC 2412, Oakley Group one uses a 768-bit prime and is not considered secure. Groups three and four use elliptic curves. Groups two and five, based on 1024 and 1536-bit primes respectively, are commonly used. RFC 3526 defines additional groups with primes up to 8192 bits and RFC 4753 provides additional elliptic curve groups.

Authentication

Authentication is used at two levels in IPsec.

First, the gateways must authenticate themselves to each other during the IKE negotiations; without that, the Diffie-Hellman key agreement protocol is not secure against man-in-the-middle attacks. This can be done in several ways:

• The RSA algorithm (named for its inventors -- Rivest, Shamir and Adleman) is a very widely used public key cryptographic technique.
• Authentication can also be done using X.509 certificates, containing either RSA or DSS signatures. This is also a public key method.
• Authentication using symmetric cryptography and a shared secret is also possible, but not widely used.

Once the connections are set up, the usual practice is to authenticate each packet with the HMAC construct; see next section.

HMAC authentication

IPsec packet authentication is done with a Hashed Message Authentication Code or HMAC. This is not just a hash of the packet data, but a more complex operation which uses both a hashing algorithm and a key. It therefore does more than a simple hash would. A simple hash only tells the receiver that the packet data was not changed in transit, or that whoever changed it also regenerated the hash. An HMAC also shows that whoever generated it knew the HMAC key. An attacker without that key cannot then alter packets without detection.

For IPsec HMAC, the output of the hash algorithm is truncated to 96 bits. This saves some space in the packets. More important, it prevents an attacker from seeing all the hash output bits and perhaps creating some sort of attack based on that knowledge.

The basic HMAC mechanism is defined in RFC 2104 "HMAC: Keyed-Hashing for Message Authentication". Other RFCs give details; see next section.

Choice of hash algorithm

The original IPsec RFCs required two hash algorithms, MD5 and SHA. Various other algorithms, such as RIPEMD and Tiger, were listed as optional. Details are in RFC 2085 "HMAC-MD5 IP Authentication with Replay Prevention", RFC 2303 "The Use of HMAC-MD5-96 within ESP and AH", RFC 2404 "The Use of HMAC-SHA-1-96 within ESP and AH", and RFC 2202 "Test Cases for HMAC-MD5 and HMAC-SHA-1".

Additional hash algorithms — SHA-256, SHA-384 and SHA-512 — have been added since to give hash strength matching the strength of AES. See RFC 4868 "Using HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512 with IPsec", RFC 4231 "Identifiers and Test Vectors for HMAC-SHA-224, HMAC-SHA-256, HMAC-SHA-384, and HMAC-SHA-512".

An AES-based hash has also been proposed; see RFC 4494.

Block ciphers

The encryption in the ESP encapsulation protocol is done with a block cipher. The only required ciphers in the original RFCs were null encryption and DES. However,since both of those are insecure, most implementations have always provided triple DES and most current implementations provide AES. The use of AES within ESP is defined in RFC 3602.

Various other ciphers are mentioned in the RFCs as optional and the protocols support extensions to use others.

Combining encryption and hashing

As originally defined, ESP makes two passes through the data, one for encryption and one for the HMAC. There is recent work on algorithms that can perform both functions in one pass; see RFC 5116 "An Interface and Algorithms for Authenticated Encryption" and RFC 5282 .

Structure of IPsec

Three protocols may be used in an IPsec implementation:

• ESP, Encapsulating Security Payload — Encrypts and/or authenticates data, RFC 2406 updated to RFC 4303
• AH, Authentication Header — Provides a packet authentication service.
• IKE, Internet Key Exchange — Negotiates connection parameters, including keys, for the other two

The term "IPsec" is slightly ambiguous. In some contexts, it includes all three of the above but in other contexts it refers only to AH and ESP.

Internet Key Exchange

The Internet Key Exchange (IKE) protocol sets up IPsec (ESP or AH) connections after negotiating appropriate parameters (algorithms to be used, keys, connection lifetimes) for them. This is done by exchanging packets on UDP port 500 between the two gateways.

The original IKE (RFC 2409) was the outcome of a long, complex process in which quite a number of protocols were proposed and debated. Oversimplifying mildly, it combined:

• ISAKMP (RFC 2408) The Internet Security Association and Key Management Protocol manages negotiation of connections and defines SAs (Security Associations) as a means of describing connection properties.
• IPsec DOI for ISAKMP (RFC 2407) A Domain Of Interpretation fills in the details necessary to turn the rather abstract ISAKMP protocol into a more tightly specified protocol, so it becomes applicable in a particular domain.
• Oakley key determination protocol (RFC 2412) Oakley creates keys using the Diffie-Hellman key agreement protocol.

For over 200 pages of details, see those RFCs. We give a summary below, but it is far from complete.

IKE has since been updated by RFC 4306, "Internet Key Exchange (IKEv2) Protocol".

Phases of IKE

IKE negotiations have two phases.

Phase one: The two gateways negotiate and set up a two-way ISAKMP SA which they can then use to handle phase two negotiations. One such SA between a pair of gateways can handle negotiations for multiple tunnels.

Phase two: Using the ISAKMP SA, the gateways negotiate IPsec (ESP and/or AH) SAs as required. IPsec SAs are unidirectional (a different key is used in each direction) and are always negotiated in pairs to handle two-way traffic. There may be more than one pair defined between two gateways.

Both of these phases use the UDP protocol and port 500 for their negotiations.

IKE phase two creates IPsec SAs to carry the protected data. These use the ESP or AH protocols. These protocols do not have ports; ports apply only to UDP or TCP.

Everything is negotiable

The IKE protocol is designed to be extremely flexible. Among the things that can be negotiated (separately for each SA) are:

• SA lifetime before rekeying
• encryption algorithm used. The RFCs say you MUST do DES, SHOULD do 3DES and MAY do various others.
• authentication algorithms. The RFCs require MD5 and SHA, with others optional.
• choice of group for Diffie-Hellman key agreement.

The protocol also allows implementations to add their own encryption algorithms, authentication algorithms or Diffie-Hellman groups.

Complications

There are a number of complications:

The gateways must be able to authenticate each other's identities before they can create a secure connection. This host authentication is part of phase one negotiations, and is a required prerequisite for packet authentication used later. Host authentication can be done in a variety of ways.

Phase one can be done in two ways. Main Mode is required by the RFCs; it uses a 6-packet exchange. Aggressive Mode is somewhat faster (only 3 packets) but reveals more to an eavesdropper. This is optional in the RFCs, and some implementations do not support it. For other implementations, however, it is the default.

A new group exchange may take place after phase one but before phase two, defining an additional group for use in the Diffie-Hellman key agreement part of phase two.

Phase two always uses Quick Mode, but there are two variants of that:

One variant provides Perfect Forward Secrecy (PFS). An attacker that obtains your long-term host authentication key does not immediately get any of your short-term packet encryption or packet authentication keys. He must conduct another successful attack each time you rekey to get the short-term keys. Having some short-term keys does not help him learn others. In particular, breaking your system today does not let him read messages he archived yesterday, assuming you've changed short-term keys in the meanwhile.

The other variant disables PFS and is therefore slightly faster. This is generally not recommended, since it is less secure. The protocol provides no way to negotiate which variant will be used. If one gateway is set for PFS and the other is not, the negotiation fails. This has proved a fairly common source of interoperation problems.

Several types of notification message may be sent by either side during either phase, or later. There is also a commit flag which may optionally be set on some messages. The errata page for the RFCs includes two changes related to this, one to clarify the description of its use and one to block a denial of service attack which uses it.

These complications can of course lead to problems, particularly when two different implementations attempt to interoperate. For example, problems the Linux FreeS/WAN team encountered getting their IPsec implementation to interoperate with others included:

• Some implementations (often products crippled by export laws) had the insecure DES algorithm as their only supported encryption method. FreeS/WAN did not implement single DES (even though the RFCs said MUST) and not all others had triple DES (even though the RFCs said SHOULD).
• Windows 2000 IPsec tried to negotiate using Aggressive Mode, optional in the RFCs, which FreeS/WAN did not support. Later on, it used the commit bit, which FreeS/WAN also did not support.
• Various implementations disabled PFS by default, while FreeS/WAN enabled it by default. You had to change one configuration.
• PGPnet, PGP Corporation's IPsec product, used notification messages which FreeS/WAN did not support.
• PGP and FreeS/WAN used slightly different variants of the RSA algorithm (see the two ways to calculate T in the RSA article), so some verifications failed.

Despite things like this, all these implementations did interoperate successfully after some sorting out.

Alternatives to IKE

A number of other protocols have been proposed to do approximately the same job as IKE; all of them are significantly simpler than IKE. Protocols which lost out to IKE in the IETF design process include SwIPe [11] and Photuris, defined in RFC 2522 and RFC 2523. A later proposal was JFK, Just Fast Keying; a PDF is available from Bellovin's page: [12].

The Authentication Header (AH)

Packet authentication can be provided separately from encryption by adding an authentication header (AH) after the IP header but before the other headers on the packet. This is the subject of this section.

Each of the several headers on a packet header contains a "next protocol" field telling the system what header to look for next. IP headers generally have either TCP or UDP in this field. When IPsec authentication is used, the packet IP header has AH in this field, saying that an Authentication Header comes next. The AH header then has the next header type -- usually TCP, UDP or encapsulated IP.

IPsec packet authentication can be added in transport mode, as a modification of standard IP transport. This is shown in this diagram from the RFC:

                 BEFORE APPLYING AH
           ----------------------------
     IPv4  |orig IP hdr  |     |      |
           |(any options)| TCP | Data |
           ----------------------------

                 AFTER APPLYING AH
           ---------------------------------
     IPv4  |orig IP hdr  |    |     |      |
           |(any options)| AH | TCP | Data |
           ---------------------------------
           ||
                except for mutable fields

Authentication can also be used in tunnel mode, encapsulating the underlying IP packet beneath AH and an additional IP header.

                        ||

IPv4 | new IP hdr* | | orig IP hdr* | | |

     |(any options)| AH | (any options) |TCP | Data |
     ------------------------------------------------
     ||
     |           in the new IP hdr                  |

This would normally be used in a gateway-to-gateway tunnel. The receiving gateway then strips the outer IP header and the AH header and forwards the inner IP packet.

The mutable fields referred to are things like the time-to-live field in the IP header. These cannot be included in authentication calculations because they change as the packet travels.

Keyed MD5 and Keyed SHA

The actual authentication data in the header is typically 96 bits and depends both on a secret shared between sender and receiver and on every byte of the data being authenticated. The technique used is HMAC, defined in RFC 2104.

The algorithms involved are the MD5 Message Digest Algorithm or SHA, the Secure Hash Algorithm. For details on their use in this application, see RFCs 2403 and 2404 respectively.

For descriptions of the algorithms themselves, see RFC 1321 for MD5 and FIPS (Federal Information Processing Standard) number 186 from NIST , the US National Institute of Standards and Technology for SHA. Applied Cryptography covers both in some detail, MD5 starting on page 436 and SHA on 442.

These algorithms are intended to make it nearly impossible for anyone to alter the authenticated data in transit. The sender calculates a digest or hash value from that data and includes the result in the authentication header. The recipient does the same calculation and compares results. For unchanged data, the results will be identical. The hash algorithms are designed to make it extremely difficult to change the data in any way and still get the correct hash.

Since the shared secret key is also used in both calculations, an interceptor cannot simply alter the authenticated data and change the hash value to match. Without the key, he or she (or even the dreaded They) cannot produce a usable hash. Sequence numbers

The authentication header includes a sequence number field which the sender is required to increment for each packet. The receiver can ignore it or use it to check that packets are indeed arriving in the expected sequence.

This provides partial protection against replay attacks in which an attacker resends intercepted packets in an effort to confuse or subvert the receiver. Complete protection is not possible since it is necessary to handle legitmate packets which are lost, duplicated, or delivered out of order, but use of sequence numbers makes the attack much more difficult.

The RFCs require that sequence numbers never cycle, that a new key always be negotiated before the sequence number reaches 2^32-1. This protects both against replays attacks using packets from a previous cycle and against birthday attacks on the the packet authentication algorithm.

In Linux FreeS/WAN, the sequence number is ignored for manually keyed connections and checked for automatically keyed ones. In manual mode, there is no way to negotiate a new key, or to recover from a sequence number problem, so we don't use sequence numbers.

Encapsulated Security Payload (ESP)

The ESP protocol is defined in RFC 2406. It provides one or both of encryption and packet authentication. It may be used with or without AH packet authentication.

Note that some form of packet authentication should always be used whenever data is encrypted. Without authentication, the encryption is vulnerable to active attacks which may allow an enemy to break the encryption. ESP should always either include its own authentication or be used with AH authentication.

The RFCs require support for only two mandatory encryption algorithms -- DES, and null encryption -- and for two authentication methods -- keyed MD5 and keyed SHA. Implementers may choose to support additional algorithms in either category.

The authentication algorithms are the same ones used in the IPsec authentication header.

IPsec modes

IPsec can connect in two modes. Transport mode is a host-to-host connection involving only two machines. In tunnel mode, the IPsec machines act as gateways and trafiic for any number of client machines may be carried.

Tunnel mode

Security gateways are required to support tunnel mode connections. In this mode the gateways provide tunnels for use by client machines behind the gateways. The client machines need not do any IPsec processing; all they have to do is route things to gateways.

Transport mode

Host machines (as opposed to security gateways) with IPsec implementations may also support transport mode. In this mode, the host does its own IPsec processing and routes some packets via IPsec. Some implementations, such as FreeS/WAN, do not support transport mode, except as a sort of degenerate tunnel mode in which a single machine acts both as a client and as its own security gateway.