Using a Linux L2TP/IPsec VPN server with Mac OS X and iPhone

I have made the following webpages on using L2TP/IPsec with Linux:

• Using a Linux L2TP/IPsec VPN server
• Using a Linux L2TP/IPsec VPN server with Windows 2000/XP
• Using a Linux L2TP/IPsec VPN server with Windows Vista
• Using a Linux server with the Microsoft L2TP/IPSec VPN Client
• Using a Linux server with third-party L2TP/IPsec clients
• Using a Linux L2TP/IPsec VPN server with Pocket PC
• Using a Linux L2TP/IPsec VPN server with Mac OS X
• Using Linux as an L2TP/IPsec client

1.2 Author

The author of this document is Jacco de Leeuw. Corrections, additions, extra information etc. are much appreciated.

• Introduction
• Contents
• Background information
• L2TP discussion
  
• Installation (Mac OS X)
• Configuration (Linux)
• Connecting with a PSK (Mac OS X 10.5 and higher)
• Connecting with a PSK (Mac OS X 10.3 and 10.4)
• Connecting with a certificate (Mac OS X 10.5 and higher)
• Connecting with a certificate (Mac OS X 10.4)
• Alternative methods of connecting with a certificate (Mac OS X 10.3 and higher)
• NAT-Traversal
• Connecting with the Apple iPhone
  
• Perfect Forward Secrecy (PFS)
• Advanced topics
  
• Troubleshooting
• Split tunnelling
  
• VPN alternatives for the Mac
  
• Acknowledgements
  
• Revision history

Mac OS X 10.3 and higher ship with an L2TP/IPsec client.�The Mac's IPsec implementation is a fork based on KAME which is known to interoperate with Openswan. I also received a report from Chris Andrews that Mac OS X's VPN client interoperates with a setup that consists of the native IPsec implementation of the Linux kernel 2.6, plus l2tpd and� ipsec-tools (racoon).

Back to Contents

The big question of course is: why would you want to use L2TP with the Mac? L2TP/IPsec has the advantage that it is an official IETF standard. Furthermore, IPsec is generally considered to be more secure than PPTP. As Apple writes, "L2TP is Mac OS X Server's preferred VPN protocol due to its superior transport encryption and its ability to be authenticated via Kerberos". These are indeed valid points: a Mac client and an Openswan server can in principle agree upon the use of strong encryption (AES, SHA-1 etc.). I have not used Kerberos on the Mac so I can't comment on that.

Some users prefer Mac OS X's L2TP/IPsec client simply because it is free. Others may prefer a third-party client, because of the support that the third-party will provide.

Back to Contents

There is no installation required. The L2TP/IPsec client is installed by default on Mac OS X 10.3 and higher.

1. Like any other L2TP/IPsec client, Mac OS X requires the use of left/rightprotoport parameters in ipsec.conf which is your Openswan configuration file. These parameters are not available in vanilla FreeS/WAN. Strongsec's X.509 patch for FreeS/WAN provides these parameters. This patch is included with Openswan and strongSwan. The problem is that Mac OS X uses a 'floating port'. Normally UDP port 1701 is used through the following Openswan parameters:
   
   left=x.x.x.x
   leftprotoport=17/1701
   right=y.y.y.y
   rightprotoport=17/1701
   
   But these do not work with Mac OS X. They result in the following error:
   
   � "cannot respond to IPsec SA request because no connection is known for x.x.x.x:17/1701...y.y.y.y:17/57937"
   

   Where 57937 (or any other number) is the ephemeral ("floating") UDP port. According to the L2TP/IPsec standard, this seems to be allowed (Apple might use this for QoS, load balancing or possibly even NAT-T?). The server has no way of knowing the UDP port in advance, so you need to to accept all possible UDP source ports. This is supported by Openswan 1.0.2+, Openswan 2.2.0+ and strongSwan 2.0.0+. You shouldn't be using versions older than that anyway. (If you want to know the details: you need at least version 0.9.38 of Strongsec's X.509 patch for Openswan 1.x/ FreeS/WAN 1.99 or version 1.5.3 for Openswan 2.x/ FreeS/WAN 2.04/2.05. Pluto's startup messages in /var/log/messages indicate which version of the X.509 patch you are using (e.g.: "Starting Pluto FreeS/WAN Version 2.04 X.509-1.5.4"). What you need is the %any port wildcard parameter:
   
   left=x.x.x.x
   leftprotoport=17/1701
   right=y.y.y.y
   rightprotoport=17/%any
   

   (This supersedes the EXPERIMENTAL patch that I made for other FreeS/WAN versions, where you needed to specify 17/0 instead of 17/%any).
   
   
2. If you are using KLIPS, you may want to run the L2TP daemon only on the internal interface using the listen-addr parameter and then use iptables to forward all packets to the daemon. But because of the floating port mentioned above, a small modification to the iptables rule is needed. You need to forward (DNAT) all ports to the interface where the L2TP server is listening (e.g. the internal network interface):
   
   iptables -t nat --append PREROUTING -i ipsec0 -p udp --dport 1701 -j DNAT --to-destination 192.168.1.98
   
   (Where 192.168.1.98 is the IP address of the internal interface). This is similar to what you can find on my main L2TP/IPsec page. The difference is the removal of --sport 1701, so that all source ports are forwarded, not just port 1701. Unfortunately this trick does not work with the kernel 2.6 IPsec implementation ("NETKEY") because NETKEY does not have ipsec0 style interfaces and NAT-after-IPsec is currently broken on vanilla kernel 2.6. There might be a few ways to solve this problem on 2.6 kernels. The first two mentioned on that link should work, although I have not tested these myself yet.
   
   
3. I noticed that Mac OS X's L2TP client does not supply a hostname during the L2TP connection negotiation (our_params.host_name[0] = 0?). This is not a problem, as long as you use at least version 8jdl of my l2tpd RPM (which contains this hostname patch). The MSL2TP client has the same issue.
   
   
4. If you happen to use l2tpns as your L2TP daemon, you may need to apply a patch to the source code because l2tpns' HELLO messages apparently confuse Mac OS X. More information can be found on this webpage by Wolfgang Hennerbichler.
   

In most cases you do not know the client's IP address in advance, so using right=y.y.y.y is out of the question then. Instead you would typically use right=%any which results in the following:

left=x.x.x.x
leftprotoport=17/1701
right=%any
rightprotoport=17/%any
rightsubnet=vhost:%priv,%no
forceencaps=yes

You may notice the parameter forceencaps=yes. This was added because of a bug in Openswan (incorrect routing) which rears its head when you use right=%any in combination with rightprotoport=17/%any. Unfortunately this is a very common configuration when you want to support Mac clients. There are four workarounds but each one has its drawbacks: A) use right=client.ip.add.ress instead of %any: this only supports 1 client with a fixed IP address, not a very attractive prospect. B) use 17/1701 instead of 17/%any: Mac clients can no longer connect. C) remove the parameter rightsubnet=vhost:%priv,%no : NAT is no longer supported D) add the parameter forceencaps=yes: this forces NAT traversal, so it introduces unnecessary overhead if clients are not behind NAT. Another drawback of option D is that Windows XP/Vista/7 clients will then need a registry modification because the server appears to be behind NAT. Nevertheless, option D is probably the easiest solution, if you happen to have Mac clients.

Back to Contents

• Open "System Preferences".
  
• Click on the "Network" icon which can be found under "Internet & Network".
• At the left hand of the window you see a list of connections. If you have upgraded from a previous version of Mac OS X you might see a connection called "VPN (L2TP)".
• Click the brown padlock at the bottom left hand side of the window. You are asked for your administrator account details. The padlock unlocks with a metalic sound.
  
• If there is no existing L2TP connection (indicated by a grey padlock with black and white striping), add one by clicking the "+" button at the bottom left corner of the window. At "Interface:", select "VPN". A select box called "VPN Type" appears. Select "L2TP over IPsec". Enter a name for the L2TP/IPsec connection, such as "VPN (L2TP)".
  
• In the list on the left hand side, click on the newly created L2TP connection.
• At the "Configuration" pull-down menu, select "Add Configuration...". Enter a name for the connection that you would to use.
• At "Server Address", enter the hostname or IP address of the VPN server.
• At "Accountname", enter the username that will be use in the PPP authentication phase.
• Click "Authentication Settings...".
  
• Mac OS X lists a number of user authentication options under "User Authentication". I would recommend that you first try CHAP, because it is easier to use. So enter your CHAP password in the "Password:" field and also configure your Linux server for CHAP.
  
• At "Machine Authentication", enter your "Shared secret". This is the IPsec PSK that you entered in ipsec.secrets on your Openswan server.
• Leave the "Group Name" blank. It is only used for Cisco style VPNs.
• Click "OK" and then click "Apply".
  
• Click "Connect".
• You may be asked to enter your Keychain password.
  
• The L2TP/IPsec connection will be set up.

    dpddelay=40
    dpdtimeout=130
    dpdaction=clear

The GUI in both Mac OS X 10.3 and 10.4 supports PSKs. Panther and Tiger are no longer supported by Apple with security updates, so I would not recommend using these old versions on the Internet as VPN clients. For archival purposes, the procedure is as follows:

• Open the Applications folder.
• Open "Internet Connect".
• If you see a padlock with the text "VPN (L2TP)", continue with the next step. If you do not see such a padlock, you have to add it: open the menu, choose "File" and then "New VPN Connection". You will be asked to choose between "L2TP over IPsec" and "PPTP". Select "L2TP over IPsec".
• Click on the padlock with the text "VPN (L2TP)".
  
• In the "Configuration" pull-down menu, select "Edit configurations...".
• Enter the "User Authentication" details for your VPN connection (Mac OS X 10.3, Mac OS X 10.4): the hostname or IP address of your Linux VPN server and the "Account Name". Mac OS X lists a number of user authentication options. I would recommend that you first try CHAP, because it is easier to use. So enter your CHAP password in the "Password:" field and also configure your Linux server for CHAP.
  
• Enter your "Shared secret". This is the IPsec PSK that you entered in ipsec.secrets on your Openswan server.
• Save settings.
• Click "Connect".
• You may be asked to enter your Keychain password.
  
• The L2TP/IPsec connection will be set up.
  

• Login password: this is the password that you use to login on the Mac. Importing machine certificates for L2TP/IPsec authentication requires an account with Administrator privileges.
  
• Certificate password: this is the password that protects the PKCS#12 file containing the machine certificate. You need the password to access the encrypted parts in the file.
  
• CHAP password: this is the "User Authentication Password" that you enter in the "Internet Connect" application. It is used in the L2TP/PPP phase of the L2TP/IPsec protocol.
  
• Preshared Key: this is something which can be regarded as a password (see the previous section). Note that PSKs are not involved when you use certificates for authentication. I'm just mentioning PSKs here for the sake of completeness.
  

• The server certificate MUST contain an "subjectAltName" ID which matches the hostname ("USER_FQDN") or IP address ("IPV4_ADDR") of the server.
• The server certificate MUST either not contain an "Extended Key Usage" (EKU) or MUST include "ikeIntermediate" (1.3.6.1.5.5.8.2.2) as an Extended Key Usage.

• Open the Applications folder.
• Open the Utilities folder.
• Open the "Keychain Access" application
• Click the orange padlock at the top left hand corner. Enter your password (importing a machine certificate requires Administrator privileges). You hear a metalic noise and the padlock is unlocked.
  
• If you don't see a "Keychains" pane in the left hand side of the window, click the symbol in the bottom left corner of the window. The "Keychains" pane should now appear and the triangle should now point upwards.
  
• Click on the "System" keychain. It will be highlighted.
• From the menu, select "File -> Import Items... " (or press Option-Shift-I).
• Select the PKCS#12 file.
• Enter your password.
• Enter the certificate password.
• Click "Always Trust".
  
• Typically, three items will be added to the System keychain at this stage: a private key (grey key symbol), a root certificate (orange symbol with a blue plus sign) and a machine certificate (blue symbol). You should not see any red crosses through any of these newly imported items. If this is not the case, check the properties of these items for reasons why they are untrusted.
• Exit the Key Access application.
  

No machine certificates found.

Certificate authentication cannot be used because your keychain does not contain any suitable certificates. Use Keychain Access to import the appropriate certificates into your keychain. If you do not have the certificates required for authentication, contact your network administrator.

• Logon to Mac OS X as a user with administrative privileges
• Open the Applications folder.
• Open the Utilities folder.
• Open the "Terminal" application
• Type the following command (all on one line) and press Enter: � sudo "/Applications/Utilities/Keychain Access.app/Contents/MacOS/Keychain Access"
• Enter your login password.
• If the bottom left button says "Hide keychains", skip to the next step. If the button says "Show keychains", click that button. You should now see a number of keychains at the left-hand top: "login", "X509-roots", "X509-certificates" and "System".
• Click on the "System" keychain. It will be highlighted.
• From the menu, select File -> Import (or press Option-Shift-I).
• Select the PKCS#12 file.
• Enter the certificate password.
  
• Enter the Keychain Access password.
  
• Typically, three items will be added to the System keychain at this stage: a private key (grey key symbol), a root certificate (orange symbol) and a machine certificate (blue symbol). If you examine the details ('summary') of the machine certificate or the root certificate, you will notice that they are reported as 'untrusted'.
• Drag the root certificate from the "System" keychain to the "X509Anchors" keychain. This is unlike the procedure for Mac OS X 10.5+. This step will change the status of the certificates from untrusted to trusted. (Unfortunately the status is not immediately updated but if you quit the Keychain Access application and start it again, you will notice that indeed they are now trusted).
• Exit the Key Access application.
  

openssl pkcs12 -in yourcrt.p12 -cacerts -out ca.pem -nokeys
openssl pkcs12 -in yourcrt.p12 -clcerts -out client.pem -nokeys
openssl pkcs12 -in yourcrt.p12 -nocerts -out key.pem -nodes
cp /System/Library/Keychains/X509Anchors $HOME/Library/Keychains/X509Anchors.bkp
cp /Library/Keychains/System.keychain $HOME/Library/Keychains/System.keychain.bkp
certtool i ca.pem k=X509anchors.bkp v
certtool i client.pem r=key.pem f=1 k=System.keychain.bkp v
sudo cp $HOME/Library/Keychains/X509Anchors.bkp /System/Library/Keychains/X509Anchors
sudo cp $HOME/Library/Keychains/System.keychain.bkp /Library/Keychains/System.keychain
rm ca.pem client.pem key.pem

"OS X creates config-files on the fly, but the main racoon.conf is not touched, instead there's a parameter in racoon.conf that says: include "/etc/racoon/remote/*.conf"� So I changed the racoon.conf just to my needs (Certificates and so on), and removed this include-line. With that, you can set the connection up via the GUI, and racoon will be called by the GUI with the correct parameters and the policies will be set correctly. This might be a problem if you have more than 1 network (different certificates) to connect to".

• PPTP with MS-CHAPv2 based user authentication.
  
• L2TP/IPsec with Preshared Key (PSK) based IPsec authentication and MS-CHAPv2 based user authentication.
• L2TP/IPsec with CryptoCard authentication, but only if the "shared secret" method is used.

• Pure IPsec (i.e. without L2TP).
  
• IPsec with certificate-based authentication.
  
• EAP-TLS (i.e. PPP user certificates or smartcards).
• L2TP/IPsec Kerberos authentication tokens.
• RSA-SecurID (EAP-RSA) authentication tokens.

• Select "General" -> "Network" -> "VPN" and then tap "Settings".
• Select "L2TP".
  
• Enter the L2TP/IPsec server's address.
• Tap "Account" and enter your username (for user authentication in the PPP phase of the VPN).
• If you want to store your password on the device, tap "Password" and enter your password.
• Tap "Secret" and enter your Preshared Key (for IPsec authentication). This is similar to the "Shared Secret" for Machine Authentication if you are familiar with Mac OS X.
  
• Tap "Save" in the upper right corner.

There are valid reasons for using PFS. If you really need PFS and you don't want to modify the default racoon.conf, you could decide to switch to IPsec without L2TP (e.g. through a third-party IPsec client or configuration utility). Then it is possible to enable PFS.

Back to Contents

Back to Contents

• Run "sudo vi /etc/racoon/racoon.conf" on the Mac OS X command line.
• Change the following line:� log debug;
• Edit your syslog configuration file: sudo vi /etc/syslog.conf
• At the end of this file, add:����� *.debug������������� /var/log/debug.log
• Reload the syslog daemon:
• ps auux | grep racoon
• sudo kill -HUP <process_ID_returned_by_previous_command>
• Stop racoon: sudo killall racoon
• Connect to the Linux VPN server (this will start racoon again).
• Examine /var/log/debug.log for errors.
  

• Find the process ID of racoon and attach a trace to it:
• ps auux | grep racoon
• sudo ktrace -p <process_ID_returned_by_previous_command>
• This will create a file called ktrace.out in your current working directory.
• Connect to the Linux VPN server.
• Wait a while for things to get written to the log.
• Decode the binary file: sudo kdump -f ktrace.out > ktrace.txt
• Examine the resulting text file: sudo less ktrace.txt
  

• Open "Internet Connect".
• From the menu, select "Connect" and then "Options".
• Uncheck "Send all traffic over VPN connection".
  

• IPsecuritas from Lobotomo Software adds a graphical front-end to the built-in IPSec core, "allowing you to setup secure communications within minutes". It is freeware and known to interoperate with Openswan, KAME (racoon) and NETKEY. It contains a Certificate Manager. A howto with screenshots for connecting with IPsecuritas to an Openswan server is available.
  
• VaporSec: Mac OS X 10.2+ contains IPsec support through KAME. It does not support L2TP/IPsec. VaporSec is a free user interface for KAME which is normally configured through ASCII configuration files. VaporSec creates IPSec policies that will allow you to connect to other IPSec devices. This may be another Mac OS X 10.2+ machine or a third-party firewall, VPN or other IPSec device. I think it has been tested with FreeS/WAN. I see no mention of compatibility with Mac OS X 10.3, but according to Patrik Tschudin, it does work.
• VPN Tracker from Equinux is a commercial IPsec client for Mac OS X. Equinux makes a couple of statements:
  
  • "Mac OS X only supports a special protocol and not the standard IPsec".
    This is not quite true. Mac OS X's GUI only supports L2TP/IPsec, which is not a "special protocol" but an official IETF standard (RFC 3193). It is simply L2TP tunnelled within IPsec. And if you are comfortable with Mac OS X commandline, you are probably capable of using standard IPsec. It's just that VPN Tracker automates that for you with a very nice GUI.
  • "Therefore it's not compatible with most third party devices out there".
    That's because not all third party devices currently support L2TP-over-IPsec, but the number is increasing. L2TP is used by Microsoft, Cisco, Checkpoint and other vendors to allow remote users to obtain an IP address from the internal network. Pure IPsec cannot do that. I'm not saying that L2TP is the best solution (IKEv2 seems much more interesting) but it is a ratified standard and it is already supported by many clients and servers.
  • "VPN Tracker on the other hand comes with predefined connection types for the majority of VPN manufacturers and extensive interoperability how-to documents are available for most set-ups".
    In a perfect world you would not have to have different connection types for different vendors because all vendors would adhere to the same standard and implement it without interoperability issues. In practice, things are different. I have not used all third party devices supported by VPN Tracker but the ones I did test are generally compliant with the standards. Nevertheless, I think it is a big plus that VPN Tracker has identified remaining issues and provides ready-to-use configurations.
  • "A complete certificate management solution with built into CA features and an import/export that enables an easy rollout for enterprise scenarios".
    I think this feature is only in the Professional version of VPN Tracker. You won't need CA functionality if you only want a VPN client and you have made your own provisions to generate your certificates. I must admit that from the screenshots it looks very nice.
    

• Mac OS X supports PPTP. This VPN protocol has its advantages and disadvantages, compared to L2TP/IPsec. In a nutshell, PPTP is not an official standard, considered to be less secure than IPsec, but also easier to use.
• Apani Mac OS X VPN Client: Known to work with Nortel Networks VPN Router (formerly Contivity) and Cisco VPN 3000. Status unknown with Openswan. Pricing and licensing terms are not known to me but I suspect there is no benefit in using this client for Openswan.
• Cisco VPN client for Mac: uses IPsec with XAUTH (extended authentication) which is supported by only a few other vendors. XAUTH support is currently in development on Openswan but I do not know if this client will work with Openswan. I also don't know the pricing and licensing terms but I suspect it may not make (financial) sense to use the Cisco client with Openswan. You need a valid CCO userid to download the Cisco VPN client from the Cisco website. Make sure you use recent versions of Mac OS X and the Cisco VPN.
  
• Checkpoint Secureclient: Status unknown with Openswan. Pricing and licensing terms are not known to me but I suspect there is no benefit in using this client for Openswan.
• DigiTunnel from Gracion: this is a PPTP client which has some extra features over the built-in Mac OS X PPTP client, such as split-routing (many companies consider this a security risk, though).
  
• OpenVPN. This is an Open Source project with clients for Windows, Linux, Mac OS X, Darwin, Solaris etc.

Back to Contents

May, 24 2010: Started section about (Snow) Leopard.
Mar 5, 2008: Started section about DHCP Inform.
Feb 26, 2008: Added section about split tunnellng.
Jan 8, 2008: iPhone compatible with the Astaro Security Gateway appliance.
Jan 4, 2008: There is a bug in Openswan which pops up when you want to support Mac clients (workarounds mentioned).
Oct 30, 2007: First tests with Mac OS X 10.5 (Leopard) are inconclusive: some users report success, others had problems.
Oct 25, 2007: Added more info about requirements for server certificates.
Sep 19, 2007: Added some remarks about the iPhone.
May 22, 2006: Mac OS X 10.4 supports PAP/CHAP (10.4.4+?). Also new: CryptoCard (10.4.6+) and EAP-TLS (10.4.4+?).
Feb 12, 2006: Mac OS X 10.4.4+ supports the IETF NAT-T standard (RFC 3947).
Dec 6, 2005: That patch is no good. Get Openswan 2.4.5 or higher.
Nov 21, 2005: Made patch (based on Peter's patch) that swaps the NAT-D hashes: needed for NAT-T.
Nov 12, 2005: Added command line procedure for importing machine certificates on Mac OS X 10.4. Update on DPD, IPCOMP.
Nov 7, 2005: Added procedure for importing machine certificates on Mac OS X 10.4.
Sep 29, 2005: (Updated) patch by Peter Van Der Beken available for NATed Mac clients.
Jun 26, 2005: Reportedly, the VPN problems in Tiger have been fixed in update 10.4.1.
May 15, 2005: s2svpnadmin ("site-to-site VPN admin") only included in Mac OS X 10.4 Server, not Client� :-(
May 3, 2005: Mac OS X 10.4 breaks several third-party VPN clients. Whoops. L2TP/IPsec (and PPTP) should work.
Apr 27, 2005: First report of Mac OS X 10.4: has a GUI for certs. NAT-T still non-standard.
Jan 16, 2005: Added mention of a method to use certificates on Mac OS X 10.3.
Jan 8, 2005: Mac OS X's NAT-T support is bogus. Apple implemented the RFC but mislabeled it.
Nov 23, 2003: Confirmed by Apple employee: no cert support yet.
Nov 16, 2003: Added screenshots and more details.
Nov 15, 2003: Succesfully tested with PSKs (had to make a FreeS/WAN patch though).
Nov 6, 2003: Page created.

Jacco de Leeuw