In this article by Eric Crist and Jan Just Keijser, the authors of the book Mastering OpenVPN, you will cover:

Types of VPNs
Comparison of VPNs

(For more resources related to this topic, see here.)

What is a VPN?

Put simply, a VPN allows an administrator to create a local network between multiple computers on varying network segments. In some instances, those machines can be on the same LAN, they can be distant from each other across the vast Internet, or they can even be connected across a multitude of connection media such as wireless uplinks, satellite, dial-up-networking, and so on. The P in VPN comes from the added protection to make that virtual network private. Network traffic that is flowing over a VPN is often referred to as inside the (VPN) tunnel, compared to all other traffic that is outside the tunnel.

In the following figure, network traffic is shown as it traditionally traverses across multiple network segments and the general Internet. Here, this traffic is relatively open to inspection and analysis. Though protected protocols such as HTTPS and SSH are less vulnerable, they are still identifiable: if an attacker is snooping network traffic, he can still see what type of connection is made from which computer to which server. When a VPN is used, the traffic inside the tunnel is no longer identifiable.

introduction-openvpn-img-0

Traffic within a VPN can be anything you would send over a local or wide-area network: web traffic, e-mail, text, graphics, and so on. Examples of some applications include the following:

Automated Teller Machines: ATMs may use a VPN to connect more securely to banking systems.
Open / Free Wi-Fi: With the proliferation of free or open wireless networks, everyday users can utilize a VPN to protect the entirety of their Internet browsing.
Corporate networks: Corporations and other organizations may use a VPN to connect multiple office locations or even entire data centers.
GeoIP / Location-based services: Some websites serve data based on geographic location using GeoIP databases and other records. A VPN can allow you to "bounce" through another machine in a location closer to the content you really want. Internet video services such as Hulu, YouTube, and Netflix are common examples.
Bypassing censorship / Political freedom: Some regimes, such as North Korea or China have extraordinarily restrictive censorship rules. The "Great Firewall of China" is one extreme example. Lock-downs of Internet access during political uprisings such as "Arab Spring" attempt to contain and control reports outside the conflict. VPNs can aid in getting outside those restrictive rules to the greater Internet.

Here is an example of traffic within a VPN. While the VPN itself is routed across the Internet like in the preceding figure, devices along the network path only see VPN traffic; those devices are completely unaware of what in being transmitted inside the private tunnel. Protected protocols, such as HTTPS and SSH, will still be protected inside the tunnel from other VPN users, but will additionally unidentifiable from outside the tunnel. A VPN not only encrypts the traffic within, it hides and protects individual data streams from those outside the tunnel.

introduction-openvpn-img-1

It should be noted that the preceding figure shows both the strength and one of the greatest threats of VPN technologies. The VPN tunnel is dug through routers and firewalls on both sides. Thus, all network traffic that is flowing via the VPN tunnel is bypassing the regular network defenses, unless special measures are taken to police the VPN traffic.

Most VPN implementations utilize some form of encryption and, additionally, authentication. Encryption of the VPN ensures that other parties that may be monitoring traffic between systems cannot decode and further analyze otherwise sensitive data. Authentication has two components, each in a different context.

First, there is user or system authentication that ensures those connecting to the service are authorized. This type of authentication may be in the form of per-user certificates, or a username/password combination. Further, rules specific to a given user can be negotiated such as specific routes, firewall rules, or other scripts and utilities. Typically, these are unique to a single instance, though even that can be configurable (when OpenVPN is used, see --duplicate-cn).

The second component of authentication is added protection to the communication stream. In this case, a method of signing each packet sent is established. Each system verifies the VPN packets it receives are properly signed before decrypting the payload. By authenticating packets that are already encrypted, a system can save processing time by not even decrypting packets that do not meet the authentication rules. In the end, this prevents a very real potential Denial of Service (DoS) attack as well as thwarts Man in the Middle (MITM) attacks, assuming the signing keys are kept secure!

Types of VPNs

There are many VPN products available on the market, both commercial and open source. Almost all of these VPN products can be separated into the four categories:

PPTP-protocol based VPNs
IPSec-protocol based VPNs
SSL-based VPNs
OpenVPN

Some people argue that OpenVPN is also an SSL-based VPN, as it uses an SSL or TLS-like protocol to establish a secure connection. However, we have created a separate category for OpenVPN, as it is different from almost every other SSL-based VPN solution.

We will now go into more detail about each of the four types of VPNs:

PPTP

One of the oldest VPN protocols is the Point-to-Point Tunneling Protocol (PPTP) developed by Microsoft and Ascend in 1999. It is officially registered as RFC2637 (see https://www.ietf.org/rfc/rfc2637.txt for the full standard). The PPTP client has been included in Windows ever since 1995 and is still included in most operating systems.

Nowadays, the PPTP protocol is considered fundamentally insecure, as the strength of the security of the connection is directly related to the strength of the authentication mechanism chosen (example, the password). Thus, an insecure password leads to an insecure VPN connection. Most PPTP setups use the MS-CHAPv2 protocol for encrypting passwords, and it is this protocol which is fundamentally broken. The security of the PPTP protocol, including the Microsoft MS-CHAPv2 extensions, has been discussed in the article available at https://www.schneier.com/paper-pptpv2.html.

It is also possible to use X.509 certificates for securing a PPTP connection, which does lead to a fairly secure connection. However, not all PPTP clients support EAP-TLS, which is needed to allow the use of X.509 certificates.

PPTP uses two channels, a control channel for setting up the connection and another channel for data transport. The control channel is initiated over TCP port 1723. The data channel uses the General Routing Encapsulation (GRE) protocol, which is IP protocol 47. For comparison, "regular" TCP/IP traffic is done using IP protocol 6 (TCP) and 17 (UDP).

PPTP clients are available on almost all operating systems, ranging from Windows to Linux and UNIX derivatives to iOS and Android devices.

IPSec

The IPSec standard is the official IEEE/IETF standard for IP security. It is officially registered as RFC2411 (see https://www.ietf.org/rfc/rfc2411.txt for the full standard). IPSec is also built into the IPv6 standard.

IPSec operates at level 2 and 3 of the OSI model of the network stack. It introduces the concept of security policies, which makes it extremely flexible and powerful, but also notoriously hard to configure and troubleshoot. Security policies allow an administrator to encrypt traffic between two endpoints based on many parameters, such as the source and destination IP address as well as the source and destination TCP or UDP ports.

IPSec can be configured to use pre-shared keys or X.509 certificates to secure the VPN connection. Additionally, it uses either X.509 certificates, one-time passwords, or username/password protocols to authenticate the VPN connection.

There are two modes of operation in IPSec: tunneling mode and transport mode. Transport mode is used most often in combination with the Level 2 Tunneling Protocol (L2TP). This L2TP protocol performs the user authentication as described in the preceding section. The IPSec clients built into most operating systems usually perform IPSec+L2TP, although it is also possible to set up an IPSec-only connection. The IPSec VPN client built into Microsoft Windows uses IPSec+L2TP by default, but it is possible to disable or bypass it. However, this involves cryptic commands and security policy changes.

Like PPTP, IPSec also uses two channels: a control channel for setting up the connection and one for data transport. The control channel is initiated over UDP port 500 or 4500. The data channel uses the Encapsulated Security Payload (ESP) protocol, which is IP protocol 50. For comparison, "regular" TCP/IP traffic is done using IP protocol 6 (TCP) and 17 (UDP). The integrity of IPSec packets is ensured using Hash-based Message Authentication Code (HMAC), which is the same method that OpenVPN uses.

One of the main disadvantages of IPSec is that many vendors have implemented extensions to the standard, which makes it hard (if not impossible) to connect two IPSec endpoints from different vendors.

IPSec software is included in almost all operating systems, as well as firewall, router, and switch firmware.

SSL-based VPNs

The most commonly used VPNs nowadays are SSL-based VPNs, which are based on the SSL/TLS protocol. SSL-based VPNs are often called client-less VPNs or web-based VPNs, although there are some vendors that provide separate client software, such as Cisco AnyConnect and Microsoft SSTP. Most SSL-based VPNs use the same network protocol as is used for secure website (HTTPS), while OpenVPN uses a custom format for encrypting and signing data traffic. This is the main reason why OpenVPN is listed as a separate VPN category.

There is no well-defined standard for SSL-based VPNs, but most use the SSL/TLS protocol to set up and secure the connection. The connection is secured using X.509 certificates in most cases, with one-time password or username/password protocols for authenticating the connection. SSL-based VPNs are very similar to connections used to secure websites (HTTPS) and the same protocol and channel (TCP and port 443) is often used.

Even though SSL-based VPNs are often called web-based or client-less, there are quite a few vendors that use a browser plugin or ActiveX control to "enhance" the VPN connection. This makes the VPN noninteroperable with unsupported browsers or operating systems.

OpenVPN

OpenVPN is often called an SSL-based VPN, as it uses the SSL/TLS protocol to secure the connection. However, OpenVPN also uses HMAC in combination with a digest (or hashing) algorithm for ensuring the integrity of the packets delivered. It can be configured to use pre-shared keys as well as X.509 certificates. These features are not typically offered by other SSL-based VPNs.

Furthermore, OpenVPN uses a virtual network adapter (a tun or tap device) as an interface between the user-level OpenVPN software and the operating system. In general, any operating system that has support for a tun/tap device can run OpenVPN. This currently includes Linux, Free/Open/NetBSD, Solaris, AIX, Windows, Mac OS as well as iOS/Android devices. For all these platforms, client software needs to be installed, which sets OpenVPN apart from client-less or web-based VPNs.

The OpenVPN protocol is not defined in an RFC standard, but the protocol is publicly available as OpenVPN is open source software. The fact that it is open source actually makes OpenVPN more secure than closed-source VPNs, as the code is continually inspected by different people. Also, there is very little chance of secret backdoors being built into OpenVPN.

OpenVPN has the notion of a control channel and a data channel, both of which are encrypted and secured differently. However, all traffic passes over a single UDP or TCP connection. The control channel is encrypted and secured using SSL/TLS, the data channel is encrypted using a custom encryption protocol.

The default protocol and port for OpenVPN is UDP and port 1194. Before IANA granted OpenVPN an official port assignment, older clients (2.0-beta16 and older) defaulted to port 5000.

Comparison of VPNs

Each of the different VPN technologies has its own characteristics, advantages, and disadvantages. Even though this article is about OpenVPN, there are use-cases where, for example, an IPSec-based VPN is more suitable, depending on the requirement of the users.

Advantages and disadvantages of PPTP

The main advantage of PPTP-based VPNs is that the VPN client software is built into most operating systems. Also, the startup time for configuring and initializing a PPTP VPN connection is quite short.

Disadvantages of PPTP-based VPNs are the lack of security and the lack of configuration options on both the client and server side. Furthermore, the EAP-TLS extension that enables the use of X.509 certificates is fully supported only on Microsoft Windows, although a patch exists for the open source pppd package to enable EAP-TLS support. The pppd package is included in almost every Linux distribution. Also, if one must resort to using EAP-TLS, then the ease of setting up a PPTP VPN is greatly diminished. This is because EAP-TLS requires setting up a public key infrastructure, just like IPSec and OpenVPN.

Another major disadvantage of PPTP is the use of the GRE protocol, which does not integrate well with NAT'ing devices.

Advantages and disadvantages of IPSec

Advantages of the IPSec protocol are its strong security, good support by different vendors and platforms, including xDSL and Wi-Fi routers, as well as the ability to use fine-grained security policies to control the flow of traffic.

The downsides of IPSec are that it is notoriously difficult to configure and troubleshoot, different IPSec implementations from different vendors do not play nicely together, and IPSec does not integrate well with NAT'ted networks. Most notably, it is not recommended and sometimes not even possible to run an IPSec server that is on a NAT'ted network.

Advantages and disadvantages of SSL-based VPNs

SSL-based VPNs or web-based VPNs have the advantage that there is no or very little client software involved. This makes installation and initialization on the client side very easy.

The disadvantage of a web-based VPN is that it is often not a full-blown VPN and allows access to a single server or set of servers. Also, it is harder to share local data with the remote site or server.

Advantages and disadvantages of OpenVPN

Advantages of OpenVPN are its ease of deployment, its configurability, and the ability to deploy OpenVPN in restricted networks, including NAT'ted networks. Also, OpenVPN includes security features that are as strong as IPSec-based solutions, including hardware token security and support for different user authentication mechanisms.

Disadvantages of OpenVPN are its current lack of scalability and its dependence on the installation of client-side software. Another disadvantage is the lack of a GUI for configuration and management. Especially, the tap interface driver for Microsoft Windows has often caused deployment issues when a new version of Windows is released.

Summary

In this article, we started out by explaining what a VPN is. We then discussed some examples of different types of VPN protocols, including PPTP, IPSec, and OpenVPN. At the end, we compared the various VPNs.