This article by Nathan Kozyra, author of Mastering Concurrency in Go, tackles one of the Internet’s most famous and esteemed challenges and attempt to solve it with core Go packages.
(For more resources related to this topic, see here.)
We’ve built a few usable applications; things we can start with and leapfrog into real systems for everyday use. By doing so, we’ve been able to demonstrate the basic and intermediate-level patterns involved in Go’s concurrent syntax and methodology.
However, it’s about time we take on a real-world problem—one that has vexed developers (and their managers and VPs) for a great deal of the early history of the Web.
In addressing and, hopefully, solving this problem, we’ll be able to develop a high-performance web server that can handle a very large volume of live, active traffic.
For many years, the solution to this problem was solely to throw hardware or intrusive caching systems at the problem; so, alternately, solving it with programming methodology should excite any programmer.
We’ll be using every technique and language construct we’ve learned so far, but we’ll do so in a more structured and deliberate way than we have up to now. Everything we’ve explored so far will come into play, including the following points:
- Creating a visual representation of our concurrent application
- Utilizing goroutines to handle requests in a way that will scale
- Building robust channels to manage communication between goroutines and the loop that will manage them
- Profiling and benchmarking tools (JMeter, ab) to examine the way our event loop actually works
- Timeouts and concurrency controls—when necessary—to ensure data and request consistency
Attacking the C10K problem
The genesis of the C10K problem is rooted in serial, blocking programming, which makes it ideal to demonstrate the strength of concurrent programming, especially in Go.
When he asked this in 1999, for many server admins and engineers, serving 10,000 concurrent visitors was something that would be solved with hardware. The notion that a single server on common hardware could handle this type of CPU and network bandwidth without falling over seemed foreign to most.
The crux of his proposed solutions relied on producing non-blocking code. Of course, in 1999, concurrency patterns and libraries were not widespread. C++ had some polling and queuing options available via some third-party libraries and the earliest predecessor to multithreaded syntaxes, later available through Boost and then C++11.
Over the coming years, solutions to the problem began pouring in across various flavors of languages, programming design, and general approaches.
Any performance and scalability problem will ultimately be bound to the underlying hardware, so as always, your mileage may vary. Squeezing 10,000 concurrent connections on a 486 processor with 500 MB of RAM will certainly be more challenging than doing so on a barebones Linux server stacked with memory and multiple cores.
It’s also worth noting that a simple echo server would obviously be able to assume more cores than a functional web server that returns larger amounts of data and accepts greater complexity in requests, sessions, and so on, as we’ll be dealing with here.
Failing of servers at 10,000 concurrent connections
When the Web was born and the Internet commercialized, the level of interactivity was pretty minimal. If you’re a graybeard, you may recall the transition from NNTP/IRC and the like and how extraordinarily rudimentary the Web was.
To address the basic proposition of [page request] → [HTTP response], the requirements on a web server in the early 1990s were pretty lenient. Ignoring all of the error responses, header reading, and settings, and other essential (but unrelated to the in → out mechanism) functions, the essence of the early servers was shockingly simple, at least compared to the modern web servers.
The first web server was developed by the father of the Web, Tim Berners-Lee.
Developed at CERN (such as WWW/HTTP itself), CERN httpd handled many of the things you would expect in a web server today—hunting through the code, you’ll find a lot of notation that will remind you that the very core of the HTTP protocol is largely unchanged. Unlike most technologies, HTTP has had an extraordinarily long shelf life.
Written in C in 1990, it was unable to utilize a lot of concurrency strategies available in languages such as Erlang. Frankly, doing so was probably unnecessary—the majority of web traffic was a matter of basic file retrieval and protocol. The meat and potatoes of a web server were not dealing with traffic, but rather dealing with the rules surrounding the protocol itself.
You can still access the original CERN httpd site and download the source code for yourself from http://www.w3.org/Daemon/. I highly recommend that you do so as both a history lesson and a way to look at the way the earliest web server addressed some of the earliest problems.
However, the Web in 1990 and the Web when the C10K question was first posed were two very different environments.
By 1999, most sites had some level of secondary or tertiary latency provided by third-party software, CGI, databases, and so on, all of which further complicated the matter. The notion of serving 10,000 flat files concurrently is a challenge in itself, but try doing so by running them on top of a Perl script that accesses a MySQL database without any caching layer; this challenge is immediately exacerbated.
By the mid 1990s, the Apache web server had taken hold and largely controlled the market (by 2009, it had become the first server software to serve more than 100 million websites).
Apache’s approach was rooted heavily in the earliest days of the Internet. At its launch, connections were initially handled first in, first out. Soon, each connection was assigned a thread from the thread pool. There are two problems with the Apache server. They are as follows:
- Blocking connections can lead to a domino effect, wherein one or more slowly resolved connections could avalanche into inaccessibility
- Apache had hard limits on the number of threads/workers you could utilize, irrespective of hardware constraints
It’s easy to see the opportunity here, at least in retrospect. A concurrent server that utilizes actors (Erlang), agents (Clojure), or goroutines (Go) seems to fit the bill perfectly. Concurrency does not solve the C10k problem in itself, but it absolutely provides a methodology to facilitate it.
The most notable and visible example of an approach to the C10K problem today is Nginx, which was developed using concurrency patterns, widely available in C by 2002 to address—and ultimately solve—the C10k problem. Nginx, today, represents either the #2 or #3 web server in the world, depending on the source.
Using concurrency to attack C10K
There are two primary approaches to handle a large volume of concurrent requests. The first involves allocating threads per connection. This is what Apache (and a few others) do.
On the one hand, allocating a thread to a connection makes a lot of sense—it’s isolated, controllable via the application’s and kernel’s context switching, and can scale with increased hardware.
One problem for Linux servers—on which the majority of the Web lives—is that each allocated thread reserves 8 MB of memory for its stack by default. This can (and should) be redefined, but this imposes a largely unattainable amount of memory required for a single server. Even if you set the default stack size to 1 MB, we’re dealing with a minimum of 10 GB of memory just to handle the overhead.
This is an extreme example that’s unlikely to be a real issue for a couple of reasons: first, because you can dictate the maximum amount of resources available to each thread, and second, because you can just as easily load balance across a few servers and instances rather than add 10 GB to 80 GB of RAM.
Even in a threaded server environment, we’re fundamentally bound to the issue that can lead to performance decreases (to the point of a crash).
First, let’s look at a server with connections bound to threads (as shown in the following diagram), and visualize how this can lead to logjams and, eventually, crashes:
This is obviously what we want to avoid. Any I/O, network, or external process that can impose some slowdown can bring about that avalanche effect we talked about, such that our available threads are taken (or backlogged) and incoming requests begin to stack up.
We can spawn more threads in this model, but as mentioned earlier, there are potential risks there too, and even this will fail to mitigate the underlying problem.
Taking another approach
In an attempt to create our web server that can handle 10,000 concurrent connections, we’ll obviously leverage our goroutine/channel mechanism to put an event loop in front of our content delivery to keep new channels recycled or created constantly.
For this example, we’ll assume we’re building a corporate website and infrastructure for a rapidly expanding company. To do this, we’ll need to be able to serve both static and dynamic content.
The reason we want to introduce dynamic content is not just for the purposes of demonstration—we want to challenge ourselves to show 10,000 true concurrent connections even when a secondary process gets in the way.
As always, we’ll attempt to map our concurrency strategy directly to goroutines and channels. In a lot of other languages and applications, this is directly analogous to an event loop, and we’ll approach it as such. Within our loop, we’ll manage the available goroutines, expire or reuse completed ones, and spawn new ones where necessary.
In this example visualization, we show how an event loop (and corresponding goroutines) can allow us to scale our connections without employing too many hard resources such as CPU threads or RAM:
The most important step for us here is to manage that event loop. We’ll want to create an open, infinite loop to manage the creation and expiration of our goroutines and respective channels.
As part of this, we will also want to do some internal logging of what’s happening, both for benchmarking and debugging our application.
Building our C10K web server
Our web server will be responsible for taking requests, routing them, and serving either flat files or dynamic files with templates parsed against a few different data sources.
As mentioned earlier, if we exclusively serve flat files and remove much of the processing and network latency, we’d have a much easier time with handling 10,000 concurrent connections.
Our goal is to approach as much of a real-world scenario as we can—very little of the Web operates on a single server in a static fashion. Most websites and applications utilize databases, CDNs (Content Delivery Networks), dynamic and uncached template parsing, and so on. We need to replicate them whenever possible.
For the sake of simplicity, we’ll separate our content by type and filter them through URL routing, as follows:
- /static/[request]: This will serve request.html directly
- /template/[request]: This will serve request.tpl after its been parsed through Go
- /dynamic/[request][number]: This will also serve request.tpl and parse it against a database source’s record
By doing this, we should get a better mixture of possible HTTP request types that could impede the ability to serve large numbers of users simultaneously, especially in a blocking web server environment.
You can find Go’s exceptional library to generate safe data-driven templating at http://golang.org/pkg/html/template/.
By safe, we’re largely referring to the ability to accept data and move it directly into templates without worrying about the sort of injection issues that are behind a large amount of malware and cross-site scripting.
For the database source, we’ll use MySQL here, but feel free to experiment with other databases if you’re more comfortable with them. Like the html/template package, we’re not going to put a lot of time into outlining MySQL and/or its variants.
Benchmarking against a blocking web server
It’s only fair to add some starting benchmarks against a blocking web server first so that we can measure the effect of concurrent versus nonconcurrent architecture.
For our starting benchmarks, we’ll eschew any framework, and we’ll go with our old stalwart, Apache.
For the sake of completeness here, we’ll be using an Intel i5 3GHz machine with 8 GB of RAM. While we’ll benchmark our final product on Ubuntu, Windows, and OS X here, we’ll focus on Ubuntu for our example.
Our localhost domain will have three plain HTML files in /static, each trimmed to 80 KB. As we’re not using a framework, we don’t need to worry about raw dynamic requests, but only about static and dynamic requests in addition to data source requests.
For all examples, we’ll use a MySQL database (named master) with a table called articles that will contain 10,000 duplicate entries. Our structure is as follows:
CREATE TABLE articles ( article_id INT NOT NULL AUTO_INCREMENT, article_title VARCHAR(128) NOT NULL, article_text VARCHAR(128) NOT NULL, PRIMARY KEY (article_id) )
With ID indexes ranging sequentially from 0-10,000, we’ll be able to generate random number requests, but for now, we just want to see what kind of basic response we can get out of Apache serving static pages with this machine.
For this test, we’ll use Apache’s ab tool and then gnuplot to sequentially map the request time as the number of concurrent requests and pages; we’ll do this for our final product as well, but we’ll also go through a few other benchmarking tools for it to get some better details.
Apache’s AB comes with the Apache web server itself. You can read more about it at http://httpd.apache.org/docs/2.2/programs/ab.html.
You can download it for Linux, Windows, OS X, and more from http://httpd.apache.org/download.cgi.
The gnuplot utility is available for the same operating systems at http://www.gnuplot.info/.
So, let’s see how we did it. Have a look at the following graph:
Ouch! Not even close. There are things we can do to tune the connections available (and respective threads/workers) within Apache, but this is not really our goal. Mostly, we want to know what happens with an out-of-the-box Apache server. In these benchmarks, we start to drop or refuse connections at around 800 concurrent connections.
More troubling is that as these requests start stacking up, we see some that exceed 20 seconds or more. When this happens in a blocking server, each request behind it is queued; requests behind that are similarly queued and the entire thing starts to fall apart.
Even if we cannot hit 10,000 concurrent connections, there’s a lot of room for improvement. While a single server of any capacity is no longer the way we expect a web server environment to be designed, being able to squeeze as much performance as possible out of that server, ostensibly with our concurrent, event-driven approach, should be our goal.
Handling requests
The Gorilla toolkit certainly makes this easier, but we should also know how to intercept the functionality to impose our own custom handler.
Here is a simple web router wherein we handle and direct requests using a custom http.Server struct, as shown in the following code:
var routes []string type customRouter struct { } func (customRouter) ServeHTTP(rw http.ResponseWriter, r *http.Request) { fmt.Println(r.URL.Path); } func main() { var cr customRouter; server := &http.Server { Addr: “:9000”, Handler:cr, ReadTimeout: 10 * time.Second, WriteTimeout: 10 * time.Second, MaxHeaderBytes: 1 << 20, } server.ListenAndServe() }
Here, instead of using a built-in URL routing muxer and dispatcher, we’re creating a custom server and custom handler type to accept URLs and route requests. This allows us to be a little more robust with our URL handling.
In this case, we created a basic, empty struct called customRouter and passed it to our custom server creation call.
We can add more elements to our customRouter type, but we really don’t need to for this simple example. All we need to do is to be able to access the URLs and pass them along to a handler function. We’ll have three: one for static content, one for dynamic content, and one for dynamic content from a database.
Before we go so far though, we should probably see what our absolute barebones, HTTP server written in Go, does when presented with the same traffic that we sent Apache’s way.
By old school, we mean that the server will simply accept a request and pass along a static, flat file. You could do this using a custom router as we did earlier, taking requests, opening files, and then serving them, but Go provides a much simpler mode to handle this basic task in the http.FileServer method.
So, to get some benchmarks for the most basic of Go servers against Apache, we’ll utilize a simple FileServer and test it against a test.html page (which contains the same 80 KB file that we had with Apache).
As our goal with this test is to improve our performance in serving flat and dynamic pages, the actual specs for the test suite are somewhat immaterial. We’d expect that while the metrics will not match from environment to environment, we should see a similar trajectory. That said, it’s only fair we supply the environment used for these tests; in this case, we used a MacBook Air with a 1.4 GHz i5 processor and 4 GB of memory.
First, we’ll do this with our absolute best performance out of the box with Apache, which had 850 concurrent connections and 900 total requests.
The results are certainly encouraging as compared to Apache. Neither of our test systems were tweaked much (Apache as installed and basic FileServer in Go), but Go’s FileServer handles 1,000 concurrent connections without so much as a blip, with the slowest clocking in at 411 ms.
Apache has made a great number of strides pertaining to concurrency and performance options in the last five years, but to get there does require a bit of tuning and testing. The intent of this experiment is not intended to denigrate Apache, which is well tested and established. Instead, it’s to compare the out-of-the-box performance of the world’s number 1 web server against what we can do with Go.
To really get a baseline of what we can achieve in Go, let’s see if Go’s FileServer can hit 10,000 connections on a single, modest machine out of the box:
ab -n 10500 -c 10000 -g test.csv http://localhost:8080/a.html
We will get the following output:
Success! Go’s FileServer by itself will easily handle 10,000 concurrent connections, serving flat, static content.
Of course, this is not the goal of this particular project—we’ll be implementing real-world obstacles such as template parsing and database access, but this alone should show you the kind of starting point that Go provides for anyone who needs a responsive server that can handle a large quantity of basic web traffic.
Routing requests
So, let’s take a step back and look again at routing our traffic through a traditional web server to include not only our static content, but also the dynamic content.
We’ll want to create three functions that will route traffic from our customRouter:serveStatic():: read function and serve a flat file serveRendered():, parse a template to display serveDynamic():, connect to MySQL, apply data to a struct, and parse a template.
To take our requests and reroute, we’ll change the ServeHTTP method for our customRouter struct to handle three regular expressions.
For the sake of brevity and clarity, we’ll only be returning data on our three possible requests. Anything else will be ignored.
In a real-world scenario, we can take this approach to aggressively and proactively reject connections for requests we think are invalid. This would include spiders and nefarious bots and processes, which offer no real value as nonusers.
