Where's the fastest place to put my server? How much does it matter?
Using my own web server accesslogs and public latency data to get a quantitative answer and why roundtrips are such a pain.
What's the fastest place to put my server? Beyond the time taken for servers to respond to requests it takes time just to traverse the internet, just to get a packet from A to B.
To estimate what the theoretical best physical place to put my own server is I've combined publicly available data on latencies with my own web server accesslogs. I'm aiming to get a rough, quantitative, answer that's based on a real data set.
Why location matters
Time taken to traverse the internet is added to the time taken to respond to a request. Even if your API can respond to a request in 1ms, if the user is in London and your API server is in California the user still has to wait ~130 milliseconds for the response.
It's a bit worse than just 130 milliseconds. Depending on what a user is doing they may end up making a number of those roundtrips. To download a web page usually requires five full roundtrips: one to resolve the domain name via DNS, one to establish the TCP connection, two more to set up an encrypted session with TLS and one, finally, for the page you wanted in the first place.
Subsequent requests can (but don't always) reuse the DNS, TCP and TLS setup but a new roundtrip is still needed each time the server is consulted, for example for an API call or a new page.
130ms sounded fast at first, but the rigmarole of just getting a page and then making an couple of API calls can easily end up taking most of a second just in terms of time waiting for the network. All the other time required: for the server to decide what response to send to your request, time downloading the thing and then rendering whatever it is in your browser - that is all extra.
The two kinds of "fast" for networks
One of the confusing things about networking is the inspecific way in which people talk of getting "faster" networking: "faster" residental broadband for example, or "fast ethernet" (100 megabits per second, no longer impressive).
This kind of "faster" is not in fact talking about speed. Greater speed would be reduced latency - so faster roundtrips. Instead "faster" networking is really about greater bandwidth: more bytes per second.
APIs or CDNs
A quantitative approach
A happy few can serve everything from their CDN. News sites, for example, show the exact same thing to everyone. Others are less lucky and can make only limited, or no, use of caching. These poor people have to pick a location for their main server to help them get their bytes to the users who want them as fast as possible. If they want to make that choice with the sole aim of reducing latency, where should they pick?
Here's what I did:
- I took my own accesslogs for a two week period in September just after I'd published something new. I got about a million requests during this period from 143k unique IPs. I excluded obvious robots (which was ~10% of requests).
- I used Maxmind's GeoIP database to geocode each IP address in those accesslogs to geographic co-ordinates.
- I then used WonderNetwork's published latency data for internet latencies between ~240 world cities.
- I mapped those cities (semi-manually, which was pretty painful) from their names to Geonames ids - which gave me co-ordinates for the cities.
- Then I loaded all of the above into a Postgres database with the PostGIS extension installed so I could do geographical queries.
- I queried to estimate how long, by percentile, requests would have taken if I'd had my server in each of the 200 cities.
In the table below I've recorded the outcome: how long users would take to complete a single roundtrip to my server if it were in each city. I've done this by percentiles so you have:
- the average ("p50")
- for three quarters of requests ("p75")
- and for 99% of requests ("p99")
All numbers are in milliseconds.
See full results as a table (click to expand)
|Salt Lake City||120||151||269|
|Ho Chi Minh City||253||287||380|
|Dar es Salaam||214||267||407|
You can also download the full results as a csv, if that's easier.
The result: east coast of North America good, right on the Atlantic better
The best places are all in North America, which is probably not a total surprise given that it's a pretty dense cluster of English speakers with another cluster not all that far away (in latency terms) in the UK/ROI and then a lot of English-as-a-second-language speakers in Europe. Being right on the Atlantic is best of all: New Jersey and New York state have many of the best places for p99 and it doesn't vary too much, at the top, between p50 and p99.
If you're wondering why small New Jersey towns like Secaucus and Piscataway are so well connected - they have big data centres used by America's financial sector.
As it stands, my server is currently in Helsinki. That's because, unusually for Finland, it was the cheapest option. I only pay about three quid a month for this server. If I moved it to somewhere in New Jersey, and spent more, users would definitely save time in aggregate: half of roundtrips would be completed in 75ms rather than 105ms, a saving of 30%. Over several roundtrips that would probably mount up to around a sixth of a second off the average of first-time page loads, which is not too bad. In case you can't tell, this website isn't hugely taxing for web browsers to render so cuts in the network wait time would make it considerably quicker.
Since I don't dynamically generate anything on this site, the truth is that I'd be best off with a CDN. That would really save a lot of time for everyone: it's nearly twice as good to be served from a CDN (~40ms) than to be in the fastest place (71ms).
How this might change over time
Latencies aren't fixed and they might improve over time. Here's a table of roundtrip latencies from London to other world cities with more than 5 million people, comparing against the theoretical maximum speed, the speed of light:
|City name||Distance (km)||Real latency||Theoretical max||Slowdown factor|
(Please note, a correction: the above table previously compared real roundtrips with theoretical straight line journeys - this has now been corrected, for more details see these two comments for discussion and more details - like how part of this is due to the nature of fibre optic cables and submarine cable curvature.)
As you can see, New York's latency is within a factor of 2 of the speed of light but routes to other places like Dhaka and Seoul are much slower: being 4 times the speed of light. There are probably understandable reasons why the London to New York route has been so well optimised though I doubt it hurts that it's mostly ocean between them, so that undersea cables can run directly. Getting to Seoul or Dhaka will be a more circuitous route.
I should probably mention that new protocols promise to reduce the number of round trips. TLS 1.3 can create an encrypted session with one round trip rather than two and HTTP3 can club together the HTTP round trip with the TLS one, meaning you now only need three: one for DNS, one single roundtrip for both a connecton and an encrypted session and then finally a third for the subject of your request.
While I think this is an interesting exercise - and hopefully indicative - I should be honest and say that the quality of the data I'm using is solidly in the "medium-to-poor" category.
Firstly, the GeoIP database's ability to predict the location of an IP address is mixed. Stated (ie: probably optimistic) accuracy ranges up to about 1000 kilometers in some cases, though for my dataset it thinks the average accuracy is 132km with a standard deviation of 276km - so not that accurate but I think still useful.
My source of latency data, WonderNetwork, are really reporting point-in-time latency from when I got it (30th November 2020) as opposed to long term data. Sometimes the internet does go on the fritz in certain places.
WonderNetwork have a lot of stations but their coverage isn't perfect. In the West it's excellent - in the UK even secondary towns (like Coventry) are represented. Their coverage worldwide is still good but more mixed. They don't have a lot of locations in Africa or South America and some of the latencies in South East Asia seem odd: Hong Kong and Shenzhen are 140ms away from each other when they're only 50km apart - that's a slowdown factor compared to the speed of light of more than a thousand times. Other mainland China pings are also strangely bad, though not on that scale. Perhaps the communists are inspecting each ICMP packet by hand?
The other problem with the latency data is that I don't have the true co-ordinates for the datacentres that the servers are in - I had to geocode that myself with some scripting and a lot of manual data entry in Excel (I've published that sheet on github to save anyone from having to redo it). I've tried hard to check these but there still might be mistakes.
By far the biggest weakness, though, is that I'm assuming that everyone is starting right from the centre of their closest city. This isn't true in practice and bias this adds can vary. Here in the UK, residental internet access is a total hack based on sending high frequency signals over copper telephone lines. My own latency to other hosts in London is about 9ms - which sounds bad for such a short distance but is still 31ms better than average. Many consumer level routers are not very good and add a lot of latency. The notorious bufferbloat problem is also a common source of latency, particularly affecting things that need a consistent latency level to work well - like videoconferencing and multiplayer computer games. Using a mobile phone network doesn't help either. 4G networks add circa 100ms of lag in good conditions but of course are much worse when the signal is poor and there are a lot of link-level retransmissions.
I did try assuming the global average latency per kilometer (about 0.03ms) to compensate for distance from the closest city but I found this just added a bunch of noise to my results as for many IPs in my dataset this is an unrealistic detour: the closest city I have for them isn't that close at all.
It's fair to wonder to what extent my results would change for a different site. It's hard to say but I suspect that the results would be approximately the same for other sites which are in English and don't have any special geographical component to them. This is because I reckon that people reading this blog are probably pretty uniformly distributed over the English speaking population of the world.
If I was writing in Russian or Italian the geographic base of readers would be pretty different and so the relative merits of different cities from a latency point of view would change.
It wasn't too hard for me to run this test and I've released all the little bits of code I wrote (mostly data loading and querying snippets) so you could easily rerun this on your own accesslogs without too much effort. Please write to me if you do that, I'd love to know what results you get.
Picking a good spot for your server only goes so far. Even in good cases you will still have nearly a hundred milliseconds of latency for each roundtrip. As I said above there can be as many as five roundtrips when you visit a page.
Having any unnecessary roundtrips will really slow things down. A single extra roundtrip would negate a fair chunk of the gains from putting your server in a fast place.
It's easy to add roundtrips accidentally. A particularly surprising
source of roundtrips are cross-origin (CORS) preflight requests. For
security reasons to do with preventing cross-site scripting attacks,
sending a request to the same url beforehand with the special
OPTIONS verb. The response to this will decide whether the
original request is allowed or not. The rules for when exactly preflighting
is done are complicated but a surprising number of requests are caught up
in the net: notably including JSON POSTs to subdomains (such as
api.foo.com when you're on
foo.com) and third
party webfonts. CORS preflighting checks use a different set of
caching headers to the rest of HTTP caching which are rarely set correctly
and anyway are only applicable for subsequent requests.
I always think it's a bit rubbish when I get a loading bar on a web page - you already sent me a page, why didn't you just send the page I wanted! One of the great ironies of the web is that while Google don't do a good job of crawling these single page apps they certainly produce a lot of them. The "search console" (the website formerly known as "webmaster tools") is particularly diabolical. I suppose Google don't need to worry overly about SEO.
Bandwidth improves quickly but latency improves slowly
Internet bandwidth just gets better and better. You can shove a lot more bytes down the line per second than you could even a few years ago. Latency improvements, however, are pretty rare and as we get closer to the speed of light the improvement will drop off completely.
100 megawhats per second is less compelling when you still have to wait the same half a second for each page to load.
Last year APNIC analysed CDN performance across the world, and concluded that 40ms is typical. I wish they'd included percentile data in this post but I can still get the vague impression that CDNs perform best in the West and less well in South American, China and Africa which is a problem given that most servers are based in the West.
While I was writing this post a number there was an outbreak of page-weight-based "clubs", like the "1MB club" and the, presumably more elite, 512K Club. I suppose I approve of the sentiment (and it's all in the name of fun I'm sure) I think they're over-emphasising the size of the stuff being transferred. If you're in London, asking for a dynamically generated page from California, it will still take a most of a second (130ms times 5 round trips) regardless of how big the thing is.
The submarine cable map is always fun to look at. If you want to see a sign of the varying importance of different places: the Channel Islands (population 170 thousand) have 8 submarine cables, including two that simply connect Guernsey and Jersey. Madagascar (population 26 million) has just four. I also think it's funny that even though Alaska and Russia are pretty close there isn't a single cable between them.
If you want to reproduce my results I've published my code and data on Github. I'm afraid that does not include my accesslogs which I can't make public for privacy reasons. Please don't expect me to have produced a repeatable build process for you: that takes a lot more time and effort so it's provided on a "some assembly required" basis. :)