In November 2011, we first published the initcwnd values of CDNs, following our blog post Tuning initcwnd for optimum performance that showed how tuning the initial congestion window parameter (initcwnd) on the server can have a significant improvement in TCP performance.
CDNs constantly tune their platform to improve performance. When we updated this article on August 27 2014, we measured higher initcwnd values for many CDNs (compared to the 2011 measurements). The most recent measurements (February 13 2017) again show some CDNs have increased the initcwnd size but interestingly for other CDNs we measured a lower value than in 2014. Also, quite a few CDNs have still not changed to be higher than the default 10 in Linux kernel.
First we show you the data, then some conclusions and finally we describe our test methodology.
CDN initcwnd values
CDN | Initcwnd value (Feb 2017) | Initcwnd value (Aug 2014) |
---|---|---|
Cachefly | 46 | 66 |
Level3 (now Lumen) | 46 | 12 |
Highwinds | 44 | 36 |
Akamai | 32 | 16 |
Edgecast (now Edgio) | 30 | 10 |
CloudFront | 25 | 10 |
CDNetworks | 22 | 10 |
Tata Communinications | 10 | 10 |
MaxCDN (now Stackpath) | 10 | 32 |
Fastly | 10 | 10 |
Cloudflare | 10 | 10 |
CDN77 | 10 | 10 |
Leaseweb | 10 | 10 |
Conclusions
Six out of the measured 15 CDNs have a initcwnd of 10. This is the default value in Linux kernel and apparently many these CDNs have found this to work well. Most the others set the initcwnd to be higher than 20 except for Limelight which is at 14. CacheFly still is at top with a value of 46. This is 20 lower than the 2014 measured value but very likely this is a direct result of how we measured initcwnd this time which is different from last time. Read on ...
Test Methodology
During the 2014 measurements, the tests were conducted on a Macbook Air in The Netherlands with an initial receive window (initrwnd) set to a high 262144.
For each CDN, we made requests to some far-away POP (US-West or APAC) to ensure a high RTT to make it easier reading tcpdumps. For each test, we made few hits to prime the cache at the edge servers. We then studied the tcpdumps, ran the entire process several times for sanity check. For the global IP Anycast CDNs like Cachefly and Cloudflare We added an extra 500ms latency using Dummynet. No dummy packet loss was added.
We used this python script to run tests and capture results using tcpdump. The dumps were manually analyzed in wireshark as described here
That was all in 2014. In 2017, we got lazy and simply used the Initcwnd Checker tool on this CDN Planet website. We measured the initcwnd of every CDN a few times. The tool works just fine. The only differences with the 2014 testing methodology are:
- the initial receive windown was 65000
- the tests ran from a AWS EC2 instance in US-East
- we did not take the trouble to figure out what the IPs are of far-away POPs and/or add latency to have higher RTTs
Differences 2 and 3 are not so relevant but the lower initial receive window is important and explains why we measured 'just' a initcwnd of 46 for Cachefly. 46 packets at ~1400 bytes each is ~65000 bytes and that is the initrwnd on the test machine. In other words, the test machine could not receive more than 65000 bytes in the first roundtrip and so Cachefly served no more than 46 packets. Had we advertised a higher initrwnd, maybe Cachefly would have sent out more packets.