Benchmarking Broadband ISP Speed Testers On a 10Gbps Line

At one point or another, most UK people will have probably attempted to run a web-based speed test on their broadband connection, but have you ever wondered how well some of the most popular speed testing services do when confronted by a fully capable 10Gbps (Gigabits per second) service? For a bit of fun, we tried to find out.

Most modern speed tests should, by now, ideally be capable of coping with today’s 1Gbps packages, although some of these services have adapted better to the demands of the gigabit era than offers. Likewise, any problems with loss of accuracy tend to grow the faster your connection goes.

Suffice to say, even if you’re fully setup to conduct the test properly (i.e. via a wired connection and with no other local network load occurring), then some testers will still struggle to deliver a reliable result for 1Gbps connections, while trying to reach the dizzy heights of 10Gbps is perhaps just asking for trouble.

All sorts of reasons can exist for this, such as the location of the server being used to conduct the test (some testers give you the option to change this), the type of test being conducted (single vs multi-thread tests can produce different results), local ISP congestion (consumer capacity is shared to keep prices affordable and thus you won’t always get the top speed) and the test server itself simply being too congested to give an accurate result.

However, problems with accuracy aren’t always centred on an under-reporting of performance. For example, a number of people on 1Gbps FTTP lines who have tried Netflix’s tester – Fast.com – have boasted about getting results of 1.2-1.4Gbps, which is despite the fact that such speeds may not even be possible on their setups due to hardware limits

Testing the Testers to 10Gbps

The purpose of this article is thus less about testing the performance of a symmetric 10Gbps broadband line and more about pushing some of the internet’s most popular speed testing providers to their limits, just to see how they cope. In order to achieve this, we teamed up with the former CEO of B4RN, Barry Forde, who kindly helped to conduct the testing.

Barry is one of the very few people in this country, that we know of, to have actually installed a consumer grade 10Gbps Fibre-to-the-Premises (FTTP) service to his home. It’ll be a few more years before the rest of us can even think about joining him, although such packages are already starting to become more common in other countries.

NOTE: If you’re covered by B4RN’s network then you can get a 10Gbps package, provided you’re willing to pay £150 a month and £360 (one-off) installation.

In this setup Barry adopted ZYXEL’s AX7501 router, which was then wired (via a 10GbE LAN port) back to an Intel x550 10G Ethernet card on his PC. All the testing was done during periods of idle load on the local (home) network and over two different time periods – off-peak (around 8am) and peak time (around 8pm). A few tests were conducted a little earlier, and some slightly later, but the results followed the same trends.

Each website speed tester was tested once in the morning and once in the evening for a full week (i.e. a total of 14 tests per site / service across 7 days). The results were then averaged (mean) out below.

Results of the 10Gbps Speedtest Stress Test

➤ B4RN’s Ookla Server (Windows 10 Speedtest.net App)

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 2 2 2
Download (Mbps) 5151 4572 5730
Upload (Mbps) 1821 1714 1929

NOTE: The first “B4RN” test above is unique from those below because it uses the Windows 10 App from Ookla (Speedtest.net) against B4RN’s own Ookla server, as opposed to the standard Speedtest.net web client that we use later on. In theory the two should roughly match, but Ookla recommend the Microsoft app for high-speed systems, which they say should give better results.

➤ Fast.com (Netflix)

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 1 1 1
Download (Mbps) 5414 5371 5457
Upload (Mbps) 1443 1400 1486

➤ Speedtest.net (Ookla)

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 2 2 2
Download (Mbps) 4061 4102 4020
Upload (Mbps) 1604 1594 1615

➤ Thinkbroadband

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 16 17 16
Download (Mbps) 2049 2052 2046
Upload (Mbps) 361 257 465

➤ Broadbandtest.which.co.uk (Which?)

  Overall Avg (Mean)
Ping (milliseconds) 11
Download (Mbps) 543
Upload (Mbps) 84

➤ Broadbandspeedchecker.co.uk

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 14 14 14
Download (Mbps) 2047 2043 2050
Upload (Mbps) 4379 4196 4562

➤ Broadbandspeedtest.org.uk

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 13 12 13
Download (Mbps) 1442 1522 1362
Upload (Mbps) 680 644 715

➤ Uswitch.com/broadband/speedtest/ (Uswitch)

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) n/a n/a n/a
Download (Mbps) 801 828 774
Upload (Mbps) 346 340 352

➤ Speedof.me

  Overall Avg (Mean) Offpeak Time Avg Peak Time Avg
Ping (milliseconds) 37 36 37
Download (Mbps) 598 603 593
Upload (Mbps) 168 176 160

One obvious realisation above is that some testers (e.g. Which?, Speedof.me, Uswitch) aren’t necessarily good enough, yet, to reasonably be used for speed testing a symmetric 1Gbps line, let alone anything faster like 10Gbps. We also saw some consistently unusual results from Broadbandspeedchecker.co.uk, which always returned double the upload rate vs download (no other tester did this).

The closest we got to 10Gbps was generally via Ookla’s testers, as well as Netflix and Broadbandspeedchecker.co.uk too (provided you ignore the upload issue). We know from different testing that Barry’s line can practically hit around 7-8Gbps when the internet is able to feed capacity to his setup at that rate, but otherwise the most capable web-based testers above would typically deliver around 4-5.7Gbps (usually less on uploads).

The fastest single test was recorded on Broadbandspeedchecker.co.uk, which hit 8011Mbps, albeit on upstream rather than downstream. By comparison, the fastest single result from B4RN’s own Ookla (Speedtest.net) server was 6951Mbps (or 5131Mbps via the regular Speedtest.net site) and Fast.com (Netflix) returned 6600Mbps – both downstream. The other testers didn’t even get remotely close to these.

A Few Caveats

One key thing to be aware of above is that hardly any of the tests provided much technical information about how they work (unless you trawl the small print and even then, it can be quite ambiguous). But this is crucial because there can be significant differences between, for example, single-thread and multi-threaded testing.

Some people suggest that a single-threaded test is perhaps more representative of normal real-world use, while multi-threaded is often perceived to be a better way of pushing a connection to its limits, but without more detail it’s difficult to give the results their full context. On the other hand, whether single or multi-thread, we believe the purpose of a tester should always be to try and return a fair indication – so far as it is possible – of what your line can actually deliver, when operating at its best.

On top of that, it was rarely ever clear if the final “average” result being reported by a tester – over the duration that a single speed test was running – had been measured as a ‘median‘ or a ‘mean‘ average, which can make a difference.

Suffice to say that our past warnings about the reliability of depending upon web-based speedtests for accuracy, particularly when testing 1Gbps or faster broadband lines, should continue to be heeded. Indeed, the problem only gets worse the further above 1Gbps you go.

However, the reality is that speeds of well over 1Gbps aren’t going to become common place in the UK for a fair few years and in the meantime those speed testers will, as they’ve always done in the past, continue to evolve and get better. Hopefully, buy the time most of us can actually take a 10Gbps package, they’ll all be better equipped to cope.

One final point to make is that this article is intended to be a bit of fun using a fairly anecdotal approach, thus to do it properly we’d ideally need to harness a wider variety of 10Gbps capable lines and over a greater variety of time periods. But since we don’t yet have ready access to more of those connections, then we’ll have to wait a few more years.

Previous post What’s the best browser for internet privacy?
Next post Crypto Possibilities Have Arrived on the Scene: Here’s Why They’re Distinct