Welcome to End Point’s blog

Ongoing observations by End Point people

Detecting Bufferbloat

Bufferbloat is topic which has been gaining broader attention, but is still not widely understood. This post will walk you through the basics of bufferbloat and how to determine if you are the victim of bufferbloat.

A Brief Synopsis of the Bufferbloat Problem

The topic of bufferbloat has been explained wide and far, but I'll add to the conversation too, focusing on brevity. This summary is based on the highly informative and technical talk Bufferbloat: Dark Buffers in the Internet, a Google Tech Talk by Jim Gettys. There is an assumption in the design of TCP that if there is network congestion, there will be timely packet loss. This packet loss triggers well designed TCP flow control mechanisms which can manage the congestion. Unfortunately, engineers designing consumer grade routers and modems (as well as all sorts of other equipment) misunderstood or ignored this assumption and in an effort to prevent packet loss added large FIFO (first-in-first-out) buffers. If users congest a network chokepoint, typically an outgoing WAN link, the device's large buffers are filled with packets by TCP and held instead of being dropped. This "bufferbloat" prevents TCP from controlling flow and instead results in significant latency.

Detecting Bufferbloat

All that's required to experience bufferbloat is to saturate a segment of your network which has one of these large FIFO buffers. Again, the outgoing WAN link is usually the easiest to do, but can also happen on low-speed WiFi links. I experienced this myself when installing Google's Music Manager, which proceed to upload my entire iTunes library in the background, at startup, using all available bandwidth. (Thanks Google!) I detected the latency using mtr. Windows and OS X does not offer such a fancy tool, so you can simply just ping your WAN gateway and see the lag.

Music Manager enabled, bufferbloat, slow ping to WAN gateway

Music Manager paused, no bufferbloat, fast ping to WAN gateway

Managing Bufferbloat

Unfortunately, there are no easy answers out there right now for many users. Often we cannot control the amount of bandwidth a piece of software will try to use or the equipment given to us by an ISP. If you are looking for a partial solution to the problem, checkout Cerowrt, a fork of the OpenWrt firmware for routers. It makes use of the best available technologies used to combat bufferbloat. Additionally, be on the look out for any software that might saturate a network segment, such as Bittorrent, Netflix streaming, or large file transfers over weak WiFi links.


Jon Jensen said...

Good to draw attention to this. It can make a home or office network pretty miserable.

On Windows machines I've used WinMTR:

And I think mtr has to be available on Mac OS X in one of the open source packaging systems there.

eas said...

Most/all of the "baked" debloating features of CeroWRT were merged with OpenWRT trunk and are in the release candidate for OpenWRT Attitude Adjustment.

The new co_del packet scheduler is the default for QOS, but to take real advantage of it you have configure it with the real-world upstream throughput of your link. The downside is that it doesn't model things like Comcast's PowerBoost (which doubles bandwidth on the first 5-10MB of an upload), so you end up capping yourself to the sustained rate of your link. It's a tradeoff I've been pretty happy with. Longer uploads for smaller files isn't a bad price to pay for keeping everything else responsive.

Anonymous said...

Is this entitled
blufferbloat rather than

for a particular reason? Just curious.

Brian Gadoury said...

Blufferboat, more like bufferboat.

Greg Sabino Mullane said...

Title fixed, thanks.