Chicago Beer Travelers

« Historical Perspectives on the Barley Wine/Old Rauchbier: Realest of Autumn Seasonals »

What's A Beer Rating Worth Anyway?

Rohit Naimpally

Pol Sci, Econ and Cricket geek who loves a good pint

I often get questions on beer rating websites: are they good sources of information, do the ratings mean anything, how much do I use them, etc. The two most popular websites seem to be Rate Beer and Beer Advocate, but I expect there are a few others as well. While I myself use these websites on numerous occasions, I rarely rate beers and find myself largely ambivalent on the entire culture engendered by beer ratings.

In my view, beer rating sites are best thought of as analogous to movie review aggregators like Metacritic and Rotten Tomatoes. For the most part, the "score" attributed to a movie can be viewed as pretty accurate: while a widely-acknowledged classic like The Godfather is rated by fans as 97% "fresh" on Rotten Tomatoes, a less distinguished production like Kazaam only meets with a 17% score. Surely this seems right- as enjoyable as Kazaam may have been (seriously, you don't find a movie with Shaq in it entertaining?), one would hardly argue that it is a greater film than The Godfather.

However, it is precisely in that subjective notion of "greatness", or "quality" that things get a little slippery. If movies are meant to entertain us, can one truly say that a cinematographic masterpiece should be rated any higher than a crowd-pleaser? I love Apocalypse Now, but I have little doubt that I would watch Wedding Crashers far more often. Yet, at some level, a part of me feels that the former is a better film.

This can get especially problematic with beers. Consider: Based on it's ratings on Rate Beer and BA, Three Floyds' Dark Lord Russian Imperial Stout is considered by tasters to be one of the best beers in the world. It garners a score of 4.35/5 on Rate Beer and having personally tasted it, I can testify to it's incredible complexity and cornucopia of flavors. However, I would be hard pressed to drink more than a pint of this at one sitting, and it would hardly be my beer of choice on a hot summer's day. One beer that I would choose on a hot summer's day would be the Paracelsus, a Zwickl from Stiegl. This unfiltered lager from Germany is a rare style and makes for a great quencher; the Paracelsus is a pretty fine example of the style (and is on tap at the Map Room quite often, including right now!) Yet, it merits a mere 3.18/5 on Rate Beer. The short version of what I am trying to say is that with beers, it is often a case of apples and oranges.

One way around this is to look at the within-style percentiles of a beer, whereby a Zwickl would only be compared to other Zwickls. Fair enough, but that still doesn't get beyond the fact that on an absolute scale, certain styles are consistently given higher scores than others. A glance at the Top 50 lists on either Beer Advocate or Rate Beer will reveal an overwhelming bias towards Imperial Stouts, with some Sours and Belgian Strong Ales also receiving representation. You would be hard-pressed to find a single Pale Ale, Vienna Lager, or other less "complex" beers.

In mentioning this, I often meet with the retort "Oh, but these ratings are meant to reflect the complexity, etc. of a beer and the amount of craft and effort that may have gone into making it." First, I think this is incorrect to a certain extent: making certain lager styles can be extremely difficult. In fact, the difficult could be higher, since the relative "clean-ness" of these beers leaves little room for leeway, with strange flavors and aromas having no place to hide. A 100 IBU, 13% beer can mask some flaws, but a crisp Bohemian Pilsener would be hard to get right precisely for it's simple requirements. This is partly why I respect Cleveland's Great Lakes Brewing as much as I do: they do a rather fine job of producing certain lager styles (the excellent Dortmunder Gold and Eliot Ness, for instance), which cannot be said for many breweries. On the other hand, there are a number of breweries that can brew a rather tasty Imperial Stout.

Second, if ratings are meant to reflect the sort of complexity that is found in a beer like Dark Lord, then we should be clear about that. I often hear people talk about a certain beer being "better" than another, based simply on it's rating; clearly, caveats aplenty need to be added. More importantly, curious beer aficionados should not take their exploring cues purely off of beer ratings. That would lead to a terribly one-dimensional, narrow beer experience indeed! Furthermore, it does a disservice to the diversity of beer to judge it within the narrow confines of a ratings scale. I think of beers in much the way I think about food- eggs make for great breakfast food, but a warm slice of pie really hits the spot after lunch. Again apples and oranges, different horses for different it what you will, but appreciating different beers within their contexts would probably make for a better beer experience in general.

Finally, there is a fundamental flaw with the manner in which most ratings are done: raters usually know the beer they are tasting beforehand, along with some basic background on it. This creates a large endogeneity problem i.e. a rater's priors would affect his impressions of a beer, rather than the other way round. In other words, we have heard so much about a beer being great that we taste it with an expectation of high quality...which inadvertently feeds into our impressions. Most raters would furiously deny this effect, and from personal experience I can admit to disliking some rather well-regarded brews. However, this bias does often exist- it is a well-established psychological fact. Of course, one solution is to engage in a blind tasting; but how often are we afforded this luxury? Or rather, how often do raters take the trouble to do this.

All that having been said, there are numerous benefits to be drawn from beer ratings sites; as I mentioned earlier, I am a frequent visitor myself. They can be a great one-stop source of information, even the unbiased non-subjective variety. See a beer name that is unfamiliar? Running a search on BA or Rate Beer will turn up a page listing the label information of the beer usually, along with details such as International Bittering Units (IBU), Alcohol by Volume (ABV), Original Gravity (OG), style, etc. In addition, just scanning through the ratings can give one a sense of what to expect from the beer. This might seem like an odd thing to advocate, given my mini-rant on the role of priors above. However, buying beer also involves a financial cost and oftentimes, I want to buy a certain type of beer. Having an idea of what to expect can be helpful in both situations. It can be especially helpful in the case of beers that have labels largely devoid of any useful information (cough*Lost Abbey*cough). An alternative would be to visit the brewers' websites, but the sheer wealth of information contained within BA and Rate Beer make them better options in this regard.

As beer explorers, we can also learn from the ratings websites about other beers that are worth seeking out (albeit with plenty more information for certain styles). I love sours, so I enjoy trawling Rate Beer to see what sours tasters have enjoyed, the flavors they identified, etc. While these may not be definitive indications of a beer's quality, they provide a good place to start one's hunt. And this goes beyond simple beer information: Beer Advocate's member forums are a great place to set up trades for beers that aren't available in your neighborhood, swap brewing tips and learn about local beer events. I had the privilege of attending a festival of funky beers in Boston last weekend, Night of the Funk. Not only did we learn about the event on Beer Advocate's forums, but the event was also organized by Beer Advocate. The forums can be useful for learning about limited release events as well.  

Finally, I think rating beers can also be a useful personal exercise. Doing so, especially in a blind tasting, allows one to truly think about a beer, refine one's own palate and maintain a record of beers that one has liked and disliked (as well as the rationale). I like to record certain beer-food pairings that I liked, since these are pairings that I would like to return to later. One could either use beer rating logs, published by some companies nowadays (I shall be reviewing one in the near future), or even a simple notebook.

I suppose the bottom line is a equivocal one then: yes, beer ratings can be helpful, but one must be aware of what exactly they are measuring and the cognitive and taste biases inherent to them. People have some pretty strong views on this matter though, so feel free to let us know how you feel.




Recent Posts


No Comments

Leave a Comment?

Some HTML is permitted: a, strong, em

What your comment will look like:


what will you say?

Related Topics

Most Active Pages Right Now on Facebook