SHANE BUETTNER BLOG

I admit I’m surprised by the controversy caused by removing the product review ratings. Most who’ve written want the ratings back and accuse HT of running scared from advertisers/manufacturers. I want to set the record straight, elaborate on the reasons we eliminated the ratings, and engage in some meaningful debate about them. I also want to note that while the ratings in their original format will never return, I’m not totally opposed to a revamped system if you readers still think it’s necessary. There are some features coming up that I think will make you miss the ratings less, and I’ll explain those below.

First and foremost, with complete support of HT’s staff I dropped the ratings because I firmly believed that the ratings system results were artificially high, allowing mediocre components to hide behind ratings that looked good numerically. As I noted in the Letters section of the July issue, of the 160+ components reviewed and rated from January 2007 through April of 2008, only a handful received overall ratings below 84 pts, indicating below average. A whopping 75% of the components were rated at 90 pts or greater, and the average overall rating was 91 pts.

For those of you still insisting that advertiser pressure was the reason behind this move, please read the above again. Manufacturers were not complaining about their ratings. At best the ratings showed some distinction between better and best. But in every instance in which a product was rated as truly exceptional that was abundantly clear reading the review.

And I don’t think this was the fault of the reviewers not being tough enough, but faulty architecture in the system and the way the performance descriptors corresponded to the numbers in place. While the ratings scale ostensibly went to 100, it wasn’t a full 100-point scale. It was really a 74-100 scale, which bunched up a lot of scores at 90 points. It was more like having something that looks like a full 5 point scale, but was really a 3-5 point scale, with of the ratings at 4.5 points and above!

I think some of these issues echo something a reader said in lauding the ratings removal- if you have to explain it every other month then something’s wrong. Frankly, it’s hard to come up with a single ratings system that really works without more explanation than necessary, and then explanations to follow explanations.

Also, while the numbers lent an air of objectivity, they were every bit as subjective as the written review. And with different reviewers, the scale was never really the same across the board as hard numbers inherently imply. And if it’s not, then what is the real value in the numbers? Don’t you have to rely on the review text and the reviewers’ personality to discern these nuances anyway?

Another factor is that with HT’s current design (which will be made over with our October issue) the graphics supporting the ratings used a lot of real estate on the page. So much that a 3 page review was limited to as little as 1400 words. That’s not a lot of space to offer meaningful analysis today’s complex products. Eliminating the ratings allowed us an immediate increase in word count over 25%. So, I saw removing the ratings as mothballing a flawed system, and providing more space for analysis and more authoritative reviews. I feel strongly that the strength of this magazine over its competitors is the knowledge and experience of our writers and their ability to convey in-depth opinion and analysis on the printed page. And I thought that removing the ratings would make this more apparent. But some of you clearly don’t agree.

Now, one significant new feature in HT that I want to ping you on debuts with the August issue, hitting newsstands and subscribers right about now. Every issue of HT will feature two pages that delineate our Top Picks in every product category, broken down by Entry Level, Midrange and High End price points. In every issue the products reviewed that make the grade will appear in Top Picks, and starting in October a Top Picks graphic will be on the opening page of the review, showing you at a glance that the product made the grade. We’re being very selective about these Picks, so they’re not just a list of what we’ve reviewed. And in October, a set of highlights will also be on the opening page of each review, telling you in a capsule what’s hot and what’s not about each product.

So, my question is, do you acknowledge what’s written above and agree? Are these new and upcoming features enough to let you say goodbye to the ratings? Or, do you feel we need to come up with a new ratings system? If a new system is desired, how do you make it meaningful and accurate given the problems I’ve brought to light above?

As I said in my first prologue, Home Theater really is your magazine. Let me know what you think.