The Great HT Ratings Debate

I admit I’m surprised by the controversy caused by removing the product review ratings. Most who’ve written want the ratings back and accuse HT of running scared from advertisers/manufacturers. I want to set the record straight, elaborate on the reasons we eliminated the ratings, and engage in some meaningful debate about them. I also want to note that while the ratings in their original format will never return, I’m not totally opposed to a revamped system if you readers still think it’s necessary. There are some features coming up that I think will make you miss the ratings less, and I’ll explain those below.

First and foremost, with complete support of HT’s staff I dropped the ratings because I firmly believed that the ratings system results were artificially high, allowing mediocre components to hide behind ratings that looked good numerically. As I noted in the Letters section of the July issue, of the 160+ components reviewed and rated from January 2007 through April of 2008, only a handful received overall ratings below 84 pts, indicating below average. A whopping 75% of the components were rated at 90 pts or greater, and the average overall rating was 91 pts.

For those of you still insisting that advertiser pressure was the reason behind this move, please read the above again. Manufacturers were not complaining about their ratings. At best the ratings showed some distinction between better and best. But in every instance in which a product was rated as truly exceptional that was abundantly clear reading the review.

And I don’t think this was the fault of the reviewers not being tough enough, but faulty architecture in the system and the way the performance descriptors corresponded to the numbers in place. While the ratings scale ostensibly went to 100, it wasn’t a full 100-point scale. It was really a 74-100 scale, which bunched up a lot of scores at 90 points. It was more like having something that looks like a full 5 point scale, but was really a 3-5 point scale, with of the ratings at 4.5 points and above!

I think some of these issues echo something a reader said in lauding the ratings removal- if you have to explain it every other month then something’s wrong. Frankly, it’s hard to come up with a single ratings system that really works without more explanation than necessary, and then explanations to follow explanations.

Also, while the numbers lent an air of objectivity, they were every bit as subjective as the written review. And with different reviewers, the scale was never really the same across the board as hard numbers inherently imply. And if it’s not, then what is the real value in the numbers? Don’t you have to rely on the review text and the reviewers’ personality to discern these nuances anyway?

Another factor is that with HT’s current design (which will be made over with our October issue) the graphics supporting the ratings used a lot of real estate on the page. So much that a 3 page review was limited to as little as 1400 words. That’s not a lot of space to offer meaningful analysis today’s complex products. Eliminating the ratings allowed us an immediate increase in word count over 25%. So, I saw removing the ratings as mothballing a flawed system, and providing more space for analysis and more authoritative reviews. I feel strongly that the strength of this magazine over its competitors is the knowledge and experience of our writers and their ability to convey in-depth opinion and analysis on the printed page. And I thought that removing the ratings would make this more apparent. But some of you clearly don’t agree.

Now, one significant new feature in HT that I want to ping you on debuts with the August issue, hitting newsstands and subscribers right about now. Every issue of HT will feature two pages that delineate our Top Picks in every product category, broken down by Entry Level, Midrange and High End price points. In every issue the products reviewed that make the grade will appear in Top Picks, and starting in October a Top Picks graphic will be on the opening page of the review, showing you at a glance that the product made the grade. We’re being very selective about these Picks, so they’re not just a list of what we’ve reviewed. And in October, a set of highlights will also be on the opening page of each review, telling you in a capsule what’s hot and what’s not about each product.

So, my question is, do you acknowledge what’s written above and agree? Are these new and upcoming features enough to let you say goodbye to the ratings? Or, do you feel we need to come up with a new ratings system? If a new system is desired, how do you make it meaningful and accurate given the problems I’ve brought to light above?

As I said in my first prologue, Home Theater really is your magazine. Let me know what you think.

Share | |
COMMENTS
Javan's picture

I'd like to make a couple comments.First, I miss the ratings. It helped to further differentiated you from other home theater publications. And the ratings of individual products were the very first thing I looked at when I opened your magazine.Second, if you do bring it back, you should consider a formula system for value. I realize that this was done for the overall score, but a formula could definately be used to tabulate value as well. If you keep a databases of the products evaluated that maintains their scores and prices, this wouldn't be difficult at all. It would also help to make it somewhat more objective.Third, having a tiered system was STUPID. Having a three different systems rate a 92 in performance even though none were equal to the other since they were in three different classes completely ruined the rating scale. It was confusing, senseless, and pointless. If you do bring back the rating scale, please just lump everything together.4th, I am no longer a subscri

Traveler's picture

Numerical rating add clarity to reviews.

Steve in Manitoba's picture

Although the rating system was a marginally useful scoring mechanism for comparison reviews in the same article, I question its value over time. As well, a rating scale converging on a narrow band close to 90 points is a waste of effort. I always found myself gravitating to the narrative to weed out the weaker component. I have to disagree with Javan about keeping historical data. Home theater technologies are evolving at such a rapid pace, any attempt to compare ratings from a 2 year old projector to one currently available is simply not practical. Shane makes a good point about space. I would rather see textual interpretation from an experienced and knowledgeable reviewer about features, abilities, and flaws, that I can use as a litmus test to include or discount potential candidates for my short list of equipment. Drop the ratings and lets try a new approach.

Shane's picture

Steve and Javan hit on some of the points I didn't even flesh out as fully. At this point in time, even a year after something is out, the scale is re-defined. And frankly, there isn't much objectivity to it, nor can there be. Not only that, with multiple reviewers, the scale simply wasn't consistent by nature. In addition, no weight whatsoever was ever given to the measured performance of products on the test bench. And really, how could it be?

Mike Mc's picture

Shane, Your feelings on the rating system mirror mine 100%. As a 10+ year subscriber to all the AV magazines, I had little or no confidence in the ratings, and I found it misleading to have a $500 unit rated as highly as a $5000. pre-pro. It only served the reader that didn't read the whole article, or to reassure the recent purchaser of new equipment. Over the last ten years, it also seemed to be an upward moving standard. The best standard would be to have the state of the art at top, but even then, you can't compare the best from 5 years ago to today. Therefore is is too arbitrary.

Shane's picture

Mike- huge point- one of my biggest concerns is that our Internet archives would become utterly obsolete and misleading over time, and it's not practical for people who didn't review the gear to go back and arbitrarily re-rate it.

Shane's picture

And BTW, I have gotten more than a few responses from people who admitted to either not reading the reviews, or only reading them if the ratings looked good. It's hard to get my head around tailoring our content around people who say straight up that they don't actually want to read the magazine!

The Flap's picture

In my experience it is hard to quantify a number to performance. It usually rewards the mediocre for having one more audio input than another. Having a review that makes you read what the review finds is much more desirable. Numbers are easier to fudge and if the metrics are known it is easy to design for the metrics, then you get an inferior product.

Colin Robertson's picture

Right on Shane! I agree 100%. I can understand why some people like a number at the end of the review, it makes it easier to flip to the highest ratings and only read those reviews, or worse, just look at the number! They don't serve to clarify anything! For a long time I considered HT mag to be a "poor man's Stereophile", harsh I know, but one of the biggest distinctions between the two publications was the numbers system in HT. Frankly, it dumbed the magazine down. I personally appreciate your efforts to improve the magazine, and I say stick to your guns. If people are still complaining a year from now, then re-evaluate the situation then. While I'm glad the scores are gone, I think a ongoing awards section, or recommended components section would be a nice middle ground. You could then give the most elevated products the attention they deserve, and help the consumers narrow their choices down so they can make the best decision.

Fred M's picture

When forced to express yourself with numbers, the only way to signal improvement is with bigger numbers. In the end, the rating system seems to have suffered, not expectantly, from subconscious grade inflation. This is best explained by Lake Effect. Specifically, the Lake Wobegon effect, where "all the women are strong, all the men are good looking, and all the children are above average."

Shane's picture

Eric- My understanding is that it's been over ten years, two or three acquisitions and I don't even know how many publishers since you worked for Home Theater. Things have changed. I would not presume to speak of how things operated when you were here, since I was not here. The fact that you don't behave with similar restraint doesn't lend credence to the idea that I've overestimated my intelligence.These are the facts: Since I took over, not a single advertiser or manufacturer has complained about the ratings, nor were any complaints passed on from the previous regime. The decision to remove the ratings was made entirely within the editorial staff, with no input or suggestion from the publishing group or anyone associated with ad sales, or any other part of this company. No one on the editorial staff of this magazine was in favor of keeping the ratings system.to be continued...

Shane's picture

Going further, no one in our publishing group was consulted on this decision, nor was anyone aware this change was even made until the blue lines came out, including the Senior VP I report to.Those in the publishing group who have spoken to me about this were not in agreement with the decision and would probably bring the ratings back if the choice was theirs. I've told them to give some time for the other format changes, the Top Picks, and some other wrinkles to take hold and see if the ratings are still missed and then re-evaluate.I have no issue with Eric or anyone else disagreeing with this decision and making suggestions. I invited that. But whether it's someone off the street or a former employee, insinuations and false accusations about motives are out of bounds. Especially when they're simply false.PS- I am a wine enthusiast with many friends in that industry. Wine Spectator's ratings are a joke to anyone serious about the hobby.

David Vaughn's picture

Shane,I've told you this privately and I'll share it in public. I cancelled my membership to Home Theater about two years ago because it had gone so far down hill. The last 3 issues have been outstanding, with or without the ratings. Keep up the good work!Regards,David

Adam's picture

I agree with the removal of the ratings. I think they encouraged too may people to simply compare products by their numerical rating, rather than which would better suit their needs and systems. You can never say that product x is better than product y because x scored a 93 and y scored a 92. Odds are, they're both good products, but you need to read the whole review to understand the subtler points. Does it sounds bright? How will that match with your speakers? Does a pre-pro have the necessary I/O for your system? Etc. And, as Shane pointed out, there is going to be some variance in each reviewer's absolute point scale. That alone makes it impossible to compare products based on a rating.

Gary's picture

I agree with the current editor in eliminating the ratings. Keeping them invites comparisons and, worse, comparisons over time, neither of which the subjective ratings can provide accurately, reliably, or validly. If readers want a quick review, the editors can write capsule reviews-- concise language in a callout box next to the full review. But ditch the meaningless numbers.

Jorge's picture

I liked the rating system and will add my 2 grains to Shane's issues. One message that may come across is that I believe your readers are more "educated" and "smarter" and can understand the ratings for what they were.1. Artificially High: As a 4 year old reader I knew the limitations of the ratings. They helped me COMPARE, products in the same category, it did not really matter what the scale was.2. Architecture: It felt esoteric and could be definitely improved. You do not have to invent the wheel as there are many systems out there that seem to work. What ever the system, it will always have limitations that readers have to recognize.3. Subjectivity: Again, we avid readers knew the personality of the reviewer, his/her likes and dislikes. We read the whole article and past reviews. But the number still helped. I know what a number from Mark compared to one from Geoff. and by the way, also with comments from reviewers from other publications.4. Much more, ran out of

jorge's picture

What wrong with Comparing?. I read reviews to keep up with technology and brands and also compare who is doing what.As "educated" readers most of us know that a rating of 2 year old product cannot compare with a recent product. If someone is interested in a product and rating I am sure they are most likely to read the whole review. The rating helps me to understand better what is written about the product.

The Flap's picture

I usually ignored the numbers. A good solid AV person or HT Junkie should be able to read between the lines to what the reviewer really thinks about a product. Numbers just obscures the real opinion

Will U. Givemeabreak's picture

Without substantive and meaningful analysis, this operation is a catalog.

Kurt Ginter's picture

I really miss the rating system, though I agree it had its flaws. Though I didn't realize the average score was a 91, after years of reading the magazine it was obvious that anything with a sub-90 score had received a less-than-stellar review. Though I, and no-doubt many other readers knew that, the inexperienced readers probably didn't. I' d like to see a new rating system, one that is as objective as possible, with a clear scale published along with it each month. Also, the new section in the back of the magazine with the list of recommended products is great, but I think it could be better if it were accompanied by an overall score for each product. Thanks for letting me put in my 2 cents.

Shane's picture

Will- I believe the substance and analysis is in the reviews, not the numbers.Kurt- we are publishing as much objective data as possible in the measurements, and even expanding that kind of material. As an example, we're working steadily toward standardizing some of the video processing performance data along the line of what we did in the June AVR video processing feature, and we're moving to include those tables in all of our AVR and surround processor reviews. And even those processing tests require human interpretation when looking at the test patterns to determine what's Excellent, Borderline or Fail.Unless something measures demonstrably out of whack, the objective measurements only tell part of a component's story. The heart of all ratings systems I know of are the reviewer's subjective analysis and I don't know a way around that.

michael k feeley's picture

Man, I miss the ratings. I am a loyal reader and suscriber. The ratings were the main reason I preferred your mag over the rest. It does seem to me that in a political way you caved to advertisers. That may not be the case, but in my mind thats the way i see it.... I will continue to suscribe but the anticipation i used to have for your mag is gone ! Thanks for doing a OK job. Hows that for taking the high road. It Sucks you guys lost the balls to tell it straight like you used to ! That was not so hard, was it ? TY

David Vaughn's picture

Frankly, until Shane took over the reigns of Home Theater, the magazine had become unreadable IMO. It's reviews were superficial and lacked focus. The past 4 issues have been very well done and I expect they will get better as time goes along. Keep up the good work Shane!

Dave's picture

I agree with you 100%, Shane. I found the ratings empty and only took up space that could be better used by a subjective reviewer's description. IMO, to compare a product using a number told me nothing IF my tastes weren't comparable to the reviewer's. There are no 'perfect' products out there (not suggesting that a ratings system said there were) so I must accept compromise and realize that one piece of equipment will do one thing better than another, yet, that other piece excels at something else. It's the reviewer's written assessment that can only tell me that. Personally, I'd like to see MORE space allowed for a written review. Keep up the good work!

Mark's picture

The ratings were the main reason I subscribed many years ago. Because of their removal, I have decided not to renew my subscription. I agree that the ratings have sufferred over the years from "inflation" for fear of upsetting advertisers. What you really needed to do was institute an objective, scientific ratings system like Consumer Reports where like products are grouped together and rated within that category. And the ratings need to be kept updated as more superior products enter the market. Something rated a 10 today, might slip in six months as the next model is released. The web is an excellent place to keep this kind of dynamic system updated. Don't eliminate a great thing like the ratings system when it gets broken due to ratings inflation, but refine it to make it even better. It's the one outstanding feature that makes your product unique from the competition.

Pascal Pedersen's picture

I still miss the ratings! Obviously many people are going to disagree on this subject, but it did define your magazine. Sound & Vision has the same thing your are attempting to do with their Sound & Visions Best pages. Why would you duplicate their magazine? I already subscripe to it, I dont need another one. I also dont want to read 25% more text to know whether or not a item is worthy of my cash. This is where I think the misunderstanding is, not everyone wants to read pages and pages to know about a product. I'm lazy, I read my mag to know whats coming out in the market and dream about what next equipment I'm going to buy. Once I see a good score, I will probably read the whole article. But if Im not shopping for a LCD TV, I'm probably not going to read the whole article. I'm a casual reader I guess you could call me. Right now, when it comes to which mag I will keep between S&V & HT, S&V has my vote. I hope your planned improvements make me want to keep both.

Peter H's picture

I encourage HT to bring back a ratings system back. I think the magazine may be "over analyzing" the issue. The goal of a ratings system should be simple: to allow the reader to quickly understand the reviewer's opinion of the product. The ratings system acts as a complement to, and not a replacement for, the substantive review. Sure, there are issues and subjectivity involved, but is that not true of the longer (more substantive) review as well? I would suggest a simplified system with a 1 to 10 "overall" rating (or 1 to 100). There is no need to share with the readers different "sub scores" (e.g., price, build, quality, etc.). These are, of necessity, part of the "overall" score. Again, I believe the readers completely understand that a ratings system is subjective and far from perfect (just like the reviews themselves). Yet, just like the reviews, they provide the reader with useful information, but in a simplified, "quicker" format than the

jerry's picture

Shane,The magazine looks outstanding, possibly, no, definitely, the best I've ever seen it. The newest issue is just superb, and accomplishes exactly what I've wanted to see: MORE REVIEWS. While i initially liked the numerical ratings, i realize now that they're really not needed--just read the damn review! You'll learn more. i think the numbers are for the ones that simply dont want to delve deeply into the reviee. Keep up the great work, and ignore that cranky ex-editor guy. His crappy attitude is exactly why he's an EX- editor.

Dennis Brandt's picture

Listing preferred components is essentially worthless unless your staff has tested every component in every category, which will never happen. Don't tell me "B" is best if you've only tested A through G.

Evan's picture

I just got my October issue and I like the new format (not too sure about your website yet it seems -- er cluttered) When the ratings first disappeared I was a bit taken aback, but I had already started to notice (only being a subscriber and reading HT for about a year) that all the ratings were too close together and seemingly above 90 consistently, which spoiled their usefulness for me. What I like in most automobile reviews is that they list comparable products. So rather than rating products, give three (max) close competitors. This would be really useful to me as a reader and consumer to look at the technical specs and reviews for these products and determine what might work best for my application. I like a lot of you writers, but I think their needs and lifestyle differ from mine. Ultimately I need to make the decision as to what will work best for me. All I ask is that HT help me by continuing to write good, in-depth reviews and comparators would be invaluable in this process. Thanks!

Pages

X
Enter your Sound & Vision username.
Enter the password that accompanies your username.
Loading
setting var node_statistics_87021 setting var node_statistics_87021