The Results Are In

It's been two weeks since I asked readers to weigh in on whether they want more audio reviews with no objective measurements or fewer reviews with measurements. I've received 58 responses so far—thanks to everyone who expressed their opinion! Many of you supported your position with additional thoughts—in fact, a rather lively debate appeared in the comments section of that blog, which offers some entertaining reading.

The results were unequivocal—of the 58 respondents, 40 want more reviews with no measurements. That's more than a two-thirds majority, enough to override even a presidential veto. (Would that Congress could get its act together with such numbers!) With so many products out there, these folks want more reviews and are willing to live without measurements, which seem to mean less to them than subjective evaluations from trusted reviewers. I even heard from some engineers who understand and appreciate measurements but for whom subjective evaluations—and a greater number of reviews—matter more.

On the other hand, blog commentator Tyler points out that objective measurements can support the trustworthiness of subjective reviews, saying, "The reason this is my favorite A/V website is because measurements have always been provided to back up the subjective reviews." He also argues that measurements encourage manufacturers to be honest in their specs: "If you don't think things like power ratings with all channels driven, frequency response, video performance, and so on have improved as a result of [independent] testing, you are deluded."

A number of interesting alternatives were proposed. For example, several people supported the idea of measuring only those audio products that exceed some to-be-determined price threshold. On the other hand, at least as many requested more reviews of lower-cost products that they might actually be able to afford, which seems to cancel out the idea of measuring only expensive items.

Of those who want measurements, several said that measuring speakers is more important than doing so with audio electronics, a position that has much merit. According to Mark Peterson, the consultant we hire to perform our audio measurements, the frequency response of modern solid-state electronics is normally close to ruler-flat, and distortion levels are generally quite low, which seems to indicate that taking such measurements isn't particularly useful.

The output power from amps and A/V receivers, however, varies more widely depending on the manufacturer—some companies publish realistic power specs that are verified by Peterson's measurements, while others' specs are wildly exaggerated beyond his results. Of course, the method used to measure output power greatly affects the outcome, and there is genuine debate over which method produces the most realistic results, making this a rather murky area of the whole measurement landscape.

Speakers tend to be much more variable than electronics in their objective performance, depending on cabinet design, driver complement, crossovers, etc. On the other hand, a speaker's subjective performance is highly dependent on the room in which it is evaluated, which makes me wonder about the value of measurements taken in a pseudo-anechoic environment. Tom Norton, UAV's über reviewer, believes that if a speaker measures well in such an environment, it's more likely to sound good in a variety of rooms, but if it measures poorly, it's bound to have trouble in most rooms.

That's not to say a poor speaker measurement guarantees poor subjective performance—for example, Michael Fremer loved the Sonus Faber Domus speaker system, even though it didn't measure very well. Does that mean his reviewing skill is suspect? Knowing Fremer as I do, I would say absolutely not. Instead, I believe the system happened to integrate well with his particular room.

This example leads me to believe that measuring speakers is important for shoppers, who should know how they perform both objectively and subjectively in order to make the most informed buying decision. Measuring electronics is less important, since most modern solid-state products tend to measure reasonably well. The one exception might be output power—ideally, shoppers should know if a receiver or power amp delivers the power it claims to.

In the end, I've decided on the following compromise—UAV will measure only the output power of AVRs and power amps and perform a full suite of tests on speakers. And we will always include a complete set of measurements in all reviews of video displays. I'm confident that we can publish a goodly number of reviews following these guidelines while providing the most important objective results to complement the subjective evaluations of our expert reviewers. That way, you will have all the info you need to make the best buying decision for your particular situation, and we'll be able to review a wide range of products without blowing the budget on extensive but ultimately unnecessary measurements.

Again, thanks to all who responded. But don't stop now—please send your comments, suggestions, and questions to me at scott.wilkinson@sourceinterlink.com. With your help, I intend to make this the go-to site for all the home-theater information you desire.

Share | |
COMMENTS
Tyler's picture

Bravo! I think you made a fantastic decision in regards to this. I am VERY glad you chose a great compromise in deciding what to measure. Thanks to this, it looks like UAV will continue to be my favorite review site. Thanks!

Andy Sullivan's picture

I also like the compromise. If I trust the reviewer then my main interest is in the conclusion. I seldom see a reviewer sum up his review with a direct overall comparison to the closest competition. For instance in your conclusion to the 71 series Samsung you could have given your preference vs. the Sony XBR4. That's the information I want from you. Same price, which would you spend your money on? Yamaha 661 or the Onkyo 605?

Scott Wilkinson's picture

You're right, I should have talked about my preference between the Samsung and Sony. It would have to be the Sony, because of the processing problem in the Samsung's AMP mode. The Sony's 120Hz motion processing has no such problem, so I'd definitely go for the XBR4.

Colin Robertson's picture

I also like this compromise. I am sure there are other sources to go to for those who need more detailed measurements than this. I know I rarely do... I will be looking forward to seeing more frequent content on this site!

Bruce in CO's picture

There are a couple of other things that would be helpful and interesting. One would be the opportunity to ask questions or start a blog on a particular review. Andy above could have started an interesting dialogue regarding Samsung/Sony preferences as it relates to the specific review. Another thing would be to do a more formal/organized review as it relates to how specific components work with other products. For example, it's great to know that receiver X sounds great with speakers A. However, if you also used receiver X with speakers B & C, then we would know how versatile a given product might be. You could also comment on combination preferences.

Scott Wilkinson's picture

All good ideas, Bruce. I'd love to see reader comments on reviews, and I've discussed it with our webmeister, but the task of monitoring all the comments all would quickly become overwhelming...basically, it's just me here, plus some freelancers. I'd also love to see "system reviews" as you suggest, but that would take a lot more time on the reviewers' part, and thus it would cost a lot more money and slow down the frequency of reviews. As we increase our page views, we could increase our budget, allowing some of these things to be implemented.

X
Enter your Sound & Vision username.
Enter the password that accompanies your username.
Loading