A Question for Readers

This week, instead of answering a reader question, I'd like to ask you a question. You can answer in one of two ways—either post a comment after this blog or send me an e-mail at scott.wilkinson@sourceinterlink.com.

Before I pose the question, let me give you a little background. Anyone who reads our reviews knows that we strive to be comprehensive in our evaluation of audio and video products—this is one of the hallmarks of UAV reviews. Another hallmark is the experience and expertise of our reviewers, all of whom have well-trained eyes and ears that allow them to form reliable subjective impressions you can count on as surrogates for your own eyes and ears.

On the objective side, all video-display reviews include a full suite of measurements, such as peak white levels, black level, peak contrast ratio, overscan, resolution, grayscale tracking, and primary and secondary color-point accuracy. All of our video reviewers have the equipment necessary to perform these measurements as part of the evaluation process, so it costs nothing extra to publish the results.

Measuring audio products, however, is a different story. Most audio reviewers do not have the necessary equipment or, in the case of speaker reviews, an appropriate acoustic space in which to perform objective measurements. As a result, we must hire a consultant to do so, which takes time and money. This is why UAV has stopped publishing measurements of audio electronics (AVRs, pre-pros, and power amps), though we still measure speakers.

As the new editor of UAV, my primary goal is to serve you, the readers, by publishing reviews that include the information you need to make informed buying decisions. On the other hand, as with all things corporate, I have a limited budget, which is the heart of the quandary and the crux of the question I wish to ask you.

So here it is: Which would you prefer, fewer audio reviews that include measurements or more reviews without measurements? This is the decision I must make, and I'm asking for your help in making it.

A follow-up question: If you would prefer to see measurements at the expense of more reviews, does that apply equally to all types of audio products, or are measurements more important with speakers or electronics?

As you ponder these questions, keep in mind that objective measurements do not always jive with subjective impressions—an audio product might measure well but sound bad or vice versa. In the end, I believe that subjective impressions ultimately trump measurements—after all, at the end of the day, you're going to listen, not measure. On the other hand, objective measurements can provide insights into a product's subjective performance, and they can certainly reveal how unrealistic many manufacturers' specifications are.

Please let me know what you think about all this; I look forward to your responses. Again, feel free to post a comment here or send me an e-mail at scott.wilkinson@sourceinterlink.com.

Share | |
Jeb's picture

More reviews. This will increase the odds that you cover something I actually want to buy.

Colin's picture

At the current rate I see reviews being done on this site, I would prefer more reviews. However if more reviews was at the expense of quality (and I do not necessarily equate quality with measurements), then by all means, take the time to write quality reviews. Perhaps there could be a compromise? Only measure components above a certain price threshold, like receivers over $1,500 or so. People paying that much are going to care more about measurements like, "am I really going to be getting the claimed 125 watts per channel?". Whereas not as many will care when it comes to a $300 receiver.

Frank's picture

A review without measurements is not really a review. But Colin seems to have come up with a reasonable compromise. Would like to see full sets of measurements on all the video players, though. Higher-end a/v equipment ($500+ or some more valid cutoff) and speakers (how much woof is that subwoofer providing...). All TVs.

Hew's picture

More frequent reviews with the caveat that upper end components have some core set of specifications tested. A value you really need to add is to chart core measurements vs other components. Data without context is frankly next to useless and the only fair comparison is of course next to the competition. "Black levels are great" ... in context to who?

Tyler's picture

To be very frank, there are already several a/v magazines/web sites that publish purely subjective reviews. It is obvious in many of these that the reviewer knows less than the average enthusiast. In many cases, reviewer integrity is also questionable. While that is certainly not the case at Ultimate A/V, I don't like where the trend is going. The reason this is my favorite a/v mag is BECAUSE measurements have always been provided to back up the quality subjective reviews. Personally, I would rather see fewer reviews if measurements can be provided. This is especially true for speakers. I was quite disappointed to discover that amplifier/receiver power measurements had been cancelled. They can be VERY revealing and have been a staple of amplifier reviewing for decades.

Wes's picture

I would prefer having more reviews with subjective content only. Your sister publication, Stereophile, is very thorough in their objective measurements, and they seem to cause more controversy than good. Half of the glowing subjective reviews are accompanied by abysmal measurements, which can confuse the reader. A better strategy for using reviews (assuming one is using them to be a buying guide) is to read the review, and if it sounds like it might be your cup of tea add it to your shortlist of components to audition yourself.

Michael Moskovitz's picture

I would like to see more reviews with out detailed measurements. What I would like to see reviews with the operational compatibility as a key criteria. I've been interested in all things audiovisual since 7th grade in 1978. I've come to the conclusion that its all about compatibility and the graphic user interface. The greatest set of features, specifications and logos don't diddley if the boxes don't interact properly and you have to buy a and you have to buy an $500-$15,000 control system to run it. Don't get me started on the idea of having to update a new fangled HD-DVD/Blu-Ray player every other week.

Tyler's picture

I strongly disagree Wes. You said "Half of the glowing subjective reviews are accompanied by abysmal measurements, which can confuse the reader." It only confuses those who don't understand that there is a lack of integrity or reviewing skills with many reviewers, and certainly those at Stereophile. Measurements do not lie. A product that measures well may not necessarily be the best sounding item, but one that measures poorly is pretty much guaranteed to sound poor or at least inaccurate. There are already numerous subjective review sites out there, I would not like to lose one of the very few who back up reviews with measurements.

Claude's picture

Tough question! I would like to see both, but if I must choose, I would prefer more reviews. I find it very frustrating in the least to look for and find no professional reviews on many new products, especially A/V receivers. I trust the reviewers on this site with their opinions backed up by many years of experience. I would say that if you can provide numbers as well as subjective opinions, go for it....otherwise more reviews please.

Fred Manteghian's picture

I tell you what I'd like I'd like more time :)

Steve in Manitoba's picture

I agree with Colin's approach. I have no home theater, no A/V receiver, and no surround. But I am building a 7.1 dedicated, Lutron light controlled, tiered seating home theater. I live in a remote enough location that seeing 2 projectors side by side or in the same space for comparison would have me checking to see if I was dreaming. And listening to decent A/V receivers or pre/pros is as unlikely as it is likely I will be told by a pimply faced teenager that the $100, 5 foot high, 20lb speakers that he is demonstrating with a clipping 50 watt receiver are all I need. I virtually have no choice but to rely on reviews by folks that have a genuine interest in quality electronics. Something I have come to expect from UAV. Leave out the measurements for the living/dining/TV room equipment and provide them for the electronics that are intended to grace a dedicated room or at least one where home theater is not an after thought.

NZ Fred's picture

Measurements are probably most important for speakers but they still don't define good sound, that is for the listener alone to decide. I'm happy to go with no measurements as long as I trust the reviewers ability. There is nothing wrong with helpful advice, but in the end we all have to make our own decisions.

MJR's picture

I'll take more reviews without measurements. The eyes and ears of an experienced reviewer can be more helpful to us. If a piece of equipment sounds forward or recessed, bright or balanced, we need to know. With the affordable stuff it's getting tougher to even get a decent demonstration these days. Plus Ultimateav does have those noble Stereophile roots.

Al's picture

More reviews please.

fuzzdehull's picture

More reviews! What good is it to get 15% of all equipment reviewed thoroughly with numbers that often don't mean that much, compared to lots of reviews that cover 85% of what we need to know...

Bruce's picture

Sorry if I repeat anything someone else said. Not enough time to read all the comments. I would like to have all the numbers but would prefer quantity if I had to choose. Also I would like to see more on reliability. Which brands have to be sent back or break more often and which don't. I like comparisons too like when you compared the JL sub to a Revel. Its a good benchmark to let me know where a piece of equipment sits amongst all the others. Thanks for all the reviews and the quality put into them. I come here and to home mag daily.

William's picture

I think you should use actual measurements in terms of speakers and audio components. Robert Heron does amazing reviews and he does not use test benching in his HDTV reviews but he is very thorough and i trust him w/o the measurements. However when it comes to audio equipment IMO you can't have the whole story w/o test benching.

Scott Wilkinson's picture

Actually, Robert Heron does measure HDTVs, at least when he reviews them for PC Magazine.

Vin Gargiulo's picture

Dear Scott, I was once in the audio industry back in the 1970s when specs were everything. But you are exactly right when you point out that many times specifications are meaningless. Does it really matter if your speakers are capable of delivery 24kHz frequencies when your ears can not detect sound in that range anyway. Perhaps we can't hear frequencies below 20Hz with our ear, but we can feel the vibration in our stomach. So how important are those specifications anyway? Ultimately, it comes down to trusting a reviewer of audio equipment. If the reviewer knows what to listen for, then his review is as good (if not sometimes better) than reading specs that, more times than not, can confuse the reader. My vote is for more reviews and fewer specs. It is obvious that you enjoy what you do and I respect your opinions on the product you review. Keep up the great work, Scott. Vin Gargiulo Little Ferry, NJ

Simon Oliver's picture

I think that benchmarking av equipment is good, but I feel that a trusted view is much stronger. Here in the uk we have a car show "top gear" they review cars and push "this car has xxx horse power" and other stats that review the cars. But the strenght of there review come from " it handles really badly" If the source is trusted then I would rather have, "I lined all these systems up, and this sounded the best!" I hope this helps Simon O

Tyler's picture

Ok, putting aside the value of seeing how a reviewer's comments correspond to the measurements (which frankly, is one of the ways I came to respect the integrity and quality of Fred and Tom's reviews), conducting measurements of equipment does one VERY important thing -- it keeps the manufacturers honest! If you don't think things like all channels driven power ratings, frequency response, video performance, etc, have improved as a result of testing, then you are deluded. But probably the most important thing measurements do is to help weed out poor or dishonest reviewers (there are MANY!). While this doesn't apply to Fred and Tom, measurements have certainly shown me who at Stereophile or other review sites is not to be trusted (a shockingly large percentage IMO).

Dan's picture

I stopped reading measurements years ago. What is important is how an audio product sounds, and that is what I rely on a reviewer to tell me. If it's a good review, I put the product on my short list of equipment to audition.

Richard's picture

I don't want you to give up measurements. I also think that speaker measurements are most important. All measurements provide valuable information, even if it is not complete information, and they do keep the suppliers honest. I would be willing to live with judicious use of measurement. I would be happy to see inexpensive equipment reviewed without measurement. I would also be happy to see some more expensive gear reviewed without measurements, but to see that the majority of gear that claims to be high-fidelity (whether audio or visual) be measured and tested.

Sean's picture

I would personally like to see more reviews. While measurements may help to illuminate a designers goals , the proof, as they say, is in the tasting. I plan on listening to any system I buy, and that is how I make my buying decisions. I've only once bought a product based on a positive review,and ended up with almost immediate buyer's remorse. The product was a pair of EPI 320 (?) loudspeakers and they had been glowingly reviewed in the now defunct High Fidelity magazine. I ended up replacing them with a pair of Paradigm 9s that I actually LISTENED to before I bought them. Now I use the reviews to narrow down the list of products I want to audition, and measurements don't really even matter to me.

Israel's picture

No question - more reviews in a more timely manner. Measurements are nice, but it's the perspective of your experienced reviewers that is really valuable. BTW: RSS feeds for your blogs would be nice too. I'd read them more often if I could subscribe.

Sam's picture

I believe measurements are the only way to get true objective analysis. Of course the reviewers opinions are valuable and can add new insight as technology changes, but subjective listening or observing has to vary with the reviewer and over time with changing expectations. Calibrated instrument readings are not subject to interpretation. We trust the expertise of the reviewer to inform us what measurements are important to consider.

John's picture

Scott, I'd prefer more reviews without the measurements and that goes for speakers and electronics. I personally feel that measurements seldom show anything that has not been pointed out by the reviewer. There are way more items on the market than you, or any other magazine, can ever hope to review, the more you can look at the more likely I/we'll have some information to work with before we start our own evaluations. Living as I do in a fairly rural area of the country, I have to travel extensively to see/hear many of the products I am interested in. Having some input from reviewers here and elsewhere can really help me to narrow choices before I climb in the car and burn more of that liquid gold.

rrs's picture

Please keep the measurements, if for no other reason than to confirm the manufacturer's claims. Assuming that test equipment does not have hidden agendas, different testers should agree within a reasonable tolerance. However, subjective reviewers have their own pet buzz phrases, palpable presence, the sound glowed (from recent KEF 2007/2 review in Stereophile) that do not translate among reviewers. For me, subjective words do not always translate into useful information. Stereophile magazine is particularly bad about this. Measurements may not tell the whole story, but subjective descriptions are little better at describing what I can expect from the product. Keeping both aspects of the review is the only way I have to vicariously experiencing its performance.

Enter your Sound & Vision username.
Enter the password that accompanies your username.