Dirty tricks or mistakes that might skew comparative listening tests

I'm starting this thread with the intention of discussing ways in which comparative listening tests can be skewed / biased / affected by tricks / mistakes / errors / conditions that prevent the components from being compared on a level playing field.

Forewarned is forearmed.

You may come across these tricks or innocent mistakes in: a hi-fi shop demo, at a friendly bake-off or even in the comfort of your own home when comparing something you've borrowed or bought.

Here's all the tricks or mistakes that I can think of. There may well be others or variations on the ones I've listed. Some of these could only be used in a sighted test. Some could be used in either a sighted or blind test.


1 Pump Up The Volume. Probably the simplest and most common way to skew a listening test. Play one component at a higher volume level than the other.

2 Nobble a Component. For example in a turntable bake-off, have a much more worn cartridge in one turntable. Or set one up with sub-optimum geometry / tracking weight.

3 The Synergy Game. For example in an amplifier bake-off comparing a 1kw ss amp against a 2 watt SET use 85db efficient speakers that dip to 1 ohm impedance played at loud volumes in a large room or use 105db speakers with a gentle impedance curve at medium volumes in a small room. Another example would be to stack-up the tonal balance: bright source, bright amp to compare bright, neutral and warm speakers.

4 Artificial Environment. For example comparing a suspended turntable against a non-suspended one with both on tables 6 inches away from the speakers – when in your home you have the flexibility to put your turntable on a wall-shelf on the opposite side of the room to your speakers. Or listening to speakers that are designed to be used in free space positioned against a wall or vice versa.

5 Hype, Brainwashing and Pre-conceptions. If you've read repeatedly in hi-fi magazines or Internet forums that a particular component has a particular type of sound you might have certain pre-conceptions that might bias your listening. Also some technologies have certain reputations so it's possible to go into a listening test expecting to hear a certain trait because of the technology used – eg horns are honky, electrostatics have poor bass, cones and domes are undynamic.

6 The Beauty and the Beast. Some people are biased towards hi-fi in prettier boxes. It's also possible to expect a component to sound a certain way because of the way it looks.

7 Price = Performance. If it's expensive, it must sound good. Surely? And if it's cheap it can't be much cop, can it?

8 Musical Hats. Only play the genre of music that most suits one set of components.

9 Let's Dance. When someone else in the room: taps their foot, hums, whistles, sings, fidgets, or dances to one component and not the other.

10 Lecture Time. When someone else confidently expresses an opinion before during or after a component has been played. Or tells you to focus on one particular aspect of the musical performance at which their preferred component excels.

 

Good list Lindsay.

More often than not i think that those tactics are used without a deliberate attempt to skew a dem. They just reflect reality and human nature IMO though everyone should be aware of them nevertheless.

 

Well, I'm 10 out of 10 for my last two bake offs; you should know Lindsay.

 

That was dancing? I thought you were having a mild stroke!

 

Different masterings of the same album. Much more subtle.

 

Here's one I forgot:

11 Under Pressure. Put the listener under pressure and they're less likely to be listening to the music in their normal way. I find that under pressure I'm more likely to concentrate on certain strands of the music, certain details, instead of letting it wash over me to get an overall impression of the music.


And the different mastering / pressings one is a good one.


I also think that it's possible to apply all of the dirty tricks or fall foul of all the innocent mistakes and for the relative merits of different components to still shine through - especially where there's relatively large differences between them.