Double blind tests

[DavidF]

I think Ian's point was aimed at cable and tweak mongers in general.

I think he makes a perfectly valid point.

........For my money an imperfect one.....

Most certainly there IS nonsense out there (I totally agree, Ian).....I don't personally count "wires" (cables?) under that banner, at least those that are produced at modest cost (I'm not expecting you to agree with that BTW...I know you don''t! ).

 

[Effem]

It seems that some posts in this debate are becoming emotionally overwrought and departing from common sense. I'd like to address the original subject calmly. I don't regard myself as a naysayer; nor do I rush out to buy things on the basis of favourable reviews. I like to form my own judgements where possible. Failing that, I look for reviewers who demonstrate a high level of objectivity, for example, http://www.tnt-audio.com/int.html

So here's my two pennyworth:

1. Blind trials are intended to prevent conscious and unconscious bias in research. Double-blind trials are a more stringent way of conducting an experiment with the aim of eliminating subjective bias on the part of both experimental subjects and the experimenters, and of achieving a higher standard of scientific rigour.

2. I don't see how anyone can argue with that as a definition, or dismiss double blind trials as an impartial method of comparing things, whether it be butter substitutes, washing powders, medicines/placebos, or audio equipment.

3. If, in audio tests, the subject can't tell the difference or prefers the cheaper equipment, and decides to buy the cheaper equipment, that's fine. The manufacturers of cheaper audio equipment may be smart enough to tailor the sound of their wares to meet the preferences of their target market. That's probably how they succeed in business.

4. If, in audio tests, the subject can't tell the difference, or prefers the sound of the cheaper equipment, and still elects to buy more expensive equipment, that's his or her choice. However, there's no sense in justifying that decision by trying to rubbish a proven empirical test method.

5. In the audio test, some people could hear a difference. Some could identify (and preferred) the more expensive equipment. Some could not. Some people have more acute hearing. End of story.

Some people have this unique knack of reading half the thread, cherry picking half of that half and then making global inaccurate statements based upon what little they have assimilated, but I won't accuse you of that Couperin by the way, your post has been objectively dispassionate and highly constructive.

I see no wrong or harm in anyone doing their own private blind testing to validate their own purchasing choices; no dispute about that at all from me and I do it myself. Where this is being erroneously translated by some into my statement about double blind ABX testing being flawed is where the argument starts. I don't intend to type out and drag up yet again what I wrote earlier in the thread, suffice to say every time I have either conducted, read and/or participated in a double blind ABX test it has always produced inconclusive results, whereas an equal numer of blind A/B tests has produced conclusive results. Same components, same environment, same listeners even yet the differing results are not to be dismissed.

The crux of the matter then is why blind A/B tests are readiy dimissed by the sceptics as being subjective and therefore invalid, whereas a double blind ABX test protocal is embraced as an objective test, knowing that it is still subjective in outcome whatever the methodology?

 

[bottleneck]

Effem, with the greatest of respect it's hard to repute a scientifically conducted DBT as an objective test.

It's equally hard to claim a sighted test can be conducted without sight-bias.

 

[Paul Ranson]

I have either conducted, read and/or participated in a double blind ABX test it has always produced inconclusive results, whereas an equal numer of blind A/B tests has produced conclusive results.

It's unclear what you mean by 'blind A/B'. IMO the test reported at the top of the thread is 'blind A/B'. They are less than completely satisfactory because the results can be ambiguous, as in this case. ABX produces clear results, and there's no need to worry about personal preferences across a range of listeners. So what's your objection to the ABX approach? Other than that the results don't concur with your beliefs?

Paul

 

[Effem]

Goodness me the prejudices are out in force in this thread aren't they? I have never once elaborated on what the "conclusive" results of the A/B tests were, not even mentioned once what was under test even, yet everyone seems to believe that they were all negative and all against what my anticipations were. You couldn't be more wrong if you tried.

 

[Effem]

Effem, with the greatest of respect it's hard to repute a scientifically conducted DBT as an objective test.

It's equally hard to claim a sighted test can be conducted without sight-bias.

I respect your view Bottleneck, but a scientifically conducted objective DBT where the outcome can only be objective then no, but using strictly objective methodologies to determine a wholly subjective outcome is rather questionable.

I agree wholly that sighted tests are difficult to conduct without some form of bias entering, but not impossible either.

 

[Paul Ranson]

You couldn't be more wrong if you tried.

Why not read the post and answer the question?

but using strictly objective methodologies to determine a wholly subjective outcome is rather questionable.

Now we get to the meat. If your cables actually sounded different then there would be an objective outcome.

Paul

 

[Lynn SkeptiK]

If your cables actually sounded different then there would be an objective outcome.

Paul

Not necessarily.

Perhaps no one on Planet Earth has sufficiently sensitive hearing to appreciate the difference.

That is, unless their hearing is supplemented by visual data, which, as we all know, greatly increases ones ability to appreciate both thrifty tweaks and expensive upgrades. Since conscious and visual recognition of any tweak or upgrade is necessary to implement it in ones system, this is surely the preferred option when auditioning such changes.

How else would one know if it's better or not?



:Quad:

 

[Effem]

Now we get to the meat. If your cables actually sounded different then there would be an objective outcome.

Paul

I have never once mentioned cables apart from as an example of components to be tested

You see, bias and prejudice is a real handicap in these debates

 

[Effem]

I'm going to bow out of this one now as there seems to be a proliferation of folks that seem to read what they want to read in postings and avoid what's actually there because it doesn't suit

 

[Paul Ranson]

Cables are a good example of components that are claimed to sound different for paranormal reasons and are therefore totally applicable examples. Given you are also a commercial manufacturer of such cables and make claims about their 'sound' the example is appropriate squared.

Anyway can you enlarge on what you mean by 'AB blind' test? And tell us about successes and failures?

that seem to read what they want to read in postings and avoid what's actually there because it doesn't suit

This is you!

Paul

 

[Effem]

Cables are a good example of components that are claimed to sound different for paranormal reasons and are therefore totally applicable examples. Given you are also a commercial manufacturer of such cables and make claims about their 'sound' the example is appropriate squared.

"Paranormal reasons"? You need a holiday methinks

Read this just one more time: I AM NOT HERE TO DISCUSS CABLES Is that clear and succinct enough for both you and anyone else?

Anyway can you enlarge on what you mean by 'AB blind' test? And tell us about successes and failures?


Paul

You sure that you have been reading the same thread as I have posted in? I have no issue with personal blind A/B testing but I have major problems with double blind ABX testing being proffered as an objective method with regard to hi-fi component evaluation. I dislike even more the cynics using it as a weapon in debates.

But if you must ask, here is the last test I carried out, firstly by controlled blind ABX testing followed immediately by a simple blind A/B test.

This is what the listeners saw:

There were 8 listeners each armed with pen and paper. 6 excerpts of music of different genres was played three times each for item A, B and X which could be either A or B in a random order from track to track. I'm not going type out the results because I have better things to do with my time but I still have the forms. The point to all this though is not what got written on the paper by the listeners, more to do with what happened during the ABX test and subsequently.

After the 'proper' formal blind ABX test with 18 excerpts of music to contend with and much switching on and off, I then changed the entire procedure by saying it will be an informal quick blind A/B test with only one music track to compare. The look on the listener's faces was priceless. The change was made once only and the results were considerably more conclusive than the formal ABX test. It was still under blind conditions.

The thing to note is that once the word "test" was uttered and the listeners were led into a room to see a sight as in the photo above, with pen and paper at the ready, then in my opinion the whole double blind ABX test was doomed on several different levels and the discussion was no longer about the abilities of component A and B any more, it is more about human reaction to stress and situations. Because the listeners are aware that "X" is going to occur several times during the test then they are virtually expecting "trickery" to happen which also skews their perceptions even when it doesn't occur. You should then be able to pick up from that information that the original hypothesis that was used to formulate the test no longer applies because it has changed from "Does component A sound better/different to component B?" to "How easy is it to confuse the listening panel?".

In total I have done around 6 test regimes like this and each has produced identical results. My view then is what purpose does a double blind ABX test protocol serve when the listeners are the variable, not because they are, but because the ABX test has actually made them that way!

Hi-Fi Choice used blind listening panels at one time and I think they were spectacular failures, mainly because they unfairly condemned perfectly good components. I have my own personal opinions of Nordost cables, but I hold some sympathy for them after the Valhalla power cord double blind ABX test fiasco they went through a while ago.

At the end of the day, I have never yet seen anyone with an ounce of common sense go through any such rigmarole over a hi-fi component purchase because a well grounded person just plugs it in and uses their ears to decide - because that is the one and only criteria it was bought for.

 

[Paul Ranson]

Now that's a more interesting post.

FWIW the classic 'ABX' test allows a listener to choose A, B or X as many times as they want before making a decision in each trial. I think the results you got from your test aren't particularly surprising.

I don't see why hifi purchasers should be doing this, it's the proponents of the components and tweaks. Cables, some amplifiers, some digital sources, supports, 'Sound has priority' labels, CD lathing, the list goes on and on. In general anything where there is no known reason for a difference in sound. The existence of a properly constructed DBT would allow the listeners to use their ears confident that there was something to hear. Or were the test to show no difference then to just buy the one they like the most (biggest wires, brightest colour, cheapest, all reasons are equal now!) without having to justify it with hifi bull.

Paul

 

[Effem]

Now that's a more interesting post.

FWIW the classic 'ABX' test allows a listener to choose A, B or X as many times as they want before making a decision in each trial. I think the results you got from your test aren't particularly surprising.

Arguably yes, but the longer the test goes on the greater the inconclusiveness (sp) sets in.

I don't see why hifi purchasers should be doing this, it's the proponents of the components and tweaks. Cables, some amplifiers, some digital sources, supports, 'Sound has priority' labels, CD lathing, the list goes on and on. In general anything where there is no known reason for a difference in sound. The existence of a properly constructed DBT would allow the listeners to use their ears confident that there was something to hear. Or were the test to show no difference then to just buy the one they like the most (biggest wires, brightest colour, cheapest, all reasons are equal now!) without having to justify it with hifi bull.

Paul

I have not disputed one word of what you have said, in fact I support wholeheartedly what you say. At the end of the day neither one of us has the sway to influence what folks spend their money on, but we can help to steer them clear of blocks of wood with mythical properties one would hope. I have faith though that the majority of people do have their head screwed on and the ones that do buy pebbles in jars and cable lifters are doing it out of curiosity because they can afford to be curious, rather than outright gullibility

 

[Jimbo]

Drivel.

I made some observations about DBT and about how those who sell products which do badly in DBTs have an unsurprising tendency to dispute the validity of the entire mechanism. I neither had a dig at anyone in particular nor mentioned any names, I was making a general point about how the tweak industry doesn't like unsighted testing.

I did have a dig at Pete's tendency to traduce those who disagree with him by implying they don't believe people should be allowed to make up their own minds, but as far as I'm aware he doesn't sell cables or anything else, so he clearly isn't the subject of my comment about cable salesmen.

I hope this is clear enough for you.

-- Ian

Dont worry all is clear to me. One thing is perfectly clear you are a moaning old git. CLEAR.

 

[sideshowbob]

If you don't have anything relevant to say about the subject of the thread, why on earth do you bother to post?

-- Ian

 

[endust4237]

If we buy hifi equipments for listening some music, and not listening equipments and sound, A/B/X testing is fundamentally flawed being blind or double blind. Sound is only a part of the music listening process, not the essence of it. Looking for hte "better" sounding equipments means we do not see the forest through the woods.

When we are listening music at home (in an ideal case) we are not focusing on the details of the sound but let the music's emotion and messages flow. Our mind and brains are working on a very different way. If we want to compare hifi equipments on a short term, using short, few minutes excerpts and switching between the candidates, our mind and brains are working on a different way, focusing on the difference in the sound not in the music reproduction capabilities. It is usually against decoding the real emotional part of the music.

This is why any short term (few hours long) testing is questionable at least. You can focus on the music reproduction capabilites of the system only, if you are living with the system for a while, let's say not less than a week. Then you got a chance to have some experience with the changed system in few different moods, music styles, etc.

It does not matter which equipment is "better". Only the music reproduction capability matters and it can not be tested if you are not in the right mood, and your brain is not switched to the music enjoyment mode. Short term testing can give you false results.

It was my experience in my 30 years of music listening, that those people making decisions based on longer term trial, not in few hours or a day or two, will change their equipments much less, than those making decisions on the other way.

 

[Couperin]

I entirely agree with endust4237 that our enjoyment of music is about far more than the sound of the audio equipment. Though I sometimes wonder if some contributors to this and other hi-fi forums spend more time listening to the equipment rather than the music.

However, the original subject of this post was the validity (or otherwise) of double blind tests. The experiment did not set out to measure all the complex variables which contribute to our enjoyment of music. It measured one parameter only - whether listeners could tell the difference between two or more pieces of audio equipment which were meant to perform the same function. Within the limited scope of that trial, a double blind test is appropriate and accurate.

It seems that the main reason for contributors becoming heated in this debate is the wide range of unsupported inferences drawn from the results of the test. In fact, there is only one indisputable conclusion to be drawn from the test - either the listeners could hear a difference or they could not. Anything beyond this is simply speculation.

 

[zanash]

well said endust4237.....

 

[sideshowbob]

If we buy hifi equipments for listening some music, and not listening equipments and sound, A/B/X testing is fundamentally flawed being blind or double blind.

Absolutely agree. People should choose based on what makes them happy, regardless of what that is. Manufacturers of equipment should use ABX as a matter of course, however, especially those who are selling products which appear to be explicable solely by voodoo.

-- Ian