Double blind tests

I'm generally against DBT, and I'm happy to trust my own ears. But the ears of others, especially self-proclaimed experts? Let me illustrate with an anecdote.

A friend who is endowed with a more high-end turntable than I took an interest when I recently installed a new tonearm. He is a VTF and VTA freak who insists that these parameters must be set by ear. First he twiddled with the VTF and both he and I could hear changes. He decided upon the best sound to his ears. Not surprisingly it was right in the middle of the manufacturer's recommended range, which was where I had it in the first place.

Then he commenced to fiddle with the VTA screw ring adjuster until he announced that he had found the optimum position. I couldn't hear the slightest difference whatever the setting this time, but I locked the adjuster and the satisfied expert departed, with comments about my deafness. Upon re-reading the set-up instructions I noted that I had omitted a crucial step, and the upshot of that was that the VTA adjuster was not actually engaged to the arm pillar. Thus his twiddling had produced no change in the VTA whatsoever.

I then set the whole thing up correctly, did my own VTA checking and heard considerable change (to my surprise) as I elevated and lowered the arm pillar, and found an optimum position that satisfied my ears. I haven't had the courage to tell my friend how badly he fooled himself.

Moral of the story: Probably several, but trusting your own ears is one of them. Being so arrogant as to believe that every physical change must produce a change in sound audible only to those with the requisite Golden Ears is another. Most importantly I wonder how reliable my friend's opinions would be in a DBT format? I'm sure he'd be the first to volunteer if one were on.

So if a significant (in the statistical sense) number of participants in a DBT were similarly afflicted with such arrogant self-belief, what reliance can we place on the "results"?

 

lol.

i'm sure we've all done something similar,dialed in a component by ear when we were actually doing nothing, then noticed later.

my best example of this.

way back, when i was a studnet i had a ghetto blaster, one night one of my mates came round and we proceeded to get all 'herbal'.

during the course of the 1st 45 minutes the radio kept going in and out of tune, so we naturally kept fiddling with the aerial and managed to tune it back in every time, this happened every five or so minutes.

we spent a while discussing radio reception and all that jazz and then bang on 45 minutes after i turned the 'radio' on. CLICK, silence, and then 5 seconds later the tape turned over and played the other side.

we'd been sat there listening to the previous nights radio broadcast that i has taped and didnt know it until the tape auto-reversed.

in our defence- we were quite stoned..

 

I have read that a few times now to make sure I fully understand what you have done. Very interesting. Thanks for sharing your experience. I think I could see what you have done and understand why to have come to your conclusion. It is absolutely correct based on what you have done. Of course it would be helpful if we could discuss about this a bit more. Maybe another day when everyone is a bit more calmer.

I do hope you are still willing to continue to investigate along this line to help all of us understanding of what is the best way to find the best hifi toys. Whether it is blind ABX, AB or the so call long term listening that audiophiles seems to still prefer. All very interesting as long as everyone don't take this too personal. It is just a hobby after all.

 

I think the human dynamics surrounding the test procedure was infinitely more interesting than the actual test protocol itself was Wolfie.

For far too long it has been argued and chewed over that the only objective appraisal of differences between components was to put it through the double blind ABX test. Fine when we are talking about a test specimen that will reliably give clear indicators for evaluation to form a conclusion from, but in this instance we have a clear dichotomy to consider; is it that the components cannot have any differences at all because the blind ABX says there isn't conclusively, or is it the test routine itself that isn't correctly formulated to provide the desired conclusive answer? Yes and no to both, hence the evidence gathered during these tests has always consistently been ending with an inconclusive outcome. To be completely honest, I would much rather be looking at reliable evidence that my perceptions have been PROVEN to be incorrect by watertight objective methods rather than repeatedly squabbling over any inconclusive result and batting percentage confidence levels to and fro.

From my side of the argument I have over the past 5 years or more received what I believe is a veritable mountain of contempraneous evidence that cables do make a difference, supported at times by end user's own blind A/B test results (yes they do them!). Empirical evidence it may be, but it would be rather foolish to wholly discounted, because it is required as a base to formulate a hypothesis to test, no? It is whichever way you look at it reliable evidence as opposed to opinion, supposition and conjecture, because if the converse were true even using the law of averages that 50% of it were not perceived to provide a tangible "improvement" in performance, then to a cable manufacturer it is no longer a viable business proposition whereby 50% of the products are being returned for non-performance. Fact. The actual commercial maximum in my opinion is 10% and yes higher percentages can be tolerated, but only by significantly (200-300%) raising purchase prices. In 5 years of trading I have had 10 returns at most and not all of those for non-performance I might add. If the naysayer's view was the accurate one, then presumably the returns rate would be a much higher percentage figure than the estimated nominal 50% at 90% or more, so their argument is entirely self- defeating because you cannot fool all of the people all of the time no matter how good a magician you are.

What is more at issue though is not that there may or may not be a tangible perceptible difference, but actually how BIG that "difference" actually is in real terms. I will use 2 examples to put this into context:

I am colour blind; my vision impairment is not unlike the way I perceive those that cannot hear cable differences. I am not making the inference that they are in any way "deaf" the same as I am not "blind" either because I cannot perceive the number 6 in the final image below.

Normal vision




I can only just see the number 3 in this image:


People without a colourblindness problem should see the number 6



I draw close parallels between colour blindness and hearing perceptions. No doubt someone will come along and dispute that. Feel free to do so.

It may be that others listen for different elements in the music they hear than I do, it may be I am focussing on what the boxes and wires are doing whereas others only criteria is to get the foot tapping and that can be prompted by a cheap transistor radio, hard bitten audiophiles may have a nervous breakdown if their LP's produce a pop, the permutations are endless and variable. It is that variability in human expectations that determines how far and how intense our own personal interest in hi-fi and music reproduction becomes, so it sets the threshold for those that don't want or care to box-swap and those that are desiring to have all their expectations met by whatever means - including tweaking.

That does not infer either that they become mindless zombies forever after their next "fix" of upgrade improvement for a better sound. Most intelligent people can discriminate what is pure bunkum like twisted toffee papers on top of speakers for a better treble sound, a jar of pebbles on your mantelpiece, or a bit of Mopani Wood dangling on a bit of string can allegedly do. Before you get too over zealous in debunking them, the mechanics of commercialism will eventually see them to their rightful resting place; I would love to see the sales and returns figures for some of this trash, or do people simply throw them in the bin?

A friend of mine recently defined the real difference between an audiophile and a music lover is the former looks for what wrong with his hi-fi and a music lover appreciates what's right with his hi-fi system. Never a truer word was said.

Another handicap to be faced is the indequacy of our language; one man's "night and day" can be another's "Can only just hear it" so when we come to putting into an understood language what is being perceived by the hearing the translation into what falls out the mouth passes through a wonky variant of Babelfish at times. Transmission and communication too can only ever function as a two-way street so a cable can for example be transmitting a particular sound, but if it's not being received exactly as sent then communication has to all intents and purposes failed completely, pretty much like an englishman and a scotsman talking in welsh to each other

A very learned man said to me recently "You are using what measuring instruments to measure what exactly?" and he was very serious indeed when he said it. He did elaborate a little by saying the primitive (his words) electronic measuring devices we currently have at our disposal cannot measure what I was describing to him. Quite.

One of the alleged improvements made by a box or cable upgrade is things like imaging and soundstaging. The "image" can be two dimensional, recessed, or forward in the mix and to the best of my knowledge there isn't a single instrument beyond the human ear that can detect it and give it a quantative and/or qualitative value. "Tighter bass" is another description often used, nut nobody yet has been able to pin an objective measurement on that aspect either and these are just two aspects with many more yet unresolved. For those that disagree that these phenomena exist then I can only say one thing and that is you really don't know what you are talking about - in the literal sense.

Next I will hopefully find the time to add more to this interesting thread which will deal with the human aspects of testing and auditioning, including some comments about the hi-fi market in general and hi-fi journalism to name but two.

 

Looks like an 8 to me

 

Then you have a different sort of colour blindness to me

 

Really? I see 12, 8, 6.

I don't think I am colour blind…

EDIT: Well I passed this with flying colours http://www.toledo-bend.com/colorblind/Ishihara.html

 

I see the 12 and 6 as well, didn't mention them as he claims they are 12 and 6, but claims middle one is 3 when it's an 8

I have had my eyes tested recently and I am not colour blind.

 

Same here. Passed with flying colours ha ha.

 

Effem,

Hi. From what you have described earlier they are 2 version of single blind tests. Still they are very informative and would be very most interesting if to you find the energy to write down actual results one day.

I understand why people think A/B comparisons are easier and better. Try asking people to describe the same toy twice. As long as you convinced them they were listening to two difference toys you will find virtually most audiophiles will genuinely believe they could heard differences. They could even describe them in very detailed words even when this shouldn't be in this case clearly. This have been demonstrated many times.

You need some proper controls to be sure the listeners and yourself have in fact not done another A/A blind test only. If you don't like ABX then you need to comes up with another way to do this.

 

according to my wife, this happens every sunday and twice on birthdays.

 

me too.

 

To control the test you could have an A/B, then A/A, B/A and B/B routine or whatever order you wish that could be just as quick and simple, the point being that the listeners are not barraged with repetitiveness fatigue or primed beforehand into knowing they are being "tricked" with A/B then an X in a series of three to contend with. It still amounts to having "X" in the test but in a way that is more definitive in outcome. It will still be 100% blind, it will still need correct identification of the components to qualify, but between each test session a break could be introduced to stop it being too formalised.

Incidentally, during one of the tests I did a couple of years ago I made fatal the mistake of not using 2 identical power cords, so the test routine inadvertantly went from ABX to ABCX and no wonder the conclusion was so vague. See, not even all kettle cords are created equal

Talking of ABCX, this is another reason why I believe the Wigwam test proved so inconclusive is because it wasn't a case of A versus B with another B or A as the control. We had Cable 1 a sooper dooper cord, Cable 2 which was a generic kettle cord, Cable 3 was 'another' generic kettle cord of unknown source and Cable 4 which was a custom homebuilt upgrade power cord. With that sort of lineup it couldn't do anything else but fail expectations

 

Effem.
Hi again. I don't understand why did you allow break during AB comparison but not for ABX?

Why do you have to make your friends had such a difficult time while doing the ABX only? That was never the point of ABX.

 

The last full ABX test routine I did was devised by someone else Wolfie and my home was only the venue for it Once that was completed in a strictly controlled environment, THEN I asked for a quick A/B test to see if more menaingful results could be obtained.

The next test I shall organise will be a refinement of the AB, AA, BB , BA, etc. system, plus I want to do a remote blind test but with a unique twist. I shall describe in words alone the totally unique sonic characterists of 3 popular generally available cable types with no clue whatsoever to their identity except the sound characteristics as described. If it is a delusion that we all suffer from, then let's see if it is the IDENTICAL delusion afflicting all audiophools worldwide. Naturally, I will also include descriptions which are NOT applicable to the subjects in the test as a control and if anyone selects these other cable descriptions it will be known to be guesswork alone.

 

I really don't think you can do 'ABX' in a play A, B then X once to an audience. The listener needs to be able to repeatedly listen to each and take as long as they want before making a choice.

Paul

 

I haven't so far found any hi-fi component that has successfully and conclusively passed a double blind test

Look at this http://www.provide.net/~djcarlst/abx_data.htm

 

Effem - why are you saying you can't find a DBT that shows a difference..... and then click to an example of DBT actually showing a difference?

Isn't that a bit bonkers?

 

Not really. Look at those numbers, are they 100%? Look at a cable double blind ABX results and are they 100%?

I have spent ages searching the web and have spoken with several people in the hi-fi industry in the past couple of days and not one could point me to an example of a conclusive double blind ABX test.

 

you'll never get 100%

If you gave 10,000 people a cabbage to eat and a mars bar with a blind fold on, there would still be a few to get it wrong.

A difference can be seen at a point of statistical significance which is significantly away from the norm (in an A/B comparison, 50%)

Or, as better put in the article you linked to -

" The different symbol means it is unlikely the percent correct score occurred by chance and thus the null hypothesis was disproven, which substantiates a real difference in sound quality. "