Double blind tests

[sideshowbob]

Well, if Tom Noisaine tests "goldeneared audiophiles" in a major publication by clearly telling them that they will listen to two different systems one of which uses "junk" electronics and a "poor" setup and another that uses expensive "audiophile" valve Amplifiers has he eliminated or minimised expectancy?

Well, I don't know the test, but if that's what he did, he did a bad job.

But I don't think ABX as a methodological principle is in danger here.

I think we're probably agreeing, for perhaps the first time ever

-- Ian

 

[wolfgang]

There is nothing specific. If you go through the research papers on perceptual coding you will find many items that indirectly relate to audibility limits. It is quite amusing how selective the ear/brain system is as what it notices and what not.

Sorry, no handy references.

Ciao T

Never mind. I do like to find out what audiophiles believe in. It seems to change every few days.

 

[3DSonics]

Hi,

Well, I don't know the test, but if that's what he did, he did a bad job.

The "test" was published in the US based "Audio" Magazine.

I cite it and Tom Noisaine (who is extremely active in the "ABX Crowd", this refering to a very vocal US based group of individuals who have been presenting themselves for decades as "guardians of the consumers" agains the "high end lunacy" and who's creed is basically "everything sounds the same") to make the point that the main visible proponents of ABCX testing have a clear agenda and misuse the test to give pseudo scientific credence to their argument which claims basically that "all well designed gear sounds the same".

But I don't think ABX as a methodological principle is in danger here.

It is, wherever the "ABX Comprator" and relevant trademarks are mentioned and where any of the people of the "ABX Crowd" are involved the lack of separation between a commercial/ideolgical/religious endeavour (the ABX Company, it's products and proponents) and a scientific principle (ABX testing as such) leads confusion and creates major FUD (Fear Uncreatainty Death).

Many people, including myself, will use the term Double Blind Testing (or DBT) to separate out the specific group of people and tests carried out in the shadow of the ABX Company in the US and often view the term of ABX testing as applying to that particular questionable subset.

Further, I think the ABX test which sparked originally the debate in this thread is more an ABX Test in the sense of belonging to the ABX Crowd subset, as opposed to the more generic and unloaded "ABX testing Methode". But this is a personal opinion.

Ciao T

 

[Couperin]

3DSonics wrote:
"When I talked about audio blind testing I forgot ONE really shining example of testing... Markus Sauer... Stereophile"

Thanks for a really interesting and illuminating post. So good to see something new in this thread. Unlike the many posts that start from personal prejudice, avoid objective consideration of other people's posts, and end in petty insults.

 

[wolfgang]

Giving the complete results of Jürgen Ackermann's experiment would be way beyond the scope of an article such as this one; besides, Ackermann has not yet completed his statistical analysis. But there are already some results that seem interesting enough to warrant a preliminary report.

Anyone know whether this research was published in a peer review journal after the final statistical analysis was carried out?

 

[johnfromnorwich]

Anyone know whether this research was published in a peer review journal after the final statistical analysis was carried out?

Nothing indexed on MIMAS/Web of Science/Web of Knowledge - although I only looked back as far as 1998. Some journals aren't indexed there, but most credible stuff is (and some indredible stuff too ). It could be earlier I suppose, but generally if a thesis is cited as a source it means the data wasn't published elsewhere at the time.

 

[Paul Ranson]

Look at this http://www.provide.net/~djcarlst/abx_data.htm

A couple of immediately interesting things can be seen from the data on this site. Regardless of whether you have a mystic approach to hifi or not and ABX that produces 'different' is informative.

One is that interconnects didn't register as different while a level difference of 0.3dB did, albeit with pink noise. The implication is that interconnects are less different than a small level difference. So any listening test at all, sighted or not, that doesn't level match to a small fraction of a dB runs the risk of the listener preferring the louder.

Two is that a single generation copy on a high quality analogue tape machine is audible. Which I found slightly surprising.

Paul

 

[Effem]

Can I also add another observation?

Some of the component choices are rather baffling too. I wouldn't call any item under test as "state of the art" and one of the speaker cables I tracked down more by luck than judgement has less than complimentry end user reviews anyway. I still don't know what a "PSACS Best" interconnct is so the result is rather meaningless.

This is another "SAME" outcome:

Audio Alchemy DDE Version 3.0 vs. Marantz CD-63 = SAME
20/40 samples = 50% No listener numbers quoted.

This one I find hard to accept and this text is even more baffling:

"The Audio Alchemy DDE Version 3.0 DA converter was driven from the digital outputs of the Marantz CD-63. The same/different comparison was between the DDE's outputs and the Marantz analog outputs, however the analog signal was put through another AD-DA conversion in a Marantz CDR-610 in record mode. This was done for level matching. The comparison was single blind same/different with listeners moved out of the room for cable changes. "

Sheesh

 

[Dev]

I agree. I describe a "in operation" problem, but one that is so common as to have in effect become a principle one.

When I talked about audio blind testing I forgot ONE really shining example of testing. I hope Markus will not begrudge me some extensive "fair use" citation from God is in the Nuances - Markus Sauer - Stereophile, Jannuary 2000:



Ciao T

I think we're probably agreeing, for perhaps the first time ever

-- Ian

Are you guys feeling OK? Do you feel the need to sit or lie down?

 

[sideshowbob]

Thorsten's being uncharacteristically reasonable. It'll wear off, I'm sure ;-)

-- Ian

 

[George Sallit]

T

Thanks for some sensible posts. I had seen but forgotten the Stereophile article. It is in my collection.

I like the differentiation between ABX and DBT. The ABX site shows a lot of quite poorly controlled tests. I was under the impression that the tests should vary one parameter a time (using whatever precision measurements are needed) and then determine if there are observed differences and then analyse this 'good' data. Putting an ABX label on a test does not make it 'scientific' anymore than using a high precision ruler with 1/10" divisions is any good for um measurements.

The idea of using a cheap parametric equaliser in an ABX test to remove frequency response variations, or an AD/DA convertor to match levels is really solving a simple problem but creating another (potentially worse) one.

 

[Paul Ranson]

ABX is a procedure, DBT is a protocol. There is no intrinsic connection except that ABX is a good way to run a listening test that is 'blind'. It avoids the need for listeners to agree on a preference, merely a difference. It allows for listeners to take as long as they like making a decision.

I suggest downloading PCABX (from the ABX guys....) with some samples and finding out what different processing sounds like.

Paul

 

[George Sallit]

ABX is a procedure, DBT is a protocol. There is no intrinsic connection except that ABX is a good way to run a listening test that is 'blind'. It avoids the need for listeners to agree on a preference, merely a difference. It allows for listeners to take as long as they like making a decision.

I suggest downloading PCABX (from the ABX guys....) with some samples and finding out what different processing sounds like.

Paul

Paul,

It is the PCABX that I object to.

It is crude, crude, crude. I get a wonderful music signal and shove it through a £5 sound board on my PC to test what???

 

[wolfgang]

Nothing indexed on MIMAS/Web of Science/Web of Knowledge - although I only looked back as far as 1998. Some journals aren't indexed there, but most credible stuff is (and some indredible stuff too ). It could be earlier I suppose, but generally if a thesis is cited as a source it means the data wasn't published elsewhere at the time.

Pity. It would be nice if the researcher own full report are available.

I find this are all very interesting. I hope we all agrees good DBT is much better then collections of anecdotal 'one-person-long-term listening' opinions.

I do hope more hobbyists could try to understand what makes a good DBT and learn from these mistakes. Just because they are difficult to get right is not a valid excuse not to do them.

It is up to us to organise better versions and most importantly publish some how ........ even in hobbyists forums.

The main manufacturers and hifi magazines obviously are not interested for reason best known to themselves. I hope we all agrees we want better toys and that is why we are here. Maybe, I am an idealist if that is what we really want then it is up to us the hobbyists to find the answers.

 

[melorib]

I hope we all agrees good DBT is much better then collections of anecdotal 'one-person-long-term listening' opinions.

What in this thread gives you any hope of that...

 

[Couperin]

Wolfgang wrote:

I find this are all very interesting. I hope we all agrees good DBT is much better then collections of anecdotal 'one-person-long-term listening' opinions.

I do admire your optimism. I have a sneaking feeling that there are contributors to this thread who would argue that earth is flat and the moon is made of green cheese - especially if any form objective test had proved otherwise. Still, we live in hope (and die in despair).

 

[Bob McC]

I wouldn't bank on it Wolfgang! There are manufacturers whose websites are full of such testimonials as some form of objective endorsement.

 

[melorib]

I have a sneaking feeling that there are contributors to this thread who would argue that earth is flat and the moon is made of green cheese - especially if any form subjective test had stated otherwise.

 

[Joe]

I have a sneaking feeling that there are contributors to this thread who would argue that earth is flat and the moon is made of green cheese - especially if any form subjective test had stated otherwise.

But surely subjective experience would lead one to believe that the earth was flat - it is only revealed as being round by objective evidence such as photographs taken from space.

 

[melorib]

I am not sure, suppose a monk with very good eye sight would climb to the top of the Everest, look into the horizon, and see that the earth was round, what would the objectivists say ???

We dont need to go that far, all the objectivists rejected the idea of a round earth when Aristotle said so 350 BC...