Double blind tests

Most cdps use the same dacs the same chipsets and same analogue opamps. Hardly surprising they sound the same. This is nothing to do with flaws in double blind testing - its to do with uniformity of performance of cdps - something double blind tests have correctly revealed.

 

strange comment ....all cdp's sound the same ....

I've not heard that many maybe fifty or sixty but none have sounded the same to me ... similar possibly, uniformity of performance no..

simple test is to fit a good power cable and a poor power cable to a cdp and hear the difference ...and thats the same cdp !

if you were to keep all the parameters and variables the same and just swap cdp's the differences tend to be obvious ....in fact I've three AA5 all sound slightly different when set up to test if they were all working. Just changing the make of one cap in the output stage of cdp will alter its sound....

So no not in my experience....nor many other people coz if it was we would all buy the same unit.

But I've no problem if you can't tell them apart...must make buying a new one a breeze.

 

Sadly not as I dislike almost all of them. Only the PT dacs and a few others sound very good to me. Not surprisingly they use their own discrete proprietary dacs and filters and high quality discrete output stages.

 

I must admit I cannot see why DBT are flawed. IMO if you listen to any kit without knowing what it is you are more likely to appreciate it's sound qualities. Having said that I know sound is not the only thing that matters and how it looks can be important but that's not relevant to how it sounds.

 

(D)BT are flawed because they don't confirm sighted tests. If only more people were prepared to actually trust their ears.

Paul

 

Hi,

A DBT as such is not flawed. However certain rules of statistics apply and usual the sample sizes are insufficiently large to draw any conclusion with any reliability. As such the commonly mentioned DB Tests in audio lack statistic power.

A funny thing. If you "batch up" a substantial number of individual small scale DB Tests on a given subject that individually seem to suggest "no difference" with a very low statistical power, combined they suggest "difference present" with substantial statistical power, however such "batching" is a little suspect as the tests are not really sufficiently identical.

Ciao T

 

Well put Dev.

But hold on a cotton pickin' minute here, why would anyone need or even want to do a blind listening test in the first place?

I defy anyone to go into a dealer's and demand the audition for a new pair of speakers or whatever to do the entire demonstration unsighted. We'll all chip in for the flowers if you like

It's a hobby and an interest to the majority of us so let's not ever forget that keystone of the entire debate. If I am buying a heart/lung machine or a suspension bridge across a gorge I want it objectively tested until I have run out of tests before I'm confident of it's performance and the last thing I would say is "I'll have the heart/lung machine in pink with a 30% discount" and "That bridge looks a bit wonky but I'll only let one vehicle across at a time to make sure, insurance will take care of the rest".

Stamp collectors cream their trousers if they find a stamp with a minor flaw for goodness sake, the camera guys spend left arm/right leg sort of money for a lens that for example gives them one more F stop to play with, car drivers get exstatic about 0-60 time at one second less than their previous car, so it puts our neuroticism about an octave of lower bass into true perspective. Whether all these uplifts in performance is worthwhile or relevant is neither here nor there in the great scheme of things IN THE CONTEXT OF A HOBBY.

The entire hi-fi chain is an illusion to begin with; it's the illusion of having the band/musican/orchestra playing in your living room and as for the stereo effect with three dimensional imaging and soundstaging, then if that isn't an illusion then I don't know what is and I have yet to see any measuring device that can give it a true qualitative value either.

Ask any true audiophile if he NEEDS a controlled DBT for any one of his purchases and he will tell you to "Piss off" or similar and quite rightly too. Why? Because all of it was paid for out of his own money for his own listening pleasure and nobody else's, so in his view what does he need some boring miserable individual bending his ear telling him what he doesn't already know about minute differences? I think it's only the sad hat's of this world driven by the politics of envy, or are too tightfisted to pay more than a tenner for any of their purchases who delight in inverse willy waving to get their rocks off through squawking about double blind testing. Good luck to you if that's your chosen path, but please don't come onto public forums widdling on everyone else's parade by saying yours is the right way and the rest of us need certifying for our stupidity, because by even inference that is exactly what you are doing

 

Zanash - I can understand increasing the number of particiants would remove bias - but what bias would you be trying to remove in a test such as this - that some people have worse hearing than others? or more ear wax? 68 seems quite a high number of participants

I have heard it said that when looking at a system of just say CDP, amp and speakers, that the latter account for 70% of the specific system sound, the amp, 25% and the CDP 5%. Since the speakers were the same in this particular test, this could account for the lack of "night and day" difference bewteen the set-ups given the huge cost difference.

I myself use a Cyrus CDP, Pathos TT amp and LSA2 speakers. I gave some of my old hifi to my in-laws and their system of Marantz CD63 CDP, NAd 3020 amp and speakers from a Sony midi system doesn't sound a million miles away for acoustic/folk music from my set-up

Of course you could say they are both crap which would be an arguement

 

kmac - the greater the size of the trial, the larger the statistical significance of the results.

 

Agreed, but even with a sample size of 68 I would have thought the differences between a system costing a few hundred quid and one costing thousands would be more marked - i.e. that a larger proportion would prefer the more expensive system.

I'm not tryiong to say more expensive is always better, but I would have hoped their high end system would have sounded clearly disernably "better" than the low cost kit.

 

Well said.


LS

 

That's like saying all cars have an engine, have 4 wheels and get you from A to B so why spend hundreds of thousands on one that claims to do a little bit better?

So, you are prepared to pay a few bob more for something better than an engine and four wheels? But back to the thread subject, would a DBT reveal your discriminating tastes? Would it really matter to you perosnally if it didn't?

This isn't having a pop at you at all BB, it's putting into context your choices and values in component selection against the insistence of somebody else to scrutinse the way you participate in your leisure activities. To me these debates are all about invading my pleasures and enjoyment in an activity that they have no right to interfere in with their negativity. I'm sure there are plenty of other forums for the cynics where they can squabble to their hearts content over the technical issues and reams of statistics until the sun goes cold.

A hi-fi forum should I think be a positive celebration of the hobby with interchanges of proven and also new ideas between like-minded people, not raking up the same old arguments time and time again for the sake of it that prove and achieve nothing. I visit another hi-fi forum that is riddled with negativity from stem to stern and I don't know why folks bother to post when they have nothing constructive or beneficial to say on the subject. I visit other forums too besides hi-fi and I have yet to see protracted 1,000 posting threads where they almost come to blows over say for example the difference between say natural wool and man-made fibres. It's only a stereo we are talking about

 

Those 'claims to do a little better' can normally be substantiated.


LS

 

hows this for an alternative testing method..

 

Before one participates in a double-blind test of whatever merit, one should be able to prove beyond statistical doubt that real differences can be discerned. Thus a preliminary same/different test must be passed before any preferences can reliably be expressed.

And one should also be able to pass a single sighted test first. I bought into the cables make a difference idea until I compared two i/c cables (one cheap and unheralded and one expensive and lauded), plugged with identical phono plugs into the dual outputs of my Arcam CD93. I knew what I was listening to but I couldn't hear any difference in both short-term trials and in seeking long-term satisfaction. Thus I'd be a write-off in an i/c DBT (if there really is a difference).

But I think I can tell the difference between some brands (not all) of competent speaker cables on a sighted basis. And I am certain that some brands of power cables are better than electric kettle cords. The music is louder to start with and clarity is improved.

 

the point of this test is that they were asked which they prefered. that's a personal choice, based on the assumption that more expensive is better to their tastes. some guys might actually like crap stereo, they just don't know it. so why shouldn't they favour the cheap system?

the only valid data from this test is that anyone expressed a preference vs no preference, and 2/3 expressed a preference, showing a significant number felt they could discern a difference between the system.

so 2/3 said there was a difference, the fact some of them liked the lower cost system could be purely down to their personal tastes, we just assume it proves that an DBT shows that more expensive is not better. it doesn't it just shows in this case that more people could discern a difference than not and in this case a roughly equal proportion liked the lower cost to the high cost.

A DBT should just ask people if they can identify a difference, and maybe if they can do it repeatedly with accuracy, it shouldn't be spoilt by asking their preference. that's a different question.

 

I have no issues at all with folks that say "I tried X and Y cd player and I heard no difference at all" and that can equally apply to amplifiers, cables, speakers, DACs, whatever, because that is their first hand account of what they actually perceived. Fine.

Where the line is drawn for me is when that same person makes a strong assertion/assumption that because of what they heard once, then automatically ALL of these components must therefore be snake oil, hallucinations, delusions, placebo, etc., etc., ad nauseum. I have no time at all for those that shout very loudly when their two comparative items cost £20 and £30 respectively. Why it should then automatically be a candidate for blind testing and christ knows what else is simply beyond my logical train of thought, because if something does nothing from me then I simply walk away and ignore it. Job done. If somebody subsequently asks for my opinion about it I say "I tried X and Y and heard no difference, you might find otherwise".

If my entire listening time was spent blindfolded, or if my hi-fi was in another room so I could never lay hand or eye on it then blind listening might feature somewhere in my purchasing choices, otherwise I'll carry on with the fleshy protuberances either side of my head thanks very much.

 

Why would you even consider wanting to participate in a double blind test for hi-fi? Double blind tests are for medical trials with a known objective outcome, whereas hi-fi listening is purely subjective with infinitely variable outcomes, hence the high degree of inconsistency time after time in the results no matter how large the sample.

To me there is only one clear choice; either you enjoy the hobby for what it is (a pleasurable pastime) to whatever degree you can afford in both time and money, or you enjoy arguing on forums because that's what you like to do. The two to me are mutually incompatible.

 

Why on earth would you see it that way?

Maybe some people really do care about the sound rather than the looks of their hi-fi and the 'hobby' of spending money on it. Maybe someone who runs a music shop franchise (or whatever) wants to know which system the majority of people who come to their shop will prefer the sound of when no-one can see the kit.

Hi-fi may be a hobby to you but it is not to everyone. There are many (more, in fact) uses that are entirely professional where the performance per £££ is the all important factor. If you spent £150,000 for a line array speaker rig for a concert tour you don't want to find out that 60% of people prefer the sound of the one that was £100,000 when they don’t look at them. No one will bloody look at them when the band is on stage!

Someone new to the hobby who has just saved up for 6 months wants a good hi-fi. They see that people like you have bought really expensive gear and say it sounds better. They would probably jump right in at the deep end and audition expensive kit. If they knew about the blind testing that shows many people prefer the more affordable gear they might have bothered to try that and found that they also preferred it and saved £5,000.

 

So how does double blind testing fit into your view of that situation?

I agree, some musicians and composers use hi-fi as a tool for their work. What do they use in their component selection processes?

Maybe that is a cheaper option in my listening room - I'll investigate further

My word, your word, a magazine reviewer's word, the word of every poster here AND a full bells and whistles blind test would make no difference whatsoever to this person and nor would I ever shove down a newbies throat that what I like kit-wise is the right choice for them to spend their money on. I have some kit here costing less than £200 that embarasses some costing £1500+ which equates to a saving of £1300 in hard cash terms, but to our allegorical newbie it might sound utter pants and look ghastly to boot, so what "saving" has been made?

We can make whatever proposals we like to others when they ask for advice, but at the end of the day it is their choice, their money, their listening pleasure and finally they that have to live with that choice, nobody else. Double blind testing isn't the answer to all that and if you think it is then who is the deluded one?