WAV compared to MP3 - by popular demand

p.s. I suggest a score of at least 80% would indicate an ability to tell which is the mp3.

 

There are well established mathematical procedures for determining statistical significance. It's 40 years since I was an undergraduate and applied these routines to experiments in perception.

Maybe someone on this forum can advise on how to frame the experiment and on correct statistical analysis.

Cos I can't remember? Can we use the t-test?

 

Keep the ideas coming.

My only concern with something complex is that people won't participate - we have difficulty getting people comparing A to B.

 

I agree. I think we should stick to comparing just 2 tracks at a time, if at all possible.

 

Lets get another one on the go then....!

 

I understand.

We can do anything for fun, so long as we realise we have not proved anything; but with the right advice we may be able to do something simple that does prove something.

 

I agree it needs to be as simple as possible. I recall one of the AVI lot started a thread somewhere, might have been pfm, and it had varying degrees of compression and the wav and those taking part were meant to sort them. I remember having a go, though I didn't actually take part - I found it overly complex and confusing trying to order four or so things so got bored and gave up. I don't think any question should have more than two options plus 'don't know', i.e. a simple File A vs. File B comparison.

Tony.

 

We could have a selection of tracks for A & B.
That woud allow for someone to compare two rock tracks, classical tracks etc.
But we'd only have one set of lossy and one set of lossless.

 

That's fine by me. It may also help indicate what types of music suffer the most / least from compression. It would not surprise me if some stuff survives the process better than others.

Tony.

 

In a simple test with just two tracks and one choice, each listener could be 100% right or 100% wrong just by luck.

One such test doesn't prove much.

A series of tests like that, say ten of them, would still generate a 50% possibilty, again just by luck.

But if someone could consistently score 80% or better hit rate then that would be impressive.

How do we keep track of any individual's score over a series of tests ???

JC.

 

the tests could have a limited length say 1.5 minutes

I don't know if I'm bothered if someone has a consistent score or not

for me its about how close they are & which gives the most enjoyment.

 

Yes John, a trivial test is not a test at all. It's just a bit of fun.

A I said there is a mathematics that allows you to say that your experimental data significantly varies from blind chance.

But you can't just pick your 80% figure out of thin air.

Another, more reliable way to do it is to present at least four files of the same music and ask individual to pick out the one that is different - better or worse. That lengthens the odds.

But, as I said, unless we are doing it just for fun we need someone who knows what they are doing to devise the test and analyse the results.

 

I think that if you run another simple test like the last one, we know that a 50/50 result will be a correct result.

This no better than chance, which is correct, since we already know that no collective group of listeners can tell the difference between mp3 and uncompressed files such as wavs. This is a well established fact despite what some individuals may think.

Since we know already that no collective group can detect the difference, further testing along those lines would seem to be pointless, other than to provide a little simple entertainment for those who consider it to be a novelty.

We no longer need to test the files, we need to test the listeners, to see whether those who do claim to be able to detect a difference can really do so in a meaningful test.

Also do not forget that some people, flatpopely for one, actually prefer compressed mp3 files to uncompressed wavs.

JC.

 

But where do the differences begin to show?

256 . . . 192 . . . 128 . . . 64?

I think 128 is just about OK and 192 is safe.

 

The test becomes statistically more reliable as the sample size increases.
So 10 people voting in a simple A//B/C (C being no difference) is too small to exclude guesses distorting the result.
100 votes makes things far more interesting though. As a cross forum poll we should do 100+ easily IMO.

 

Would you allow multiple voting?

 

Would make things over complex IMO, but of course it isn't just a poll, so listeners can comment in the thread.

WRT bit rate, I'd say 256 is a good place to start as it represents the iTunes+ standard and will be what many are using.

 

iTunes doesn't sell mp3, it sells aac+, which is a hell of a lot better imo.

 

We can encode to that if people prefer.

 

You could try all three, - wav, mp3, aac. I don't think anyone will perceive much of a difference at 256k.