Conclusion: double-blind research, does it matter?
Contents
With this article, I hope there will be a little more insight and understanding of what is involved if you would like to organise an ‘objective’ listening test in audio. I have no illusions that this will stop the comments and emails, but I can always refer to this article in response.
I will save the real kicker for last. Suppose we at Alpha Audio magically manage to do a scientifically valid double-blind study on preferences or differences. As a reader, what would you then gain from the results of that study? I will tell you: very little.
The study then tells you that it has been shown that a percentage of listeners prefer A to B, or hear a difference. But that says nothing about your experience when you sit on the sofa at home. Because maybe you are someone who would prefer B over A if you participated in the study.
In short: always listen yourself and choose what you find most pleasant. You should listen to it, not the largest group of people.
This conclusion is not scientifically based, but it is so logical that I hope you understand that “double-blind or it’s not true” in (consumer) audio is a straw man argument with which you only want to inflame or stifle the debate about a test.
… even worse:
If you manage to successfully differentiate between let’s say some cables or amplifiers in your controlled environment, this may not at all reflect how these samples behave in a different environment. Talking different floors (cable risers?), different qualities of AC from the wall, and so on.
Maybe two power cables with a specific amp sound the very same in one lab, but manage to sound different from each other in another environment.
In the end to me it seems that the “double-blind-A/B/X police” constantly fails in explaining why two different cables/amplifiers/whatever should sound indistinguishable from each other. Their standard phrase then is “if it sounds different, it’s broken”. 😉
Thank you! Now, we could use a similar discussion of why statements about cables like, “It has no coloration of its own and just lets the sound of the components through,” or “It recreates just what it was like in the studio,.” reflect fundamental misunderstanding of how cables and audio recordings are developed and created.
To add to my previous comment, I would say that in audio a double-blind study method is also unnecessary, since if one has a helper and they follow a method blind to the listener that’s sufficient, i.e., single blind. But if one doesn’t, or it takes some time to switch, then the effect of the physical — time and/or energy expenditure — come into play.
For audio, however, your point about the absence of a placebo is something of a straw man, since the question is difference not so much effect (measurable or scalable symptoms). Affect is a different matter and that’s really a Yes or No query — I like it or don’t, or some gradient of that. Then the question becomes which do I like more (and then other questions, such as function needed and cost).
Finally, and I think this is the “bottom line,” the person who feels they need a blinded method has two big problems on the face of it: hearing/listening ability, and honesty with themselves. For that person, the only answer is to judge on affect, which one likes better. If it’s a toss up, then go with the cheaper or most functional for one’s purpose.
To quote from the article: “ A/B is testing two variants of a test subject against each other. […] After all, the results then do not tell you whether A or B is better, but whether there is a clear preference in the listening group between A or B.“