Intro
Contents
Whenever we do a comparative listening test, comments are bound to appear below the article or in emails to us, saying “why didn’t you perform a double-blind test?”.
It is a kind of a mantra: double-blind testing. But is it useful for what we do? I have a viewpoint that I would like to share.
A little background
This article is not going to be a thorough, scientifically irrefutable position paper. I try to write it down in (somewhat) understandable language. In a previous life, I was a researcher at an economics faculty and my specialism was economics, statistical modelling and research methodology. I do not know anything about the field of audio research, but I do know about methodology.
Research methodology is the discipline that tries to prescribe which research methods are fit for what type of research, so it can lead to sensible evidence that your research question is either proven, or to genuinely state your research hypothesis is disproven. In research in the social sciences, economics being one of them, the outcome is 256 shades of grey and never black or white. It is quite difficult to arrive at ‘proof’.
Is double-blind testing useful to us? Then I must first explain something about conducting double-blind research and what this method requires.
… even worse:
If you manage to successfully differentiate between let’s say some cables or amplifiers in your controlled environment, this may not at all reflect how these samples behave in a different environment. Talking different floors (cable risers?), different qualities of AC from the wall, and so on.
Maybe two power cables with a specific amp sound the very same in one lab, but manage to sound different from each other in another environment.
In the end to me it seems that the “double-blind-A/B/X police” constantly fails in explaining why two different cables/amplifiers/whatever should sound indistinguishable from each other. Their standard phrase then is “if it sounds different, it’s broken”. 😉
Thank you! Now, we could use a similar discussion of why statements about cables like, “It has no coloration of its own and just lets the sound of the components through,” or “It recreates just what it was like in the studio,.” reflect fundamental misunderstanding of how cables and audio recordings are developed and created.
To add to my previous comment, I would say that in audio a double-blind study method is also unnecessary, since if one has a helper and they follow a method blind to the listener that’s sufficient, i.e., single blind. But if one doesn’t, or it takes some time to switch, then the effect of the physical — time and/or energy expenditure — come into play.
For audio, however, your point about the absence of a placebo is something of a straw man, since the question is difference not so much effect (measurable or scalable symptoms). Affect is a different matter and that’s really a Yes or No query — I like it or don’t, or some gradient of that. Then the question becomes which do I like more (and then other questions, such as function needed and cost).
Finally, and I think this is the “bottom line,” the person who feels they need a blinded method has two big problems on the face of it: hearing/listening ability, and honesty with themselves. For that person, the only answer is to judge on affect, which one likes better. If it’s a toss up, then go with the cheaper or most functional for one’s purpose.
To quote from the article: “ A/B is testing two variants of a test subject against each other. […] After all, the results then do not tell you whether A or B is better, but whether there is a clear preference in the listening group between A or B.“