A practical example: System Audio
Contents
In one of the interesting podcasts, Ole Witthøft talks about a study they performed together with a Danish research institute for testing two different designs of a new speaker.
Here I revert to the subject of placebo and the conditions for double-blind. In this case, System Audio was able to build speakers that looked the same on the outside but were different on the inside. The listeners could not tell which speaker they were listening to.
But from a puristic viewpoint, they are not testing with a placebo, but they are doing an A/B test. A/B is testing two variants of a test subject against each other. From an audio manufacturer’s point of view, this is a sensible study. After all, the results then do not tell you whether A or B is better, but whether there is a clear preference in the listening group between A or B. If you sell speakers, that is valuable information.
Also interesting: Ole says they learned a lot from the research, but also that it costs a lot of money and is not doable for everything they make.
… even worse:
If you manage to successfully differentiate between let’s say some cables or amplifiers in your controlled environment, this may not at all reflect how these samples behave in a different environment. Talking different floors (cable risers?), different qualities of AC from the wall, and so on.
Maybe two power cables with a specific amp sound the very same in one lab, but manage to sound different from each other in another environment.
In the end to me it seems that the “double-blind-A/B/X police” constantly fails in explaining why two different cables/amplifiers/whatever should sound indistinguishable from each other. Their standard phrase then is “if it sounds different, it’s broken”. 😉
Thank you! Now, we could use a similar discussion of why statements about cables like, “It has no coloration of its own and just lets the sound of the components through,” or “It recreates just what it was like in the studio,.” reflect fundamental misunderstanding of how cables and audio recordings are developed and created.
To add to my previous comment, I would say that in audio a double-blind study method is also unnecessary, since if one has a helper and they follow a method blind to the listener that’s sufficient, i.e., single blind. But if one doesn’t, or it takes some time to switch, then the effect of the physical — time and/or energy expenditure — come into play.
For audio, however, your point about the absence of a placebo is something of a straw man, since the question is difference not so much effect (measurable or scalable symptoms). Affect is a different matter and that’s really a Yes or No query — I like it or don’t, or some gradient of that. Then the question becomes which do I like more (and then other questions, such as function needed and cost).
Finally, and I think this is the “bottom line,” the person who feels they need a blinded method has two big problems on the face of it: hearing/listening ability, and honesty with themselves. For that person, the only answer is to judge on affect, which one likes better. If it’s a toss up, then go with the cheaper or most functional for one’s purpose.
To quote from the article: “ A/B is testing two variants of a test subject against each other. […] After all, the results then do not tell you whether A or B is better, but whether there is a clear preference in the listening group between A or B.“