Can I Have My Money Back?

Rabbi Fremer says: The Peter, Paul & Mary file comparison story has created quite an uproar. Most of you took it for what it was—an informal but reasonable comparison of two unidentified files that could have been the same or different. 79% of you thought they were different files. 21% though they were the same file posted twice.

The majority was correct. You were then asked to vote for which you preferred. There the voting was closer: 43% to 36%, with more preferring "A", which was the brighter of the two according to the comments posted before the results were revealed.

No, this wasn't a double blind A/B/X but I thought it made a pretty good case for cables sounding different at least in the critical application of tonearm cable, which passes very low voltage. The cable deniers had other ideas.

One reader said my conclusion was misleading and wrong because the poll says the results were 50-50. Were they? NO. Which people preferred was 50-50 but that the files were not identical was about 80-20!

Another reader who went by the last name "Assmilk" wrote Chances were Mikey did something to one of the clips, or else why post the article? This isn't difficult to understand. He then went on to completely change the subject and write "And the even split between those who claimed to have heard a difference shows the expensive upgrade doesn't improve the sound in an obvious way.".

Was this a test of "improved" sound? No. Does different and more expensive sometimes mean "worse"? Yes of course! Mr. Assmilk changed the subject because sonic differences among cables is outside of his belief system. And that's fine with me.

But his charge that I "did something to one of the clips" is not fine with me. Note, he didn't say I changed a source variable and then recorded the clip. He charged me with "doing something" to one of the clips, which would mean some post-production manipulation. I don't think it can be read any other way. I think his intentions are clear: he's accusing me of cheating.

I don't like that. So I removed his account from this site. Does that mean I treat readers poorly? That's the accusation of the same individual who wrote that my conclusions were "misleading and wrong."

Another reader who goes by the name "Hats Domino" agrees! I was going to answer him in the comments section but I decided to post it on the home page because his post includes another outright falsehood that I wish to refute.

So here is what "Hats Domino" wrote:

"Bravo, fjhuerta!!!

I also thought the banishment of this professional (despite his rather unprofessional username) was out of line. Reminds me of how the Steve Hoffman forum systematically bans professionals like Ken Scott, Ted Jensen, Rob LoVerde and many others for voicing opinions that don't fall within the group-think of the audiophile world. Just as with religion, people get angry when you question their beliefs with the threat of scientific facts.

If Mr. Fremer was really interested in testing to see if cables make an audible difference he should post Cable A, Cable B and a control. I'd bet the results would fall within the boundaries of guessing. He's backed out of tests like this before (my boldface, ed.). I'm sure we all know why. I certainly hope Mr. Fremer doesn't go to doctors who don't believe in scientific testing methods. Can you imagine a doctor telling you the medication hasn't gone though blind testing, but it's okay because it costs a lot so he firmly believes it'll work."

And here is my response:

You are accusing me of banning someone for "voicing an opinion"? I have NEVER done that. Ever. I banned him for impugning my integrity. For claiming I "fixed" that test. He did that because the results short-circuited a long cherished belief so the only way out was to call me a cheat.

As for your charge that I "backed out" of a cable test, that is a lie perpetrated by the "Amazing Randi". So let me clear that up for you. Randi proposed a 'cable challenge' because he thinks audiophiles are loonies and that cable manufacturers are frauds. So he said he'd pay a million dollars to anyone who could distinguish cables in a double blind ABX test that his people would supervise.

I said I'd take the challenge. I understand that these kinds of tests are really circuses but I really don't care one way or the other because I'm already called a "fraud" or a cheat by guys like Jamie (Assmilk). So what's the difference?

A company called Pear Cables was set to supply their cables, which I had never heard nor had I ever heard of the company. Interestingly Randi did not wish to compare them to lampcord but to "Monster Cable" (which of course is meaningless since Monster has a wide variety of cables, but whatever). Because even Randi was willing to admit that Monster Cable (by which I believe he meant "thicker" cable) sounded better than lamp cord.

But then Pear backed out. PEAR BACKED OUT, GOT ME "HATS"? Pear was afraid I'd fail the test. I emailed and told them they were being stupid because they were playing into Mr. Randi's hands. But they chickened out. NOT ME.

Next morning I go onto the "Amazing" Randi's site to read that I HAD BACKED OUT! That was a COMPLETE lie. How big a lie? So big that Randi started looking really bad to his acolytes. Those who were paying attention pointed out to Randi that he was engaging in very poor behavior attacking me. They saw with their own eyes just what a deceptive guy was Mr. Randi the "truth seeker" who was really a serial demonizer.

Randi was caught "red handed" by his peeps and so what did he do? He was forced to use the usual scoundrel's fallback: he claimed to have had a "medical emergency" that confused him or caused him to misread or misinterpret what had happened. He went so far as to post a picture of his E.R wristband! UGH!

Eventually we started negotiating again and as I remember it he wanted to use some ultra-expensive Transparent Cable that has the network boxes. I said that wouldn't be a good test since the box is like adding another component to the test and that we should just compare CABLES.

Ultimately it turned into a big J.O. session and talks broke down. But I didn't back out. Were I to have been compensated for the amount of time I spent on this "challenge" it would have run into many thousands of dollars.

So now that I've dispensed with that, let's look at the rest of what you've written. You would like to see double blind A/B/X testing because you are SURE of the results.

You know the conclusion. You are not really interested in the experiment. That's why when this little experiment, which doesn't meet your standards, seemed to disprove your cherished beliefs, rather than saying maybe there's something here, you chose to deny its validity and to suggest a different test. Maybe we will do just that in the future.

And were I to take and pass your test, I am certain you'd do the same thing again. You'd move the goalposts. Or if a large group took the test and only I got it right, my result would be tossed and I'd be declared a "lucky coin," which is what happened in an amplifier double blind test I took at an AES.

The test was set up specifically for me to fail because someone challenged ME to a double blind test. Someone who claimed that all amplifiers that measure the same sound the same.

Five amplfiiers were chosen (not by me) and used for a test set up by someone else: the person who challenged me to take his test.

I got 5 of 5 identifications correct. Stereophile editor John Atkinson got 4 of five correct. But among the many members of the AES who took the test the results were statistically insignificant. Did that prove the contention that all amplifiers that measure the same sound the same?

Well, actually not because for some reason the test producer chose to include a Crown DC-300 known as an "ear-bleeder" and a VTL 300 a warm, vacuum tube amplifier. These amplifiers sound and measure very differently yet the AES members, most of whom were audio engineers could not distinguish between them!

Now if the test creator's point was to show that I'm a bad listener his test backfired, since I was easily able to hear those two.

Though I passed the test I got no credit and my detractors said "well there weren't enough samples. It wasn't a valid test. You don't understand statistics, etc." Of course I didn't create the test! And had I failed it do you think those same people would have said "There weren't enough samples, it wasn't statistically valid?" Of course not! They'd have said "You failed. All amplifiers do sound the same." That's how it works.

The test also backfired because what it really proved was that double blind testing in audio can lead to STUPID results. They can lead to STUPID results because while double blind DRUG testing is useful, double blind audio testing can produce confusion, especially among less experienced listeners. Drug trials and evaluating audio gear are not remotely comparable. How about "double blind" automobile testing? After all when you see the BMW logo, rational thought ends. So what you do is, you blindfold the driver and send him out on the track and…

Or double blind wife testing. You cut holes large enough for lips to fit through in a giant sheet and line up five women behind the sheet, one of whom is your wife. You then have to walk down the line and kiss each set of lips. If you can't tell which is your wife it "proves" all women are the same and it doesn't matter who you marry.

That is how we got MP3s (and this is a simplified version of the process). The inventors used double blind listening and inexperienced listeners to remove information from CD quality audio it claimed was superflous because it was "masked" by adjacent information. After removing it, the participants "A/B"'d the CD and the lossy file and if they couldn't hear the difference that was "proof" it was inaudible. This continued until the CODEC was complete. A great deal of information was removed and the inexperienced listeners couldn't hear the difference as each bit of information was shaved off, so therefore, concluded the inventors, MP3s are transparent to the full resolution source, but require a fraction of the bits.

But are MP3s indistinguishable from CDs? NO!!!!!!!! And yes, I know this is a gross simplification of that process but my bottom line is that evaluating audio gear using double blind A/B/X testing can lead to stupid results as the five amplifier test I took (and passed) proves.

Those who discount observational audio reviewing claim there's "reviewer bias" and too many variables and that a 1dB level difference between components can produce skewed results or that champagne faceplates can make you think it sounds warm and brushed aluminum bright, etc. Experienced reviewers and listeners take all of that into account. When I review speakers in Stereophile I try to predict what the measured response will be and more often than not I get it right. Sometimes I confuse a dip in one region with a peak in the next, but that's essentially the same sonic observation.

Finally, if you look at the responses to this story from those who are certain cables don't "sound" you can see that the result of the experiment so unsettled them they either completely mis-interpreted the results claiming it was "50-50" when it really was "80-20", or they changed the subject claiming the much closer listener preference "proved" the "expensive upgrade doesn't necessarily improve the sound" or they accuse me of cheating.

Back to your medical analogy, I can point to acupuncture, which absolutely does work. I've seen it work on my dogs and they certainly don't go into the needle pushing sessions with 'pre-conceived notions" of its efficacy nor do they walk out relieved of pain because of "the placebo effect". Acupuncture has been around for thousands of years. It was dismissed out of hand by western medicine until very recently and now blind tests have shown it does work yet there are still doctors who refuse to believe even the tests. And for those who have gotten excellent results, do they care if it has been double blind tested? No. And why should they? And how many drugs that have been blind tested and released by the FDA have proven to be disasters? Many. And we're not talking about life and death here, we're talking about audio!

Differences of opinion here are welcomed. Accusations of cheating are not. So I chose to delete that individual's account. Did I treat him poorly? Or did he treat me poorly?