MP3 and the Perils of Testing

Walt Crawford, 1999

One or two of you might have seen an essay hereabouts on MP3, posted June 6, 1999 and taken down June 7 or 8. It's been restored (on June 12, 1999), and you can see it here.

The original essay had a much harsher commentary based on my preliminary tests of two Simon & Garfunkel songs (ripped at 160Kbps) and two Gordon Lightfoot songs (ripped at 128Kbps). Here are some relevant paragraphs from that commentary, quoted with the notation that these are wrong.

Surprise, surprise: the MP3 versions did not sound the same. For one thing, it proved to be impossible to match volume levels precisely. There appear to be two reasons for that. First, the balance of low, medium, and high-frequency sound is different: thus, when some instruments are the same volume, others (and voices) will be louder. Second, the MP3 version messed up the stereo image, causing shifts in placement that also caused problems in volume matching. (I think MP3 may also be compressing the dynamic range, but that’s harder to tell with pop music: as noted, this first test was designed to be easy for MP3.)

I felt that the innards of the music were missing. Art Garfunkel’s voice sounded more like a generic tenor than like Garfunkel; the ethnic string instruments on El Condor Pasa sounded a little like banjos. The high frequencies that I can hear were there, all right, but they seemed exaggerated.

The biggest problem was vocal placement: it wasn’t there. On most good vocal recordings, the voice sits squarely centered between the two loudspeakers. In surround mode, the Altec Lansing’s have a "center" volume control; when turned all the way down, it should give a near-karaoke effect, with the vocals almost disappearing. That’s how it worked on CD playback; on MP3 playback, dropping the center volume all the way had no effect on the vocal volume—which, in regular stereo or surround, was coming from some cloudy area encompassing the entire soundstage rather than from the center.

If this sounds like crazed audiophile talk, it isn’t. In stereo mode, I could tell that something was wrong. The surround mode made it possible to nail the problem: somehow, the stereo image had been damaged quite badly. That was true on both Simon & Garfunkel cuts. When I tried two Gordon Lightfoot songs the next day (recording at 128Kbps this time), the same was true: the singer wasn’t centered and had lost much of his vocal identity. The recordings were hollow—much worse than the fairly mediocre originals.

When I started out to write this up, I was going to say that the good news was that the music was still enjoyable. It was still good enough that I could see using it as background while writing—and certainly good enough to use while jogging or on an airplane. The latter may be true. But I tried using the four recorded MP3 tracks as background music while I was doing computer work, and found the hollow vocals and confused stereo distracting and, in the end, unpleasant. So for me, at least, for these tracks and under these conditions, MP3 is a non-starter: I’d just as soon play the CDs.

Shortly after writing that, I received the free CD from mp3.com, "103 of the best songs you've never heard." It pretty consistently had the same problem, with the stereo image messed up fairly badly.

Well, OK: maybe I was too optimistic. Still, I found it hard to believe that even sloppy and half-deaf journalists could consider this to be acceptable sound. Was it possible that something else was going on?

Something Else: Creative Lab's "Spatial" Control

I was reasonably certain that it wasn't just that my PC's sound chip was incompetent. If that was the case, it was (and is) a problem for most home PCs: after all, my PC is a contemporary Gateway, using the same Creative Labs SoundBlaster PCI 64 sound support that almost all Gateways do. I believe that's really the Ensoniq chipset, and while it's not the best on the market, it's also not a $10 noname special. I was also certain that the audio CD output was going through the SoundBlaster circuitry, or at least part of it: to wit, I can control CD sound using the SoundBlaster 64 mixer. That wouldn't be possible of the CD output was bypassing the sound circuitry. (I would say "sound card," but on my PC it's on the motherboard. Details, details.)

But then, as I considered the mixer, I noticed something else.

Under "Wave," there's a control called "Spatial." It was lit--and the setting was "On." (I had adjusted all these settings for the best overall sound--particularly from Synth [MIDI]--when I first got the computer.) Surely that couldn't be causing a problem--or could it?

I switched it to "Off" and played back the MP3 cuts again. Since this setting only affects the playback of wavetable sound, it should not have affected the "ripping" (which doesn't involve the soundcard). But it would affect the "103 best songs..." CD, because that's a CD-ROM: MusicMatch is playing back the cuts as wavetable sound. They're not coming through as CD output.

Well well well...

I substituted a placeholder article as soon as I heard the difference. The MP3 cuts still weren't perfect, but they did have centered vocals and reasonably stable stereo images. The "cloud of sound" was replaced by stereo, and I found that the MP3 versions of favorite CD songs worked just fine as background music.

This was a "D'oh" experience, and it's tempting to pretend it didn't happen. I've revised the commentary and reposted it June 12, 1999. I'll be doing more testing, all with the Wave Spatial option turned off. The egg on my face is like that tree falling in the forest: based on site-visit counts, I don't think more than one or two people saw the erroneous commentary. But it's not quite that simple.

Tied to the Tracks: The Perils of PC Testing

This wasn't really an isolated incident. That same week, I read two reviews of new "Office" suites in the local paper, both done by the same reviewer. He found more bugs and glitches in MS Office 2000 than most other reviewers had found, and even commented that it could be his machines--but he didn't think so. Except that he also reviewed Corel WordPerfect Office Suite 2000, and found bugs of such magnitude that I can't believe Corel would have released the product (which has been out for some time).

[Note: I've been using CorelDraw and Corel Ventura for some time, and I know that Corel sometimes releases products that should be beta versions--but these were bugs at a fundamental alpha-release level.]

As a reader, I immediately questioned his findings. If he was having that much trouble with both products, is it possible that he'd mangled his Registry or other Windows setup rather badly? Or had he, gasp, left an odd little setting that shouldn't be there...sort of like leaving on an audio setting that might disrupt the stereo image to widen it?

Nah. Couldn't happen. But it does. I get nervous when I test CD-ROMs, because I'm never entirely sure whether failures are from the product or from my PCs--and I try to indicate that in my review comments. Then again, I've seen reviews by better-known reviewers that assail a product for a defect that's clearly a problem with their machine, not the CD.

So it goes. The only real answer is to start out with a completely clean new PC for each software test, but that's ridiculous as well as implausible.

In the real world, you do the best you can. Sometimes, that means that you find fault with a product when the fault is rather within yourself. Fortunately, sometimes you spot the problem and can correct it.

Meanwhile, of course, MP3 at 128Kbps still doesn't offer what I would call CD-quality sound. But that's another story, told in the revised ongoing essay.

Posted June12, 1999; layout modified July 18, 1999