MP3 and the Perils of Testing
Walt Crawford, 1999
One or two of you might have seen an essay hereabouts on MP3, posted June 6,
1999 and taken down June 7 or 8. It's been restored (on June 12, 1999), and you can see it here.
The original essay had a much harsher commentary based on my preliminary tests
of two Simon & Garfunkel songs (ripped at 160Kbps) and two Gordon Lightfoot songs (ripped at 128Kbps). Here
are some relevant paragraphs from that commentary, quoted with the notation that these
Surprise, surprise: the MP3 versions did not sound the same. For one thing, it proved to be impossible to match volume levels precisely.
There appear to be two reasons for that. First, the balance of low, medium, and high-frequency sound is different:
thus, when some instruments are the same volume, others (and voices) will be louder. Second, the MP3 version messed
up the stereo image, causing shifts in placement that also caused problems in volume matching. (I think MP3 may
also be compressing the dynamic range, but that’s harder to tell with pop music: as noted, this first test was
designed to be easy for MP3.)
I felt that the innards of the music were missing. Art Garfunkel’s voice sounded
more like a generic tenor than like Garfunkel; the ethnic string instruments on El
Condor Pasa sounded a little like banjos. The high frequencies that I
can hear were there, all right, but they seemed exaggerated.
The biggest problem was vocal placement: it wasn’t there. On most good vocal
recordings, the voice sits squarely centered between the two loudspeakers. In surround mode, the Altec Lansing’s
have a "center" volume control; when turned all the way down, it should give a near-karaoke effect, with
the vocals almost disappearing. That’s how it worked on CD playback; on MP3 playback, dropping the center volume
all the way had no effect on
the vocal volume—which, in regular stereo or surround, was coming from some cloudy area encompassing the entire
soundstage rather than from the center.
If this sounds like crazed audiophile talk, it isn’t. In stereo mode, I could
tell that something was wrong. The surround mode made it possible to nail the problem: somehow, the stereo image
had been damaged quite badly. That was true on both Simon & Garfunkel cuts. When I tried two Gordon Lightfoot
songs the next day (recording at 128Kbps this time), the same was true: the singer wasn’t centered and had lost
much of his vocal identity. The recordings were hollow—much worse than the fairly mediocre originals.
When I started out to write this up, I was going to say that the good news was
that the music was still enjoyable. It was still good enough that I could see using it as background while writing—and
certainly good enough to use while jogging or on an airplane. The latter may be true. But I tried using the four
recorded MP3 tracks as background music while I was doing computer work, and found the hollow vocals and confused
stereo distracting and, in the end, unpleasant. So for me, at least, for
these tracks and under these conditions, MP3 is a non-starter: I’d just
as soon play the CDs.
Shortly after writing that, I received the free CD from mp3.com, "103 of
the best songs you've never heard." It pretty consistently had the same problem, with the stereo image messed
up fairly badly.
Well, OK: maybe I was too optimistic. Still, I found it hard to believe that
even sloppy and half-deaf journalists could consider this to be acceptable sound. Was it possible that something
else was going on?
Something Else: Creative Lab's "Spatial" Control
I was reasonably certain that it wasn't just that
my PC's sound chip was incompetent. If that was the case, it was (and is) a problem for most home PCs: after all,
my PC is a contemporary Gateway, using the same Creative Labs SoundBlaster
PCI 64 sound support that almost all Gateways do. I believe that's really the Ensoniq chipset, and while it's not
the best on the market, it's also not a $10 noname special. I was also certain that the audio CD output was going
through the SoundBlaster circuitry, or at least part of it: to wit, I can control CD sound using the SoundBlaster
64 mixer. That wouldn't be possible of the CD output was bypassing the sound circuitry. (I would say "sound
card," but on my PC it's on the motherboard. Details, details.)
But then, as I considered the mixer, I noticed something else.
Under "Wave," there's a control called "Spatial." It was
lit--and the setting was "On." (I had adjusted all these settings for the best overall sound--particularly
from Synth [MIDI]--when I first got the computer.) Surely that couldn't be causing a problem--or could it?
I switched it to "Off" and played back the MP3 cuts again. Since this
setting only affects the playback of wavetable sound, it should not have affected the "ripping" (which
doesn't involve the soundcard). But it would affect the "103 best songs..." CD, because that's a CD-ROM: MusicMatch is playing back
the cuts as wavetable sound. They're not coming through as CD output.
Well well well...
I substituted a placeholder article as soon as I heard the difference. The MP3
cuts still weren't perfect, but they did have centered vocals and reasonably stable stereo images. The "cloud
of sound" was replaced by stereo, and I found that the MP3 versions of favorite CD songs worked just fine
as background music.
This was a "D'oh" experience, and it's tempting to pretend it didn't
happen. I've revised the commentary and reposted it June 12, 1999. I'll be doing more testing, all with the Wave
Spatial option turned off. The egg on my face is like that tree falling in the forest: based on site-visit counts,
I don't think more than one or two people saw the erroneous commentary. But it's not quite that simple.
Tied to the Tracks: The Perils of PC Testing
This wasn't really an isolated incident. That same week, I read two reviews of
new "Office" suites in the local paper, both done by the same reviewer. He found more bugs and glitches
in MS Office 2000 than most other reviewers had found, and even commented that it could be his machines--but he
didn't think so. Except that he also reviewed Corel WordPerfect Office Suite 2000, and found bugs of such magnitude
that I can't believe Corel would have released the product (which has been out for some time).
[Note: I've been using CorelDraw and Corel Ventura for some time, and I know that Corel sometimes releases products
that should be beta versions--but these were bugs at a fundamental alpha-release level.]
As a reader, I immediately questioned his findings. If he was having that much
trouble with both products, is it possible that he'd mangled his Registry or other Windows setup rather badly?
Or had he, gasp, left an odd little setting that shouldn't be there...sort of like leaving on an audio setting
that might disrupt the stereo image to widen it?
Nah. Couldn't happen. But it does. I get nervous when I test CD-ROMs, because
I'm never entirely sure whether failures are from the product or from my PCs--and I try to indicate that in my
review comments. Then again, I've seen reviews by better-known reviewers that assail a product for a defect that's
clearly a problem with their
machine, not the CD.
So it goes. The only real answer is to start out with a completely clean new
PC for each software test, but that's ridiculous as well as implausible.
In the real world, you do the best you can. Sometimes, that means that you find
fault with a product when the fault is rather within yourself. Fortunately, sometimes you spot the problem and
can correct it.
Meanwhile, of course, MP3 at 128Kbps still
doesn't offer what I would call CD-quality sound. But that's another
story, told in the revised ongoing essay.
Posted June12, 1999; layout modified July 18, 1999