Vorbis dev - Feb 2002 - Comparison between Ogg Vorbis and LAME

Guys (and girls),

I\'ve been testing Ogg Vorbis quality at home, and I thought I\'d share
the
results with you. For those who want the conclusion, Vorbis 1.0 RC3 sounds 
better than LAME 3.70 vbr/cbr (which I\'m told now that it\'s broken for
VBR), at
less bitrate, however, it\'s not perfect (and probably never will, since
it\'s a
lossy audio codec) and could use some help.  Further testing will yield better 
results.

Background:

Let me tell you that I\'ve been to discos and concerts and I think all that
was
bad to my ears since I can honestly tell that my hearing has diminished.
I\'m 22
however, and I can tell which one of the three TVs is on the minute I come 
home, even when they\'re muted.  I can also listen to 19 KHz sine waves, 
although at a rather loud volume that would burn the speakers on 100 Hz sine 
waves.  I have to ask people to repeat themselves in some cases, though. So I 
consider I have average ears, and in the spirit of open source and free 
software, I share these results with the hope they help.

Gear:

A self-made with off-the-shelf parts computer, with a Sound Blaster Live! OEM.  
I turned all audio input sources off since they introduce hiss, motorbike sound 
(CD input) and distortion, leaving only Wave selected.  I tried using a pair of 
$100 Technics headphones directly hooked to the headphone out of the sound 
card, but found that listening to the samples through an Aiwa NSX-V30 (el 
cheapo 30W sound system) was both easier and sounded moere natural, which made 
distinguishing sounds better.

After the test, I reconnected the rear speakers (which I didn\'t use),
powered
by a 1975 Sansui A-M7 amplifier.  I should have used the Sansui, if only 
because at its loudest volume you can hear NO hiss, while on the Aiwa you do 
(the Aiwa speakers are better than the Sansui\'s, with more frequency range,
but
they don\'t withstand as much power).  It\'s probably because of RF
interference
with the analog CD cable wired into the Aiwa\'s mainboard (it has a
mainboard
and the internal CD changer connects by a cable just like a CDROM audio 
cable!!!!).

I also have a Technics SU-Z2 1977 amp, which I\'d like to use but the
amplifier
is burnt, and when you raise the volume the left channel sounds terrible.  The 
speakers that came with that are fine, though, and sound amazingly good (and 
much louder, at 1.20 M tall with a 28cm woofer).  I might use the Sansui amp to 
power them.

Method:

For graphical analysis, I selected the entire waves in Cool Edit Pro 1.2 
(Alt+Z) and I scanned them with a window of 1024 and a Blackmann-Harris 
function.  I hope these are right.
For hearing, blind testing consisting of listening to the original sound, then 
listening to a randomized Winamp playlist of 10 samples, 5 originals and 5 
encoded, without seeing the list, and writing results down on a piece of paper.

I used a sample of the first 30 seconds of Aqua - Doctor Jones, directly ripped 
from the CD with EAC using C2 capabilities.  I encoded this sample using the 
default VBR mode in LAME 3.70 and quality levels 1, 3, 4, 5 and 10 in oggenc.

I tried to listen to the first part which consists of the sound of the forest, 
and at 5 seconds a string instrument attacks for the next 15 seconds.  It\'s
there that I notice the differences.  Later on the song, when the dance part 
comes, I can\'t tell squat.  I\'ll try and describe the differences.

Results:

Let me tell you that it\'s possible to tell that the VBR MP3 is quite
damaged in
comparison to the original, missing high frequencies and having problems 
with \"underwater\" wavy sound.

The VBR MP3 done with -V 2 suffers from the same problems, except for the 
underwater sound.  Both MP3 sound like recorded on high-quality tape (a little 
wobbly due to the tape engine, and loss of high frequencies that is 
unavoidable).

Oggenc Q1 sounds distinctly like a 96 kbps Fraunhofer MP3, with a bit of the 
choppiness produced by stretching a sound on WaveLab without altering pitch.  
Oggenc beats MP3 on space.

Oggenc Q3 I could tell 10 out of 10 on the listening test.  It has a lack of 
definition both of the sound of the forest and the string instrument.  I 
noticed that on that part, bitrate is higher than on the dance parts (probably 
due to masking of frequencies on the loud dance part, that allows the encoder 
to \"throw away\" more music as it gets more complex and louder).

Oggenc Q4 I could tell 9 out of 10 times (three times in a row).  It has a very 
tiny lack of attack on the string instrument, but no difference on the ambience 
forest sound.  It\'s not perfect though.  It beats LAME hands down, at 133
kbps
it sounds extremely better than an MP3.  I could archive at this quality and be 
happy, but I can tell the difference, and I\'m perfectionist.

Oggenc Q5 I told an average 7 out of 10 times (6/10,8/10,6/10) getting 
dangerously close to the \"toss a coin\" statistics (barring
Murphy\'s laws of
course).  Which makes Q5 a decent choice, except for the bitrate (170 kbps).  
Why does it need that much space on such a small quality increment?  If it 
weren\'t for the space, I\'d be encoding at this rate.  I expected like
145 kbps
on this setting!

For Q10, I could have fared better at guessing on a baccarat table at the 
casino.  Good job!
What makes me wonder is why at q10 Oggenc does take some high frequencies away 
(graph analysis), which at 320 Kbps LAME does not, and the graph follows the 
original strictly.

Okay.  The graph analysis show that all Oggenc and LAME streams eat high 
frequencies for dinner, but Oggenc seems to do the task with surgical precision 
and a purpose.  See, I haven\'t told you yet that the song has a 17 KHz 
continuous sound (sort of like a string ensemble) at the first 20 seconds, so I 
suppose oggenc finds out that this sound masks all other high frequencies and 
throws them away.  I wonder if this takes \"fullness\" out of sound. 
I didn\'t
feel like it did.  

Conclusions:
* I\'m using Q5 for my encoding needs, but it takes too much space.  
Nevertheless, I had been using CBR 160 KBPS on LAME before that.
* Oggenc beats lame 3.70, both in quality and in bitrate.  I have to test 3.91.

<p>One thing: although MP3Pro is marketed as the next audio solution, I
steer away
from a \"We cut high frequencies then boost them on playback with a filter
and a
stream of hints\" solution.  I can tell too, and people are getting fooled.
Correct me if I\'m wrong

One more thing: is there a lobby to exert pressure on hardware manufacturers to 
build Ogg Vorbis support into their hardware?  I\'d like to know what\'s
taking
them so long, and I\'m sure Thomson Multimedia had a lobby too.

Luck, great work and many thanks for this amazing software and the effort for 
ridding the world of software patents,

<p>   Manuel Amador (Rudd-O)
   http://www.usm.edu.ec/~amadorm/

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

Quoting Karl Heyes <karl@pts.tele2.co.uk>:
> 
> Just to let you know, the difference with the q5 is that at q5 lossless
> 
> coupling
> (aka MP3 joint stereo) is used,  retry at q 4.99 to get the best
> channel
> coupling mode.   It has already been noted that there is a jump in
> bitrate when
> using q5 because of this.
Thanks a lot! I\'ll follow up on this.
> 
> There is, by all accounts, some more tweaks to go into the lower quality
> modes
> and more support in the lower sample rate modes, although that may not
> be
> what you are looking for.
> 
> karl.
> 
> 
> 
> 
<p><p>   Manuel Amador (Rudd-O)
   http://www.usm.edu.ec/~amadorm/

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

----- Original Message -----
From: "Manuel Amador" <amadorm@zeus.usm.edu.ec>
To: <vorbis-dev@xiph.org>
Sent: Wednesday, February 13, 2002 5:42 PM
Subject: [vorbis-dev] Comparison between Ogg Vorbis and LAME

<p>> Let me tell you that I\'ve been to discos and concerts and I
think all
that was
> bad to my ears since I can honestly tell that my hearing has diminished.
I\'m 22
> however, and I can tell which one of the three TVs is on the minute I come
> home, even when they\'re muted.  I can also listen to 19 KHz sine
waves,
> although at a rather loud volume that would burn the speakers on 100 Hz
sine
> waves.  I have to ask people to repeat themselves in some cases, though.
So I
> consider I have average ears, and in the spirit of open source and free
> software, I share these results with the hope they help.
A little note: your ability to hear a 'standalone' 19kHz wave has
little bearing on your actual ability to hear those frequencies
in music. If you want to get a more interesting test, pick a normal
song, and lowpass filter it (make sure to use a hq one to avoid aliasing
artifacts). I bet you'll have great trouble detecting even a 17kHz
lowpass in a blind test.
> For graphical analysis, I selected the entire waves in Cool Edit Pro 1.2
> (Alt+Z) and I scanned them with a window of 1024 and a Blackmann-Harris
> function.  I hope these are right.
They are wrong. Any _graphical_ analysis of a _perceptual_ codec
is wrong. It's what you hear that matters, not what you see. And not
what you _think_ you hear for that matter.
> For hearing, blind testing consisting of listening to the original sound,
then
> listening to a randomized Winamp playlist of 10 samples, 5 originals and 5
> encoded, without seeing the list, and writing results down on a piece of
paper.

This is acceptable, although 10 trials doesn't really provide much
statistical certainty in the general case.
Read up a little on ABX testing - you'll find it usefull for proper blind
testing.
> I used a sample of the first 30 seconds of Aqua - Doctor Jones, directly
ripped
> from the CD with EAC using C2 capabilities.  I encoded this sample using
the
> default VBR mode in LAME 3.70 and quality levels 1, 3, 4, 5 and 10 in
oggenc.
>
> I tried to listen to the first part which consists of the sound of the
forest,
> and at 5 seconds a string instrument attacks for the next 15 seconds.
It\'s
> there that I notice the differences.  Later on the song, when the dance
part
> comes, I can\'t tell squat.  I\'ll try and describe the
differences.
Note that one sample is not saying much. You certainly shouldn't be making
any general-sounding conclusions from it.
> The VBR MP3 done with -V 2 suffers from the same problems, except for the
> underwater sound.  Both MP3 sound like recorded on high-quality tape (a
little
> wobbly due to the tape engine, and loss of high frequencies that is
> unavoidable).
MP3 VBR with LAME is tricky business. As you probably have noted by
now, LAME 3.70 is way outdated. Moreover, getting the best settings
is a tough job. To see the best MP3 is capable of right now, try
LAME 3.91 with the setting '--alt-preset standard' (and nothing
else) If you notice any imperfections with that setting, please mail
again. Many people will be quite interested :)
> Oggenc Q1 sounds distinctly like a 96 kbps Fraunhofer MP3, with a bit of
the
> choppiness produced by stretching a sound on WaveLab without altering
pitch.
> Oggenc beats MP3 on space.
Not sure what you mean here. Either you compare quality at equal bitrates,
or bitrates at equal quality. What are you doing here?
> Oggenc Q4 I could tell 9 out of 10 times (three times in a row).
In other words, you scored 27/30. There is a big statitical difference
between 9/10 and 27/30 :)
> Oggenc Q5 I told an average 7 out of 10 times (6/10,8/10,6/10) getting
> dangerously close to the \"toss a coin\" statistics (barring
Murphy\'s
laws of
> course).
20/30 I'm too lazy to do the math, but this doesn't look like a
sufficiently
significant result. (Someone correct me if...)
> I wonder if this takes \"fullness\" out of sound.  I didn\'t
> feel like it did.
If you couldn't tell the ogg from the original, it obviously didn't.

If you are interested in doing listening tests, I'd advise you to take a
look at http://ff123.net and http://hydrogenaudio.org
We regularly organize listening test, doing the best we can to make them
as accurate and objective as possible, and we can always need more trained
listeners.


--
GCP

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.