Vorbis dev - Jun 2003 - lossless ogg encoding with good compression rate

I think that there is a very simple way to encode audio files with lossless
compression around 35%.
(the compression may be improved if we can lose some lsb bits around 30% for 2
and 25% for 3)
Other lossless audio encoder I've seen reach at maximum a compression of 48%
(*.flac, *.ape)

So the questions are:
1- Is someone interested in another encoder?
2- If true n.1 may it be included in ogg format?
3- Is there some programmer that can help me, I don't know C a lot :(
4- It'll be not too much cpu consuming.

hopefully someone 'll answer to me.

Follow the description of how I've tested it:
1) create a wav file [A.wav]:
2) encode it in ogg format (my best result with high 7) [B.ogg]
3) decode the ogg into a wav again [C.wav] (the size will be the same as A.wav)
4) create a file with the diff of each sample from A.wav with B.wav [D.tmp] also
with the  same size at least of the header.
5) bzip2 the D.tmp [D.tmp.bz2]
6) keep B.ogg and D.tmp.bz2 (the size will be around 17% of the original for
each file)

imply is't it?

I think it's simple to recreate the a wave from this.

If it's possible to embed this in the creation process of a ogg file the
procedure can become still more simple.

Francesco Riosa

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

> -----Original Message-----
> From: owner-vorbis-dev@xiph.org 
> [mailto:owner-vorbis-dev@xiph.org] On Behalf Of General purpose mail
> Sent: Thursday, 26 June 2003 10:41 AM
> To: vorbis-dev@xiph.org
> Subject: [vorbis-dev] lossless ogg encoding with good compression rate
> 
> 
> I think that there is a very simple way to encode audio files 
> with lossless compression around 35%.
> (the compression may be improved if we can lose some lsb bits 
> around 30% for 2 and 25% for 3)
> Other lossless audio encoder I've seen reach at maximum a 
> compression of 48% (*.flac, *.ape)
> 
...
> 
> Follow the description of how I've tested it:
> 1) create a wav file [A.wav]:
> 2) encode it in ogg format (my best result with high 7) [B.ogg]
> 3) decode the ogg into a wav again [C.wav] (the size will be 
> the same as A.wav)
> 4) create a file with the diff of each sample from A.wav with 
> B.wav [D.tmp] also with the  same size at least of the header.
I think you mean C.wav not B.wav
> 5) bzip2 the D.tmp [D.tmp.bz2]
> 6) keep B.ogg and D.tmp.bz2 (the size will be around 17% of 
> the original for each file)
> 
Very interesting way to do it, but if you're only getting around %35 and
FLAC is getting around %48 then why reinvent the wheel.

Perhaps I am misunderstanding your numbers. Are you talking a percentage
of original size OR percentage of original size saved. If it's the
former read on. If it's the latter ignore the rest of this message.

Have you encoded with FLAC on the source you are using? I have found the
FLAC compression estimates seem a little optimistic for a lot of my
stuff. I get more like %66 of the original size (in your terms %33
compression?)

Either way, there's probably more CPU required than simply decoding an
OGG since you have to do decode the OGG, bunzip the diff AND apply the
diff...

My 2c....

CH

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On 2003-06-26 at 02:40:30 General purpose mail wrote:
> 1) create a wav file [A.wav]:
> 2) encode it in ogg format (my best result with high 7) [B.ogg]
> 3) decode the ogg into a wav again [C.wav] (the size will be the same as
A.wav)
> 4) create a file with the diff of each sample from A.wav
> with B.wav [D.tmp] also with the  same size at least of the
> header.
> 5) bzip2 the D.tmp [D.tmp.bz2]
> 6) keep B.ogg and D.tmp.bz2 (the size will be around 17% of the original
for each file)
> simply is't it?
I'm afraid this will not work, except in some special cases. It's
probably best to go read some books on data compression and
information theory. ;)

For example:

1) I start with test.wav here (5:59.200/63,362,924 bytes). It's just a
music track, nothing weird.
2) Encoding to ogg with -q7 gives me test.ogg, 10,472,702 bytes.
3) Decoding this ogg gives me test2.wav, 63,362,924 bytes again.
4) I make a diff with Cool Edit, by mix pasting an inverted copy of
test2.wav into test.wav. This gives diff.wav, also 63,362,924 bytes.
5) However, compressing diff.wav with bzip2 -9, results in
diff.wav.bz2, which is 39,858,199 bytes, so only ~62.9% compression.
6) The end size is thus: 10,472,702 bytes ogg + 39,858,199 bytes bz2,
totaling 50,330,901 bytes, so ~79.4% compression.
7) This is much worse than FLAC, for example, which manages to
compress the original test.wav to 37,728,889 bytes, so ~59.5%
compression. Monkey's Audio even manages to get it to 34,971,640
bytes, so ~55.2% compression.

The problem is that you cannot guarantee that step 5) will compress
the diff so much. If it were so simple, I guess FLAC/APE/etc would
already be using this for a long time. :)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/pgp-signature
Size: 184 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/vorbis-dev/attachments/20030626/8f144869/part-0001.pgp

Just to add my 2 cents, surely the difference will always be quite small 
, thus you don't need the full 16-bits for each one in the PCM file. 
i.e. say each difference could be 8-bits, thus halfing the difference 
file size. Although just a passing though and might not work in practice.

Charles

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Wed, 2003-06-25 at 20:40, General purpose mail wrote:
> 
> Follow the description of how I've tested it:
> 1) create a wav file [A.wav]:
> 2) encode it in ogg format (my best result with high 7) [B.ogg]
> 3) decode the ogg into a wav again [C.wav] (the size will be the same as
A.wav)
> 4) create a file with the diff of each sample from A.wav with B.wav [D.tmp]
also with the  same size at least of the header.
> 5) bzip2 the D.tmp [D.tmp.bz2]
> 6) keep B.ogg and D.tmp.bz2 (the size will be around 17% of the original
for each file)
> 
> simply is't it?
> 
> I think it's simple to recreate the a wave from this.
Besides the difficulty in reproducing your astonishing results that
others have mentioned on this thread, there is another problem with your
approach. Different decoders (e.g. on different platforms) are not
guaranteed to produce the same wav from the same Ogg Vorbis file. In
order for your approach to be lossless, you need the exact same Vorbis
decoder on the decode side as the one that was used for calculating the
difference signal on the encode side. Unfortunately, this requirement
places your idea anywhere between impractical and impossible.

Best regards,

Carsten Haese.

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

The way I see it, the combination of ogg and diff file can never be encoded
as easily (or as compact) as the original wave file itself.

The reason is that the ogg encoding is in part based on a quantization process.
Quantization always creates a quantization error which by nature is noise.
This noise is present in the decoded ogg file - though inaudible, because the
noise is nicely hidden below the ear's masking threshold by the
psychoacoustic
model.

When you form the diff between the decoded ogg and the original wave file,
this quantization noise will be imprimted on the diff file also. So what you
get is a diff with increased irrelevance (noise). Information theory (Claude
Shannon) teaches us that irrelevance is really hard to encode - basicall not
compressible.

Christian

<p>--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.