ogg dev - Jun 2009 - Fixing ogg vorbis corruption caused by bad metadata

I have over 1,000 ogg vorbis files that are no longer playable or 
editable. I believe the main cause was a MediaMonkey plugin that 
automatically fetched album art and added it to the header. The plugin 
itself just uses the generic MediaMonkey metadata facility, but 
something happened -- perhaps when the album art bitmap image was too 
large -- that caused the files to no longer work. Unfortunately, I don't 
have backups.

I have noticed a handful of unresolved threads about ogg corruption on 
various message boards, so I suspect I am not alone in having, for one 
reason or another, unplayable oggs.

I have tried rewriting the files with oggz-sort (both last release and 
latest from git) and also with hogg reconstruct (including latest 
version built from source). I have also tried dumping the files with 
oggz-dump and swapping in either packetno 1 or 1,2 or 1,2,3 from a known 
good ogg file, replacing the serialnos, and then recreating the ogg with 
oggz-dump -r. None of these changes work. vlc will "play" the file,
but
all I hear is silence.

In a few instances, I have had ogg files in this state where they played 
but I couldn't edit the metadata with any tool. hogg reconstruct fixed 
those files so they seem to be fully working again, but most of my 
corrupted files don't fall into this category.

oggz-validate reports on an example file:

serialno 1225743615: Terminal header page contains non-header packet

after attempting to splice in good headers, on the same file:

serialno 1225743615: Terminal header page has non-zero granulepos

I have posted a couple of sample files, one before and one after running 
through oggz-sort, here:

http://adam.rosi-kessel.org/bugs/liboggz/484/

I'm not sure if all of my files have the same problem, so I can share 
others if it would be helpful.

I'd appreciate any assistance/suggestions in fixing these files. I don't
see any reason why the corruption should extend beyond just the metadata 
part of the file, but I could of course be wrong.

Hi,

(I was discussing this with Adam earlier today, and he has also mailed
vorbis at xiph.org about this issue in March).

This sounds like it needs a new tool specifically for fixing broken
Ogg files. Suggestions so far (tried by Adam, or suggested to him in
email/IRC):

  * oggz-sort, hogg sort

These rewrite the file, sorting the pages by presentation time. This
usually fixes muxing problems, but will not change the actual page
contents, so are of little use here where header data is corrupted.

  * hogg reconstruct

This reconstructs a file "by doing a full packet demux". Here, hogg
actually keeps track of page boundaries, ie. what segments each packet
consists of. The reconstruction then involves writing new pages with
the same segmentation. This will fix some paging issues (sequence
numbering, granulepos) and re-calculate checksums; this may have fixed
some files where some bad metadata pages were randomly inserted at the
start.

  * oggz-dump, [edit], oggz-dump -r

This is a fun hack, but doesn't preserve any segmentation info so is
unlikely to help here.

  * oggz-chop -s 0

(a clever suggestion I saw from ogg.k.ogg.k ;-) This will usually
rewrite the headers and flush them correctly, so it can fix some
header segmentation problems, but like oggz-sort it doesn't modify the
contents.

So basically it looks like these files are corrupted in a new and
interesting way! The current tools won't work, and I'm unlikely to
have time to write a fixer for these in the short term. This sounds
like the perfect challenge for someone who wants to dive deeper into
Ogg hacking (or a good tool for the recent C++ libraries :-)

If you want to try with liboggz, a good place to start would be the
files in src/examples, such as fix-eos.c or rewrite-pages.c. I could
also help anyone who wants to hack on hogg, where it's probably just a
matter of writing a short helper along the lines of:

sort :: [OggPage] -> [OggPage]
sort = sortHeaders . listMerge . demux

Anyway, an interesting problem :-)

Conrad.

2009/6/18 Adam Rosi-Kessel <adam at
rosi-kessel.org>:
> I have over 1,000 ogg vorbis files that are no longer playable or
> editable. I believe the main cause was a MediaMonkey plugin that
> automatically fetched album art and added it to the header. The plugin
> itself just uses the generic MediaMonkey metadata facility, but
> something happened -- perhaps when the album art bitmap image was too
> large -- that caused the files to no longer work. Unfortunately, I
don't
> have backups.
>
> I have noticed a handful of unresolved threads about ogg corruption on
> various message boards, so I suspect I am not alone in having, for one
> reason or another, unplayable oggs.
>
> I have tried rewriting the files with oggz-sort (both last release and
> latest from git) and also with hogg reconstruct (including latest
> version built from source). I have also tried dumping the files with
> oggz-dump and swapping in either packetno 1 or 1,2 or 1,2,3 from a known
> good ogg file, replacing the serialnos, and then recreating the ogg with
> oggz-dump -r. None of these changes work. vlc will "play" the
file, but
> all I hear is silence.
>
> In a few instances, I have had ogg files in this state where they played
> but I couldn't edit the metadata with any tool. hogg reconstruct fixed
> those files so they seem to be fully working again, but most of my
> corrupted files don't fall into this category.
>
> oggz-validate reports on an example file:
>
> serialno 1225743615: Terminal header page contains non-header packet
>
> after attempting to splice in good headers, on the same file:
>
> serialno 1225743615: Terminal header page has non-zero granulepos
>
> I have posted a couple of sample files, one before and one after running
> through oggz-sort, here:
>
> http://adam.rosi-kessel.org/bugs/liboggz/484/
>
> I'm not sure if all of my files have the same problem, so I can share
> others if it would be helpful.
>
> I'd appreciate any assistance/suggestions in fixing these files. I
don't
> see any reason why the corruption should extend beyond just the metadata
> part of the file, but I could of course be wrong.
> _______________________________________________
> ogg-dev mailing list
> ogg-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/ogg-dev
>
>

On 6/18/09, Conrad Parker <conrad at metadecks.org>
wrote:
> This sounds like it needs a new tool specifically for fixing broken
> Ogg files.
I see two solutions:

1) the new vcut which I reckon will fix the problem since it has to
split and rebuild the audio in a new Ogg

2) an Hex editor, even if just to check what's wrong in the metadata section

-Ivo