ogg dev - Jun 2009 - Fixing ogg vorbis corruption caused by bad metadata

2009/6/18 Adam Rosi-Kessel <adam at
rosi-kessel.org>:
> On Thu, Jun 18, 2009 at 11:12:30PM +0900, Conrad Parker wrote:
>> yow, that's pretty badly corrupted, the vorbis codebooks (usually
the
>> third packet) is completely gone. It looks like you'd need to throw
>> away the first 16 packets, start with fresh headers, and copy in the
>> rest of the data (packet by packet, or page by page?)
>
> Maybe the metadata writer somehow spilled over the second packet into the
> third? In any case, is this something I can do myself -- insert fresh
> headers, and then start with packetno 17? I tried doing this with the
> oggz-dump trick, and the resulting file validated but wouldn't play. I
> tried running the resulting file through the various other tools
> (oggz-sort, hogg reconstruct) with no luck.
2009/6/19 ogg.k.ogg.k at googlemail.com <ogg.k.ogg.k at
googlemail.com>:
> FWIW, yesterday I tried to dump the file, and replace the comments
> and codebooks packets with some the ones from a known good file,
> and dump -r it, but since you say dump -r doesn't preserve segmentation
> (not sure what you meant there), that might be why it did not work. I
> also don't know how much the codebooks header would change between
> files, I didn't try to work out which rate/channels/etc to match with a
> codebooks packet from a matching stream.
oggz-dump -r will at least create an Ogg file with the desired
packets, and it's good that that at least validated. By "not
preserving segmentation" I meant that it won't respect the page
boundaries of the original file -- but now I see that that doesn't
matter here, as the issue is really that the packet data is corrupted.

Hopefully at some point the vorbis data in the file becomes valid.
Perhaps we just need to know the original encoding settings to create
a new file with valid codebooks and splice them together: oggz-dump -r
should be ok for that, just take the first 3 packets of the new file,
and packets 17 onwards (or 100 onwards) of the original file ...

vi is a great audio editor ;-)

Conrad.

Conrad Parker wrote, on 6/18/2009 3:15 PM:
> Hopefully at some point the vorbis data in the file becomes valid.
> Perhaps we just need to know the original encoding settings to create
> a new file with valid codebooks and splice them together: oggz-dump -r
> should be ok for that, just take the first 3 packets of the new file,
> and packets 17 onwards (or 100 onwards) of the original file ...
Is there some magic to the codebooks? Do I need to find something ripped 
with exactly the same settings (e.g. quality level) and/or the same 
ripper? Or will any ogg vorbis file work?

I tried this with a good file with the same libvorbis string as the 
corrupted file -- "libVorbis I 20050304" -- suggesting, I think, that
I
ripped it around the same time and probably using the same tool with the 
same settings. No luck. oggz-validate 0.9.9 reports serialno "Terminal 
header page has non-zero granulepos." It also plays silently.

Even taking out the 100 packets, same result. Or swapping in the first 
100 packets from the "good" file in place of the 100 packets from the 
"bad" file (fixing serialnos of course).

What is the meaning of "terminal header page has non-zero granulepos"?
Is that actually the clue to fix it?

Adam

Adam Rosi-Kessel wrote, on 6/18/2009 3:35 PM:
> Conrad Parker wrote, on 6/18/2009 3:15 PM:
>> Hopefully at some point the vorbis data in the file becomes valid.
>> Perhaps we just need to know the original encoding settings to create
>> a new file with valid codebooks and splice them together: oggz-dump -r
>> should be ok for that, just take the first 3 packets of the new file,
>> and packets 17 onwards (or 100 onwards) of the original file ...
>
> Is there some magic to the codebooks? Do I need to find something ripped
> with exactly the same settings (e.g. quality level) and/or the same
> ripper? Or will any ogg vorbis file work?
>
> I tried this with a good file with the same libvorbis string as the
> corrupted file -- "libVorbis I 20050304" -- suggesting, I think,
that I
> ripped it around the same time and probably using the same tool with the
> same settings. No luck. oggz-validate 0.9.9 reports serialno "Terminal
> header page has non-zero granulepos." It also plays silently.
>
> Even taking out the 100 packets, same result. Or swapping in the first
> 100 packets from the "good" file in place of the 100 packets from
the
> "bad" file (fixing serialnos of course).
>
> What is the meaning of "terminal header page has non-zero
granulepos"?
> Is that actually the clue to fix it?
Actually, this seems to be a problem with the oggz-dump method:

# oggz-validate in.ogg
# oggz-dump in.ogg | oggz-dump -o out.ogg -r -
# oggz-validate out.ogg
out.ogg: Error:
serialno 1404133069: Terminal header page has non-zero granulepos

So just running it back and forth through oggz-dump generates that error.

Conrad Parker wrote, on 6/18/2009 3:15 PM:
> Hopefully at some point the vorbis data in the file becomes valid.
> Perhaps we just need to know the original encoding settings to create
> a new file with valid codebooks and splice them together: oggz-dump -r
> should be ok for that, just take the first 3 packets of the new file,
> and packets 17 onwards (or 100 onwards) of the original file ...
I've been able to replicate the corruption reliably now. It appears to 
be caused by the metadata tagger trying to embed too-large album art. I 
have a "before and after" ogg that should only be different by the 
metadata album art tagging. I tried swapping in the "before" header
into
the "after" file, but still couldn't play the fixed
"after" file.

I've put hex dumps here:

http://adam.rosi-kessel.org/bugs/liboggz/484/original.hexdump.bz2
http://adam.rosi-kessel.org/bugs/liboggz/484/corrupted.hexdump.bz2

Does this shed any light on exactly what the corruption is and/or how to 
fix it?

Adam

Adam Rosi-Kessel wrote, on 6/19/2009 3:56 PM:
> http://adam.rosi-kessel.org/bugs/liboggz/484/original.hexdump.bz2
> http://adam.rosi-kessel.org/bugs/liboggz/484/corrupted.hexdump.bz2
> Does this shed any light on exactly what the corruption is and/or how to
> fix it?
I should note that, perhaps oddly, after the first couple of packets 
with the metadata, the "original" and "corrupted" file are
identical.
But then a bit later they no longer match.

Adam

Adam Rosi-Kessel wrote, on 6/19/2009 3:56 PM:
> Conrad Parker wrote, on 6/18/2009 3:15 PM:
>> Hopefully at some point the vorbis data in the file becomes valid.
>> Perhaps we just need to know the original encoding settings to create
>> a new file with valid codebooks and splice them together: oggz-dump -r
>> should be ok for that, just take the first 3 packets of the new file,
>> and packets 17 onwards (or 100 onwards) of the original file ...
>
> I've been able to replicate the corruption reliably now. It appears to
> be caused by the metadata tagger trying to embed too-large album art. I
> have a "before and after" ogg that should only be different by
the
> metadata album art tagging. I tried swapping in the "before"
header into
> the "after" file, but still couldn't play the fixed
"after" file.
>
> I've put hex dumps here:
>
> http://adam.rosi-kessel.org/bugs/liboggz/484/original.hexdump.bz2
> http://adam.rosi-kessel.org/bugs/liboggz/484/corrupted.hexdump.bz2
>
> Does this shed any light on exactly what the corruption is and/or how to
> fix it?
How about another approach -- is there a tool that will verify the page 
checksums? oggz-validate doesn't seem to do this. If the page checksums 
are okay, then doesn't that suggest the streams are also recoverable?

Adam