ogg dev - Nov 2008 - [Schrodinger-devel] ogg dirac granulepos in oggz tools

2008/11/14 David Flynn <davidf+nntp at woaf.net>:
> On 2008-11-13, Conrad Parker <conrad at metadecks.org> wrote:
>> I'm wondering if the Dirac granulepos parsing in liboggz and
display
>> in the oggz tools is currently correct, as I'd like to do a release
of
>> these soon.
>
> I believe it is -- although if correct support in the rest of the
> liboggz tools is required, a little more work may need to happen.
Ok, let's at least make sure the tools for inspecting streams are
working first, ie. oggz dump, info, and validate.

oggz-dump seems to be correct now (with your last dt patch).
oggz-info works.
I've fixed up oggz-validate with your suggestions below.

As for the other tools:

oggz-rip works ok.

Should oggz-comment be modified to disallow modification of Dirac
streams, ie. does Ogg Dirac never contain VorbisComment metadata?

It seems oggz chop, merge and sort will need some attention to deal
with the Dirac granulepos and dependency ordering, so let's leave them
for the next release.
>> Here's some oggz output that seems a bit fishy. Using
>> http://diracvideo.org/download/test-streams/ogg/sage-640x360.ogg and
>> current liboggz svn, http://svn.annodex.net/liboggz/trunk/ :
>
> The only niggle i have with this file is that the EOS page has a bizare
> timestamp.  imho, because there is no picture in the eos page, there is
> no meaningful time for it, so the granulepos should be -1 on the eos
> page on the dirac logical stream.
ok, as it seems the EOS granulepos is under discussion I've not made any
checks
specific to that.

As of changeset:3782, oggz-validate reports only this one error for that file:

conrad at chichai:~/share/dirac$ oggz validate sage-640x360.ogg
sage-640x360.ogg: Error:
00:02:36.077: serialno 1763535876: Granulepos decreasing within track

>> conrad at chichai:~/share/dirac$ oggz validate sage-640x360.ogg
>> sage-640x360.ogg: Error:
>> 00:00:00.000: serialno 1763535876: Granulepos decreasing within track
>> 00:00:00.166: serialno 1763535876: Granulepos decreasing within track
>
> These two are understood -- infact, the fact it has errored twice
> possibly points to an error in oggz-validate treating granulepos as
> signed.  The value of GP64 is increasing between the two packets, it
> then overflows past zero for the third packet.
ok, I modified that to allow negative granulepos immediately after
headers, in changeset:3781.

As for previously showing that error twice: it was effectively
remembering the higher value that it had seen before, not just the
immediately previous value.
>> If it is ok for packets in an Ogg Dirac bitstream to be decreasing,
>> which oggz-validate reports as out of order -- I assume/recall that
>> this is if they are sequenced in dependency order, not display order
>> -- then how should oggz-validate be modified to handle this properly?
>
> It should check that DT (GP64) doesn't decrease ("Granulepos
decreasing
> within track")
>
> It should not check for 'out of order' packets based on GPH+L for a
> dirac stream, since by definition they are.
Ok, so the GP64 check remains.

I modified the packet ordering check to not take the timestamp of
Dirac packets into account (changeset:3782).

So effectively, the GP64 check ensures that the ordering within the
Dirac track is ok, but the timestamps of the Dirac packets do not
affect the check for multiplexing order (ie. don't affect the checking
of Vorbis packets).

Conrad.
checking packet ordering.

On 2008-11-14, Conrad Parker <conrad at metadecks.org>
wrote:
> Should oggz-comment be modified to disallow modification of Dirac
> streams, ie. does Ogg Dirac never contain VorbisComment metadata?
Correct; there is no metadata handling capability in the current
mapping spec.
> It seems oggz chop, merge and sort will need some attention to deal
> with the Dirac granulepos and dependency ordering, so let's leave them
> for the next release.
ok. -- may be worth having them 'warn' if they are operating on
something containing oggdirac.
>
> As of changeset:3782, oggz-validate reports only this one error for that
file:
>
> conrad at chichai:~/share/dirac$ oggz validate sage-640x360.ogg
> sage-640x360.ogg: Error:
> 00:02:36.077: serialno 1763535876: Granulepos decreasing within track
Ok, that is due to the invalid granulepos in the EOS page (the actual
value doesn't make sense -- and is infact invalid (see in a moment))

The mapping spec sets two bits (8 & 30) to be zero.  It may be worth
checking for them (if the GP64 isn't -1).  The GP64 of the EOS page
in the sage- stream doesn't pass this test.

Infact, i was wondering why that granulepos looks so wrong:
00:02:36.077: -362377 -> (pt:7484,dt:0,dist:65399,delay:7484)

the way i calculated dt in oggz is actually redundant.  dt is defined
to be: dt=pt-delay; and is contained in the top 32bits of GP64, so
if GP64 is very -ve, why is dt=0?

Answer: because pt="GPH+L">>9 relies on the two zero bits being
set
so that any overflow in adding dist_l and dist_h together don't affect
pt.

To summarise, the value of dt calculated in oggz_tools should be
identical to the top 32 bits of GP64.  If that fails, The two
protection bits havn't been cleared correctly.
> I modified the packet ordering check to not take the timestamp of
> Dirac packets into account (changeset:3782).
Ok, thanks.

On Sat, Nov 15, 2008 at 07:16:58AM +0900, Conrad Parker
wrote:
> Should oggz-comment be modified to disallow modification of Dirac
> streams, ie. does Ogg Dirac never contain VorbisComment metadata?
Currently, no.  Depending on the requirements, it could be encapsulated
in a Dirac Auxiliary data unit.  I'm not a big fan of doing this,
since Ogg/Dirac is almost always going to be paired with Vorbis,
Speex, or Flac.



dave...

On Fri, Nov 14, 2008 at 11:02:29PM +0000, David Flynn
wrote:
> > As of changeset:3782, oggz-validate reports only this one error for
that file:
> >
> > conrad at chichai:~/share/dirac$ oggz validate sage-640x360.ogg
> > sage-640x360.ogg: Error:
> > 00:02:36.077: serialno 1763535876: Granulepos decreasing within track
> 
> Ok, that is due to the invalid granulepos in the EOS page (the actual
> value doesn't make sense -- and is infact invalid (see in a moment))
Um, sorry, that's me smoking crack.  Here's the oggz-dump for a more
recent attempt:

00:00:03.629: serialno 0655859631, granulepos 86016|3099, packetno 86:
16.162 kB
00:00:03.546: serialno 0655859631, granulepos 87040|28, packetno 87: 739
bytes
00:00:03.588: serialno 0655859631, granulepos 88064|29, packetno 88: 784
bytes
00:00:03.713: serialno 0655859631, granulepos 89088|2078, packetno 89:
15.023 kB
00:00:03.671: serialno 0655859631, granulepos 90112|31, packetno 90: 713
bytes
00:00:03.671: serialno 0655859631, granulepos 90112|31, packetno 91 ***
eos: 13 bytes



dave...

On 11/14/08, David Flynn <davidf+nntp at woaf.net>
wrote:
> Correct; there is no metadata handling capability in the current
> mapping spec.
I'd suggest at some point to look into separate stream solution for
metadata, perhaps M3F [1].

-Ivo

[1] http://wiki.xiph.org/index.php/M3F

2008/11/15 David Flynn <davidf+nntp at woaf.net>:
> On 2008-11-14, Conrad Parker <conrad at metadecks.org> wrote:
>> Should oggz-comment be modified to disallow modification of Dirac
>> streams, ie. does Ogg Dirac never contain VorbisComment metadata?
>
> Correct; there is no metadata handling capability in the current
> mapping spec.
ok, as of changeset:3798, oggz-comment refuses to handle comments for
Dirac tracks. It Warns if a Dirac track is present but the user did not
explictly specify to operate on it (ie. if comments from all tracks are
being listed or edited), and returns an error if the user specified a
Dirac track, by serialno or content type.

Conrad.

2008/11/15 David Flynn <davidf+nntp at woaf.net>:
> On 2008-11-14, Conrad Parker <conrad at metadecks.org> wrote:
>> It seems oggz chop, merge and sort will need some attention to deal
>> with the Dirac granulepos and dependency ordering, so let's leave
them
>> for the next release.
>
> ok. -- may be worth having them 'warn' if they are operating on
> something containing oggdirac.
I started patching oggz sort and merge, and discovered they were very nearly
working. After adding a custom function to convert granulepos to time for
Dirac, rather than just going by the result of the granuleshift, it seems to
be working. You can review the changeset here:

http://trac.annodex.net/changeset/3801

please test :-) For example you can start by ripping an existing Dirac+Vorbis
stream apart:

$ oggz rip -c vorbis input.ogv -o vorbis.ogg
$ oggz rip -c dirac input.ogv -o dirac.ogv

Then merge them back together:

$ oggz merge dirac.ogv vorbis.ogg -o output.ogv

I think the output is properly merged, please test in case there are any
problems I have missed.

Tested with liboggz r3806.

Conrad.