Vorbis dev - Aug 2001 - Ogg MIDI proposal

On Sat, Aug 25, 2001 at 06:34:30PM -0600, Jack Moffitt
wrote:
> This is a proposal for Ogg MIDI streams.  It will also help other 
> stream additions to Vorbis, as they should follow a similar model.
> Ralph has done much of the work on this, and also is working on MNG
> Ogg streams.
[...]
> There are three foramts of standard MIDI files, Format 0, 1, and 2.
> Format 2 is not a song, but multiple idependent patterns, so is
> irrelevant to this discussion.  Format 0 files are one track, and 
> Foramt 1 files are multiple tracks (note that these are logical 
> divisions for sequencers, and all the data of a format 1 file could be
>  represented in one track).  Since tracks are stored serially, which
> breaks streaming, Ogg MIDI will use only format 0 MIDI files, which means
> any Format 1 files will have to be collapsed into Format 0.
How about allowing multiple Format 0 streams (thus representing Format
1 and even (partially?) Format 2) in an Ogg stream, so long as the
pages are small enough and spaced properly so everything is
simultaneous (if the application requires it)?
> Changes to Vorbis
> -----------------
> libvorbisfile will need to be modified to recognize substreams and ignore
> all but the first valid Vorbis stream.
I think it's about time to get the full Ogg stream interleave format
together. I have not seen the MNG work (Ralph, could you commit it to
CVS?), but I suspect this has already addressed some of the issues
I'll propose for getting this together:

* generic Ogg codec interface
  - whether codec is video or audio or MIDI or metadata or what
  - functions that player program must call to submit pages and read
    out data in the appropriate format
  - how to seek with that codec, or in general how to tell what time
    position corresponds to which page
* interleave library
  - to replace / extend libvorbisfile -- a liboggfile or something
    like that, to handle encoding, decoding, cutting, splicing,
    tagging, seeking, etc. in an Ogg stream of multiple codecs
  - this would include all logic to play back and seek like
    vorbisfile, and encode like vorbisenc, and manipulating like vcut,
    vorbiscomment, ogginfo, or possibly leave these as separate
    utilities
* table of contents header
  - to provide information about what codecs are used in the stream,
    for player selection or desktop environments -- more of a
    future-proofing than actually needed now
* metadata
  - this has been discussed in some detail in the past; I'll only
    mention that it's about time to get this standardized.

One final concern: will hardware players using libvorbis 1.0 or even
rc1 be able to play back these future Ogg streams, or at least
intelligently skip past what they don't know about and just play the
audio?

I have done only preliminary work on the generic Ogg codec interface;
it'll need a lot more thought and work before it becomes
usable. Anyone else doing any of the other things already?

comments? suggestions?


-- 
Kenneth Arnold <ken@arnoldnet.net> / kcarnold / Linux user #180115
http://arnoldnet.net/~kcarnold/



<HR NOSHADE>
<UL>
<LI>application/pgp-signature attachment: stored
</UL>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/octet-stream
Size: 233 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/vorbis-dev/attachments/20010825/24931ceb/part-0001.obj

This is a proposal for Ogg MIDI streams.  It will also help other 
stream additions to Vorbis, as they should follow a similar model.
Ralph has done much of the work on this, and also is working on MNG
Ogg streams.

Background
----------
I was contracted to do this work, so even if you can't think of a use
for this, there is one ;)  Basically with midi in Ogg, we can have
a synchronized midi+vorbis stream where vorbis plays audio and MIDI
controls some external device (synthesizer, digital piano, lights, etc).
There is also a provision of MIDI that allows lyrics, and this might be 
a good use also.

Goals
-----
- tool for combining Vorbis and a MIDI into on Ogg file (oggmerge)
- streamable
- tool for playing back said Ogg file

Current Tradeoffs
-----------------
There are three foramts of standard MIDI files, Format 0, 1, and 2.
Format 2 is not a song, but multiple idependent patterns, so is
irrelevant to this discussion.  Format 0 files are one track, and 
Foramt 1 files are multiple tracks (note that these are logical 
divisions for sequencers, and all the data of a format 1 file could be
 represented in one track).  Since tracks are stored serially, which
breaks streaming, Ogg MIDI will use only format 0 MIDI files, which means
any Format 1 files will have to be collapsed into Format 0.

File Format
----------
A file with Vorbis and Midi will look like:

<Vorbis header pages><MIDI header page><Vorbis data
page><MIDI data page>...

MIDI pages will be placed in sync with vorbis pages (immediately following
their corresponding Vorbis page).  Since MIDIs are tiny, it is likely
there will be very few pages of MIDI data.

Each MIDI Ogg packet will contain rougly 512 bytes aligned to MIDI event
boundaries.  This means approximately 8 MIDI packets will normally be place
in an Ogg page.

MIDI header packet format
-------------------------
The MIDI header will be:

'OggMIDI\000'
version (1 byte) - this is the OggMIDI version supported, starts from 0
time format (1 byte) - this is the time format, 1 for ticks per
        quarternote; 24,25,29,30 for SMTPE time
timebase (2 bytes) - subdivisions per SMTPE frame or ticks per quarternote

MIDI packet format
------------------
<midi event><midi event>...

Changes to Vorbis
-----------------
libvorbisfile will need to be modified to recognize substreams and ignore
all but the first valid Vorbis stream.

Comments, suggestions?

jack.

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Saturday, August 25, 2001, at 07:34 , Jack Moffitt wrote:

A file with Vorbis and Midi will look like:

<Vorbis header pages><MIDI header page><Vorbis data
page><MIDI data page>...

that should be <vorbis header page><MIDI header page>...etc.

According to the ogg specification, all the header pages (marked by the bos
flag) must occur together, before the first non-header page of any of the
logical bitstreams.

MIDI pages will be placed in sync with vorbis pages (immediately following
their corresponding Vorbis page).  Since MIDIs are tiny, it is likely
there will be very few pages of MIDI data.

Immediately following the vorbis page corresponding to the start or the end of
the midi data? I'd suggest the start, since the midi data will be cheaper to
buffer. :)

You also don't mention how the granule_pos field of the ogg page will be set
(called 'pcm absolute position' in framing.html). I propose it contain
the accumulated tick number in the timebase units the midi data uses. That means
the multiplexer has to read the timebase header of the midi stream *and* keep
track of the current tempo to maintain sync. Maybe we should instead specify
(only) that the midi data must come *before* the corresponding vorbis data. That
way the decoder needn't worry about midi underflow, and a dumb encoder can
just stick the whole midi stream at the beginning. The head ordering requirement
will remain in place, and the page flush requirement will ensure this
arrangement is possible.

Each MIDI Ogg packet will contain rougly 512 bytes aligned to MIDI event
boundaries.  This means approximately 8 MIDI packets will normally be place
in an Ogg page.

I picked 512 out of the air, and would be happy to hear any arguments for
different numbers.

The MIDI header will be:

'OggMIDI\000'
version (1 byte) - this is the OggMIDI version supported, starts from 0

Is the \000 part of the magic string, or the value of the version byte?

time format (1 byte) - this is the time format, 1 for ticks per
        quarternote; 24,25,29,30 for SMTPE time

0 is a better flag for 'per quarternote'.

timebase (2 bytes) - subdivisions per SMTPE frame or ticks per quarternote

This field is written little-endian, like the rest of ogg. The midi format
itself is big endian. We won't muck with the ordering within the event
stream (data packets) itself.

MIDI packet format
------------------
<midi event><midi event>...

Note that the midi data packets are sections of the raw event bytestream chopped
at an event boundary, and have no OggMIDI-specific header. You  just concatenate
them to get the midi track back again. We rely on ogg for all framing and sync.
I proposed that the packet boundaries should also occur before a non-zero
timestamp.

We also talked about expanding the context of all midi events to simplify random
access. (MIDI lets you omit the event status byte when you're only updating
a parameter on the 'current' note.) Do we want to drop that since there
are other issues with starting midi playback mid-stream?

Comments, suggestions?

Guess you should have let me review :)

 -r

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Saturday, August 25, 2001, at 05:01 , Kenneth Arnold wrote:
> How about allowing multiple Format 0 streams (thus representing Format
> 1 and even (partially?) Format 2) in an Ogg stream, so long as the
> pages are small enough and spaced properly so everything is
> simultaneous (if the application requires it)?
Jack said the format 1 -> format 0 conversion could be done without loss of
generality, so I
generally agreed the '0 only' spec made sense. I'm not entirely
clear on whether this imposes a 16
'device' limitation for Ogg MIDI, or not, and how onerous that actually
is.

In theory yes, you should be able to group multiple midi streams. But they might
be alternate
tracks instead of tracks to be mixed in. Same would go for multiple vorbis
streams, and so on,
which brings us again to the metadata can of worms. See below. :)
>> Changes to Vorbis
>> -----------------
>> libvorbisfile will need to be modified to recognize substreams and
ignore
>> all but the first valid Vorbis stream.
> 
> I think it's about time to get the full Ogg stream interleave format
> together. I have not seen the MNG work (Ralph, could you commit it to
> CVS?), but I suspect this has already addressed some of the issues
Not really. Most of this is still at the design stage.

A lot of what you mention is important work from the tools point of view; mostly
we've been
thinking at the spec level, which has something to do with why we missed your
concerns. At the ogg
and vorbis levels, we already have everything we need.

Figuring out how to properly interleave all these bitstream types is also *hard*
in a way that had
me assuming much of it belonged at the application level. And there will be so
many different
types. I do expect oggmerge will eventually have either plugins or a nice
interface for compiling
in support for various formats, but I haven't seen how we can have a unified
library-level
architecture that handles everything. Can you explain your idea?
> [...]
> * table of contents header
>   - to provide information about what codecs are used in the stream,
>     for player selection or desktop environments -- more of a
>    future-proofing than actually needed now
Right, I maintain we don't need this yet. The 'principle use first'
doctrine (why the vorbis
header must come first in an OggMIDI file) takes care of simple player selection
and mimetype
issues. Anything sophisticated enough to understand that this file has both
vorbis and midi data
can search the headers just like the players do. ogginfo could also be invoked
to implement this
for scripts.

My primary concern is that we don't have a complete requirements list for
this stream-description
metadata, and I don't want this 'future proofing' to turn into
baggage. Defining everything
implicitly, as in this proposal, avoids the issue in a way that is elegantly
compatible with
simpler implementations.

Requirements gathering could certainly be happening faster. The two main things
I wanted to see
happen is a video player implementation that supports all the overlay and
alternate track features
we've discussed (dvd done right) and some kind of support for structured
text data (my infamous
xml proposal). Audio, Video, and Text are the three pillars of our grand unified
theory of
multimedia, and I'd want us to have a experience and a plan for each of them
before we write this
stream description format.

Now, anyone could slap an ogg packet format around the experimental tarkin or
w3d codecs and start
playing with the video issues; that's ready anytime. Metadata I think needs
to be connected to an
external database, so more thinking should happen there. Or we could settle on
'musicbrainz is
good enough' for now.
> One final concern: will hardware players using libvorbis 1.0 or even
> rc1 be able to play back these future Ogg streams, or at least
> intelligently skip past what they don't know about and just play the
> audio?
Yes, this is a real problem. Current libvorbisfile WILL choke on anything but a
degenerate vorbis
stream and that's a bug! It will cause much grief if we don't fix it and
we must make clear that
handling multiplexed streams is part of the ogg vorbis spec. Jack mentioned this
in his post, and
that should be all the future proofing we need.

IMHO,
 -r

BTW, sorry about not mentioning the Ogg MNG code earlier. My normal mailhost is
down and the
announcement got queued.

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

Jack Moffitt wrote:
> 
> File Format
> ----------
> A file with Vorbis and Midi will look like:
> 
> <Vorbis header pages><MIDI header page><Vorbis data
page><MIDI data page>...
> 
> MIDI pages will be placed in sync with vorbis pages (immediately following
> their corresponding Vorbis page).  Since MIDIs are tiny, it is likely
> there will be very few pages of MIDI data.
> 
> Each MIDI Ogg packet will contain rougly 512 bytes aligned to MIDI event
> boundaries.  This means approximately 8 MIDI packets will normally be place
> in an Ogg page.
I'm a bit concerned here about packet ordering in non-degenerate Ogg
streams. We could specify that a MIDI packet comes before the Vorbis
packet that should be played at the same time, but what if we have more
streams? Let's say we want to multiplex a Vorbis audio stream, Tarkin
video, XML subtitles, and DMX (?) data to control lights in the cinema.
How would we do that?

Why not just specify that a packet which should start being played at
time x is not allowed to come before any packets that end before or on
time x, and that packets should be ordered within that constraint in
order of size (ie smallest packet first) to make buffering most
efficient.

Did I miss anything?

Lourens

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.