ogg dev - Aug 2008 - liboggplay and overlay video

Hi,

I've eventually released the first version of my rendering library for
Kate streams,
and I've recently been hacking away at adding it to liboggplay. That was the
missing "overlay" mode to the browser_plugin patch I sent a while ago.

Now, I'm not too sure how to expose those overlays to liboggplay users: the
obvious solution is to add a RGBA video type, and expose the render of this,
which the client can then overlay onto the video. I've done that, and it
works
(modulo bugs). However, it's slow, both because of the two renders (once to
render the overlay in liboggplay, the second to overlay it onto the video in the
client code), and because libtiger (my rendering lib) only draws what's
needed,
whereas liboggplay exposes a full size buffer (would need at least a bounding
rectangle).

Another solution would be to let liboggplay overlay Kate tracks onto the video
directly - but this would seem contrary to the design, where each track feeds
data to the client. Here, a track would pipe its data onto another track's
data.

I suppose I'm asking for input on whether this would be OK, or if someone
can
think of another better way.

For the interested, here's a Theora render of libtiger rendering the
demo.kate
stream from the libkate distribution:
http://rapidshare.com/files/138181866/tiger.ogv.html

Thanks

I think libkate should be able to render in the yuv colour space. A
lot of players use hardware accelerated colour space conversion and
scaling, so in addition to your point about saving unneccesary
conversions, they are likely to be able to offload the compositing
step as well. This means that libtiger should also be able to do the
compositing in yuv.

I think adding an RGB/RGBA frame type to liboggplay should *also*
happen. yuv2rgb is tedious at best and hard to get right at worst, so
being able to get frames (even theora frames) in sRGB is something I
think we should provide. Many 2D graphics libraries handle alpha
compositing, but only in RGB space; anyone using one of those for
overlay would want this.

FWIW,
 -r

> I think libkate should be able to render in the yuv colour space. A
I thought of that a while ago, but there are various problems that would
crop up and need some kind of solution.

I use Cairo and Pango for rendering, and these use RGBA. I do not
think (though this isn't based on experimentation) that passing in YUV
colors in RGB values would work well. Additionally, YUV typically uses
downsampled UV, another thing which would not play nicely with Cairo
and Pango.

I briefly added YUV palettes to kate_palette, but reverted due to those.
Besides, I'd need YUVA, which is less commonly found. Also, alpha
doesn't usually play nice with subsampled planes as colors tend to
bleed (or so I hear).

Which brings me to an interesting idea I had: instead of placing bitmaps
(or text) in regions, I could have placed a Theora video - YUV. Since
the mapping for this is already done and the streams muxed in, it could
be quite easy to not render the videos, but for libtiger to pull the images
so they could have all the xforms that kate regions can have.

To make this really useful, I'd need to actually get around to implementing
continuations though (essentially adding to an event's motions from data
in subsequent packets), and this has been moved to the back burner. But
you could manipulate videos in 2D, switching them on and off, etc.

Anyway, I do ramble.
> lot of players use hardware accelerated colour space conversion and
> scaling, so in addition to your point about saving unneccesary
> conversions, they are likely to be able to offload the compositing
> step as well. This means that libtiger should also be able to do the
> compositing in yuv.
Yes, I did not really think of hardware acceleration here.
> I think adding an RGB/RGBA frame type to liboggplay should *also*
> happen. yuv2rgb is tedious at best and hard to get right at worst, so
> being able to get frames (even theora frames) in sRGB is something I
> think we should provide. Many 2D graphics libraries handle alpha
> compositing, but only in RGB space; anyone using one of those for
> overlay would want this.
oggplay_yuv2rgb is a testament to this :)

Which makes me remember - I had to #if 0 the MMX/SSE/etc code in
liboggplay, it didn't build for me i686-pc-linux-gnu, x86_64-pc-linux-gnu.
I could send the (pages of) errors, this had slipped my mind.

Thanks

> I think adding an RGB/RGBA frame type to liboggplay should *also*
> happen. yuv2rgb is tedious at best and hard to get right at worst, so
I'd added an RGBA type, but if this is going to also carry video,
it might be better to merge this with the existing video type, and
have a 'type' enum, or just several pointers, only one set being
non NULL (makes client code simpler). Would it be annoying to
users if the exposed struct was being changed (since I hear that
liboggplay is being used by Firefox) ? At the moment, changes
are limited to additions to the API.

Thanks