Vorbis dev - Jan 2001 - video compression using textured polygons?

I was just thinking last night, how about this:

For each frame, detect edges and divide the frame into a 
bunch of non-overlapping polygons. Extract textures from 
the first frame, then save changes in polygon vertex 
coordinates only, unless the textures change too much in 
which case new textures (which could be wavelet compressed) 
would be stored.

Also, if the movements of the vertices would be taken 
across multiple frames they could be expressed as splines, 
causing more compression.

I realise that this is quite a lot like mpeg 4, with its 
sprites and body objects, but I think it's more generic. 
And it has the added bonus of being able to do playback 
hardware-accelerated on standard 3D hardware.

Problems I foresee: edges may look too sharp, or if the 
polygons don't have enough edges too straight. Also the 
frame-to-frame matching of polygons absolutely needs to be 
correct or things go wrong spectacularly.

Well, any thoughts are welcome.

Lourens

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.

On Sat, Jan 06, 2001 at 01:43:20PM +0330, jsr@dds.nl
wrote:
> 
> I was just thinking last night, how about this:
Weird, so was I :) and I'm not just saying that to sound
smart... actually I have evidence saying that I thought about this
Thursday at 10:05 AM EST, but that's very much besides the point.
> For each frame, detect edges and divide the frame into a 
> bunch of non-overlapping polygons. Extract textures from 
> the first frame, then save changes in polygon vertex 
> coordinates only, unless the textures change too much in 
> which case new textures (which could be wavelet compressed) 
> would be stored.
> 
> Also, if the movements of the vertices would be taken 
> across multiple frames they could be expressed as splines, 
> causing more compression.
The problem here is of course the encoder. Do you know any algorithms
that could possibly come close to doing this in anything near
real-time? That's what discourages me from putting it in the core of a
video spec (though you could have one polygon filling the entire
screen and the texture changing, but that would not compress well in
the proposed scheme).

The textures themselves could change, but in relatively predictable
ways. For example, consider (as Sean Lynch was talking to me about) a
road with cars passing by. Every time a car moves, a conventional
video codec would have to retransmit the road underneath. A
polygon- (or generic object) based codec could realize that the road
is now mostly the same as it was before the car passed, and just tell
the player that object "car" moved, and (if it doesn't already
know)
polygon "road"'s texture has changed to <what the road was
underneath>
> I realise that this is quite a lot like mpeg 4, with its 
> sprites and body objects, but I think it's more generic. 
> And it has the added bonus of being able to do playback 
> hardware-accelerated on standard 3D hardware.
I don't know how MPEG4 works here, but I will say that a simple
polygon would not cut it in all cases. The encoder will probably not
be able to handle it, but the codec should provide for entirely
arbitrary 3d worlds (awesome for recording a quake level to show your
friends :), though in a way that doesn't overly harm the compressino
for the rest of the encoder. Same for any other encoder extensions. I
will be playing with this sort of stuff soon in a prototype
renderer-based player (encode by hand for a while) and keep you posted
on how things work out.
> Problems I foresee: edges may look too sharp, or if the 
> polygons don't have enough edges too straight. Also the 
> frame-to-frame matching of polygons absolutely needs to be 
> correct or things go wrong spectacularly.
Encoder issues... the player for this would be absurdly simple on a
system with 3d accel (or a port of Mesa).
> Well, any thoughts are welcome.
Chirplets? Can anyone go beyond the buzzword and tell me what's going
on with them? Everything I've seen looks promising but isn't enough to
write an actual implementation or even do something by hand with them.
> Lourens
Always happy to talk to smart people.


-- 
Kenneth Arnold <ken@arnoldnet.net> / kcarnold / Linux user #180115
http://arnoldnet.net/~kcarnold/



<HR NOSHADE>
<UL>
<LI>application/pgp-signature attachment: stored
</UL>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: part
Type: application/octet-stream
Size: 233 bytes
Desc: not available
Url :
http://lists.xiph.org/pipermail/vorbis-dev/attachments/20010107/62382e4c/part-0001.obj

[snip]
> Also, if the movements of the vertices would be taken
> across multiple frames they could be expressed as splines,
> causing more compression.
[snip]

That's nice ofcourse, but how about taking this to 3D? The problem would
be equivalent to finding surfaces in a voxel volume, for which there are
already algorithms available. The textures become two-axis-aligned
surfaces in the volume. And if we use surfaces we will also have the
motion data encoded straight away, making matching of triangles a lot
easier, which should speed up the decoder a lot. We could also chop the
volume if a totally new image appears (ie an I-frame (I think? one that
totally replaces the previous one, like when the video switches from one
camera to another one)). On MTV-style videos with lots of camera
switching this would do worse though. Perhaps only cut if the resulting
slices are thick enough.

Lourens

--- >8 ----
List archives:  http://www.xiph.org/archives/
Ogg project homepage: http://www.xiph.org/ogg/
To unsubscribe from this list, send a message to
'vorbis-dev-request@xiph.org'
containing only the word 'unsubscribe' in the body.  No subject is
needed.
Unsubscribe messages sent to the list will be ignored/filtered.