On 06/10/2010 09:25 PM, Chris Pearce wrote:> I looked at Benjamin's proposal, it does indeed produce much better > compression. However I decided not to use it because I felt it was too > complex.For the record, the Skeleton 4.0, as implemented by OggIndex, currently requires a multi-round iteration to produce. The encoder produces the index, then shifts the file content downstream to make room for the index packet ... but doing so changes the byte offsets, forcing the index to be recalculated. This potentially changes the length of the index, which forces the output to be shifted again, and changes the byte offsets again... Skeleton A-mod solves this problem, separately from the issue of the variable-length coding of the index, by coding the initial offset in fixed length. What I'm trying to say is: the complexity is mixed bag.> 4. The existing OggIndex format only takes up a fraction of a percent > of the media size. Over a network capable of playing video, it > won't be noticed.For a two hour movie with frequent keyframes, like on DVD, the index could easily be in excess of 50 KB, maybe even 100 KB. This would not be a large fraction of the total size (probably at least several hundred MB) but because the index comes at the front of the file it would delay the arrival of video data. Because of TCP connection ramp-up, this could introduce a noticeable additional delay to begin playback, even on connections with plenty of excess bandwidth. I should repeat that I think Chris's Skeleton 4.0 is a totally acceptable index implementation, and I'm not sure how to weigh the various tradeoffs here. --Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: OpenPGP digital signature Url : http://lists.xiph.org/pipermail/ogg-dev/attachments/20100610/60300621/attachment.pgp
On 11/06/2010 1:48 p.m., Benjamin M. Schwartz wrote:> On 06/10/2010 09:25 PM, Chris Pearce wrote: >> I looked at Benjamin's proposal, it does indeed produce much better >> compression. However I decided not to use it because I felt it was too >> complex. > For the record, the Skeleton 4.0, as implemented by OggIndex, currently > requires a multi-round iteration to produce. The encoder produces the > index, then shifts the file content downstream to make room for the index > packet ... but doing so changes the byte offsets, forcing the index to be > recalculated. This potentially changes the length of the index, which > forces the output to be shifted again, and changes the byte offsets again... >Right, this is an implementation decision. You need to encode the index usually twice, in rare cases three times. It is cheap and easy to just recall the function to recalculate the index. I'd rather do that than to mess around figuring out the perfect size up front. There's nothing stopping me from doing something cleverer here, except my desire to apply my cleverness elsewhere.> Skeleton A-mod solves this problem, separately from the issue of the > variable-length coding of the index, by coding the initial offset in fixed > length.Right, so I could encode the offset of the first keypoint with extra empty bytes using my unicode-style-variable-byte-encoding, in order to remove one potential re-encode. I decided there was no reason to bother with this, since re-encoding the index was so cheap and easy.> > For a two hour movie with frequent keyframes, like on DVD, the index could > easily be in excess of 50 KB, maybe even 100 KB.As a data point here, I indexed a 2 hour 55 minute theora+vorbis movie I have here with a 2000ms keyframe index interval, the index used 38404 bytes, for an overhead of 0.00184199%. I doubt any network capable of playing this video in real time would notice an extra 37KB at the start of the file.> I should repeat that I think Chris's Skeleton 4.0 is a totally acceptable > index implementation, and I'm not sure how to weigh the various tradeoffs > here. >Thanks. As you correctly guessed, I want to wrap this up. We've spent enough time on this. We have something that's good enough, let's ship it! All the best, Chris P.
On Fri, Jun 11, 2010 at 1:03 PM, Chris Pearce <chris at pearce.org.nz> wrote:> ?On 11/06/2010 1:48 p.m., Benjamin M. Schwartz wrote: >> On 06/10/2010 09:25 PM, Chris Pearce wrote: >>> I looked at Benjamin's proposal, it does indeed produce much better >>> compression. However I decided not to use it because I felt it was too >>> complex. >> For the record, the Skeleton 4.0, as implemented by OggIndex, currently >> requires a multi-round iteration to produce. ?The encoder produces the >> index, then shifts the file content downstream to make room for the index >> packet ... but doing so changes the byte offsets, forcing the index to be >> recalculated. This potentially changes the length of the index, which >> forces the output to be shifted again, and changes the byte offsets again... >> > > Right, this is an implementation decision. You need to encode the index > usually twice, in rare cases three times. It is cheap and easy to just > recall the function to recalculate the index. I'd rather do that than to > mess around figuring out the perfect size up front. There's nothing > stopping me from doing something cleverer here, except my desire to > apply my cleverness elsewhere.I regard this as an implementation issue, too, and nothing fundamental to the skeleton 4 format. However, I wonder if Benjamin's implementation was cleverer for this particular part and it might be useful to extract just this part of his implementation, seeing as it's already implemented? Note that this has nothing to do with the choice of skeleton formats.>> For a two hour movie with frequent keyframes, like on DVD, the index could >> easily be in excess of 50 KB, maybe even 100 KB. > > As a data point here, I indexed a 2 hour 55 minute theora+vorbis movie I > have here with a 2000ms keyframe index interval, the index used ?38404 > bytes, for an overhead of 0.00184199%. I doubt any network capable of > playing this video in real time would notice an extra 37KB at the start > of the file.I agree with Chris' prioritisation of simplicity over an overhead that is in the small KB range. I know the ffmpeg guys might care, but in reality it doesn't make a difference to the application unless you are on a low-bandwidth network and then you have other issues to worry about with video. Also, I believe that on small devices decoding with a simpler algorithm is preferable over bandwidth savings, since they usually have to deal with a less capable CPU, while networks are now fast enough. I'm happy with Chris shipping the spec - if possible, the OggIndex implementation could be improved with Benjamin's code. Also, I would suggest that we can keep Benjamin's more complex indexing algorithm for a later version where we may see a need to compress skeleton more. Incidentally, what is the typical overhead on a vorbis or speex file? Cheers, Silvia.