As an optimisation to reading Ogg encapsulated files, I've been thinking about a strategy of reading entire packets out of the stream, and sending them to the decoder. This has the advantages that: * The packet is just a linear array of bits in memory, with length known in advance. * The routine which gets bits out of the packet is then trivial and possibly inlined. * There is no possibility of blocking reads in the middle of a packet. However, I'm not sure what the upper bound of packet size is. I see there's a "4KB rule of thumb" mentioned in the spec, but does this apply to every packet? Even so, 4KB is still rather large, and I haven't seen packet sizes that large in any file generated by oggenc 1.0. Is there even a maximum? - John Ripley --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
John Ripley wrote on 2003-06-04:> As an optimisation to reading Ogg encapsulated files, I've been thinking > about a strategy of reading entire packets out of the stream, and sending > them to the decoder. This has the advantages that: > > * The packet is just a linear array of bits in memory, with length known in > advance. > * The routine which gets bits out of the packet is then trivial and possibly > inlined. > * There is no possibility of blocking reads in the middle of a packet. > > However, I'm not sure what the upper bound of packet size is.There is no bound whatsoever on packets. There is a limit on the size of a page - just below 64KB. Packets can span multiple pages, which is the only limitation to reading entire packets. I/O should be much slower than moving around memory anyway.> I see there's a "4KB rule of thumb" mentioned in the spec, but does > this apply to every packet?This applies to pages, not packets. Which of them do you want to talk about?> Even so, 4KB is still rather large, and I haven't seen packet sizes > that large in any file generated by oggenc 1.0. >Most packets are much smaller but that's the typical page size. However the initial setup packets can be larger. I don't know the codebook sizes but the comment packet can be up to 2^32 bytes. But that much has never seen in the wild ;-).> Is there even a maximum? >Multiplying max page size by page sequence number limit, you can get a theoretical limit of packet size but that would be ridiculously big. Ogg intentionally supports arbitrarily big packets. -- Beni Cherniavsky <cben@users.sf.net> The Three Laws of Copy-Protechnics: http://www.technion.ac.il/~cben/threelaws.html --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
> From: Beni Cherniavsky <cben@techunix.technion.ac.il> > Sent: 04 June 2003 20:02 > > John Ripley wrote on 2003-06-04: > > > As an optimisation to reading Ogg encapsulated files, I've > been thinking > > about a strategy of reading entire packets out of the > stream, and sending > > them to the decoder. This has the advantages that: > > > > * The packet is just a linear array of bits in memory, with > length known in > > advance. > > * The routine which gets bits out of the packet is then > trivial and possibly > > inlined. > > * There is no possibility of blocking reads in the middle > of a packet. > > > > However, I'm not sure what the upper bound of packet size is. > > There is no bound whatsoever on packets. There is a limit on the size > of a page - just below 64KB. Packets can span multiple pages, which > is the only limitation to reading entire packets. I/O should be much > slower than moving around memory anyway.That's the point, I/O is slow and may block, which then involves a context switch. If you want the ability to context switch, you suddenly need a separate stack (or thread if you're running a big OS). If you take all the I/O out of the decode path, it won't ever need to context switch.> > Even so, 4KB is still rather large, and I haven't seen packet sizes > > that large in any file generated by oggenc 1.0. > > > Most packets are much smaller but that's the typical page size. > However the initial setup packets can be larger. I don't know the > codebook sizes but the comment packet can be up to 2^32 bytes. But > that much has never seen in the wild ;-).I worked out the maximum header size once and found it was something like 1TB. I'm not worried about the size of header packets because those aren't in the "normal" decode path.> > Is there even a maximum? > > Multiplying max page size by page sequence number limit, you can get a > theoretical limit of packet size but that would be ridiculously big. > Ogg intentionally supports arbitrarily big packets.The "unknown" and "arbitrary" upper bounds of Vorbis streams is something I don't particularly like. However, if you assumed stereo (I'm throwing away anything with more), 8192 block size, and worst case entropy (all codes are 32 bits), I bet it would still be reasonably small. I'll calculate this when I have some spare time. - John Ripley. --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.
> From: Beni Cherniavsky <cben@techunix.technion.ac.il> > Sent: 05 June 2003 13:39 > > John Ripley wrote on 2003-06-05: > > > Beni Cherniavsky wrote: > > > > > > There is no bound whatsoever on packets. There is a limit on the > > > size of a page - just below 64KB. Packets can span multiple > > > pages, which is the only limitation to reading entire packets. > > > I/O should be much slower than moving around memory anyway. > > > > That's the point, I/O is slow and may block, which then involves a > > context switch. If you want the ability to context switch, you > > suddenly need a separate stack (or thread if you're running a big > > OS). If you take all the I/O out of the decode path, it won't ever > > need to context switch. > > > How do you take all I/O out of the decode path? I'm afraid I missed > you intent, please explain again.You separate the tasks of reading I/O and decoding packets. Reading the stream can be done with non-blocking I/O as one task, with the aim to read an entire packet into memory. This entire packet is then passed to the decoder, which will never have to worry about blocking I/O, seeing as all the data it will ever need is already there. That way you can run the file reads, the decoder, and whatever else is running in your system as a pure state machine. This means you don't need any threads. The trouble with an API which only supports blocking I/O via callbacks is that it needs its own context (thread) to run in, or it blocks everything else. I'll probably end up not bothering with the method of reading entire packets and then passing them to the decoder. It's simpler, but it's not actually any faster, and I still have to support other codecs with similar APIs to libvorbis. I'll probably arrange for the codec to be a state machine without callbacks, though.> > > Multiplying max page size by page sequence number limit, > you can get a > > > theoretical limit of packet size but that would be > ridiculously big. > > > Ogg intentionally supports arbitrarily big packets. > > > > The "unknown" and "arbitrary" upper bounds of Vorbis > streams is something I > > don't particularly like. However, if you assumed stereo > (I'm throwing away > > anything with more), 8192 block size, and worst case > entropy (all codes are > > 32 bits), I bet it would still be reasonably small. I'll > calculate this when > > I have some spare time. > > > It might make some sense to place bounds on packets size in specific > use areas. It most certainly doesn't belong in the Ogg framing > standard, I think it's wonderful that it does not place constraints.I agree it doesn't belong in Ogg. But for Vorbis, it makes sense to know in advance what the structure size limits are. As far as I can tell, there's no way you can implement a decoder which will play any arbitrary, valid 1.0 spec stream. The upper bounds on the size of things like the codebooks are in gigabytes (or was it terabytes)? I could certainly arrange for a perfectly valid stereo 44.1kHz stream which oggdec 1.0 couldn't play, for example. It would be good, in my opinion, to actually place some more constraints in the spec. It's hard to advertise "Supports Ogg Vorbis 1.0" when you know that's not true in all cases. What you actually meant is "Supports OggEnc 1.0 files, and YMMV". - John Ripley. --- >8 ---- List archives: http://www.xiph.org/archives/ Ogg project homepage: http://www.xiph.org/ogg/ To unsubscribe from this list, send a message to 'vorbis-dev-request@xiph.org' containing only the word 'unsubscribe' in the body. No subject is needed. Unsubscribe messages sent to the list will be ignored/filtered.