Ralph, The problem is that there is no clear advantage, at least in terms of multiple cores, to the approach you're asking about. In order to allow each stage of the codec to overlap, you need smart buffering between each stage. That adds code and complexity which isn't there currently. So you end up making the system do more work in the hopes that there will be some overlap. Basically, later stages get blocked waiting for their input buffer to fill, which means that you're not really getting very much overlap at all, but plenty of multi- threading overhead. At least that's the predicted result - I admit that nobody has tried this, to my knowledge. Brian Willoughby Sound Consulting On Sep 7, 2007, at 18:25, Ralph Giles wrote: On Fri, Sep 07, 2007 at 04:59:50PM -0700, Josh Coalson wrote:> it actually is complicated. the libFLAC api is not suited to a > multithreaded design because the i/o is stream-based, not file- > based. flac(.exe) is the file-based wrapper around libFLAC that > allows it to work on files. the way libFLAC buffers data is also > impossible to parallelize without significantly changing the api.It seems like buffering (especially compressed) blocks and writing them to the stream in sequence wouldn't be a problem. Is there something in the way the blocking decisions are made that makes it hard to divide the input audio this way? -r
2007/9/8, Brian Willoughby <brianw@sounds.wa.com>:> > Ralph, > > The problem is that there is no clear advantage, at least in terms of > multiple cores, to the approach you're asking about. In order to > allow each stage of the codec to overlap, you need smart buffering > between each stage. That adds code and complexity which isn't there > currently. So you end up making the system do more work in the hopes > that there will be some overlap. Basically, later stages get blocked > waiting for their input buffer to fill, which means that you're not > really getting very much overlap at all, but plenty of multi- > threading overhead. At least that's the predicted result - I admit > that nobody has tried this, to my knowledge.this is because of the limitations/design problem of FLAC API in particular. When the developers had made a smart decision and based everything on file based I/O you would get a HUGE performance boos when using multiple threads divided between multiple cores, because they only thing to do was to split the file output in different threads. But it's not clear to me why everything was based on streams... Harry Brian Willoughby> Sound Consulting > > > On Sep 7, 2007, at 18:25, Ralph Giles wrote: > > On Fri, Sep 07, 2007 at 04:59:50PM -0700, Josh Coalson wrote: > > > it actually is complicated. the libFLAC api is not suited to a > > multithreaded design because the i/o is stream-based, not file- > > based. flac(.exe) is the file-based wrapper around libFLAC that > > allows it to work on files. the way libFLAC buffers data is also > > impossible to parallelize without significantly changing the api. > > It seems like buffering (especially compressed) blocks and writing them > to the stream in sequence wouldn't be a problem. Is there something in > the way the blocking decisions are made that makes it hard to divide the > input audio this way? > > -r > > _______________________________________________ > Flac-dev mailing list > Flac-dev@xiph.org > http://lists.xiph.org/mailman/listinfo/flac-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/flac-dev/attachments/20070908/72e8c009/attachment.htm
No, streams should stay. Audio is NOT a file based process -- it's a stream. You can't listen to an entire song simultaneously. You organize it into files for later use, but you listen and record from a stream. Stream-based storage is practically REQUIRED for an audio codec. It's not random access, it's sequential. You can put wrappers around it to make it convenient for file storage and conversions, but the codec itself must be streams. Multi-core support may not be practical for a variable-length encoding. How would you know where to write the next block when you don't know what size the first block is going to be? The functionality for that is not trivial and is not currently implemented in the API. Maybe somebody will write a multi-core file-based wrapper for you, or maybe you could try writing one yourself. Or if you disagree with Josh about the direction of FLAC you can write your own codec. But your nattering on and on about how you think the API isn't right doesn't help at all and is very annoying. _____ From: flac-dev-bounces@xiph.org [mailto:flac-dev-bounces@xiph.org] On Behalf Of Harry Sack Sent: Saturday, September 08, 2007 6:06 AM To: Brian Willoughby Cc: flac-dev@xiph.org Subject: Re: [Flac-dev] Re: multiple core support 2007/9/8, Brian Willoughby <brianw@sounds.wa.com>: Ralph, The problem is that there is no clear advantage, at least in terms of multiple cores, to the approach you're asking about. In order to allow each stage of the codec to overlap, you need smart buffering between each stage. That adds code and complexity which isn't there currently. So you end up making the system do more work in the hopes that there will be some overlap. Basically, later stages get blocked waiting for their input buffer to fill, which means that you're not really getting very much overlap at all, but plenty of multi- threading overhead. At least that's the predicted result - I admit that nobody has tried this, to my knowledge. this is because of the limitations/design problem of FLAC API in particular. When the developers had made a smart decision and based everything on file based I/O you would get a HUGE performance boos when using multiple threads divided between multiple cores, because they only thing to do was to split the file output in different threads. But it's not clear to me why everything was based on streams... Harry Brian Willoughby Sound Consulting On Sep 7, 2007, at 18:25, Ralph Giles wrote: On Fri, Sep 07, 2007 at 04:59:50PM -0700, Josh Coalson wrote:> it actually is complicated. the libFLAC api is not suited to a > multithreaded design because the i/o is stream-based, not file- > based. flac(.exe) is the file-based wrapper around libFLAC that > allows it to work on files. the way libFLAC buffers data is also > impossible to parallelize without significantly changing the api.It seems like buffering (especially compressed) blocks and writing them to the stream in sequence wouldn't be a problem. Is there something in the way the blocking decisions are made that makes it hard to divide the input audio this way? -r _______________________________________________ Flac-dev mailing list Flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/flac-dev/attachments/20070908/04eb758c/attachment.html
On Sat, Sep 08, 2007 at 03:05:32PM +0200, Harry Sack wrote:> But it's not clear to me why everything was based on streams...Streams, at least in the variety we seem to be describing, aren't seekable. This is a considerable simplification and allows for more flexibility in other areas, like network delivery. This can't be quite correct of course, because there's some header information that requires updating at the end. -r