Hmm, You're actually correct, when you put it that way. Because the audio blocks are coded independently, you could speed things by having the encoder (or decoder) do a little up-front processing on the metadata, then use the overall settings to divide the file into sections and run the FLAC process in parallel. However, I think it is generally frowned upon in some circles to create one thread per processor, rather than making the division dependent upon some other factor that is actually independent of the number of processors. Apart from that caveat, someone could certainly develop an encoder or decoder which uses the FLAC library API to implement a threaded encoder/decoder - well, assuming that the API can write to subsets of a file, instead of requiring that the entire file be written from start to finish - and also assuming that the API can run in multiple threads and write to the same file. By your method, decoding would be faster, too. Not everybody decodes just to listen. Some of us keep original masters in FLAC format for archival purposes, but need the whole thing decoded before we can mix and master. Anything which speeds up decoding would help get work done faster. On the flip side, another thing to consider is that not all encoding is done with files that already exist. If you are recording live to FLAC, you cannot multi-thread the encoder because you don't have any audio besides the current block. So, decoding for playback and encoding for recording are two examples which cannot be multi-threaded. But file format conversion could be developed to take advantage of multiple processors. It would require a lot of tweaks to the library, perhaps. Note: PC games are multi-threaded because they're already doing several different things at once. Sometimes the complexity is actually reduced by dedicating a thread to each part of the game logic. And precisely because there are so many different things going on, multi-threading can speed things up when they overlap in unpredictable ways. Games would be harder to write in a single- threaded way. Meanwhile, encoding/decoding FLAC is easier to write in a single-threaded design. Brian W. On Sep 6, 2007, at 16:42, Harry Sack wrote: yes, i totally agree but I'm talking about the ENcoder :) You can perfectly 'cut' the pcm stream 'in half' and let 1 cpu encode one part , the other 2nd part (or with 4 cpu's in 4 parts, ...), because it's just a stream one frame after the other, as far as i know (correct me if i'm wrong). Many pc games synchronise multiple threads on 2 (even 4!) cores (much More complicated then reading frames, compressing, writing compressed frames like the encoder does), so i think making the encoder support 2 cpu's is perfectly possible. of course adding multi-core support to the decoder is silly because it's not necessary (any single core cpu is fast enough to play 1 file and it would only slow it down)! 2007/9/7, Brian Willoughby <brianw@sounds.wa.com>:> Any software which supports multiple processors must be multi- > threaded. The process of designing multi-threaded code adds > complexity to the software, so there must be a good reason to go > through all the trouble. The procedure for decoding a single file > could only benefit marginally from being multi-threaded, because most > operations would need to wait until the previous operation was > completed. One section of the decoder might be able to process > several blocks, but the overall result would still need to wait until > all blocks are completed. Besides, managing dynamic buffers between > each step of the decoding process would actually require the decoder > to do more, meaning it would slow down to an extent. There would > pretty much be no difference in the total time, whether single > processor or multiple processors. > > The best way to take advantage of multiple processors is to decode > multiple FLAC files at the same time. This will take full advantage > of your system, provided that the disk bandwidth and memory bandwidth > can keep up. > > Brian W.
it's really not complicated I think: only api changes to write on any position in the file if that's not possible already with existing function. I'm not sure if decoding can have multi-core support: you need an api for writing pcm files in different parts then and this is maybe more difficult to check if it's valid pcm data since the decoder can only check for valid flac streams , not pcm streams i think. you can make recording also support multi cores: just let cpu 1 fill a pcm data buffer while cpu 2 encodes it. In this case you have a more reliable encoding with less cpu usage of your system (it's spread among 2 cpu's) 2007/9/7, Brian Willoughby <brianw@sounds.wa.com>:> Hmm, > > You're actually correct, when you put it that way. Because the audio > blocks are coded independently, you could speed things by having the > encoder (or decoder) do a little up-front processing on the metadata, > then use the overall settings to divide the file into sections and > run the FLAC process in parallel. However, I think it is generally > frowned upon in some circles to create one thread per processor, > rather than making the division dependent upon some other factor that > is actually independent of the number of processors. Apart from that > caveat, someone could certainly develop an encoder or decoder which > uses the FLAC library API to implement a threaded encoder/decoder - > well, assuming that the API can write to subsets of a file, instead > of requiring that the entire file be written from start to finish - > and also assuming that the API can run in multiple threads and write > to the same file. > > By your method, decoding would be faster, too. Not everybody decodes > just to listen. Some of us keep original masters in FLAC format for > archival purposes, but need the whole thing decoded before we can mix > and master. Anything which speeds up decoding would help get work > done faster. > > On the flip side, another thing to consider is that not all encoding > is done with files that already exist. If you are recording live to > FLAC, you cannot multi-thread the encoder because you don't have any > audio besides the current block. > > So, decoding for playback and encoding for recording are two examples > which cannot be multi-threaded. But file format conversion could be > developed to take advantage of multiple processors. It would require > a lot of tweaks to the library, perhaps. > > Note: PC games are multi-threaded because they're already doing > several different things at once. Sometimes the complexity is > actually reduced by dedicating a thread to each part of the game > logic. And precisely because there are so many different things > going on, multi-threading can speed things up when they overlap in > unpredictable ways. Games would be harder to write in a single- > threaded way. Meanwhile, encoding/decoding FLAC is easier to write > in a single-threaded design. > > Brian W. > > > On Sep 6, 2007, at 16:42, Harry Sack wrote: > > yes, i totally agree but I'm talking about the ENcoder :) > You can perfectly 'cut' the pcm stream 'in half' and let 1 cpu encode > one part , the other 2nd part (or with 4 cpu's in 4 parts, ...), > because it's just a stream one frame after the other, as far as i > know (correct me if i'm wrong). > Many pc games synchronise multiple threads on 2 (even 4!) cores (much > More complicated then reading frames, compressing, writing compressed > frames like the encoder does), so i think making the encoder support 2 > cpu's is perfectly possible. > > of course adding multi-core support to the decoder is silly because > it's not necessary (any single core cpu is fast enough to play 1 file > and it would only slow it down)! > > 2007/9/7, Brian Willoughby <brianw@sounds.wa.com>: > > Any software which supports multiple processors must be multi- > > threaded. The process of designing multi-threaded code adds > > complexity to the software, so there must be a good reason to go > > through all the trouble. The procedure for decoding a single file > > could only benefit marginally from being multi-threaded, because most > > operations would need to wait until the previous operation was > > completed. One section of the decoder might be able to process > > several blocks, but the overall result would still need to wait until > > all blocks are completed. Besides, managing dynamic buffers between > > each step of the decoding process would actually require the decoder > > to do more, meaning it would slow down to an extent. There would > > pretty much be no difference in the total time, whether single > > processor or multiple processors. > > > > The best way to take advantage of multiple processors is to decode > > multiple FLAC files at the same time. This will take full advantage > > of your system, provided that the disk bandwidth and memory bandwidth > > can keep up. > > > > Brian W. > >
On 9/6/07, Harry Sack <tranzedude@gmail.com> wrote:> it's really not complicated I think: only api changes to write on anyPlease get started writing a patch, immediately. -- avuton -- Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
I really should have just said that it will require some testing to make sure the FLAC API can handle writing the same file from multiple threads. It may not turn out to be complicated at all. The FLAC decoder has its own code for writing PCM files already. Tweaking this to support multi-threaded writes would not be too difficult. It's simply a matter of building the PCM file, setting aside the data area for the PCM samples, and then providing each thread with information needed to write at different offsets. FLAC never needs to "check for valid PCM streams" since it is creating the PCM stream from scratch every time. Recording always involves a PCM data buffer. You're correct that there might be a 2 CPU split, automatically, depending upon the system. However, I was simply stating that going to an arbitrary number of threads would not be possible when recording, since you're only getting one block at a time, and the block is where the division between CPUs would be made. Brian W. On Sep 6, 2007, at 17:24, Harry Sack wrote: it's really not complicated I think: only api changes to write on any position in the file if that's not possible already with existing function. I'm not sure if decoding can have multi-core support: you need an api for writing pcm files in different parts then and this is maybe more difficult to check if it's valid pcm data since the decoder can only check for valid flac streams , not pcm streams i think. you can make recording also support multi cores: just let cpu 1 fill a pcm data buffer while cpu 2 encodes it. In this case you have a more reliable encoding with less cpu usage of your system (it's spread among 2 cpu's)