flac dev - May 2011 - Can a libFLAC encoder be initialize and called from inside a libFLAC decoder callback?

Thanks for the tip, Brian.  I did have a version that does everything in
memory, but only had enough memory to get six threads going.  (Although my
system has 16GB, I have not taken the time to create 64-bit libraries for
wxWidgets, TagLib, libFLAC, etc.)

Based on the direction you are pointing me in, I assume that encoding from
within the decoder is not permitted.

I really like your idea of the FIFO, and will probably go in that direction.
What do you think of implementing the FIFO via a pipe?  That might simplify
the implementation.

Thanks for the advice.

Kindest regards to you and the flac-dev community.

David

-----Original Message-----
From: Brian Willoughby [mailto:brianw at sounds.wa.com] 
Sent: Tuesday, May 24, 2011 1:16 PM
To: David Troendle
Cc: flac-dev at xiph.org
Subject: Re: [Flac-dev] Can a libFLAC encoder be initialize and called from
inside a libFLAC decoder callback?

Have you tried decoding to memory, and encoding from that memory?   
This technique would avoid file I/O.

If you are reluctant to allocate enough memory for the full file (and  
why not, with a 64-bit system?), then you could implement a FIFO.   
You would need a semaphore to keep track of the decoder's progress, and
multithreaded coding.  The decoder would pause, whenever the FIFO gets full,
until the encoder empties out the samples so far.

It's also possible that you have made a mistake in InitializeEncoder
() where the new file format does not match the incoming Buffer blocks.

Brian Willoughby
Sound Consulting

On Tue, May 24, 2011 at 3:11 PM, David Troendle <david at troendle.org>
wrote:
> Thanks for the tip, Brian. ?I did have a version that does everything in
> memory, but only had enough memory to get six threads going. ?(Although my
> system has 16GB, I have not taken the time to create 64-bit libraries for
> wxWidgets, TagLib, libFLAC, etc.)
>
> Based on the direction you are pointing me in, I assume that encoding from
> within the decoder is not permitted.
>
> I really like your idea of the FIFO, and will probably go in that
direction.
> What do you think of implementing the FIFO via a pipe? ?That might simplify
> the implementation.
Pipes on *nix systems are generally not very big, so be careful.
Historically they were limited to about 5kB.

On May 24, 2011, at 12:11, David Troendle wrote:
> Thanks for the tip, Brian.  I did have a version that does  
> everything in
> memory, but only had enough memory to get six threads going.   
> (Although my
> system has 16GB, I have not taken the time to create 64-bit  
> libraries for
> wxWidgets, TagLib, libFLAC, etc.)
What's wrong with only 6 threads?  Is that just 3 FLAC recodes (3  
decode, 3 encode)?  With a queuing layer driving the threads, you  
could schedule hundreds of file recodes and each one would start as  
soon as a thread opens up.

> Based on the direction you are pointing me in, I assume that  
> encoding from
> within the decoder is not permitted.
I have no idea.  My assumption before your email would have been that  
the individual encoder/decoder handles would allow reentrancy, but  
perhaps there are pieces of libFLAC that are not reentrant even when  
working on separate encoder/decoder objects.  The only way to know  
for sure is if Josh Coalson pipes up, or someone takes the time to  
look at the source for a thorough code review.  There's a possibility  
that your code has a bug, and what you want to do may actually be  
possible.  But, again, I have no idea.

> I really like your idea of the FIFO, and will probably go in that  
> direction.
> What do you think of implementing the FIFO via a pipe?  That might  
> simplify
> the implementation.
You don't strictly need a FIFO.  A pipe might be a good choice - at  
the very least it would allow you to link two processes instead of  
being limited to two threads within a process.  The pipe could also  
allow you to decode on one machine and encode on another over a  
network connection - assuming that's of any use to you.

Brian Willoughby
Sound Consulting

Sorry for ambiguity.  Each thread does a decode followed by the encode.  The
threads are driven by a queue of files for recoding, and synced with a
critical section.  When the queue is empty the threads exit.

It is not all that bad of an approach, but  the technique does use a lot of
I/O and only drives the processor at about 25%.  I thought this change in
approach would reduce the I/O and increase processor usage.  It has also
been a good tool to get some experience using libFLAC.

Regards.

David

-----Original Message-----
From: Brian Willoughby [mailto:brianw at sounds.wa.com] 
Sent: Tuesday, May 24, 2011 2:31 PM
To: David Troendle
Cc: flac-dev at xiph.org
Subject: Re: [Flac-dev] Can a libFLAC encoder be initialize and called from
inside a libFLAC decoder callback?


On May 24, 2011, at 12:11, David Troendle wrote:
> Thanks for the tip, Brian.  I did have a version that does everything 
> in
> memory, but only had enough memory to get six threads going.   
> (Although my
> system has 16GB, I have not taken the time to create 64-bit libraries 
> for wxWidgets, TagLib, libFLAC, etc.)
What's wrong with only 6 threads?  Is that just 3 FLAC recodes (3 decode, 3
encode)?  With a queuing layer driving the threads, you could schedule
hundreds of file recodes and each one would start as soon as a thread opens
up.

> Based on the direction you are pointing me in, I assume that encoding 
> from within the decoder is not permitted.
I have no idea.  My assumption before your email would have been that the
individual encoder/decoder handles would allow reentrancy, but perhaps there
are pieces of libFLAC that are not reentrant even when working on separate
encoder/decoder objects.  The only way to know for sure is if Josh Coalson
pipes up, or someone takes the time to look at the source for a thorough
code review.  There's a possibility that your code has a bug, and what you
want to do may actually be possible.  But, again, I have no idea.

> I really like your idea of the FIFO, and will probably go in that 
> direction.
> What do you think of implementing the FIFO via a pipe?  That might 
> simplify the implementation.
You don't strictly need a FIFO.  A pipe might be a good choice - at the very
least it would allow you to link two processes instead of being limited to
two threads within a process.  The pipe could also allow you to decode on
one machine and encode on another over a network connection - assuming
that's of any use to you.

Brian Willoughby
Sound Consulting