I have been running a lot of large compression tests to see how well
Martijn van Beurden's new presets do and was once again reminded how
real the fragmentation problem still is.
To have decent speed it's necessary to run multiple encoders in
parallel. In my setup FLAC was compressing four files at the same time
and each instance writes tiny bits of data to disk at once. A
fragmentation analyzing tool reported that my files were in 300
fragments on average. Thanks to this decoding the files happened in a
fraction of the speed that I should be getting.
There would be one very simple solution to help against this issue, the
use of setvbuf() command to buffer the file writes to larger chunks.
Unfortunately it looks like the call has to be done inside libflac. The
simple test change shown at the end of the mail reduced average
fragmentation to two fragments per file and decode speeds were as good
as without fragmentation.
I don't know how big issue this is in *nix environments but added
buffering would definitely be a nice change for the Windows frontend.
diff --git a/src/libFLAC/stream_encoder.c b/src/libFLAC/stream_encoder.c
index 6f46d78..bd95634 100644
--- a/src/libFLAC/stream_encoder.c
+++ b/src/libFLAC/stream_encoder.c
@@ -1288,6 +1288,8 @@ static FLAC__StreamEncoderInitStatus
init_FILE_internal_(
if(file == stdout)
file = get_binary_stdout_(); /* just to be safe */
+ setvbuf(file, NULL, _IOFBF, 10*1024*1024); /* 10MB output buffer to
help reduce disk fragmentation */
+
encoder->private_->file = file;
encoder->private_->progress_callback = progress_callback;
On Tue, Sep 23, 2014 at 06:42:16PM +0300, cse at sci.fi wrote:> I don't know how big issue this is in *nix environments but added > buffering would definitely be a nice change for the Windows frontend.It depends on the file system being used. With almost every Linux distro, you can choose which FS to use at install time. Most of them have solved most of the problems of fragmentation, at the OS level. Whereas NTFS seems to actually make the problem worse, by allocating blocks in ways that make fragmentation a bigger problem. What I have done for testing that involves lots of files, is create a partition just for the test files, and reformat it each time. If it's small enough, it could be formatted as FAT for added speed. -- -Dec. --- (no microsoft products were used to create this message) "Mosaic is going to be on every computer in the world." - Marc Andreessen, 1994
Janne Hyv?rinen wrote:> I don't know how big issue this is in *nix environments but added > buffering would definitely be a nice change for the Windows frontend.The patch got a bit mangled by either your email client or mine, but since it was simple it was east to apply by hand. Thanks. Cheers, Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/
On Tue, Sep 23, 2014 at 06:42:16PM +0300, Janne Hyv?rinen wrote:> I don't know how big issue this is in *nix environments but added > buffering would definitely be a nice change for the Windows frontend. > > > diff --git a/src/libFLAC/stream_encoder.c b/src/libFLAC/stream_encoder.c > index 6f46d78..bd95634 100644 > --- a/src/libFLAC/stream_encoder.c > +++ b/src/libFLAC/stream_encoder.c > @@ -1288,6 +1288,8 @@ static FLAC__StreamEncoderInitStatus > init_FILE_internal_( > if(file == stdout) > file = get_binary_stdout_(); /* just to be safe */ > > + setvbuf(file, NULL, _IOFBF, 10*1024*1024); /* 10MB output buffer to > help reduce disk fragmentation */ > +Hm, does this mean that every process encoding FLAC files will now need extra 10MB of memory? Is that ok for small devices with limited memory? -- Miroslav Lichvar
Miroslav Lichvar wrote:> Hm, does this mean that every process encoding FLAC files will now > need extra 10MB of memory? Is that ok for small devices with limited > memory?What small device do you have in mind? The smallest device I could possibly imagine *encoding* FLAC on is Rasberry Pi which has 512Meg of RAM. If you really are concerned about this (and knowing that it has little benefit on Linux) maybe we could wrap it in a Windows only #ifdef. Erik -- ---------------------------------------------------------------------- Erik de Castro Lopo http://www.mega-nerd.com/