Hi, I work for a company which makes meteor and wind radar (http://www.gsoft.com.au). On occasion (ie during meteor showers such as the Leonids) we configure the system to save raw data as it comes out of the acquisition system, the data rate for this varies (depends on acquisition parameters and number of coherent integrations etc), but usually it is around 600kb/sec. The data consists of 16 bit samples, one for each digital channel (in phase and quad for each receiver) at each sampled height. Typically there are 10 channels and 30 range gates, so for a single transmitter pulse tick you get 10 * 30 = 300 16 bit samples. In an effort to compress the data I looked at lossless audio compressors (as well as gzip etc) and I ended up trying flac and shorten. Program Compressed Size Compression time Decompression time gzip 358 Mb (84.9%) 147.96 sec 21.28 sec bzip2 330 Mb (78.2%) 589.01 sec 232.13 sec shorten -c 10 326 Mb (77.4%) 91.39 sec 55.89 sec shorten -c 300 313 Mb (74.3%) 102.08 sec 53.04 sec flac 5chn 328 Mb (77.8%) 131.22 sec 66.54 sec flac 8chn 332 Mb (78.7%) 139.57 sec 66.74 sec flac 10chn 327 Mb (77.5%) 126.32 sec n/a sec Note the 10 channel version was done by me changing the format.h file and recompiling, but alas I couldn't get decompression to go :( The command line I used for flac was /usr/bin/time flac --lax --endian=little --channels=5 --bps=16 --sample-rate=64320 --sign=signed --force-raw-format reallyraw -o reallyraw.flac-5chn I needed the --lax otherwise it generated a FLAC__STREAM_ENCODER_NOT_STREAMABLE error. The shorten command line was /usr/bin/time shorten -c 10 -ts16lh reallyraw reallyraw.short.10 I am basically letting you guys know about a possibly more interesting than normal application for your code and soliciting any tips you may have :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 9A8C 569F 685A D928 5140 AE4B 319B 41F4 5D17 FDD5
On Wed, 2002-11-20 at 16:56, Josh Coalson wrote:> I mean). FLAC only supports 8 channels, but I would bet that > the compression would be a lot better if you separated the 300 > channels stream into several streams (<=8 channels each) and > compressed them with FLAC. That may not be practical for your > application.Hmm, I just acquired some data that has 2 gates and 4 channels (for 8 independent channels), and shorten still beat flac. skiymet:/tmp>ll really* -rw-r--r-- 1 radar wheel 516384 Nov 20 07:43 reallyraw -rw-r--r-- 1 radar wheel 195712 Nov 20 07:43 reallyraw.flac-8chn -rw-r--r-- 1 radar wheel 193049 Nov 20 07:44 reallyraw.short.8 Weird! -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 9A8C 569F 685A D928 5140 AE4B 319B 41F4 5D17 FDD5
--- Daniel O'Connor <doconnor@gsoft.com.au> wrote:> Hi, > I work for a company which makes meteor and wind radar > (http://www.gsoft.com.au). > > On occasion (ie during meteor showers such as the Leonids) we > configure > the system to save raw data as it comes out of the acquisition > system, > the data rate for this varies (depends on acquisition parameters and > number of coherent integrations etc), but usually it is around > 600kb/sec. > > The data consists of 16 bit samples, one for each digital channel (in > phase and quad for each receiver) at each sampled height. Typically > there are 10 channels and 30 range gates, so for a single transmitter > pulse tick you get 10 * 30 = 300 16 bit samples. > > In an effort to compress the data I looked at lossless audio > compressors > (as well as gzip etc) and I ended up trying flac and shorten. > > > Program Compressed Size Compression time > Decompression time > gzip 358 Mb (84.9%) 147.96 sec 21.28 > sec > bzip2 330 Mb (78.2%) 589.01 sec 232.13 > sec > shorten -c 10 326 Mb (77.4%) 91.39 sec 55.89 > sec > shorten -c 300 313 Mb (74.3%) 102.08 sec 53.04 > sec > flac 5chn 328 Mb (77.8%) 131.22 sec 66.54 > sec > flac 8chn 332 Mb (78.7%) 139.57 sec 66.74 > sec > flac 10chn 327 Mb (77.5%) 126.32 sec n/a > secIn every case I've ever seen, default FLAC compresses more than default shorten. The anomaly here is due to the number of channels; it appears that you really do have 300 uncorrelated channels of data (uncorrelated according to the linear predictor I mean). FLAC only supports 8 channels, but I would bet that the compression would be a lot better if you separated the 300 channels stream into several streams (<=8 channels each) and compressed them with FLAC. That may not be practical for your application.> Note the 10 channel version was done by me changing the format.h file > and recompiling, but alas I couldn't get decompression to go :(Yes, the hard limit of 8 channels is binding everywhere; it would take some more hacking at the code to make it handle more channels.> The command line I used for flac was > /usr/bin/time flac --lax --endian=little --channels=5 --bps=16 > --sample-rate=64320 --sign=signed --force-raw-format reallyraw -o > reallyraw.flac-5chn > > I needed the --lax otherwise it generated a > FLAC__STREAM_ENCODER_NOT_STREAMABLE error.Yes, by default the encoder tries to encode according to a subset of parameters that makes for easier decoding. You need --lax to turn that off.> I am basically letting you guys know about a possibly more > interesting > than normal application for your code and soliciting any tips you may > have :)You could try splitting the signal like I mentioned. It might take some experimenting to find channel combinations that have some exploitable correlation. See the format page for the kinds of inter-channel and intra-channel decorrelation FLAC does, it might give you some more ideas: http://flac.sourceforge.net/format.html Josh __________________________________________________ Do you Yahoo!? Yahoo! Web Hosting - Let the expert host your site http://webhosting.yahoo.com
On Wed, 2002-11-20 at 16:56, Josh Coalson wrote:> In every case I've ever seen, default FLAC compresses more > than default shorten. The anomaly here is due to the number > of channels; it appears that you really do have 300 uncorrelated > channels of data (uncorrelated according to the linear predictor > I mean). FLAC only supports 8 channels, but I would bet that > the compression would be a lot better if you separated the 300 > channels stream into several streams (<=8 channels each) and > compressed them with FLAC. That may not be practical for your > application.Hmm, well it would be a bit of a pain :) Might be an interesting thing to try though.> > I needed the --lax otherwise it generated a > > FLAC__STREAM_ENCODER_NOT_STREAMABLE error. > > Yes, by default the encoder tries to encode according to a subset > of parameters that makes for easier decoding. You need --lax to > turn that off.Ahh, I see.> You could try splitting the signal like I mentioned. It might > take some experimenting to find channel combinations that have > some exploitable correlation. See the format page for the kinds > of inter-channel and intra-channel decorrelation FLAC does, it > might give you some more ideas:OK, the format description explains a bit more. Thanks :) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 9A8C 569F 685A D928 5140 AE4B 319B 41F4 5D17 FDD5