flac dev - Oct 2024 - C API: How to get a seektable for very long files?

Op di 15 okt 2024 om 21:27 schreef Alistair Buxton <a.j.buxton at
gmail.com>:
>
> I would like to see this kind of thing put into a secondary metadata block
aimed specifically at SDR. This could be completely ignored by regular audio
players - these files are not meant to be listened to anyway. I could probably
figure out how to implement that, I even started looking into it once, but I
realised that 1. nobody would adopt it if it is just me behind it and 2. I
don't know enough to make it suitable for all use cases. So I cannot and
should not do this alone, as it would just be yet another half-baked adhoc
"standard" that causes more problems than it solves. For example I had
not even considered the idea of using multiple files.
>
But I don't know a thing about SDR, so I can't make a standard either.
I can propose an approach however.

First of all, it seems a good idea to make this extensible. I'd say
assigning a application metadata block ID to contain XML (either
general or specific to SDR) and setting up some form of standard is
relatively straightforward: there are a lot of XML parsers available
for a lot of languages, and XML is very well extensible.

You can also use vorbis comments. That would mean no structure, but
easier to do one step at a time. For example, one could set a tag
SAMPLE_RATE=40000000 to convey a sample rate of 40 MHz. Same thing for
the number of samples and actual bit depth (before padding). Of
course, structured things like seektables won't work that way, but
(compressed) XML would. On the other hand, seeing the seektable sizes
you mention, it seems quite a burden to XML parsers.

In your case, where you need to skip a lot, I'd say you need to only
store the frame sizes, compress them (streaming) with something like
deflate or xz, and use the data (decompressed in a streaming fashion
as well) to instruct skip ahead frame by frame.

Op wo 16 okt 2024 om 00:53 schreef Stefan Oltmanns <stefan-oltmanns at
gmx.net>:
>
> I think I have an even better idea now that definitely won't break any
> decoder:
> -Create a normal last seekpoint pointing to the very last frame. Total
> samples should be that seekpoint + frame length
>
> To indicate that the last seekpoint is pointing to the last frame, there
> should be two placeholders:
> First has the offset set to ASCII "TOTALSMP", the next one all
zeros.
> That some encoder adds placeholder seekpoints with that pattern just by
> chance is almost 0.
>
> When the decoder scans the seekpoints, the total_samples is 0 and if it
> detects that two placeholders it will set total_samples to that value.
>
> I don't see an issue with that? Old files are treated by new decoder
> just like before, new files are treated by old decoder also just like
> before, only new file and new decoder will result in that feature.
> And this only affects files with more than
>
But how should such a number be presented to the libFLAC user? You
suggested overwriting the streaminfo total_samples number, but
streaminfo always precedes the seektable, so the streaminfo metadata
block is already parsed by the application before the seektable is
even read. Also, I think it is quite hacky to not pass the actual
streaminfo metadata block to the application but 'patch' it first.
Furthermore, applications might rely on the total number of samples
being 36 bits, storing the number of seconds in a 32-bit int after
dividing it by the samplerate for example. All in all, this seems like
a bad idea.
> > So, adding an API function that applications need to adopt is a
feature,
> > not a bug. I really don't want to modify existing behavior when it
is not
> > strictly necessary.
> >
>
> I'm not saying a new API call is a bug. But as this is such a niche
> case, that I don't think many programs will incorporate that feature,
> because the need is not seen (and being able to open the file in an
> audio player for quick check and random points is a great feature). If
> libflac just returns the correct total sample value you can just
> recompile libflac on your system and it will work.
> Additionally I think there is one issue: there are only 31 bits for the
> frame number, with a frame size of 4096 this is "just" 2^43
samples,
> which maybe not enough in every case (SDR with higher sample rate?), and
> smaller frame sizes are possible. And for variable block length it's
> again 2^36.
> Of course in combination with the last seekpoint this could work, but
> without a seektable it's again problematic.
>
I still don't understand **why** you want FLAC files containing radio
frequency signals to be parsable by audio applications. I get that it
is convenient for debugging, but is there any use other than that?
Would you like to listen to such signals as if they were audio?

I'd prefer a SDR-specific metadata standard. That would mean setting
the sample rate to 0 (which is defined by the FLAC spec to mean: does
not contain audio), and set that samplerate in either an application
metadata block or vorbis comment fields. That way audio players won't
attempt to playback such non-audio files, which I think is a good
thing.

Am 16.10.24 um 15:15 schrieb Martijn van Beurden:
>
> But how should such a number be presented to the libFLAC user? You
> suggested overwriting the streaminfo total_samples number, but
> streaminfo always precedes the seektable, so the streaminfo metadata
> block is already parsed by the application before the seektable is
> even read. Also, I think it is quite hacky to not pass the actual
> streaminfo metadata block to the application but 'patch' it first.
> Furthermore, applications might rely on the total number of samples
> being 36 bits, storing the number of seconds in a 32-bit int after
> dividing it by the samplerate for example. All in all, this seems like
> a bad idea.
Yes, overwriting the streaminfo total_samples number is a bit of a hack,
but it would only affect those files with more than 2^36 samples that
contains the special seek point, so no difference for the average flac user.

Not sure how the applications actually read flac files, if they start
using the total_samples number before the seektable is parsed or if
libflac can indicate that the streaminfo is not yet complete.

The worst thing that could happen:
-An application uses the total_samples number before parsing of
seektable is done -> no difference to now
-An application uses a 32 bit number for seconds: At 40 kHz indicated
sample rate that is still more than 2^47 samples. At 40 MHz real sample
rate this is still more than 40 days.
If that's really a concern libflac could check the number first if the
seconds fit in a 32 bit int and otherwise not set it
>>
>> I'm not saying a new API call is a bug. But as this is such a niche
>> case, that I don't think many programs will incorporate that
feature,
>> because the need is not seen (and being able to open the file in an
>> audio player for quick check and random points is a great feature). If
>> libflac just returns the correct total sample value you can just
>> recompile libflac on your system and it will work.
>> Additionally I think there is one issue: there are only 31 bits for the
>> frame number, with a frame size of 4096 this is "just" 2^43
samples,
>> which maybe not enough in every case (SDR with higher sample rate?),
and
>> smaller frame sizes are possible. And for variable block length
it's
>> again 2^36.
>> Of course in combination with the last seekpoint this could work, but
>> without a seektable it's again problematic.
>>
>
> I still don't understand **why** you want FLAC files containing radio
> frequency signals to be parsable by audio applications. I get that it
> is convenient for debugging, but is there any use other than that?
> Would you like to listen to such signals as if they were audio?
>
> I'd prefer a SDR-specific metadata standard. That would mean setting
> the sample rate to 0 (which is defined by the FLAC spec to mean: does
> not contain audio), and set that samplerate in either an application
> metadata block or vorbis comment fields. That way audio players won't
> attempt to playback such non-audio files, which I think is a good
> thing.
The main reason is not listening to those files (but that's not only for
debugging, but very helpful just to quick check if there is a signal or
just silence/noise), but that you can use audio tools for inspecting
(waveform or spectrum/fft), cutting, low/high/band pass etc.

I would suspect most audio tools would refuse to work if sample rate is
set to 0 and that is definetly a bad idea, because even though the
content is not audio, it is a sampled signal with a fixed sample rate
like audio and not other kind of data, so using audio tools makes
perfect sense.

That's why I would want all audio tools to be able to work with long
flac files.

Best regards
Stefan