flac dev - Oct 2024 - C API: How to get a seektable for very long files?

Am 15.10.24 um 19:03 schrieb Martijn van Beurden:
>
> No, seeking to a specific sample can take a while because of all the
> back-and-forth, but seeking to almost the end of the file is very quick.
>
> I know this is not the cleanest way, but as this only for the rare cases
Ah, I see, because the frame header also include either the sample
number (variable frame size) or the frame number (fixed frame size). I
assumed the stream would have to be decoded from the point of the last
seekpoint to the end
>
> There are already a lot of applications reading FLAC, with and without
> libFLAC. Changing behavior and the spec like this is going to break some of
> them, because the changes you propose also affect files with fewer samples.
> total samples being 0 can and does happen for files that were created by a
> streaming encoder for example.
>
I think I have an even better idea now that definitely won't break any
decoder:
-Create a normal last seekpoint pointing to the very last frame. Total
samples should be that seekpoint + frame length

To indicate that the last seekpoint is pointing to the last frame, there
should be two placeholders:
First has the offset set to ASCII "TOTALSMP", the next one all zeros.
That some encoder adds placeholder seekpoints with that pattern just by
chance is almost 0.

When the decoder scans the seekpoints, the total_samples is 0 and if it
detects that two placeholders it will set total_samples to that value.

I don't see an issue with that? Old files are treated by new decoder
just like before, new files are treated by old decoder also just like
before, only new file and new decoder will result in that feature.
And this only affects files with more than
> So, adding an API function that applications need to adopt is a feature,
> not a bug. I really don't want to modify existing behavior when it is
not
> strictly necessary.
>
I'm not saying a new API call is a bug. But as this is such a niche
case, that I don't think many programs will incorporate that feature,
because the need is not seen (and being able to open the file in an
audio player for quick check and random points is a great feature). If
libflac just returns the correct total sample value you can just
recompile libflac on your system and it will work.
Additionally I think there is one issue: there are only 31 bits for the
frame number, with a frame size of 4096 this is "just" 2^43 samples,
which maybe not enough in every case (SDR with higher sample rate?), and
smaller frame sizes are possible. And for variable block length it's
again 2^36.
Of course in combination with the last seekpoint this could work, but
without a seektable it's again problematic.

Best regards
Stefan

Op di 15 okt 2024 om 21:27 schreef Alistair Buxton <a.j.buxton at
gmail.com>:
>
> I would like to see this kind of thing put into a secondary metadata block
aimed specifically at SDR. This could be completely ignored by regular audio
players - these files are not meant to be listened to anyway. I could probably
figure out how to implement that, I even started looking into it once, but I
realised that 1. nobody would adopt it if it is just me behind it and 2. I
don't know enough to make it suitable for all use cases. So I cannot and
should not do this alone, as it would just be yet another half-baked adhoc
"standard" that causes more problems than it solves. For example I had
not even considered the idea of using multiple files.
>
But I don't know a thing about SDR, so I can't make a standard either.
I can propose an approach however.

First of all, it seems a good idea to make this extensible. I'd say
assigning a application metadata block ID to contain XML (either
general or specific to SDR) and setting up some form of standard is
relatively straightforward: there are a lot of XML parsers available
for a lot of languages, and XML is very well extensible.

You can also use vorbis comments. That would mean no structure, but
easier to do one step at a time. For example, one could set a tag
SAMPLE_RATE=40000000 to convey a sample rate of 40 MHz. Same thing for
the number of samples and actual bit depth (before padding). Of
course, structured things like seektables won't work that way, but
(compressed) XML would. On the other hand, seeing the seektable sizes
you mention, it seems quite a burden to XML parsers.

In your case, where you need to skip a lot, I'd say you need to only
store the frame sizes, compress them (streaming) with something like
deflate or xz, and use the data (decompressed in a streaming fashion
as well) to instruct skip ahead frame by frame.

Op wo 16 okt 2024 om 00:53 schreef Stefan Oltmanns <stefan-oltmanns at
gmx.net>:
>
> I think I have an even better idea now that definitely won't break any
> decoder:
> -Create a normal last seekpoint pointing to the very last frame. Total
> samples should be that seekpoint + frame length
>
> To indicate that the last seekpoint is pointing to the last frame, there
> should be two placeholders:
> First has the offset set to ASCII "TOTALSMP", the next one all
zeros.
> That some encoder adds placeholder seekpoints with that pattern just by
> chance is almost 0.
>
> When the decoder scans the seekpoints, the total_samples is 0 and if it
> detects that two placeholders it will set total_samples to that value.
>
> I don't see an issue with that? Old files are treated by new decoder
> just like before, new files are treated by old decoder also just like
> before, only new file and new decoder will result in that feature.
> And this only affects files with more than
>
But how should such a number be presented to the libFLAC user? You
suggested overwriting the streaminfo total_samples number, but
streaminfo always precedes the seektable, so the streaminfo metadata
block is already parsed by the application before the seektable is
even read. Also, I think it is quite hacky to not pass the actual
streaminfo metadata block to the application but 'patch' it first.
Furthermore, applications might rely on the total number of samples
being 36 bits, storing the number of seconds in a 32-bit int after
dividing it by the samplerate for example. All in all, this seems like
a bad idea.
> > So, adding an API function that applications need to adopt is a
feature,
> > not a bug. I really don't want to modify existing behavior when it
is not
> > strictly necessary.
> >
>
> I'm not saying a new API call is a bug. But as this is such a niche
> case, that I don't think many programs will incorporate that feature,
> because the need is not seen (and being able to open the file in an
> audio player for quick check and random points is a great feature). If
> libflac just returns the correct total sample value you can just
> recompile libflac on your system and it will work.
> Additionally I think there is one issue: there are only 31 bits for the
> frame number, with a frame size of 4096 this is "just" 2^43
samples,
> which maybe not enough in every case (SDR with higher sample rate?), and
> smaller frame sizes are possible. And for variable block length it's
> again 2^36.
> Of course in combination with the last seekpoint this could work, but
> without a seektable it's again problematic.
>
I still don't understand **why** you want FLAC files containing radio
frequency signals to be parsable by audio applications. I get that it
is convenient for debugging, but is there any use other than that?
Would you like to listen to such signals as if they were audio?

I'd prefer a SDR-specific metadata standard. That would mean setting
the sample rate to 0 (which is defined by the FLAC spec to mean: does
not contain audio), and set that samplerate in either an application
metadata block or vorbis comment fields. That way audio players won't
attempt to playback such non-audio files, which I think is a good
thing.