flac dev - Oct 2024 - C API: How to get a seektable for very long files?

Op di 15 okt. 2024 16:18 schreef Stefan Oltmanns <stefan-oltmanns at
gmx.net>:
>
> I see, but that would require changes in the software using libflac and
> require the file to be easily seekable to be able to skip to the end and
> depending on how far away the seek points are, it could take a while.
>
No, seeking to a specific sample can take a while because of all the
back-and-forth, but seeking to almost the end of the file is very quick.

I know this is not the cleanest way, but as this only for the rare
cases
> with more than 2^36 samples, this should not affect a lot of people.
>
There are already a lot of applications reading FLAC, with and without
libFLAC. Changing behavior and the spec like this is going to break some of
them, because the changes you propose also affect files with fewer samples.
total samples being 0 can and does happen for files that were created by a
streaming encoder for example.

So, adding an API function that applications need to adopt is a feature,
not a bug. I really don't want to modify existing behavior when it is not
strictly necessary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/flac-dev/attachments/20241015/01a97b66/attachment.htm>

Another SDR user here. It was me who reported the bug where total samples
wraps around on overflow.

FLAC performs extremely well on SDR samples, both speed and compression
ratio. In my testing it outperforms any other free lossless codec by a
large margin, being 20% smaller and 10% faster than the next best (which
was ffv1). The problem is the metadata, and not just total samples. We also
can't put true values in the sample rate field because it doesn't have
enough bits (I have files with 35468950MHz nominal sample rate for
example), and there is no way to record that samples have been padded eg
from 10 bits to 16 bits, which seems to be very common in SDR applications.
These are just two examples off the top of my head - there are probably
more.

The problems around total samples and seek table allocation could be
alleviated by using multiple files as mentioned previously, but that
introduces a new problem: how do you know if you have the full set of
files? How do you know which file contains the nth sample? This would still
require extra metadata somewhere.

I would like to see this kind of thing put into a secondary metadata block
aimed specifically at SDR. This could be completely ignored by regular
audio players - these files are not meant to be listened to anyway. I could
probably figure out how to implement that, I even started looking into it
once, but I realised that 1. nobody would adopt it if it is just me behind
it and 2. I don't know enough to make it suitable for all use cases. So I
cannot and should not do this alone, as it would just be yet another
half-baked adhoc "standard" that causes more problems than it solves.
For
example I had not even considered the idea of using multiple files.

On the topic of seeking, it is also a problem for my specific use case. I
am interested in digital signals (teletext) hidden in analog video (CVBS
samples), and they just aren't sequential in the same way that audio and
video usually are. I need to seek to an arbitrary video line as fast as
possible, preferably in constant time. A line is usually 2048 samples but
can vary between 1000 and 3000 samples long (always fixed size within a
given file, but can vary depending on the hardware used to record). A
typical recording will have 32 lines per frame (because the picture has
already been discarded), 25 frames per second, and be up to 12 hours long 35
million lines and 70 billion samples. That would result in a ~700MB seek
table. And making the block size = 1 line also has a negative effect on
compression ratio. Finally since the files are huge I am storing them on
cheap USB HDDs which have terrible seek and read times to begin with. Note
though that very high resolution seeking is not really necessary for most
SDR uses and is mainly a quirk of the fact that teletext was one of the
earliest attempts at digital data transmission and has very small and fixed
size data packets that map directly to video lines. As such this would
probably be best handled with a custom, app-specific seek table in a
separate file, built on-demand and possibly stored on a faster disk.

-- 
Alistair Buxton
a.j.buxton at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.xiph.org/pipermail/flac-dev/attachments/20241015/3f6533e5/attachment.htm>

Am 15.10.24 um 19:03 schrieb Martijn van Beurden:
>
> No, seeking to a specific sample can take a while because of all the
> back-and-forth, but seeking to almost the end of the file is very quick.
>
> I know this is not the cleanest way, but as this only for the rare cases
Ah, I see, because the frame header also include either the sample
number (variable frame size) or the frame number (fixed frame size). I
assumed the stream would have to be decoded from the point of the last
seekpoint to the end
>
> There are already a lot of applications reading FLAC, with and without
> libFLAC. Changing behavior and the spec like this is going to break some of
> them, because the changes you propose also affect files with fewer samples.
> total samples being 0 can and does happen for files that were created by a
> streaming encoder for example.
>
I think I have an even better idea now that definitely won't break any
decoder:
-Create a normal last seekpoint pointing to the very last frame. Total
samples should be that seekpoint + frame length

To indicate that the last seekpoint is pointing to the last frame, there
should be two placeholders:
First has the offset set to ASCII "TOTALSMP", the next one all zeros.
That some encoder adds placeholder seekpoints with that pattern just by
chance is almost 0.

When the decoder scans the seekpoints, the total_samples is 0 and if it
detects that two placeholders it will set total_samples to that value.

I don't see an issue with that? Old files are treated by new decoder
just like before, new files are treated by old decoder also just like
before, only new file and new decoder will result in that feature.
And this only affects files with more than
> So, adding an API function that applications need to adopt is a feature,
> not a bug. I really don't want to modify existing behavior when it is
not
> strictly necessary.
>
I'm not saying a new API call is a bug. But as this is such a niche
case, that I don't think many programs will incorporate that feature,
because the need is not seen (and being able to open the file in an
audio player for quick check and random points is a great feature). If
libflac just returns the correct total sample value you can just
recompile libflac on your system and it will work.
Additionally I think there is one issue: there are only 31 bits for the
frame number, with a frame size of 4096 this is "just" 2^43 samples,
which maybe not enough in every case (SDR with higher sample rate?), and
smaller frame sizes are possible. And for variable block length it's
again 2^36.
Of course in combination with the last seekpoint this could work, but
without a seektable it's again problematic.

Best regards
Stefan