flac dev - Jun 2021 - Autocorrelation precision insufficient

Hi Martijn,

Great work!

I concur with David that compression should work the same on all 
platforms. There can be SIMD optimizations for some routines, but they 
should produce the same result as the non-SIMD versions. If outdated 
optimizations cannot be updated, better drop them instead of keeping 
versions that produce different results than the new reference code.

Regarding VSX, you could try to contact Anton Blanchard to ask whether 
he can update the routines. Otherwise, it should be possible to emulate 
VSX with QEMU. You won't be able to verify performance that way, of 
course, but should be able to verify correctness.

Best,
Robert

Am 25.06.2021 um 19:55 schrieb David Willmore:
> Martijn,
>
> Just a lone voice here, but I would opt for the best compression on 
> the most platforms.? As I am one of the people who always run FLAC at 
> the placebo max compression, my vote will always be for best 
> compression over any other resource use.
>
> I also can make ARMv7 hardware available to you or I can do any 
> testing you want on a number of machines.
>
> Thank you for your work!
>
> Cheers,
> David
>
> On Fri, Jun 25, 2021 at 3:48 AM Martijn van Beurden <mvanb1 at
gmail.com>
> wrote:
>
>     Op do 24 jun. 2021 om 09:17 schreef Martijn van Beurden
>     <mvanb1 at gmail.com>:
>     > - Switch to autoc[] as doubles, but remove current SSE and
>     disable VSX
>     > intrinsics for someone to update them later (I don't have any
POWER8
>     > or POWER9 hardware to test). This means all platforms will get the
>     > same compression, but some (with only SSE but not SSE2 or with
VSX)
>     > will see a large slowdown.
>
>     I see now that besides routines with SSE intrinsics (which I rewrote
>     into SSE2) and with VSX intrinsics (which I don't have hardware
for)
>     there is also a open pull request for routines with ARM intrinsics. I
>     am willing and able to rewrite those if this change is accepted and
>     merged. I have access to ARMv8 with 32-bit OS, ARMv8 with 64-bit OS,
>     ARMv6 and I might be able to get hold of ARMv7 hardware.
>     _______________________________________________
>     flac-dev mailing list
>     flac-dev at xiph.org
>     http://lists.xiph.org/mailman/listinfo/flac-dev
>
>
> _______________________________________________
> flac-dev mailing list
> flac-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/flac-dev

Op vr 25 jun. 2021 om 23:22 schreef Robert Kausch <robert.kausch at
freac.org>:
> Regarding VSX, you could try to contact Anton Blanchard to ask whether
> he can update the routines. Otherwise, it should be possible to emulate
> VSX with QEMU. You won't be able to verify performance that way, of
> course, but should be able to verify correctness.
Using QEMU is an excellent suggestion, I was surprised how easy
user-mode emulation works. However, cross-compiling to powerpc64le and
using qemu results in a binary that doesn't pass make check. I don't
know whether that is a bug in qemu or the cross-compiler, but it
doesn't seem to be a bug in FLAC, because it runs make check fine in
Travis CI. Maybe I'll try building something and relying on Travis to
check my work. Apparently building for ppc64le target is sponsored by
IBM and thus free.

Playing Devil's Advocate here, but isn't the point of compression
standards like FLAC that encoders can vary quite a bit, so long as decompression
always produces the correct results? In other words, I think it might be taking
it too far to require that the encoder produce identical results on every
processor, at least if the decompressed audio is correct.

To counter myself on this topic, I do think it's important to maintain any
distinction between levels (and I'm not sure how that's defined). I have
the Sound Device 702 recorder, and it can only play back FLAC files with low to
moderate compression. The recorder itself can only compress moderately -
certainly nowhere near --best -9

I'd say that it's important to maintain compatibility with players that
have limited capabilities. I don't actually know which levels of compression
the SD702 supports, and which levels are too extreme, but whatever the threshold
is, then I would expect that future versions of FLAC would not stop being
compatible with the SD702 when the same compression level is used.

My apologies that I do not have more details on player limitations, but my point
is that compatibility with players should be required, while producing identical
compressed bit streams might not necessarily be required.

I've also never understood the mathematics behind a compression format where
the encoder can vary immensely in quality - even to the point of exceeding
original compression ratios - but the decoder remains compatible. Witness MP3,
where they improved the algorithm of the encoder and produced smaller files, all
without requiring an update to the decoder. I'm assuming that FLAC fits that
same model.

Brian Willoughby


On Jun 25, 2021, at 14:15, Robert Kausch <robert.kausch at freac.org>
wrote:
> I concur with David that compression should work the same on all platforms.
There can be SIMD optimizations for some routines, but they should produce the
same result as the non-SIMD versions. If outdated optimizations cannot be
updated, better drop them instead of keeping versions that produce different
results than the new reference code.
> 
> Regarding VSX, you could try to contact Anton Blanchard to ask whether he
can update the routines. Otherwise, it should be possible to emulate VSX with
QEMU. You won't be able to verify performance that way, of course, but
should be able to verify correctness.
> 
> Best,
> Robert
> 
> Am 25.06.2021 um 19:55 schrieb David Willmore:
>> Martijn,
>> 
>> Just a lone voice here, but I would opt for the best compression on the
most platforms.  As I am one of the people who always run FLAC at the placebo
max compression, my vote will always be for best compression over any other
resource use.
>> 
>> I also can make ARMv7 hardware available to you or I can do any testing
you want on a number of machines.
>> 
>> Thank you for your work!
>> 
>> Cheers,
>> David
>> 
>> On Fri, Jun 25, 2021 at 3:48 AM Martijn van Beurden <mvanb1 at
gmail.com> wrote:
>>>    Op do 24 jun. 2021 om 09:17 schreef Martijn van Beurden
>>>    <mvanb1 at gmail.com>:
>>>    > - Switch to autoc[] as doubles, but remove current SSE and
>>>    disable VSX
>>>    > intrinsics for someone to update them later (I don't
have any POWER8
>>>    > or POWER9 hardware to test). This means all platforms will
get the
>>>    > same compression, but some (with only SSE but not SSE2 or
with VSX)
>>>    > will see a large slowdown.
>>> 
>>>    I see now that besides routines with SSE intrinsics (which I
rewrote
>>>    into SSE2) and with VSX intrinsics (which I don't have
hardware for)
>>>    there is also a open pull request for routines with ARM
intrinsics. I
>>>    am willing and able to rewrite those if this change is accepted
and
>>>    merged. I have access to ARMv8 with 32-bit OS, ARMv8 with 64-bit
OS,
>>>    ARMv6 and I might be able to get hold of ARMv7 hardware.
>