flac dev - Apr 2012 - Git branch with compiling fixes for win32

From: Erik de Castro Lopo <mle+la at
mega-nerd.com>
>To: flac-dev at xiph.org 
>Cc: Josh Coalson <xflac at yahoo.com> 
>Sent: Wednesday, April 25, 2012 4:42 PM
>Subject: Re: [flac-dev] [Flac-dev] Git branch with compiling fixes for win32
> 
>Josh Coalson wrote:
>
>> But regardless of submitter, any patch that affects encoding must be
>> reviewed very carefully, preferably by several other people and
definitely
>> me.
>
>Is there any way of encoding this manual review process in the test suite
>so that people hacking on FLAC can immediately see that soemthing is wrong
>before even attempting to submit a patch (assuming they run the test suite)?
>
>Erik

Part of the reason the current test suite is so long is to try and discover
those problems automatically.? But it's not possible to be exhaustive
simply because new code may not be covered by the test suite.

Encoder bugs can be very subtle and sometimes it takes someone
with good knowledge of the format to notice.

Josh Coalson wrote:
> From: Erik de Castro Lopo <mle+la at mega-nerd.com>
> >To: flac-dev at xiph.org 
> >Cc: Josh Coalson <xflac at yahoo.com> 
> >Sent: Wednesday, April 25, 2012 4:42 PM
> >Subject: Re: [flac-dev] [Flac-dev] Git branch with compiling fixes for
win32
> > 
> >Josh Coalson wrote:
> >
> >> But regardless of submitter, any patch that affects encoding must
be
> >> reviewed very carefully, preferably by several other people and
definitely
> >> me.
> >
> >Is there any way of encoding this manual review process in the test
suite
> >so that people hacking on FLAC can immediately see that soemthing is
wrong
> >before even attempting to submit a patch (assuming they run the test
suite)?
> >
> >Erik
> 
> 
> Part of the reason the current test suite is so long is to try and discover
> those problems automatically.? But it's not possible to be exhaustive
> simply because new code may not be covered by the test suite.
> 
> Encoder bugs can be very subtle and sometimes it takes someone
> with good knowledge of the format to notice.
What exactly is the problem? New versions of the encoder producing files
that can't be decoded by older decoders or something else?

Erik
-- 
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

Largest risks are:

1) encoding errors in a scenario not exercised by the tests
2) violations of the spec that are tolerated by the decoder.  I've seen this
before with, e.g. x264, where a bug is repeated in the encoder and decoder and
hence not caught by any tests.

On Apr 26, 2012, at 2:49 AM, Erik de Castro Lopo <mle+la at mega-nerd.com>
wrote:
> Josh Coalson wrote:
> 
>> From: Erik de Castro Lopo <mle+la at mega-nerd.com>
>>> To: flac-dev at xiph.org 
>>> Cc: Josh Coalson <xflac at yahoo.com> 
>>> Sent: Wednesday, April 25, 2012 4:42 PM
>>> Subject: Re: [flac-dev] [Flac-dev] Git branch with compiling fixes
for win32
>>> 
>>> Josh Coalson wrote:
>>> 
>>>> But regardless of submitter, any patch that affects encoding
must be
>>>> reviewed very carefully, preferably by several other people and
definitely
>>>> me.
>>> 
>>> Is there any way of encoding this manual review process in the test
suite
>>> so that people hacking on FLAC can immediately see that soemthing
is wrong
>>> before even attempting to submit a patch (assuming they run the
test suite)?
>>> 
>>> Erik
>> 
>> 
>> Part of the reason the current test suite is so long is to try and
discover
>> those problems automatically.  But it's not possible to be
exhaustive
>> simply because new code may not be covered by the test suite.
>> 
>> Encoder bugs can be very subtle and sometimes it takes someone
>> with good knowledge of the format to notice.
> 
> What exactly is the problem? New versions of the encoder producing files
> that can't be decoded by older decoders or something else?
> 
> Erik
> -- 
> ----------------------------------------------------------------------
> Erik de Castro Lopo
> http://www.mega-nerd.com/
> _______________________________________________
> flac-dev mailing list
> flac-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/flac-dev

On 12-04-25 5:11 PM, Josh Coalson wrote:
> Part of the reason the current test suite is so long is to try and discover
> those problems automatically.  But it's not possible to be exhaustive
> simply because new code may not be covered by the test suite.
Coverage of the test suite is something the test suite could be extended
to measure.
> Encoder bugs can be very subtle and sometimes it takes someone
> with good knowledge of the format to notice.
Sadly, this remains true anyway. It's best to have both automatic and
human checks. Both are fallible.

 -r