theora dev - Sep 2009 - Request for input on the preparation of a subjective video quality comparative study

Hi there,

First of all THANK YOU for putting all the effort on developing Theora! and
all the other projects of xiph, of course.

I am planning to conduct together with my students a subjective video
quality comparative study of video codecs.

The plan is to ask our friends, family and others to participate so that
we will be looking towards a minimum of 100 participants.

The experiments are going to start sometime towards the beginning of November.
I suppose we will be able to use Theora 1.1 final by then, if not we will use
a build from SVN.

I would like to have your input on:

1. which codecs would you be interested in including in the study,
2. what encoding parameters should be used,
3. advise on source video material,
4. software that can be used for running the tests,
5. anything else you would like to give me input about.

If you want to include in the study alternative implementations or setups of
the theora encoder I will be able to include them.


The way I envision it is that we will distribute DVDs to the participants
which will include the test video clips and an executable that will provide
a GUI for going through the experimental procedure so that the participants
can run the test on their own computer setups. This will introduce an
uncontrolled variable but it will be nevertheless more realistic than
running all the experiments in one of our labs with the specific computer
setup that we happen to use for our purposes.

I would be happy to have your opinions, advice, views or anything else you
wish.


John

John Pisokas wrote:
> I would be happy to have your opinions, advice, views or anything else you
> wish.
I assume you are consulting documents such as ITU-R BT.500-11
"Methodology for the subjective assessment of the quality of television
pictures," which contain advice on setting up and administering the
tests, selection of test material, etc. ITU-T SG 12 COM 12-67 "VQEG
subjective test plan" may also be informative. DSCQS is generally
regarded as the most reliable evaluation method, and typically involves
short, 10s test clips. A wide variety of suitable clips are available at
http://media.xiph.org/video/derf/ and more will be added soon.

Varying viewing conditions is going to make comparison of results much
more difficult, and will require many more subjects to get statistically
significant results, if that is even possible. Even using the same
software on multiple platforms can produce very different results
(famously, even on the same platform, if you open a video in Windows
Media Player, pause it, and then open a second copy of the same video,
the second copy will often have brightness and contrast levels very
different from the first).

If you are comparing multiple codecs, you must also ensure that the
software is configured appropriately to ensure a fair comparison. For
example, post-processing is seldom enabled for Theora (the current
implementation is mostly unoptimized and fairly expensive at high
resolutions), but often enabled for MPEG4 codecs. Enabling
post-processing is not generally useful for evaluating the quality of a
format, as most codecs could equally well take advantage of the
post-processing methods of another codec. One possibility is to use only
pre-decoded, uncompressed videos produced with a carefully controlled
toolchain, however this will have difficulty fitting on a single DVD for
even a moderate number of sequences and codecs.

We did a similar, but very subjective test like this for the LiVES
project a couple of years ago. It would interesting to see if your
results correlate with our findings.

We are particularly interested in comparing free codecs against
proprietary ones.

I would suggest at least the following codecs for video:

- theora
- h264
- dirac
- snow
- wmv
- xvid

and for audio:

- vorbis
- mp3
- aac

if you are interested in the container formats, I would suggest:

- avi
- ogg
- matroska
- nut


Regards,
Salsaman.
http://lives.sourceforge.net



On Sun, Sep 20, 2009 at 2:40 PM, John Pisokas <john.pisokas at gmail.com>
wrote:
> Hi there,
>
> First of all THANK YOU for putting all the effort on developing Theora! and
> all the other projects of xiph, of course.
>
> I am planning to conduct together with my students a subjective video
> quality comparative study of video codecs.
>
> The plan is to ask our friends, family and others to participate so that
> we will be looking towards a minimum of 100 participants.
>
> The experiments are going to start sometime towards the beginning of
November.
> I suppose we will be able to use Theora 1.1 final by then, if not we will
use
> a build from SVN.
>
> I would like to have your input on:
>
> 1. which codecs would you be interested in including in the study,
> 2. what encoding parameters should be used,
> 3. advise on source video material,
> 4. software that can be used for running the tests,
> 5. anything else you would like to give me input about.
>
> If you want to include in the study alternative implementations or setups
of
> the theora encoder I will be able to include them.
>
>
> The way I envision it is that we will distribute DVDs to the participants
> which will include the test video clips and an executable that will provide
> a GUI for going through the experimental procedure so that the participants
> can run the test on their own computer setups. This will introduce an
> uncontrolled variable but it will be nevertheless more realistic than
> running all the experiments in one of our labs with the specific computer
> setup that we happen to use for our purposes.
>
> I would be happy to have your opinions, advice, views or anything else you
> wish.
>
>
> John
> _______________________________________________
> theora-dev mailing list
> theora-dev at xiph.org
> http://lists.xiph.org/mailman/listinfo/theora-dev
>