theora - Aug 2009 - Sampling encoder timings between libtheora-1.0 & libtheora-1.1beta3

As requested by Ralph in my introduction e-mail to this list [1], I've
gone ahead and done some sample test timings for encoding between
libtheora-1.0 and libtheora-1.1beta3, as available from the Theora
website [2].

So far, I've tested with 4 different files that I generated with content
I already have.  The files, in brief, are the following:

   1. planet-earth-360x240.yuv4mpeg - A clip from the Discovery/BBC
      series Planet Earth from DVD 1, the "Mountains" chapter - the
      first 5000 frames (I dropped half the fields to get non-interlaced
      video, then rescaled)
   2. kuala-lumpur-downtime-timelapse.yuv4mpeg - A time-lapse video clip
      I created from 307 individual frames of the downtown skyline of
      Kuala Lumpur, Malaysia I took while staying there.  The resolution
      is "hd720" in FFmpeg parlance, or 1280x720, a.k.a. 720p.
   3. kuala-lumpur-afternoon-1-vga-30fps.yuv4mpeg - Another time-lapse
      video clip, this time from 1500 individual frames, taken from the
      balcony of the apartment I stayed in, again in Kuala Lumpur.  The
      resolution is 640x480.
   4. canon-powershot-a550.yuv4mpeg - A video clip taken with my Canon
      PowerShot A550 of the Gong Badak stadium in Kuala Terengganu
      shortly after it happened [3].  The video is 640x480.  The format
      was originally MJPEG with the quality boosted by the CHDK firmware
      add-on to about 49,386 kb/s.

All clips were converted to yuv4mpeg format and then copied to /dev/shm
to remove, as much as possible, any potential outside influences on
encoder performance.  This limited me to working with clips whose size
in yuv4mpeg format would be under approximately 750MB.  Also, no sound
was included, thus limiting this test solely to Theora encoding.

Commands used were of the form:

time ~/packages/libtheora-1.0/examples/encoder_example --output
/dev/null --video-quality 7 canon-powershot-A550.yuv4mpeg

I am presenting the best time out of three trials for each combination
of input file, video quality setting, and encoder version.  Time taken
is the first value returned by the "time" command, which is total
elapsed time from command start to command end.

All tests were run under Fedora 11, 32 bit, booted into single user mode
[4].

The hardware used was a Dell Inspiron 8600 with a 2GHz Pentium M
processor and 1,544,420 kB (according to /proc/meminfo) of RAM.

Okay okay!  Enough pedantism!  On to the results!

1. planet-earth-360x240.yuv4mpeg --video-quality 10

libtheora-1.0 - 1m11.486s
libtheora-1.1beta3 - 1m39.108s

2. planet-earth-360x240.yuv4mpeg --video-quality 7

libtheora-1.0 - 0m53.652s
libtheora-1.1beta3 - 1m21.661s

3. kuala-lumpur-downtown-timelapse-yuv4mpeg

libtheora-1.0 - 36.566s
libtheora-1.1beta3 - 48.982s

4. kuala-lumpur-balcony-afternoon-1-vga-30fps.yuv4mpeg

libtheora-1.0 - 0m33.185s
libtheora-1.1beta3 - 1m7.169s

5. canon-powershot-a550.yuv4mpeg

libtheora-1.0 - 1m17.176s
libtheora-1.1beta3 - 1m32.746s

I apologize for not presenting these in the best possible way - I simple
don't know what that would be!  Suggestions are welcome, though.  Also,
if anyone would like the samples I used (aside from the first file,
which I cannot legally share...yet), I am happy to do so.

Interesting to note are tests 4 & 5.  I don't know why the spread is so
great in #4 and so little in #5.  #4, on first appearances, is a lot
like #3, in that the bottom is mostly static and the top consists of
clouds moving.  #5 is taken from a tripod and consists of some close-up
and zoomed-out pans of the collapsed stadium.  Picture quality on #5 is
pretty good as it was quite a sunny day and the viewing conditions were
quite clear.

If there's a lot of interest in this testing methodology, I might put
together some scripts to make automating, timing, & reporting more
systematic.  These runs were all basically done manually on the
command-line.  I even hand-wrote the results down.  ;)

I intended to put up more of an analysis on my blog, but I am not sure
if I'll do that yet, as I'm planning to migrate my site to a new server,
so I prefer not to have new uploads just yet.

Constructive feedback will be most appreciated!

[1] http://lists.xiph.org/pipermail/theora/2009-August/002678.html
[2] http://www.theora.org/downloads/
[3]
http://www.basilgohar.com/blog/2009/06/05/sultan-mizan-zainal-abidin-stadium-aka-gong-badak-stadium-collapses
[4]
http://docs.fedoraproject.org/install-guide/f11/en-US/html/s1-rescuemode-booting-single.html

-- 
      Basil Mohamed Gohar
abu_hurayrah at hidayahonline.org
http://www.basilgohar.com/blog
basilgohar on irc.freenode.net
GPG Key Fingerprint:  5AF4B362

Thanks for your testing and reporting.  It would be helpful for me to know, in 
addition to the time it took to encode each sample, the time it takes to play 
each sample, the size of each sample prior to encoding, and the sizes of 
resulting files after encoding.
Regards,
John Kintree

On Saturday 29 August 2009 09:11 pm, Basil Mohamed Gohar
wrote:
> As requested by Ralph in my introduction e-mail to this list [1], I've
> gone ahead and done some sample test timings for encoding between
> libtheora-1.0 and libtheora-1.1beta3, as available from the Theora
> website [2].
>
> So far, I've tested with 4 different files that I generated with
content
> I already have.  The files, in brief, are the following:
>
>    1. planet-earth-360x240.yuv4mpeg - A clip from the Discovery/BBC
>       series Planet Earth from DVD 1, the "Mountains" chapter -
the
>       first 5000 frames (I dropped half the fields to get non-interlaced
>       video, then rescaled)
>    2. kuala-lumpur-downtime-timelapse.yuv4mpeg - A time-lapse video clip
>       I created from 307 individual frames of the downtown skyline of
>       Kuala Lumpur, Malaysia I took while staying there.  The resolution
>       is "hd720" in FFmpeg parlance, or 1280x720, a.k.a. 720p.
>    3. kuala-lumpur-afternoon-1-vga-30fps.yuv4mpeg - Another time-lapse
>       video clip, this time from 1500 individual frames, taken from the
>       balcony of the apartment I stayed in, again in Kuala Lumpur.  The
>       resolution is 640x480.
>    4. canon-powershot-a550.yuv4mpeg - A video clip taken with my Canon
>       PowerShot A550 of the Gong Badak stadium in Kuala Terengganu
>       shortly after it happened [3].  The video is 640x480.  The format
>       was originally MJPEG with the quality boosted by the CHDK firmware
>       add-on to about 49,386 kb/s.
>
> All clips were converted to yuv4mpeg format and then copied to /dev/shm
> to remove, as much as possible, any potential outside influences on
> encoder performance.  This limited me to working with clips whose size
> in yuv4mpeg format would be under approximately 750MB.  Also, no sound
> was included, thus limiting this test solely to Theora encoding.
>
> Commands used were of the form:
>
> time ~/packages/libtheora-1.0/examples/encoder_example --output
> /dev/null --video-quality 7 canon-powershot-A550.yuv4mpeg
>
> I am presenting the best time out of three trials for each combination
> of input file, video quality setting, and encoder version.  Time taken
> is the first value returned by the "time" command, which is total
> elapsed time from command start to command end.
>
> All tests were run under Fedora 11, 32 bit, booted into single user mode
> [4].
>
> The hardware used was a Dell Inspiron 8600 with a 2GHz Pentium M
> processor and 1,544,420 kB (according to /proc/meminfo) of RAM.
>
> Okay okay!  Enough pedantism!  On to the results!
>
> 1. planet-earth-360x240.yuv4mpeg --video-quality 10
>
> libtheora-1.0 - 1m11.486s
> libtheora-1.1beta3 - 1m39.108s
>
> 2. planet-earth-360x240.yuv4mpeg --video-quality 7
>
> libtheora-1.0 - 0m53.652s
> libtheora-1.1beta3 - 1m21.661s
>
> 3. kuala-lumpur-downtown-timelapse-yuv4mpeg
>
> libtheora-1.0 - 36.566s
> libtheora-1.1beta3 - 48.982s
>
> 4. kuala-lumpur-balcony-afternoon-1-vga-30fps.yuv4mpeg
>
> libtheora-1.0 - 0m33.185s
> libtheora-1.1beta3 - 1m7.169s
>
> 5. canon-powershot-a550.yuv4mpeg
>
> libtheora-1.0 - 1m17.176s
> libtheora-1.1beta3 - 1m32.746s
>
> I apologize for not presenting these in the best possible way - I simple
> don't know what that would be!  Suggestions are welcome, though.  Also,
> if anyone would like the samples I used (aside from the first file,
> which I cannot legally share...yet), I am happy to do so.
>
> Interesting to note are tests 4 & 5.  I don't know why the spread
is so
> great in #4 and so little in #5.  #4, on first appearances, is a lot
> like #3, in that the bottom is mostly static and the top consists of
> clouds moving.  #5 is taken from a tripod and consists of some close-up
> and zoomed-out pans of the collapsed stadium.  Picture quality on #5 is
> pretty good as it was quite a sunny day and the viewing conditions were
> quite clear.
>
> If there's a lot of interest in this testing methodology, I might put
> together some scripts to make automating, timing, & reporting more
> systematic.  These runs were all basically done manually on the
> command-line.  I even hand-wrote the results down.  ;)
>
> I intended to put up more of an analysis on my blog, but I am not sure
> if I'll do that yet, as I'm planning to migrate my site to a new
server,
> so I prefer not to have new uploads just yet.
>
> Constructive feedback will be most appreciated!
>
> [1] http://lists.xiph.org/pipermail/theora/2009-August/002678.html
> [2] http://www.theora.org/downloads/
> [3]
> http://www.basilgohar.com/blog/2009/06/05/sultan-mizan-zainal-abidin-stadiu
>m-aka-gong-badak-stadium-collapses [4]
> http://docs.fedoraproject.org/install-guide/f11/en-US/html/s1-rescuemode-bo
>oting-single.html

On Sun, Aug 30, 2009 at 10:11:59AM +0800, Basil Mohamed Gohar
wrote:
>    1. planet-earth-360x240.yuv4mpeg - A clip from the Discovery/BBC
>       series Planet Earth from DVD 1, the "Mountains" chapter -
the
>       first 5000 frames (I dropped half the fields to get non-interlaced
>       video, then rescaled)
You can properly deinterlace the clip with e.g. the yadif or pp=lb filters of
mplayer/mencoder.
> All clips were converted to yuv4mpeg format and then copied to /dev/shm
> to remove, as much as possible, any potential outside influences on
> encoder performance.
It's not necessary to use /dev/shm on Linux if you use the time command.

It's also possible to separate conversion to yuv4mpeg and theora compression
into separate processes. Just time the theora compression process then. No
problem.
>  Also, no sound
> was included, thus limiting this test solely to Theora encoding.
Good. I think that's what we want.
> All tests were run under Fedora 11, 32 bit, booted into single user mode
> [4].
Again, it's not necessary to do the tests in single user mode on Linux if
you
use the time command.
> If there's a lot of interest in this testing methodology, I might put
> together some scripts to make automating, timing, & reporting more
> systematic.  These runs were all basically done manually on the
> command-line.  I even hand-wrote the results down.  ;)
Of course there is interest. Go ahead. It will make testing of different files,
processors and compressor settings and versions by different people much more
easier. Even the simple fact that you'll probably have to do some more tests
will justify scripting!

Thank you.

Bernard.
-- 
http://home.euphonynet.be/bjung/