I have run a few more test at different bitrates and 1.1 is looking even worse in terms of speed compared to previous versions. I have shared a google sheet which has the raw data and charts for 6,16 and 32 kbps. Unfortunately you cannot show proper error bars on Google sheets but the standard deviation is in the data if you want to look. You can see that the profile for 1.1 is a lot different from 0.9.14 and 1.0.3. I will probably do another run at 64kbps and then find a music sample and repeat the operation. I then will re-compile as fixed point and see what that looks like. Link for the spreadsheet is here: https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing <https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing> Best Regards, Stuart Marsden Tactical Communications Consultant FinMars Consulting Ltd UK: +441865589833 Finland: +358453046287 On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk>wrote:> Cliff, > > Yes it would be good, but very hard to get a figure for the quality. > > At 6kbps I assume it does not bother trying to figure what mode to use as > at that rate it can only use SILK. When I run some other bitrates it may > get a bit slower trying to decide whether it is voice or music. > > I started with low bit rate because I am only really interested in Voice > and very low bit rate. > > I think there are so many variables it is hard to get that useful metrics. > For instance I cannot really hear the difference between complexity 0 or 10 > on this sample. It may be however that 10 would be much better with a > poorer input from a cheap microphone with lots of background noise. I also > have yet to look at the lost packet tolerance is that affected by the > complexity? For realtime applications on most hardware it seems you could > just go for the default complexity 10 and hope for the best. For low power > devices or microcontrollers however the speed difference could be crucial. > > At the moment on the Pi, which is admittedly quite an old ARM > architecture, the promised speed boost for 1.1 on ARM is not present. > Unless you compare old complexity 10 with new complexity 5 which I > understand may be equivalent. So it is just something to be aware of. > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk> wrote: > >> Hi All, >> >> What would be interesting would be a plot of complexity versus subjective >> or >> object audio quality. >> >> I've not had a chance to look at the new analysis code in 1.1 so maybe in >> the case of a 6kbps compression you could clarify what decisions would it >> be >> making that would justify the extra complexity? >> >> Best Regards >> >> Cliff Parris >> >> -----Original Message----- >> From: opus-request at xiph.org >> Sent: Thursday, December 19, 2013 6:07 PM >> To: opus at xiph.org >> Subject: opus Digest, Vol 59, Issue 21 >> >> Send opus mailing list submissions to >> opus at xiph.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> http://lists.xiph.org/mailman/listinfo/opus >> or, via email, send a message with subject or body 'help' to >> opus-request at xiph.org >> >> You can reach the person managing the list at >> opus-owner at xiph.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of opus digest..." >> >> >> Today's Topics: >> >> 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart Marsden) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Thu, 19 Dec 2013 20:06:57 +0200 >> From: Stuart Marsden <stuartmarsden at finmars.co.uk> >> Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi >> To: "opus at xiph.org" <opus at xiph.org> >> Cc: Gregory Maxwell <gmaxwell at gmail.com> >> Message-ID: >> <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO=RsTVb+qt04mKsQw at mail.gmail.com> >> Content-Type: text/plain; charset="iso-8859-1" >> >> I wanted to roughly benchmark how the different version of libopus >> performed at each complexity level for a 6kbit/s output opus file. This >> was >> conducted on a Raspberry Pi so it is a constant hardware platform. This >> was >> done on an early Pi so only 256MB RAM but it was never used up so should >> not make a difference. >> >> I compiled the three final versions of each major release of libopus so >> that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively on the >> machine using the current repo version of gcc 4.6.3 and with these >> optimisation flags: >> >> -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard >> >> >> These were compiled with floating point enabled. I will look at the fixed >> point version separately later. >> >> I used a clip of speech from a librevox recording which was resampled from >> 44.1khz to 48khz within audacity. The clip is 2 minutes long. I wrote a >> simple bash script that ran the encode at each complexity level and >> repeated 10 times to try and get a good average. >> >> The results can be seen in this graph >> http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN >> >> [image: Inline images 1] >> >> This clearly shows what I had discovered in what appears to be a >> regression >> for complexity 7,8,9 and 10. From what Gregory said earlier then in fact >> this is because these levels are producing more quality than they did >> before. It is still good to know this profile though if you only have a >> little CPU to play with such as in embedded applications. The thing I >> cannot graph on this is encode quality. My ears are not good enough to >> hear >> the difference and unless there is an automated way to score it we will >> just have to assume that each complexity level does increase the quality. >> >> The graph also suggests that on this platform at least complexity level 9 >> is pointless as it was slower than 10 and presumably produces worse >> results. This could of course have been some background task kicking in on >> the OS when this ran and the error bars are quite large so I will see if >> this maintained over other runs. >> >> All these speeds were taken from opusenc outputs and I used version 0.1.2 >> of opus-tools which was compatible with all three versions of the library. >> I am running another test using 0.1.8 at the moment but it will only work >> with libopus 1.0.3 and 1.1. I think I observed that it was slightly slower >> but we will see if the results will bear that out. >> >> I also will run some tests at different bit rates and with music instead >> of >> voice as well and share the charts here. If anyone wants I can share the >> OpenOffice spreadsheet with the raw numbers and the bash script I used >> (though you have to do all the compiling yourself). >> >> Hope this is helpful. >> >> Best Regards, >> >> Stuart Marsden >> >> Tactical Communications Consultant >> FinMars Consulting Ltd >> UK: +441865589833 >> Finland: +358453046287 >> >> >> On 18 December 2013 00:14, Stuart Marsden >> <stuartmarsden at finmars.co.uk>wrote: >> >> > Gregory, >> > >> > That is good to know and if therefore the true apples to apples >> comparison >> > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. My ears are >> > not good enough to hear the difference so for speed I would target comp >> 5 >> > or lower. >> > >> > I just did a quick test and 0.9.14 at comp 10 was 3.872 >> > 1.1 at comp 5 was 5.218 >> > >> > So if the output is comparable then we do in fact see a speed >> improvement. >> > >> > Thanks for pointing this out. Is it documented? I admit I have only read >> > some of the documentation. >> > >> > >> > Best Regards, >> > >> > Stuart Marsden >> > >> > >> > >> > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com> wrote: >> > >> >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden >> >> <stuartmarsden at finmars.co.uk> wrote: >> >> > I have just started trying Opus with a view to using it in a >> project. I >> >> am >> >> > interested in embedded hardware and tried it on the Raspberry Pi >> using >> >> the >> >> > raspbian distro. >> >> > >> >> > The version of libopus in the repos is 0.9.14. I installed this and >> >> tried >> >> > encoding 2 minutes of speech from a librevox recording. It managed >> this >> >> at a >> >> > respectable pace for complexity 10: >> >> >> >> Complexity 10 is new analysis code that didn't exist in prior >> >> versions, setting complexity 5 gets you basically the same analysis >> >> that the 1.0 version had. >> >> >> >> On x86 and modern arm cores with fast FPUs the other speedups are >> >> enough that complexity 10 is about the same speed in the old software >> >> or the new software (but with much higher and consistent VBR quality). >> >> But on chips with slow FPUs the new analysis code is much slower, in >> >> particular because it has not been entirely converted to fixed point >> >> (e.g. in the fixed point builds) which is what I believe you're seeing >> >> here. >> >> >> > >> > >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> >> http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.htm >> -------------- next part -------------- >> A non-text attachment was scrubbed... >> Name: not available >> Type: image/png >> Size: 41961 bytes >> Desc: not available >> Url : >> >> http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.png >> >> ------------------------------ >> >> _______________________________________________ >> opus mailing list >> opus at xiph.org >> http://lists.xiph.org/mailman/listinfo/opus >> >> >> End of opus Digest, Vol 59, Issue 21 >> ************************************ >> >> _______________________________________________ >> opus mailing list >> opus at xiph.org >> http://lists.xiph.org/mailman/listinfo/opus >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131221/8fd34776/attachment-0001.htm
It might be good to use the (uncompressed) samples on the opus page, as a common starting point? http://www.opus-codec.org/examples/ On Dec 21, 2013, at 9:43 AMEST, Stuart Marsden wrote:> I have run a few more test at different bitrates and 1.1 is looking even worse in terms of speed compared to previous versions. > > I have shared a google sheet which has the raw data and charts for 6,16 and 32 kbps. Unfortunately you cannot show proper error bars on Google sheets but the standard deviation is in the data if you want to look. You can see that the profile for 1.1 is a lot different from 0.9.14 and 1.0.3. > > I will probably do another run at 64kbps and then find a music sample and repeat the operation. > > I then will re-compile as fixed point and see what that looks like. > > Link for the spreadsheet is here: https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk> wrote: > Cliff, > > Yes it would be good, but very hard to get a figure for the quality. > > At 6kbps I assume it does not bother trying to figure what mode to use as at that rate it can only use SILK. When I run some other bitrates it may get a bit slower trying to decide whether it is voice or music. > > I started with low bit rate because I am only really interested in Voice and very low bit rate. > > I think there are so many variables it is hard to get that useful metrics. For instance I cannot really hear the difference between complexity 0 or 10 on this sample. It may be however that 10 would be much better with a poorer input from a cheap microphone with lots of background noise. I also have yet to look at the lost packet tolerance is that affected by the complexity? For realtime applications on most hardware it seems you could just go for the default complexity 10 and hope for the best. For low power devices or microcontrollers however the speed difference could be crucial. > > At the moment on the Pi, which is admittedly quite an old ARM architecture, the promised speed boost for 1.1 on ARM is not present. Unless you compare old complexity 10 with new complexity 5 which I understand may be equivalent. So it is just something to be aware of. > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk> wrote: > Hi All, > > What would be interesting would be a plot of complexity versus subjective or > object audio quality. > > I've not had a chance to look at the new analysis code in 1.1 so maybe in > the case of a 6kbps compression you could clarify what decisions would it be > making that would justify the extra complexity? > > Best Regards > > Cliff Parris > > -----Original Message----- > From: opus-request at xiph.org > Sent: Thursday, December 19, 2013 6:07 PM > To: opus at xiph.org > Subject: opus Digest, Vol 59, Issue 21 > > Send opus mailing list submissions to > opus at xiph.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.xiph.org/mailman/listinfo/opus > or, via email, send a message with subject or body 'help' to > opus-request at xiph.org > > You can reach the person managing the list at > opus-owner at xiph.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of opus digest..." > > > Today's Topics: > > 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart Marsden) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Dec 2013 20:06:57 +0200 > From: Stuart Marsden <stuartmarsden at finmars.co.uk> > Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi > To: "opus at xiph.org" <opus at xiph.org> > Cc: Gregory Maxwell <gmaxwell at gmail.com> > Message-ID: > <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO=RsTVb+qt04mKsQw at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > I wanted to roughly benchmark how the different version of libopus > performed at each complexity level for a 6kbit/s output opus file. This was > conducted on a Raspberry Pi so it is a constant hardware platform. This was > done on an early Pi so only 256MB RAM but it was never used up so should > not make a difference. > > I compiled the three final versions of each major release of libopus so > that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively on the > machine using the current repo version of gcc 4.6.3 and with these > optimisation flags: > > -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard > > > These were compiled with floating point enabled. I will look at the fixed > point version separately later. > > I used a clip of speech from a librevox recording which was resampled from > 44.1khz to 48khz within audacity. The clip is 2 minutes long. I wrote a > simple bash script that ran the encode at each complexity level and > repeated 10 times to try and get a good average. > > The results can be seen in this graph > http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN > > [image: Inline images 1] > > This clearly shows what I had discovered in what appears to be a regression > for complexity 7,8,9 and 10. From what Gregory said earlier then in fact > this is because these levels are producing more quality than they did > before. It is still good to know this profile though if you only have a > little CPU to play with such as in embedded applications. The thing I > cannot graph on this is encode quality. My ears are not good enough to hear > the difference and unless there is an automated way to score it we will > just have to assume that each complexity level does increase the quality. > > The graph also suggests that on this platform at least complexity level 9 > is pointless as it was slower than 10 and presumably produces worse > results. This could of course have been some background task kicking in on > the OS when this ran and the error bars are quite large so I will see if > this maintained over other runs. > > All these speeds were taken from opusenc outputs and I used version 0.1.2 > of opus-tools which was compatible with all three versions of the library. > I am running another test using 0.1.8 at the moment but it will only work > with libopus 1.0.3 and 1.1. I think I observed that it was slightly slower > but we will see if the results will bear that out. > > I also will run some tests at different bit rates and with music instead of > voice as well and share the charts here. If anyone wants I can share the > OpenOffice spreadsheet with the raw numbers and the bash script I used > (though you have to do all the compiling yourself). > > Hope this is helpful. > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 18 December 2013 00:14, Stuart Marsden > <stuartmarsden at finmars.co.uk>wrote: > > > Gregory, > > > > That is good to know and if therefore the true apples to apples comparison > > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. My ears are > > not good enough to hear the difference so for speed I would target comp 5 > > or lower. > > > > I just did a quick test and 0.9.14 at comp 10 was 3.872 > > 1.1 at comp 5 was 5.218 > > > > So if the output is comparable then we do in fact see a speed improvement. > > > > Thanks for pointing this out. Is it documented? I admit I have only read > > some of the documentation. > > > > > > Best Regards, > > > > Stuart Marsden > > > > > > > > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com> wrote: > > > >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden > >> <stuartmarsden at finmars.co.uk> wrote: > >> > I have just started trying Opus with a view to using it in a project. I > >> am > >> > interested in embedded hardware and tried it on the Raspberry Pi using > >> the > >> > raspbian distro. > >> > > >> > The version of libopus in the repos is 0.9.14. I installed this and > >> tried > >> > encoding 2 minutes of speech from a librevox recording. It managed this > >> at a > >> > respectable pace for complexity 10: > >> > >> Complexity 10 is new analysis code that didn't exist in prior > >> versions, setting complexity 5 gets you basically the same analysis > >> that the 1.0 version had. > >> > >> On x86 and modern arm cores with fast FPUs the other speedups are > >> enough that complexity 10 is about the same speed in the old software > >> or the new software (but with much higher and consistent VBR quality). > >> But on chips with slow FPUs the new analysis code is much slower, in > >> particular because it has not been entirely converted to fixed point > >> (e.g. in the fixed point builds) which is what I believe you're seeing > >> here. > >> > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.htm > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/png > Size: 41961 bytes > Desc: not available > Url : > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.png > > ------------------------------ > > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus > > > End of opus Digest, Vol 59, Issue 21 > ************************************ > > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus > > > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131221/4408d400/attachment-0001.htm
Marc, I will use the music sample when I try that next. The speech sample is way to short for benchmarking though at only 10 seconds. This makes it hard to get a metric with it as it completes quickly and also whatever the initialisation and start up times are will become very significant and swamp the actual encoding time. Would welcome a definitive speech sample to use but it really needs to be over a minute. If I wrote a test shim and called the encode from C then I could probably get better results even on short samples but I have not got that far yet. I have changed the way I am benchmarking now. Instead of parsing the x realtime from the opusenc output I am using the unix time command to get the time the process ran for. This is more reliable and cleaner to get a result. This flips the graphs because lower is now faster but the pattern is still clear. I needed to do this as I have run the same tests on my PC and that goes so fast I was getting exponential numbers in the output. Best Regards, Stuart Marsden Tactical Communications Consultant FinMars Consulting Ltd UK: +441865589833 Finland: +358453046287 On 22 December 2013 01:58, Marc Lindahl <marc at bowery.com> wrote:> It might be good to use the (uncompressed) samples on the opus page, as a > common starting point? > http://www.opus-codec.org/examples/ > > On Dec 21, 2013, at 9:43 AMEST, Stuart Marsden wrote: > > I have run a few more test at different bitrates and 1.1 is looking even > worse in terms of speed compared to previous versions. > > I have shared a google sheet which has the raw data and charts for 6,16 > and 32 kbps. Unfortunately you cannot show proper error bars on Google > sheets but the standard deviation is in the data if you want to look. You > can see that the profile for 1.1 is a lot different from 0.9.14 and 1.0.3. > > I will probably do another run at 64kbps and then find a music sample and > repeat the operation. > > I then will re-compile as fixed point and see what that looks like. > > Link for the spreadsheet is here: > https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing <https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing> > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk>wrote: > >> Cliff, >> >> Yes it would be good, but very hard to get a figure for the quality. >> >> At 6kbps I assume it does not bother trying to figure what mode to use as >> at that rate it can only use SILK. When I run some other bitrates it may >> get a bit slower trying to decide whether it is voice or music. >> >> I started with low bit rate because I am only really interested in Voice >> and very low bit rate. >> >> I think there are so many variables it is hard to get that useful >> metrics. For instance I cannot really hear the difference between >> complexity 0 or 10 on this sample. It may be however that 10 would be much >> better with a poorer input from a cheap microphone with lots of background >> noise. I also have yet to look at the lost packet tolerance is that >> affected by the complexity? For realtime applications on most hardware it >> seems you could just go for the default complexity 10 and hope for the >> best. For low power devices or microcontrollers however the speed >> difference could be crucial. >> >> At the moment on the Pi, which is admittedly quite an old ARM >> architecture, the promised speed boost for 1.1 on ARM is not present. >> Unless you compare old complexity 10 with new complexity 5 which I >> understand may be equivalent. So it is just something to be aware of. >> >> Best Regards, >> >> Stuart Marsden >> >> Tactical Communications Consultant >> FinMars Consulting Ltd >> UK: +441865589833 >> Finland: +358453046287 >> >> >> On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk> wrote: >> >>> Hi All, >>> >>> What would be interesting would be a plot of complexity versus >>> subjective or >>> object audio quality. >>> >>> I've not had a chance to look at the new analysis code in 1.1 so maybe in >>> the case of a 6kbps compression you could clarify what decisions would >>> it be >>> making that would justify the extra complexity? >>> >>> Best Regards >>> >>> Cliff Parris >>> >>> -----Original Message----- >>> From: opus-request at xiph.org >>> Sent: Thursday, December 19, 2013 6:07 PM >>> To: opus at xiph.org >>> Subject: opus Digest, Vol 59, Issue 21 >>> >>> Send opus mailing list submissions to >>> opus at xiph.org >>> >>> To subscribe or unsubscribe via the World Wide Web, visit >>> http://lists.xiph.org/mailman/listinfo/opus >>> or, via email, send a message with subject or body 'help' to >>> opus-request at xiph.org >>> >>> You can reach the person managing the list at >>> opus-owner at xiph.org >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of opus digest..." >>> >>> >>> Today's Topics: >>> >>> 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart Marsden) >>> >>> >>> ---------------------------------------------------------------------- >>> >>> Message: 1 >>> Date: Thu, 19 Dec 2013 20:06:57 +0200 >>> From: Stuart Marsden <stuartmarsden at finmars.co.uk> >>> Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi >>> To: "opus at xiph.org" <opus at xiph.org> >>> Cc: Gregory Maxwell <gmaxwell at gmail.com> >>> Message-ID: >>> <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO=RsTVb+qt04mKsQw at mail.gmail.com> >>> Content-Type: text/plain; charset="iso-8859-1" >>> >>> I wanted to roughly benchmark how the different version of libopus >>> performed at each complexity level for a 6kbit/s output opus file. This >>> was >>> conducted on a Raspberry Pi so it is a constant hardware platform. This >>> was >>> done on an early Pi so only 256MB RAM but it was never used up so should >>> not make a difference. >>> >>> I compiled the three final versions of each major release of libopus so >>> that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively on the >>> machine using the current repo version of gcc 4.6.3 and with these >>> optimisation flags: >>> >>> -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard >>> >>> >>> These were compiled with floating point enabled. I will look at the fixed >>> point version separately later. >>> >>> I used a clip of speech from a librevox recording which was resampled >>> from >>> 44.1khz to 48khz within audacity. The clip is 2 minutes long. I wrote a >>> simple bash script that ran the encode at each complexity level and >>> repeated 10 times to try and get a good average. >>> >>> The results can be seen in this graph >>> http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN >>> >>> [image: Inline images 1] >>> >>> This clearly shows what I had discovered in what appears to be a >>> regression >>> for complexity 7,8,9 and 10. From what Gregory said earlier then in fact >>> this is because these levels are producing more quality than they did >>> before. It is still good to know this profile though if you only have a >>> little CPU to play with such as in embedded applications. The thing I >>> cannot graph on this is encode quality. My ears are not good enough to >>> hear >>> the difference and unless there is an automated way to score it we will >>> just have to assume that each complexity level does increase the quality. >>> >>> The graph also suggests that on this platform at least complexity level 9 >>> is pointless as it was slower than 10 and presumably produces worse >>> results. This could of course have been some background task kicking in >>> on >>> the OS when this ran and the error bars are quite large so I will see if >>> this maintained over other runs. >>> >>> All these speeds were taken from opusenc outputs and I used version 0.1.2 >>> of opus-tools which was compatible with all three versions of the >>> library. >>> I am running another test using 0.1.8 at the moment but it will only work >>> with libopus 1.0.3 and 1.1. I think I observed that it was slightly >>> slower >>> but we will see if the results will bear that out. >>> >>> I also will run some tests at different bit rates and with music instead >>> of >>> voice as well and share the charts here. If anyone wants I can share the >>> OpenOffice spreadsheet with the raw numbers and the bash script I used >>> (though you have to do all the compiling yourself). >>> >>> Hope this is helpful. >>> >>> Best Regards, >>> >>> Stuart Marsden >>> >>> Tactical Communications Consultant >>> FinMars Consulting Ltd >>> UK: +441865589833 >>> Finland: +358453046287 >>> >>> >>> On 18 December 2013 00:14, Stuart Marsden >>> <stuartmarsden at finmars.co.uk>wrote: >>> >>> > Gregory, >>> > >>> > That is good to know and if therefore the true apples to apples >>> comparison >>> > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. My ears >>> are >>> > not good enough to hear the difference so for speed I would target >>> comp 5 >>> > or lower. >>> > >>> > I just did a quick test and 0.9.14 at comp 10 was 3.872 >>> > 1.1 at comp 5 was 5.218 >>> > >>> > So if the output is comparable then we do in fact see a speed >>> improvement. >>> > >>> > Thanks for pointing this out. Is it documented? I admit I have only >>> read >>> > some of the documentation. >>> > >>> > >>> > Best Regards, >>> > >>> > Stuart Marsden >>> > >>> > >>> > >>> > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com> wrote: >>> > >>> >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden >>> >> <stuartmarsden at finmars.co.uk> wrote: >>> >> > I have just started trying Opus with a view to using it in a >>> project. I >>> >> am >>> >> > interested in embedded hardware and tried it on the Raspberry Pi >>> using >>> >> the >>> >> > raspbian distro. >>> >> > >>> >> > The version of libopus in the repos is 0.9.14. I installed this and >>> >> tried >>> >> > encoding 2 minutes of speech from a librevox recording. It managed >>> this >>> >> at a >>> >> > respectable pace for complexity 10: >>> >> >>> >> Complexity 10 is new analysis code that didn't exist in prior >>> >> versions, setting complexity 5 gets you basically the same analysis >>> >> that the 1.0 version had. >>> >> >>> >> On x86 and modern arm cores with fast FPUs the other speedups are >>> >> enough that complexity 10 is about the same speed in the old software >>> >> or the new software (but with much higher and consistent VBR quality). >>> >> But on chips with slow FPUs the new analysis code is much slower, in >>> >> particular because it has not been entirely converted to fixed point >>> >> (e.g. in the fixed point builds) which is what I believe you're seeing >>> >> here. >>> >> >>> > >>> > >>> -------------- next part -------------- >>> An HTML attachment was scrubbed... >>> URL: >>> >>> http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.htm >>> -------------- next part -------------- >>> A non-text attachment was scrubbed... >>> Name: not available >>> Type: image/png >>> Size: 41961 bytes >>> Desc: not available >>> Url : >>> >>> http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.png >>> >>> ------------------------------ >>> >>> _______________________________________________ >>> opus mailing list >>> opus at xiph.org >>> http://lists.xiph.org/mailman/listinfo/opus >>> >>> >>> End of opus Digest, Vol 59, Issue 21 >>> ************************************ >>> >>> _______________________________________________ >>> opus mailing list >>> opus at xiph.org >>> http://lists.xiph.org/mailman/listinfo/opus >>> >> >> > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus > > > > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131222/dff486a6/attachment-0001.htm
Hello Marc, does the complexity changes a lot with different sample files? Christian Hoene Von: opus-bounces at xiph.org [mailto:opus-bounces at xiph.org] Im Auftrag von Marc Lindahl Gesendet: Sonntag, 22. Dezember 2013 00:59 An: opus at xiph.org Betreff: Re: [opus] Benchmarks on Pi It might be good to use the (uncompressed) samples on the opus page, as a common starting point. http://www.opus-codec.org/examples/ On Dec 21, 2013, at 9:43 AMEST, Stuart Marsden wrote: I have run a few more test at different bitrates and 1.1 is looking even worse in terms of speed compared to previous versions. I have shared a google sheet which has the raw data and charts for 6,16 and 32 kbps. Unfortunately you cannot show proper error bars on Google sheets but the standard deviation is in the data if you want to look. You can see that the profile for 1.1 is a lot different from 0.9.14 and 1.0.3. I will probably do another run at 64kbps and then find a music sample and repeat the operation. I then will re-compile as fixed point and see what that looks like. Link for the spreadsheet is here: <https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VV dZRUtJRU16bnc&usp=sharing> https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVd ZRUtJRU16bnc&usp=sharing Best Regards, Stuart Marsden Tactical Communications Consultant FinMars Consulting Ltd UK: +441865589833 Finland: +358453046287 On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk> wrote: Cliff, Yes it would be good, but very hard to get a figure for the quality. At 6kbps I assume it does not bother trying to figure what mode to use as at that rate it can only use SILK. When I run some other bitrates it may get a bit slower trying to decide whether it is voice or music. I started with low bit rate because I am only really interested in Voice and very low bit rate. I think there are so many variables it is hard to get that useful metrics. For instance I cannot really hear the difference between complexity 0 or 10 on this sample. It may be however that 10 would be much better with a poorer input from a cheap microphone with lots of background noise. I also have yet to look at the lost packet tolerance is that affected by the complexity? For realtime applications on most hardware it seems you could just go for the default complexity 10 and hope for the best. For low power devices or microcontrollers however the speed difference could be crucial. At the moment on the Pi, which is admittedly quite an old ARM architecture, the promised speed boost for 1.1 on ARM is not present. Unless you compare old complexity 10 with new complexity 5 which I understand may be equivalent. So it is just something to be aware of. Best Regards, Stuart Marsden Tactical Communications Consultant FinMars Consulting Ltd UK: +441865589833 <tel:%2B441865589833> Finland: +358453046287 <tel:%2B358453046287> On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk> wrote: Hi All, What would be interesting would be a plot of complexity versus subjective or object audio quality. I've not had a chance to look at the new analysis code in 1.1 so maybe in the case of a 6kbps compression you could clarify what decisions would it be making that would justify the extra complexity? Best Regards Cliff Parris -----Original Message----- From: opus-request at xiph.org Sent: Thursday, December 19, 2013 6:07 PM To: opus at xiph.org Subject: opus Digest, Vol 59, Issue 21 Send opus mailing list submissions to opus at xiph.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.xiph.org/mailman/listinfo/opus or, via email, send a message with subject or body 'help' to opus-request at xiph.org You can reach the person managing the list at opus-owner at xiph.org When replying, please edit your Subject line so it is more specific than "Re: Contents of opus digest..." Today's Topics: 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart Marsden) ---------------------------------------------------------------------- Message: 1 Date: Thu, 19 Dec 2013 20:06:57 +0200 From: Stuart Marsden <stuartmarsden at finmars.co.uk> Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi To: "opus at xiph.org" <opus at xiph.org> Cc: Gregory Maxwell <gmaxwell at gmail.com> Message-ID: <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO=RsTVb+qt04mKsQw at mail.gmail.com <mailto:RsTVb%2Bqt04mKsQw at mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" I wanted to roughly benchmark how the different version of libopus performed at each complexity level for a 6kbit/s output opus file. This was conducted on a Raspberry Pi so it is a constant hardware platform. This was done on an early Pi so only 256MB RAM but it was never used up so should not make a difference. I compiled the three final versions of each major release of libopus so that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively on the machine using the current repo version of gcc 4.6.3 and with these optimisation flags: -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard These were compiled with floating point enabled. I will look at the fixed point version separately later. I used a clip of speech from a librevox recording which was resampled from 44.1khz to 48khz within audacity. The clip is 2 minutes long. I wrote a simple bash script that ran the encode at each complexity level and repeated 10 times to try and get a good average. The results can be seen in this graph http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN [image: Inline images 1] This clearly shows what I had discovered in what appears to be a regression for complexity 7,8,9 and 10. From what Gregory said earlier then in fact this is because these levels are producing more quality than they did before. It is still good to know this profile though if you only have a little CPU to play with such as in embedded applications. The thing I cannot graph on this is encode quality. My ears are not good enough to hear the difference and unless there is an automated way to score it we will just have to assume that each complexity level does increase the quality. The graph also suggests that on this platform at least complexity level 9 is pointless as it was slower than 10 and presumably produces worse results. This could of course have been some background task kicking in on the OS when this ran and the error bars are quite large so I will see if this maintained over other runs. All these speeds were taken from opusenc outputs and I used version 0.1.2 of opus-tools which was compatible with all three versions of the library. I am running another test using 0.1.8 at the moment but it will only work with libopus 1.0.3 and 1.1. I think I observed that it was slightly slower but we will see if the results will bear that out. I also will run some tests at different bit rates and with music instead of voice as well and share the charts here. If anyone wants I can share the OpenOffice spreadsheet with the raw numbers and the bash script I used (though you have to do all the compiling yourself). Hope this is helpful. Best Regards, Stuart Marsden Tactical Communications Consultant FinMars Consulting Ltd UK: +441865589833 <tel:%2B441865589833> Finland: +358453046287 <tel:%2B358453046287> On 18 December 2013 00:14, Stuart Marsden <stuartmarsden at finmars.co.uk>wrote:> Gregory, > > That is good to know and if therefore the true apples to apples comparison > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. My ears are > not good enough to hear the difference so for speed I would target comp 5 > or lower. > > I just did a quick test and 0.9.14 at comp 10 was 3.872 > 1.1 at comp 5 was 5.218 > > So if the output is comparable then we do in fact see a speed improvement. > > Thanks for pointing this out. Is it documented? I admit I have only read > some of the documentation. > > > Best Regards, > > Stuart Marsden > > > > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com> wrote: > >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden >> <stuartmarsden at finmars.co.uk> wrote: >> > I have just started trying Opus with a view to using it in a project. I >> am >> > interested in embedded hardware and tried it on the Raspberry Pi using >> the >> > raspbian distro. >> > >> > The version of libopus in the repos is 0.9.14. I installed this and >> tried >> > encoding 2 minutes of speech from a librevox recording. It managed this >> at a >> > respectable pace for complexity 10: >> >> Complexity 10 is new analysis code that didn't exist in prior >> versions, setting complexity 5 gets you basically the same analysis >> that the 1.0 version had. >> >> On x86 and modern arm cores with fast FPUs the other speedups are >> enough that complexity 10 is about the same speed in the old software >> or the new software (but with much higher and consistent VBR quality). >> But on chips with slow FPUs the new analysis code is much slower, in >> particular because it has not been entirely converted to fixed point >> (e.g. in the fixed point builds) which is what I believe you're seeing >> here. >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachmen t.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 41961 bytes Desc: not available Url : http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachmen t.png ------------------------------ _______________________________________________ opus mailing list opus at xiph.org http://lists.xiph.org/mailman/listinfo/opus End of opus Digest, Vol 59, Issue 21 ************************************ _______________________________________________ opus mailing list opus at xiph.org http://lists.xiph.org/mailman/listinfo/opus _______________________________________________ opus mailing list <mailto:opus at xiph.org> opus at xiph.org <http://lists.xiph.org/mailman/listinfo/opus> http://lists.xiph.org/mailman/listinfo/opus -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131222/e3667257/attachment-0001.htm
I have to admit that I am impressed by your results -- making 1.1 look slower than 1.0 is by no means an easy task. On the other hand, it's a great tutorial on how not to use Opus, so for the benefit of everyone, this is a summary of what we learned in this exercise: 1) When running on ARM, the fixed-point build is usually faster than floating point. This is true on the majority of ARM archs (on x86, float is faster). 2) Turning on assembly is useful too, though that requires 1) first (writing float assembly on ARM would have been pointless). 3) Complexity settings cannot be directly compared between versions because we keep adding things. For example, the Opus 1.1 encoder at complexity 5 does more advanced analysis (and has higher quality) than the 1.0 encoder at complexity 10. 4) Using 44.1 kHz is a bad idea for benchmarking since it also counts the cost of the resampling done in opus-tools. 5) Using different opus-tools version just adds to the randomness. Jean-Marc On 12/21/2013 09:43 AM, Stuart Marsden wrote:> I have run a few more test at different bitrates and 1.1 is looking even > worse in terms of speed compared to previous versions. > > I have shared a google sheet which has the raw data and charts for 6,16 > and 32 kbps. Unfortunately you cannot show proper error bars on Google > sheets but the standard deviation is in the data if you want to look. > You can see that the profile for 1.1 is a lot different from 0.9.14 and > 1.0.3. > > I will probably do another run at 64kbps and then find a music sample > and repeat the operation. > > I then will re-compile as fixed point and see what that looks like. > > Link for the spreadsheet is > here: https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing > <https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing> > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 > Finland: +358453046287 > > > On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk > <mailto:stuartmarsden at finmars.co.uk>> wrote: > > Cliff, > > Yes it would be good, but very hard to get a figure for the quality. > > At 6kbps I assume it does not bother trying to figure what mode to > use as at that rate it can only use SILK. When I run some other > bitrates it may get a bit slower trying to decide whether it is > voice or music. > > I started with low bit rate because I am only really interested in > Voice and very low bit rate. > > I think there are so many variables it is hard to get that useful > metrics. For instance I cannot really hear the difference between > complexity 0 or 10 on this sample. It may be however that 10 would > be much better with a poorer input from a cheap microphone with lots > of background noise. I also have yet to look at the lost packet > tolerance is that affected by the complexity? For realtime > applications on most hardware it seems you could just go for the > default complexity 10 and hope for the best. For low power devices > or microcontrollers however the speed difference could be crucial. > > At the moment on the Pi, which is admittedly quite an old ARM > architecture, the promised speed boost for 1.1 on ARM is not > present. Unless you compare old complexity 10 with new complexity 5 > which I understand may be equivalent. So it is just something to be > aware of. > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 <tel:%2B441865589833> > Finland: +358453046287 <tel:%2B358453046287> > > > On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk > <mailto:cliff at espico.co.uk>> wrote: > > Hi All, > > What would be interesting would be a plot of complexity versus > subjective or > object audio quality. > > I've not had a chance to look at the new analysis code in 1.1 so > maybe in > the case of a 6kbps compression you could clarify what decisions > would it be > making that would justify the extra complexity? > > Best Regards > > Cliff Parris > > -----Original Message----- > From: opus-request at xiph.org <mailto:opus-request at xiph.org> > Sent: Thursday, December 19, 2013 6:07 PM > To: opus at xiph.org <mailto:opus at xiph.org> > Subject: opus Digest, Vol 59, Issue 21 > > Send opus mailing list submissions to > opus at xiph.org <mailto:opus at xiph.org> > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.xiph.org/mailman/listinfo/opus > or, via email, send a message with subject or body 'help' to > opus-request at xiph.org <mailto:opus-request at xiph.org> > > You can reach the person managing the list at > opus-owner at xiph.org <mailto:opus-owner at xiph.org> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of opus digest..." > > > Today's Topics: > > 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart Marsden) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Dec 2013 20:06:57 +0200 > From: Stuart Marsden <stuartmarsden at finmars.co.uk > <mailto:stuartmarsden at finmars.co.uk>> > Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi > To: "opus at xiph.org <mailto:opus at xiph.org>" <opus at xiph.org > <mailto:opus at xiph.org>> > Cc: Gregory Maxwell <gmaxwell at gmail.com <mailto:gmaxwell at gmail.com>> > Message-ID: > <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO=RsTVb+qt04mKsQw at mail.gmail.com > <mailto:RsTVb%2Bqt04mKsQw at mail.gmail.com>> > Content-Type: text/plain; charset="iso-8859-1" > > I wanted to roughly benchmark how the different version of libopus > performed at each complexity level for a 6kbit/s output opus > file. This was > conducted on a Raspberry Pi so it is a constant hardware > platform. This was > done on an early Pi so only 256MB RAM but it was never used up > so should > not make a difference. > > I compiled the three final versions of each major release of > libopus so > that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively > on the > machine using the current repo version of gcc 4.6.3 and with these > optimisation flags: > > -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp > -mfloat-abi=hard > > > These were compiled with floating point enabled. I will look at > the fixed > point version separately later. > > I used a clip of speech from a librevox recording which was > resampled from > 44.1khz to 48khz within audacity. The clip is 2 minutes long. I > wrote a > simple bash script that ran the encode at each complexity level and > repeated 10 times to try and get a good average. > > The results can be seen in this graph > http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN > > [image: Inline images 1] > > This clearly shows what I had discovered in what appears to be a > regression > for complexity 7,8,9 and 10. From what Gregory said earlier then > in fact > this is because these levels are producing more quality than > they did > before. It is still good to know this profile though if you only > have a > little CPU to play with such as in embedded applications. The > thing I > cannot graph on this is encode quality. My ears are not good > enough to hear > the difference and unless there is an automated way to score it > we will > just have to assume that each complexity level does increase the > quality. > > The graph also suggests that on this platform at least > complexity level 9 > is pointless as it was slower than 10 and presumably produces worse > results. This could of course have been some background task > kicking in on > the OS when this ran and the error bars are quite large so I > will see if > this maintained over other runs. > > All these speeds were taken from opusenc outputs and I used > version 0.1.2 > of opus-tools which was compatible with all three versions of > the library. > I am running another test using 0.1.8 at the moment but it will > only work > with libopus 1.0.3 and 1.1. I think I observed that it was > slightly slower > but we will see if the results will bear that out. > > I also will run some tests at different bit rates and with music > instead of > voice as well and share the charts here. If anyone wants I can > share the > OpenOffice spreadsheet with the raw numbers and the bash script > I used > (though you have to do all the compiling yourself). > > Hope this is helpful. > > Best Regards, > > Stuart Marsden > > Tactical Communications Consultant > FinMars Consulting Ltd > UK: +441865589833 <tel:%2B441865589833> > Finland: +358453046287 <tel:%2B358453046287> > > > On 18 December 2013 00:14, Stuart Marsden > <stuartmarsden at finmars.co.uk > <mailto:stuartmarsden at finmars.co.uk>>wrote: > > > Gregory, > > > > That is good to know and if therefore the true apples to > apples comparison > > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. > My ears are > > not good enough to hear the difference so for speed I would > target comp 5 > > or lower. > > > > I just did a quick test and 0.9.14 at comp 10 was 3.872 > > 1.1 at comp 5 was 5.218 > > > > So if the output is comparable then we do in fact see a speed > improvement. > > > > Thanks for pointing this out. Is it documented? I admit I have > only read > > some of the documentation. > > > > > > Best Regards, > > > > Stuart Marsden > > > > > > > > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com > <mailto:gmaxwell at gmail.com>> wrote: > > > >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden > >> <stuartmarsden at finmars.co.uk > <mailto:stuartmarsden at finmars.co.uk>> wrote: > >> > I have just started trying Opus with a view to using it in > a project. I > >> am > >> > interested in embedded hardware and tried it on the > Raspberry Pi using > >> the > >> > raspbian distro. > >> > > >> > The version of libopus in the repos is 0.9.14. I installed > this and > >> tried > >> > encoding 2 minutes of speech from a librevox recording. It > managed this > >> at a > >> > respectable pace for complexity 10: > >> > >> Complexity 10 is new analysis code that didn't exist in prior > >> versions, setting complexity 5 gets you basically the same > analysis > >> that the 1.0 version had. > >> > >> On x86 and modern arm cores with fast FPUs the other speedups are > >> enough that complexity 10 is about the same speed in the old > software > >> or the new software (but with much higher and consistent VBR > quality). > >> But on chips with slow FPUs the new analysis code is much > slower, in > >> particular because it has not been entirely converted to > fixed point > >> (e.g. in the fixed point builds) which is what I believe > you're seeing > >> here. > >> > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.htm > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: image/png > Size: 41961 bytes > Desc: not available > Url : > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.png > > ------------------------------ > > _______________________________________________ > opus mailing list > opus at xiph.org <mailto:opus at xiph.org> > http://lists.xiph.org/mailman/listinfo/opus > > > End of opus Digest, Vol 59, Issue 21 > ************************************ > > _______________________________________________ > opus mailing list > opus at xiph.org <mailto:opus at xiph.org> > http://lists.xiph.org/mailman/listinfo/opus > > > > > > _______________________________________________ > opus mailing list > opus at xiph.org > http://lists.xiph.org/mailman/listinfo/opus >
Jean-Marc, Thank you for taking the time to reply. First of all a big thanks to everyone who has worked on Opus, yourself included. I love open source and the fact I can just download and use such an excellent codec at no cost. In my benchmarking I was just trying to understand how the profile varies between versions and it was not intended in any way as a criticism of 1.1 because the additional features and quality are great. As you can see from me just joining the mailing list, I am taking my first steps with Opus. If my missteps can help inform others then that is hopefully useful. I think in 99.9% of cases the fact that in some very select cases it may appear slower is irrelevant. In general people will be getting greater quality and if their encode takes 1.2 Seconds instead of 1 for a 2 minute sample then who cares. What drove me to benchmark was that I was interested in the low power device environment and the one I had to hand was a Raspberry Pi which a specific and quite dated ARM core. What this exercise has shown to me is that if the fact that complexity 10 on 1.1 is slightly slower is a problem, then I can switch to complexity 5. I now know I will be getting at least as good quality as complexity 10 on 1.0.3 and actually getting it a bit quicker. This was not clear to me as an Opus newbie by just reading the release notes and other documents. On your points: 1) My poor overworked Pi is still going through a run on the floating point version. When it is done I will set it going over Christmas doing the same run with fixed point. My initial play, on this device at least, did not seem to show a great difference between fixed or floating point implementation but I will have some hard data to inform myself soon. I will share for those that are interested but of course it only applies to this device as others may have a very different profile. 2) I think it said during configure that it did not have any assembly for that architecture but I will soon find out. 3) This has been the main and useful learning point for me. It may be worth while capturing this more clearly in the documentation so people do not make the same assumptions as I. 4) I re-encoded the sample myself so that opus did not have to. That being said does the encoder have issues with up-sampled data? I guess I should try and find an original speech sample at 48khz to try that. Or record myself (but I hate the sound of my own voice ;-) 5) I did run originally with just 0.1.2 as it could use all libs but just wanted to see if 0.1.8 made a difference. It did not seem to have an effect and the main comparison is between 1.0.3 and 1.1 which are both using 0.1.8 anyway. I will probably not bother with 0.9.14 any more anyway as it is so close to 1.0.3. Once again many thanks for you work on this. I will continue to learn and enjoy using 1.1 Best Regards, Stuart Marsden On 22 December 2013 19:40, Jean-Marc Valin <jmvalin at jmvalin.ca> wrote:> I have to admit that I am impressed by your results -- making 1.1 look > slower than 1.0 is by no means an easy task. On the other hand, it's a > great tutorial on how not to use Opus, so for the benefit of everyone, > this is a summary of what we learned in this exercise: > > 1) When running on ARM, the fixed-point build is usually faster than > floating point. This is true on the majority of ARM archs (on x86, float > is faster). > 2) Turning on assembly is useful too, though that requires 1) first > (writing float assembly on ARM would have been pointless). > 3) Complexity settings cannot be directly compared between versions > because we keep adding things. For example, the Opus 1.1 encoder at > complexity 5 does more advanced analysis (and has higher quality) than > the 1.0 encoder at complexity 10. > 4) Using 44.1 kHz is a bad idea for benchmarking since it also counts > the cost of the resampling done in opus-tools. > 5) Using different opus-tools version just adds to the randomness. > > Jean-Marc > > > On 12/21/2013 09:43 AM, Stuart Marsden wrote: > > I have run a few more test at different bitrates and 1.1 is looking even > > worse in terms of speed compared to previous versions. > > > > I have shared a google sheet which has the raw data and charts for 6,16 > > and 32 kbps. Unfortunately you cannot show proper error bars on Google > > sheets but the standard deviation is in the data if you want to look. > > You can see that the profile for 1.1 is a lot different from 0.9.14 and > > 1.0.3. > > > > I will probably do another run at 64kbps and then find a music sample > > and repeat the operation. > > > > I then will re-compile as fixed point and see what that looks like. > > > > Link for the spreadsheet is > > here: > https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing > > < > https://docs.google.com/spreadsheet/ccc?key=0AmUOhhaYBtrKdEJaUGFmZkxzaUE0VVdZRUtJRU16bnc&usp=sharing > > > > > > Best Regards, > > > > Stuart Marsden > > > > Tactical Communications Consultant > > FinMars Consulting Ltd > > UK: +441865589833 > > Finland: +358453046287 > > > > > > On 20 December 2013 18:24, Stuart Marsden <stuartmarsden at finmars.co.uk > > <mailto:stuartmarsden at finmars.co.uk>> wrote: > > > > Cliff, > > > > Yes it would be good, but very hard to get a figure for the quality. > > > > At 6kbps I assume it does not bother trying to figure what mode to > > use as at that rate it can only use SILK. When I run some other > > bitrates it may get a bit slower trying to decide whether it is > > voice or music. > > > > I started with low bit rate because I am only really interested in > > Voice and very low bit rate. > > > > I think there are so many variables it is hard to get that useful > > metrics. For instance I cannot really hear the difference between > > complexity 0 or 10 on this sample. It may be however that 10 would > > be much better with a poorer input from a cheap microphone with lots > > of background noise. I also have yet to look at the lost packet > > tolerance is that affected by the complexity? For realtime > > applications on most hardware it seems you could just go for the > > default complexity 10 and hope for the best. For low power devices > > or microcontrollers however the speed difference could be crucial. > > > > At the moment on the Pi, which is admittedly quite an old ARM > > architecture, the promised speed boost for 1.1 on ARM is not > > present. Unless you compare old complexity 10 with new complexity 5 > > which I understand may be equivalent. So it is just something to be > > aware of. > > > > Best Regards, > > > > Stuart Marsden > > > > Tactical Communications Consultant > > FinMars Consulting Ltd > > UK: +441865589833 <tel:%2B441865589833> > > Finland: +358453046287 <tel:%2B358453046287> > > > > > > On 20 December 2013 17:20, Cliff Parris <cliff at espico.co.uk > > <mailto:cliff at espico.co.uk>> wrote: > > > > Hi All, > > > > What would be interesting would be a plot of complexity versus > > subjective or > > object audio quality. > > > > I've not had a chance to look at the new analysis code in 1.1 so > > maybe in > > the case of a 6kbps compression you could clarify what decisions > > would it be > > making that would justify the extra complexity? > > > > Best Regards > > > > Cliff Parris > > > > -----Original Message----- > > From: opus-request at xiph.org <mailto:opus-request at xiph.org> > > Sent: Thursday, December 19, 2013 6:07 PM > > To: opus at xiph.org <mailto:opus at xiph.org> > > Subject: opus Digest, Vol 59, Issue 21 > > > > Send opus mailing list submissions to > > opus at xiph.org <mailto:opus at xiph.org> > > > > To subscribe or unsubscribe via the World Wide Web, visit > > http://lists.xiph.org/mailman/listinfo/opus > > or, via email, send a message with subject or body 'help' to > > opus-request at xiph.org <mailto:opus-request at xiph.org> > > > > You can reach the person managing the list at > > opus-owner at xiph.org <mailto:opus-owner at xiph.org> > > > > When replying, please edit your Subject line so it is more > specific > > than "Re: Contents of opus digest..." > > > > > > Today's Topics: > > > > 1. Opus Major Version Benchmarks on Raspberry Pi (Stuart > Marsden) > > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Thu, 19 Dec 2013 20:06:57 +0200 > > From: Stuart Marsden <stuartmarsden at finmars.co.uk > > <mailto:stuartmarsden at finmars.co.uk>> > > Subject: [opus] Opus Major Version Benchmarks on Raspberry Pi > > To: "opus at xiph.org <mailto:opus at xiph.org>" <opus at xiph.org > > <mailto:opus at xiph.org>> > > Cc: Gregory Maxwell <gmaxwell at gmail.com <mailto: > gmaxwell at gmail.com>> > > Message-ID: > > <CALPi7JckeXBzKuE2M4iG5iH91M=joj6uNO> RsTVb+qt04mKsQw at mail.gmail.com > > <mailto:RsTVb%2Bqt04mKsQw at mail.gmail.com>> > > Content-Type: text/plain; charset="iso-8859-1" > > > > I wanted to roughly benchmark how the different version of > libopus > > performed at each complexity level for a 6kbit/s output opus > > file. This was > > conducted on a Raspberry Pi so it is a constant hardware > > platform. This was > > done on an early Pi so only 256MB RAM but it was never used up > > so should > > not make a difference. > > > > I compiled the three final versions of each major release of > > libopus so > > that was 0.9.14, 1.0.3 & 1.1. These were all compiled natively > > on the > > machine using the current repo version of gcc 4.6.3 and with > these > > optimisation flags: > > > > -O2 -pipe -march=armv6j -mtune=arm1176jzf-s -mfpu=vfp > > -mfloat-abi=hard > > > > > > These were compiled with floating point enabled. I will look at > > the fixed > > point version separately later. > > > > I used a clip of speech from a librevox recording which was > > resampled from > > 44.1khz to 48khz within audacity. The clip is 2 minutes long. I > > wrote a > > simple bash script that ran the encode at each complexity level > and > > repeated 10 times to try and get a good average. > > > > The results can be seen in this graph > > http://ubuntuone.com/2gOdUG3h3MyjLY7gSYseRN > > > > [image: Inline images 1] > > > > This clearly shows what I had discovered in what appears to be a > > regression > > for complexity 7,8,9 and 10. From what Gregory said earlier then > > in fact > > this is because these levels are producing more quality than > > they did > > before. It is still good to know this profile though if you only > > have a > > little CPU to play with such as in embedded applications. The > > thing I > > cannot graph on this is encode quality. My ears are not good > > enough to hear > > the difference and unless there is an automated way to score it > > we will > > just have to assume that each complexity level does increase the > > quality. > > > > The graph also suggests that on this platform at least > > complexity level 9 > > is pointless as it was slower than 10 and presumably produces > worse > > results. This could of course have been some background task > > kicking in on > > the OS when this ran and the error bars are quite large so I > > will see if > > this maintained over other runs. > > > > All these speeds were taken from opusenc outputs and I used > > version 0.1.2 > > of opus-tools which was compatible with all three versions of > > the library. > > I am running another test using 0.1.8 at the moment but it will > > only work > > with libopus 1.0.3 and 1.1. I think I observed that it was > > slightly slower > > but we will see if the results will bear that out. > > > > I also will run some tests at different bit rates and with music > > instead of > > voice as well and share the charts here. If anyone wants I can > > share the > > OpenOffice spreadsheet with the raw numbers and the bash script > > I used > > (though you have to do all the compiling yourself). > > > > Hope this is helpful. > > > > Best Regards, > > > > Stuart Marsden > > > > Tactical Communications Consultant > > FinMars Consulting Ltd > > UK: +441865589833 <tel:%2B441865589833> > > Finland: +358453046287 <tel:%2B358453046287> > > > > > > On 18 December 2013 00:14, Stuart Marsden > > <stuartmarsden at finmars.co.uk > > <mailto:stuartmarsden at finmars.co.uk>>wrote: > > > > > Gregory, > > > > > > That is good to know and if therefore the true apples to > > apples comparison > > > is 0.9.14 at comp 10 and 1.1 at comp 5 then things are fine. > > My ears are > > > not good enough to hear the difference so for speed I would > > target comp 5 > > > or lower. > > > > > > I just did a quick test and 0.9.14 at comp 10 was 3.872 > > > 1.1 at comp 5 was 5.218 > > > > > > So if the output is comparable then we do in fact see a speed > > improvement. > > > > > > Thanks for pointing this out. Is it documented? I admit I have > > only read > > > some of the documentation. > > > > > > > > > Best Regards, > > > > > > Stuart Marsden > > > > > > > > > > > > On 17 December 2013 23:50, Gregory Maxwell <gmaxwell at gmail.com > > <mailto:gmaxwell at gmail.com>> wrote: > > > > > >> On Mon, Dec 16, 2013 at 5:03 AM, Stuart Marsden > > >> <stuartmarsden at finmars.co.uk > > <mailto:stuartmarsden at finmars.co.uk>> wrote: > > >> > I have just started trying Opus with a view to using it in > > a project. I > > >> am > > >> > interested in embedded hardware and tried it on the > > Raspberry Pi using > > >> the > > >> > raspbian distro. > > >> > > > >> > The version of libopus in the repos is 0.9.14. I installed > > this and > > >> tried > > >> > encoding 2 minutes of speech from a librevox recording. It > > managed this > > >> at a > > >> > respectable pace for complexity 10: > > >> > > >> Complexity 10 is new analysis code that didn't exist in prior > > >> versions, setting complexity 5 gets you basically the same > > analysis > > >> that the 1.0 version had. > > >> > > >> On x86 and modern arm cores with fast FPUs the other speedups > are > > >> enough that complexity 10 is about the same speed in the old > > software > > >> or the new software (but with much higher and consistent VBR > > quality). > > >> But on chips with slow FPUs the new analysis code is much > > slower, in > > >> particular because it has not been entirely converted to > > fixed point > > >> (e.g. in the fixed point builds) which is what I believe > > you're seeing > > >> here. > > >> > > > > > > > > -------------- next part -------------- > > An HTML attachment was scrubbed... > > URL: > > > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.htm > > -------------- next part -------------- > > A non-text attachment was scrubbed... > > Name: not available > > Type: image/png > > Size: 41961 bytes > > Desc: not available > > Url : > > > http://lists.xiph.org/pipermail/opus/attachments/20131219/6320e394/attachment.png > > > > ------------------------------ > > > > _______________________________________________ > > opus mailing list > > opus at xiph.org <mailto:opus at xiph.org> > > http://lists.xiph.org/mailman/listinfo/opus > > > > > > End of opus Digest, Vol 59, Issue 21 > > ************************************ > > > > _______________________________________________ > > opus mailing list > > opus at xiph.org <mailto:opus at xiph.org> > > http://lists.xiph.org/mailman/listinfo/opus > > > > > > > > > > > > _______________________________________________ > > opus mailing list > > opus at xiph.org > > http://lists.xiph.org/mailman/listinfo/opus > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.xiph.org/pipermail/opus/attachments/20131223/106b8794/attachment-0001.htm