llvm dev - Jun 2016 - FireFox-46.0.1 build with interprocedural register allocation enabled

Hello,

I build FireFox-46.0.1 source with llvm to test interprocedural register
allocation.
The build was successful with out any runtime faliures, here are few stats:

Measure W/O IPRA WITH IPRA
======= ======== ========Total Build Time 76 mins 82.3 mins 8% increment
Octane v2.0 JS Benchmark Score (higher is better) 18675.69  19665.16 5%
improvement
Kraken JS Benchmark time (lower is better) 1416.2 ms 1421.3 ms 0.35%
regression
JetStream JS Benchmark Score (higer is better) 110.10 112.88 2.52%
improvement

Any suggestions are welcome on how to effectively measure performance
improvement!

Sincerely,
Vivek
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160620/11e5d661/attachment.html>

Hi Vivek,

[Dropping firefox-dev, since I don't want to spam them]

vivek pandya via llvm-dev wrote:
 > Measure W/O IPRAWITH IPRA
 > ======================= > Total Build Time76 mins82.3 mins8% increment
 > Octane v2.0 JS Benchmark Score (higher is better)18675.69 19665.165%
improvement
 > Kraken JS Benchmark time (lower is better)1416.2 ms 1421.3 ms 0.35%
regression
 > JetStream JS Benchmark Score (higer is better)110.10112.882.52%
improvement

This is great!

Do you have a sense of how much noise these benchmarks have?  For
instance, if you run the benchmark 10 times, what is standard
deviation in the numbers?

It'd also be great to have a precise, non-measurement oriented view of
the benchmarks.  Is it possible to collect some statistics on how much
you've improved the register allocation in these workloads?  Perhaps
you could count the instances where you preserved a register over a
call site that wouldn't have been possible without IPRA?

Thanks!
-- Sanjoy

On Mon, Jun 20, 2016 at 4:14 AM, Sanjoy Das <sanjoy at
playingwithpointers.com>
wrote:
> Hi Vivek,
>
> [Dropping firefox-dev, since I don't want to spam them]
>
> vivek pandya via llvm-dev wrote:
> > Measure W/O IPRAWITH IPRA
> > =======================> > Total Build Time76 mins82.3 mins8%
increment
> > Octane v2.0 JS Benchmark Score (higher is better)18675.69 19665.165%
> improvement
> > Kraken JS Benchmark time (lower is better)1416.2 ms 1421.3 ms 0.35%
> regression
> > JetStream JS Benchmark Score (higer is better)110.10112.882.52%
> improvement
>
> This is great!
>
> Do you have a sense of how much noise these benchmarks have?  For
> instance, if you run the benchmark 10 times, what is standard
> deviation in the numbers?
>
Hello Sanjoy,

For Octane and Kraken I have run them 4 times and above result is geometric
mean. For Octane standard deviation (SD) is 918.54 (NO_IPRA) and 597.82
(With_IPRA). For Kraken unfortunately I don't have readings any more but
there was very minor change in each run. JetStream it self runs the test
for 3 times reports the result. From next time onwards I will run test at
least for 10 times
>
> It'd also be great to have a precise, non-measurement oriented view of
> the benchmarks.
I don't understand this point actually all these benchmarks are suggested
by firefox-devs and they wanted the results back.
> Is it possible to collect some statistics on how much
> you've improved the register allocation in these workloads?
Actually while testing single source test case  with debug build I used to
compare results of -stats with regalloc keyword but while building such a
huge software I prefer release build of llvm. And also I don't know if
there is any way in llvm to generate stats for the whole build.
> Perhaps
> you could count the instances where you preserved a register over a
> call site that wouldn't have been possible without IPRA?
>
Yes this seems a good idea and I think this is implementable, after
calculating new regmask when inserting data into immutable pass two
regmasks can be compared to calculate improvements. I will work on this and
let you know the progress.
>
> Thanks!
> -- Sanjoy
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160620/9872cb64/attachment.html>

On Sun, Jun 19, 2016 at 11:41 AM, vivek pandya via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Hello,
>
> I build FireFox-46.0.1 source with llvm to test interprocedural register
> allocation.
> The build was successful with out any runtime faliures, here are few stats:
This is very good, thanks for working on this.
>
> Measure W/O IPRA WITH IPRA
> ======= ======== ========> Total Build Time 76 mins 82.3 mins 8%
increment
> Octane v2.0 JS Benchmark Score (higher is better) 18675.69  19665.16 5%
> improvement
This speedup is kind of amazing, enough to make me a little bit
suspicious.
>From what I can see, Octane is not exactly a microbenchmark but tries
to model complex/real-world web applications, so, I think you might
want to analyze where this speedup is coming from? Also, "score" might
be a misleading metric, can you actually elaborate what that means?
How does that relate to, let's say, runtime performance improvement?

Thanks!

-- 
Davide

"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare

On Mon, Jun 20, 2016 at 10:06 PM, Davide Italiano <davide at freebsd.org>
wrote:
> On Sun, Jun 19, 2016 at 11:41 AM, vivek pandya via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > Hello,
> >
> > I build FireFox-46.0.1 source with llvm to test interprocedural
register
> > allocation.
> > The build was successful with out any runtime faliures, here are few
> stats:
>
> This is very good, thanks for working on this.
>
> >
> > Measure W/O IPRA WITH IPRA
> > ======= ======== ========> > Total Build Time 76 mins 82.3 mins
8% increment
> > Octane v2.0 JS Benchmark Score (higher is better) 18675.69  19665.16
5%
> > improvement
>
> This speedup is kind of amazing, enough to make me a little bit suspicious.
> From what I can see, Octane is not exactly a microbenchmark but tries
> to model complex/real-world web applications, so, I think you might
> want to analyze where this speedup is coming from?
Hi Davide,

I don't understand much about browser benchmarks but what IPRA is trying to
do is reduce spill code , and trying to keep values in register where ever
it can so speed up is comping from improved code quality. But with current
infrastructure it is hard to tell which particular functions in browser
code is getting benefitted.
> Also, "score" might
> be a misleading metric, can you actually elaborate what that means?
> How does that relate to, let's say, runtime performance improvement?
>
Octane score considers 2 things execution speed and latency (pause during
execution) . You can find more information here
https://developers.google.com/octane/faq

-Vivek
>
> Thanks!
>
> --
> Davide
>
> "There are no solved problems; there are only problems that are more
> or less solved" -- Henri Poincare
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160621/8dee5c9d/attachment.html>

On 6/19/16 2:41 PM, vivek pandya via llvm-dev wrote:
> Octane v2.0 JS Benchmark Score (higher is better)18675.69 19665.165%
> improvement
What was the breakdown across the sub-benchmarks, if I might ask?

-Boris

On Tue, Jun 21, 2016 at 2:00 AM, Boris Zbarsky <bzbarsky at mit.edu>
wrote:
> On 6/19/16 2:41 PM, vivek pandya via llvm-dev wrote:
>
>> Octane v2.0 JS Benchmark Score (higher is better)18675.69 19665.165%
>> improvement
>>
>
> What was the breakdown across the sub-benchmarks, if I might ask?
Hey Boris,
Sorry but I haven't kept details I just noted the final score. I will keep
it from next time.
-Vivek
>
>
> -Boris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160621/6230a7ce/attachment.html>

Hi,

the best way to move forward, and I guess the answer to your question, 
is to get your work to build and test on try.

And now I'm going to flood you with jargon, sorry. Instead of trying to 
document that all here, I suggest you ask people for help on irc. 
Notably, #taskcluster, and #build. Folks there are your target audience 
for compiler changes, too.

First read is https://wiki.mozilla.org/ReleaseEngineering/TryServer, 
just to get the first piece of jargon out.

I suspect that the next steps are:

I speculate that you can use your version of llvm if you can create a 
docker image that is able to compile Firefox. Please confirm that with 
#taskcluster, and get some help there on how to do that.

Then you'd want to make sure that you can actually compile/run 
mozilla-central. In particular wrt to taskcluster, there's been various 
file moves and changes, so working off of release is going to make your 
path rockier. I understand that mozilla-central is more of a moving 
target, but that shouldn't be that much of a problem for you in 
practice, I hope.

I suspect that just focusing on linux x64 is good for you for now?

So jargon-thought-train:

Validate my assumptions about all of this ;-) .

Make your setup work with mozilla-central.

Work in docker.

Make your mozilla-central's taskcluster tasks for linux x64 pick your 
docker image to build on try.

Push to try for linux x64, with all tests and perf tests.

Use treeherder and perfherder (web uis) to see tests and performance.


If you want to focus on your toolchain impact over time instead of 
mozilla-central, I think you should be able to keep the same base 
version of your patch and just update the docker image, and then use 
perfherder to compare the results.

HTH

Axel


On 19/06/16 20:41, vivek pandya wrote:
> Hello,
>
> I build FireFox-46.0.1 source with llvm to test interprocedural 
> register allocation.
> The build was successful with out any runtime faliures, here are few 
> stats:
>
> Measure W/O IPRAWITH IPRA
> =======================> Total Build Time76 mins82.3 mins8% increment
> Octane v2.0 JS Benchmark Score (higher is better)18675.69 19665.165% 
> improvement
> Kraken JS Benchmark time (lower is better)1416.2 ms 1421.3 ms 0.35% 
> regression
> JetStream JS Benchmark Score (higer is better)110.10112.882.52% 
> improvement
>
> Any suggestions are welcome on how to effectively measure performance 
> improvement!
>
> Sincerely,
> Vivek
>
>
>
> _______________________________________________
> firefox-dev mailing list
> firefox-dev at mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160621/8854fba6/attachment.html>

On Tue, Jun 21, 2016 at 1:59 PM, Axel Hecht <l10n at mozilla.com> wrote:
> Hi,
>
> the best way to move forward, and I guess the answer to your question, is
> to get your work to build and test on try.
>
> And now I'm going to flood you with jargon, sorry. Instead of trying to
> document that all here, I suggest you ask people for help on irc. Notably,
> #taskcluster, and #build. Folks there are your target audience for compiler
> changes, too.
>
> First read is https://wiki.mozilla.org/ReleaseEngineering/TryServer, just
> to get the first piece of jargon out.
>
> I suspect that the next steps are:
>
> I speculate that you can use your version of llvm if you can create a
> docker image that is able to compile Firefox. Please confirm that with
> #taskcluster, and get some help there on how to do that.
>
> Then you'd want to make sure that you can actually compile/run
> mozilla-central. In particular wrt to taskcluster, there's been various
> file moves and changes, so working off of release is going to make your
> path rockier. I understand that mozilla-central is more of a moving target,
> but that shouldn't be that much of a problem for you in practice, I
hope.
>
> I suspect that just focusing on linux x64 is good for you for now?
>
> So jargon-thought-train:
>
> Validate my assumptions about all of this ;-) .
>
> Make your setup work with mozilla-central.
>
> Work in docker.
>
> Make your mozilla-central's taskcluster tasks for linux x64 pick your
> docker image to build on try.
>
> Push to try for linux x64, with all tests and perf tests.
>
> Use treeherder and perfherder (web uis) to see tests and performance.
>
>
> If you want to focus on your toolchain impact over time instead of
> mozilla-central, I think you should be able to keep the same base version
> of your patch and just update the docker image, and then use perfherder to
> compare the results.
>
> Hello Axel,
This seems good plan but I think I would require time to setup all these,
so I prefer to do this during next weekend.

Thanks,
Vivek
> HTH
>
> Axel
>
> On 19/06/16 20:41, vivek pandya wrote:
>
> Hello,
>
> I build FireFox-46.0.1 source with llvm to test interprocedural register
> allocation.
> The build was successful with out any runtime faliures, here are few stats:
>
> Measure W/O IPRA WITH IPRA
> ======= ======== ========> Total Build Time 76 mins 82.3 mins 8%
increment
> Octane v2.0 JS Benchmark Score (higher is better) 18675.69  19665.16 5%
> improvement
> Kraken JS Benchmark time (lower is better) 1416.2 ms 1421.3 ms 0.35%
> regression
> JetStream JS Benchmark Score (higer is better) 110.10 112.88 2.52%
> improvement
>
> Any suggestions are welcome on how to effectively measure performance
> improvement!
>
> Sincerely,
> Vivek
>
>
>
> _______________________________________________
> firefox-dev mailing listfirefox-dev at
mozilla.orghttps://mail.mozilla.org/listinfo/firefox-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160621/2d31a9f0/attachment.html>