llvm dev - Oct 2015 - RFC: Adding background workers to LNT

One limitation LNT has is that all processing must take place within a request.
This limits our ability to do anything really compute heavy, we are always bound
by page load time.

I’d like to add a background processing feature to LNT.  This would allow a
request to queue some work to be processed later by a subprocess.  Things I’d
like to use this feature on: generating caches to speed up page load times,
calculating derived metrics (like geomean and SPEC score) during submission,
calculating field changes during submission, running much more compute heavy
submission processing to correlate detected regressions.

The defacto standard for doing background processing in Flask is Celery.  I
don’t think that is a great option for LNT. It is a big package dependency, and
requires Redis/Rabbitmq or another database as a communication mechansim. We
want LNT to still be easy to use from the command line.  I propose we use
Python’s built in multiprocessing package, to setup a pool of workers.  This
adds no dependencies; though it will lack some of the cool features of Celery
like debugging and monitoring.

Are there other opportunities for background work anyone can think of?  Are
there other task queuing solutions that might be more appropriate?  Thanks