I think it would be super useful to give LNT a RESTful interface. As LNT grows, it would be nice to be able to update some of the pages with Ajax(or json). Especially for pages with long load times, it makes sense to be able to speed up load times, by fetching the data after page load. It would also allow us to build more dynamic pages, and allow non-web clients better structured access to LNT information. We have been doing this to a degree already by adding &json=True parameters to some of the pages. This approach is ad-hoc, serialization and exploitability have been a big pain. I think it is time to formalize the non-web interface, and use some technology to make it as rich and useful as possible. Since we are not in the business of reinventing web technologies, I think we should use one of the preexisting flask frameworks for this. I have experimented with two: Eve + eve-sqlalchmey and flask-RESTful. http://python-eve.org <http://python-eve.org/> https://flask-restful.readthedocs.org/en/0.3.3/ <https://flask-restful.readthedocs.org/en/0.3.3/> If anyone in the community has some ideas on this I’d love to hear them. Thus far, I hacked on Eve for an afternoon and could not get it to understand LNTs (admittedly obscure) database structure. Flask-RESTful looked simple, but more work to build the actual interfaces, though it seems like it might be a more natural fit for LNTs URL schema. If no one has any options, I will try and integrate flaskRESTful into LNT. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150630/7f166af7/attachment.html>
This sounds very interesting . I’m all for not reinventing the wheel. I don’t have experience with frameworks to implement REST interfaces so can’t comment on which one is best suited for LNT. Just a few questions off the top of my head: Do you already have ideas for what the REST API roughly will look like? Very close to the database structure, or something slightly higher-level? Should the current existing URL scheme be extended to a full REST API; or do you think it’s best for the REST API to live in a separate “namespace”? Thanks! Kristof From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Chris Matthews Sent: 30 June 2015 19:53 To: LLVM Developers Mailing List Subject: [LLVMdev] Proposal: REST and Ajax in LNT I think it would be super useful to give LNT a RESTful interface. As LNT grows, it would be nice to be able to update some of the pages with Ajax(or json). Especially for pages with long load times, it makes sense to be able to speed up load times, by fetching the data after page load. It would also allow us to build more dynamic pages, and allow non-web clients better structured access to LNT information. We have been doing this to a degree already by adding &json=True parameters to some of the pages. This approach is ad-hoc, serialization and exploitability have been a big pain. I think it is time to formalize the non-web interface, and use some technology to make it as rich and useful as possible. Since we are not in the business of reinventing web technologies, I think we should use one of the preexisting flask frameworks for this. I have experimented with two: Eve + eve-sqlalchmey and flask-RESTful. http://python-eve.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__python-2Deve.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=2w-zKuHO6XKGe45Mp98iJPvAYkGZV_1z40LUQrrC8GY&s=m3f5xCrrNlH1r4TY3q-pzKYdpMp3dy4hzKs4REnVe0c&e=> https://flask-restful.readthedocs.org/en/0.3.3/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__flask-2Drestful.readthedocs.org_en_0.3.3_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=2w-zKuHO6XKGe45Mp98iJPvAYkGZV_1z40LUQrrC8GY&s=6rrEYWcF4BibyeSK23X9xK_FXUfGlfwRY6zlb03TdC4&e=> If anyone in the community has some ideas on this I’d love to hear them. Thus far, I hacked on Eve for an afternoon and could not get it to understand LNTs (admittedly obscure) database structure. Flask-RESTful looked simple, but more work to build the actual interfaces, though it seems like it might be a more natural fit for LNTs URL schema. If no one has any options, I will try and integrate flaskRESTful into LNT. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150630/c563d167/attachment.html>
On 30 June 2015 at 19:53, Chris Matthews <chris.matthews at apple.com> wrote:> I think it would be super useful to give LNT a RESTful interface. As LNT > grows, it would be nice to be able to update some of the pages with Ajax(or > json). Especially for pages with long load times, it makes sense to be able > to speed up load times, by fetching the data after page load. It would also > allow us to build more dynamic pages, and allow non-web clients better > structured access to LNT information.Hi Chris, As to what technology we use, I'm not particularly fussy, but REST can be a double edge sword, especially if your server is already fully loaded, like LNT. I'm not against it, by all means, I do believe it's the simplest and most efficient of web interfaces, particularly for dynamic websites like you propose. But we need to solve the stability problem first, or REST+Ajax will only make it worse. It may get the bare page quicker, but it will also load the server for longer, and multiply the number of database connections to the server, which is our main problem. Once we solve the slow queries problem, and increase the number of concurrent connections to a limit that we know we can handle, I'm all up for using whatever RESTful technology you like. :) cheers, --renato
That is a good question. I think the API should live outside the current url namesapce, but have a similar schema (maybe /api/database/resource). Where it makes sense, lets match the url scheme for the web interface. I think for non-web clients, they probably want basically JSONed versions of pretty much what is in the interface now (at least until they know better). The use cases I have seen are writing scripts to pull data for custom reporting dashboards, or pulling data into iPython Notebook. I think for Ajaxing the front end, we are going to want more DB like resources, probably with query parameters. My current motivation is getting access to the FieldChange database entries, to dynamically render them in a new interface. I would want FieldChanges + URI to anything they link to, I would also like to be able to limit/filter the queries and paginate the results. I also want to be able to construct a graph interface where data sets can be dynamically added and removed, that would be collecting sample sets, + exploring run machine data. For this in envisioned /api/database/FieldChanges with url or post parameters to set limits and filters.> On Jun 30, 2015, at 1:29 PM, Kristof Beyls <kristof.beyls at arm.com> wrote: > > This sounds very interesting . > > I’m all for not reinventing the wheel. I don’t have experience with frameworks to implement REST interfaces so can’t comment on which one is best suited for LNT. > > Just a few questions off the top of my head: > Do you already have ideas for what the REST API roughly will look like? > Very close to the database structure, or something slightly higher-level? > Should the current existing URL scheme be extended to a full REST API; or do you think it’s best for the REST API to live in a separate “namespace”? > > Thanks! > > Kristof > > From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On Behalf Of Chris Matthews > Sent: 30 June 2015 19:53 > To: LLVM Developers Mailing List > Subject: [LLVMdev] Proposal: REST and Ajax in LNT > > I think it would be super useful to give LNT a RESTful interface. As LNT grows, it would be nice to be able to update some of the pages with Ajax(or json). Especially for pages with long load times, it makes sense to be able to speed up load times, by fetching the data after page load. It would also allow us to build more dynamic pages, and allow non-web clients better structured access to LNT information. > > We have been doing this to a degree already by adding &json=True parameters to some of the pages. This approach is ad-hoc, serialization and exploitability have been a big pain. I think it is time to formalize the non-web interface, and use some technology to make it as rich and useful as possible. > > Since we are not in the business of reinventing web technologies, I think we should use one of the preexisting flask frameworks for this. I have experimented with two: Eve + eve-sqlalchmey and flask-RESTful. > > http://python-eve.org <https://urldefense.proofpoint.com/v2/url?u=http-3A__python-2Deve.org&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=2w-zKuHO6XKGe45Mp98iJPvAYkGZV_1z40LUQrrC8GY&s=m3f5xCrrNlH1r4TY3q-pzKYdpMp3dy4hzKs4REnVe0c&e=> > https://flask-restful.readthedocs.org/en/0.3.3/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__flask-2Drestful.readthedocs.org_en_0.3.3_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=2w-zKuHO6XKGe45Mp98iJPvAYkGZV_1z40LUQrrC8GY&s=6rrEYWcF4BibyeSK23X9xK_FXUfGlfwRY6zlb03TdC4&e=> > > If anyone in the community has some ideas on this I’d love to hear them. Thus far, I hacked on Eve for an afternoon and could not get it to understand LNTs (admittedly obscure) database structure. Flask-RESTful looked simple, but more work to build the actual interfaces, though it seems like it might be a more natural fit for LNTs URL schema. > > If no one has any options, I will try and integrate flaskRESTful into LNT.-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150630/be11b729/attachment.html>
I don’t want to block progress on this because of a bug on the llvm.org <http://llvm.org/> LNT server. That just needs to be fixed. I don’t think the issues of scale are a problem with LNT inheriently. We run a private LNT server instance on smaller hardware that is running 5X the amount of work the llvm.org server is. It has very good uptime and no request failures, in fact I would say it could still handle 10x the load it is running at right now, maybe even 20x. In the past we have had issues with bad workloads breaking the server, and if you sit down and have access to the logs and the running server it take almost no time to sort the issue out. The llvm.org <http://llvm.org/> LNT server has a bug with long running requests, and that needs to be fixed. I agree that we should be careful with the queries and number of requests. But I think in the long run this would provide a net win. We can start to build interfaces that don’ t need to pull as much data out of the database. Lots of short fast requests. In LNT we have consistently made the tradeoff of presenting all the data, since breaking up requests requires far more complicated interface design. With these apis, we can start to undo that mess!> On Jun 30, 2015, at 2:09 PM, Renato Golin <renato.golin at linaro.org> wrote: > > On 30 June 2015 at 19:53, Chris Matthews <chris.matthews at apple.com> wrote: >> I think it would be super useful to give LNT a RESTful interface. As LNT >> grows, it would be nice to be able to update some of the pages with Ajax(or >> json). Especially for pages with long load times, it makes sense to be able >> to speed up load times, by fetching the data after page load. It would also >> allow us to build more dynamic pages, and allow non-web clients better >> structured access to LNT information. > > Hi Chris, > > As to what technology we use, I'm not particularly fussy, but REST can > be a double edge sword, especially if your server is already fully > loaded, like LNT. > > I'm not against it, by all means, I do believe it's the simplest and > most efficient of web interfaces, particularly for dynamic websites > like you propose. But we need to solve the stability problem first, or > REST+Ajax will only make it worse. > > It may get the bare page quicker, but it will also load the server for > longer, and multiply the number of database connections to the server, > which is our main problem. > > Once we solve the slow queries problem, and increase the number of > concurrent connections to a limit that we know we can handle, I'm all > up for using whatever RESTful technology you like. :) > > cheers, > --renato-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150630/01286fec/attachment.html>