Tobias Grosser via llvm-dev
2017-Aug-03 04:11 UTC
[llvm-dev] [LNT] new server instance http://lnt.llvm.org seems unstable
On Wed, Aug 2, 2017, at 21:55, Tobias Grosser via llvm-dev wrote:> This started since about 1-2 weeks, I guess. I am not sure about it > either, but seems unfortunate. Maybe we run into some kind of timeout?And another one :( http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio 2017-08-03 00:24:44 CRITICAL: Results were not obtained from submission. nt /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-0/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-1/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-2/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-3/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-4/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-5/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-6/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-7/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-8/report.simple.csv /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-9/report.simple.csv error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <title>500 Internal Server Error</title> <h1>Internal Server Error</h1> <p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p> 2017-08-03 00:24:44 INFO: Running process cleanup. program finished with exit code 1 elapsedTime=20725.395111 Best, Tobias> > Best, > Tobias > > On Wed, Aug 2, 2017, at 21:51, Matthias Braun via llvm-dev wrote: > > Actually I just remember LNT has a page for viewing the log: > > > > http://lnt.llvm.org/log > > > > ... > > OperationalError: (OperationalError) (2006, 'MySQL server has gone away') > > 'SELECT `NT_Run`.`ID` AS `NT_Run_ID`, `NT_Run`.`MachineID` AS > > `NT_Run_MachineID`, `NT_Run`.`OrderID` AS `NT_Run_OrderID`, > > `NT_Run`.`ImportedFrom` AS `NT_Run_ImportedFrom`, `NT_Run`.`StartTime` AS > > `NT_Run_StartTime`, `NT_Run`.`EndTime` AS `NT_Run_EndTime`, > > `NT_Run`.`SimpleRunID` AS `NT_Run_SimpleRunID`, `NT_Run`.`Parameters` AS > > `NT_Run_Parameters` \nFROM `NT_Run` \nWHERE `NT_Run`.`ID` = %s' (62996L,) > > ... > > > > Not sure what is causing that or what to do about it though. > > > > - Matthias > > > > > On Aug 2, 2017, at 12:47 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > > > > > > It's not super urgent, just wanted to make sure it does not get lost. > > > I CC Chris to make sure he has a full inbox when coming back from > > > vacation. ;-) > > > > > > @Chris, hope you have a good and relaxed stay. > > > > > > Best, > > > Tobias > > > > > > On Wed, Aug 2, 2017, at 21:46, Matthias Braun wrote: > > >> Chris is on vacation this week. Not sure if someone else has access to > > >> the logs... > > >> > > >> - Matthias > > >> > > >>> On Aug 2, 2017, at 12:45 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > > >>> > > >>> OK, who is in charge? > > >>> > > >>> Best, > > >>> Tobias > > >>> > > >>> On Wed, Aug 2, 2017, at 21:45, Matthias Braun via llvm-dev wrote: > > >>>> > > >>>>> On Aug 2, 2017, at 12:43 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > > >>>>> > > >>>>> On Tue, Aug 1, 2017, at 00:33, Matthias Braun via llvm-dev wrote: > > >>>>>> The run page problem were triggered by one of my commits (sorry) and > > >>>>>> should be mitigated now, see the thread at > > >>>>>> http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html > > >>>>>> <http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html> > > >>>>>> > > >>>>>> I don't know about the submission problems, could they just an occasional > > >>>>>> network problem or are they a common phenomenon? Chris did some > > >>>>>> improvements to LNT to report back problems from the server side rather > > >>>>>> than a generic "500 internal server error" in case of errors. This should > > >>>>>> become active the next time lnt.llvm.org <http://lnt.llvm.org/> is > > >>>>>> updated. > > >>>>> > > >>>>> Just FYI, I still see such errors on the most recent bui;lds: > > >>>> > > >>>> Seems it is something else then, someone needs to inspect the server logs > > >>>> to find out more about what is going on. > > >>>> > > >>>> - Matthias > > >>>> > > >>>> _______________________________________________ > > >>>> LLVM Developers mailing list > > >>>> llvm-dev at lists.llvm.org > > >>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >> > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Chris Matthews via llvm-dev
2017-Aug-07 23:13 UTC
[llvm-dev] [LNT] new server instance http://lnt.llvm.org seems unstable
I noticed some strange stuff going on in the background workers because of the eventlets. Very much like what is discussed here: https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth <https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth> I have disabled that feature in Gunicorn. It could impact db level socket stuff too. Please let me know if you continue to see issues now.> On Aug 2, 2017, at 9:11 PM, Tobias Grosser via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > On Wed, Aug 2, 2017, at 21:55, Tobias Grosser via llvm-dev wrote: >> This started since about 1-2 weeks, I guess. I am not sure about it >> either, but seems unfortunate. Maybe we run into some kind of timeout? > > And another one :( > > http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio <http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio> > > 2017-08-03 00:24:44 CRITICAL: Results were not obtained from submission. > nt > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-0/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-1/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-2/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-3/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-4/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-5/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-6/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-7/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-8/report.simple.csv > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-9/report.simple.csv > error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> > <title>500 Internal Server Error</title> > <h1>Internal Server Error</h1> > <p>The server encountered an internal error and was unable to complete > your request. Either the server is overloaded or there is an error in > the application.</p> > > 2017-08-03 00:24:44 INFO: Running process cleanup. > program finished with exit code 1 > elapsedTime=20725.395111 > > Best, > Tobias > >> >> Best, >> Tobias >> >> On Wed, Aug 2, 2017, at 21:51, Matthias Braun via llvm-dev wrote: >>> Actually I just remember LNT has a page for viewing the log: >>> >>> http://lnt.llvm.org/log >>> >>> ... >>> OperationalError: (OperationalError) (2006, 'MySQL server has gone away') >>> 'SELECT `NT_Run`.`ID` AS `NT_Run_ID`, `NT_Run`.`MachineID` AS >>> `NT_Run_MachineID`, `NT_Run`.`OrderID` AS `NT_Run_OrderID`, >>> `NT_Run`.`ImportedFrom` AS `NT_Run_ImportedFrom`, `NT_Run`.`StartTime` AS >>> `NT_Run_StartTime`, `NT_Run`.`EndTime` AS `NT_Run_EndTime`, >>> `NT_Run`.`SimpleRunID` AS `NT_Run_SimpleRunID`, `NT_Run`.`Parameters` AS >>> `NT_Run_Parameters` \nFROM `NT_Run` \nWHERE `NT_Run`.`ID` = %s' (62996L,) >>> ... >>> >>> Not sure what is causing that or what to do about it though. >>> >>> - Matthias >>> >>>> On Aug 2, 2017, at 12:47 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: >>>> >>>> It's not super urgent, just wanted to make sure it does not get lost. >>>> I CC Chris to make sure he has a full inbox when coming back from >>>> vacation. ;-) >>>> >>>> @Chris, hope you have a good and relaxed stay. >>>> >>>> Best, >>>> Tobias >>>> >>>> On Wed, Aug 2, 2017, at 21:46, Matthias Braun wrote: >>>>> Chris is on vacation this week. Not sure if someone else has access to >>>>> the logs... >>>>> >>>>> - Matthias >>>>> >>>>>> On Aug 2, 2017, at 12:45 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: >>>>>> >>>>>> OK, who is in charge? >>>>>> >>>>>> Best, >>>>>> Tobias >>>>>> >>>>>> On Wed, Aug 2, 2017, at 21:45, Matthias Braun via llvm-dev wrote: >>>>>>> >>>>>>>> On Aug 2, 2017, at 12:43 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017, at 00:33, Matthias Braun via llvm-dev wrote: >>>>>>>>> The run page problem were triggered by one of my commits (sorry) and >>>>>>>>> should be mitigated now, see the thread at >>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html >>>>>>>>> <http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html> >>>>>>>>> >>>>>>>>> I don't know about the submission problems, could they just an occasional >>>>>>>>> network problem or are they a common phenomenon? Chris did some >>>>>>>>> improvements to LNT to report back problems from the server side rather >>>>>>>>> than a generic "500 internal server error" in case of errors. This should >>>>>>>>> become active the next time lnt.llvm.org <http://lnt.llvm.org/> is >>>>>>>>> updated. >>>>>>>> >>>>>>>> Just FYI, I still see such errors on the most recent bui;lds: >>>>>>> >>>>>>> Seems it is something else then, someone needs to inspect the server logs >>>>>>> to find out more about what is going on. >>>>>>> >>>>>>> - Matthias >>>>>>> >>>>>>> _______________________________________________ >>>>>>> LLVM Developers mailing list >>>>>>> llvm-dev at lists.llvm.org >>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>>> >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> llvm-dev at lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170807/5d57a11d/attachment.html>
Tobias Grosser via llvm-dev
2017-Aug-08 06:00 UTC
[llvm-dev] [LNT] new server instance http://lnt.llvm.org seems unstable
Thank you, I will keep an eye on the servers. Best, Tobias On Tue, Aug 8, 2017, at 01:13, Chris Matthews via llvm-dev wrote:> I noticed some strange stuff going on in the background workers because > of the eventlets. Very much like what is discussed here: > > https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth > <https://stackoverflow.com/questions/14736766/why-does-gevent-socket-break-multiprocessing-connections-auth> > > I have disabled that feature in Gunicorn. It could impact db level > socket stuff too. > > Please let me know if you continue to see issues now. > > > > > On Aug 2, 2017, at 9:11 PM, Tobias Grosser via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > > On Wed, Aug 2, 2017, at 21:55, Tobias Grosser via llvm-dev wrote: > >> This started since about 1-2 weeks, I guess. I am not sure about it > >> either, but seems unfortunate. Maybe we run into some kind of timeout? > > > > And another one :( > > > > http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio <http://lab.llvm.org:8011/builders/perf-x86_64-penryn-O3-polly/builds/1986/steps/lnt.nightly-test/logs/stdio> > > > > 2017-08-03 00:24:44 CRITICAL: Results were not obtained from submission. > > nt > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-0/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-1/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-2/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-3/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-4/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-5/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-6/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-7/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-8/report.simple.csv > > /home/grosser/buildslave/perf-x86_64-penryn-O3-polly/tests/nt/build/sample-9/report.simple.csv > > error: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> > > <title>500 Internal Server Error</title> > > <h1>Internal Server Error</h1> > > <p>The server encountered an internal error and was unable to complete > > your request. Either the server is overloaded or there is an error in > > the application.</p> > > > > 2017-08-03 00:24:44 INFO: Running process cleanup. > > program finished with exit code 1 > > elapsedTime=20725.395111 > > > > Best, > > Tobias > > > >> > >> Best, > >> Tobias > >> > >> On Wed, Aug 2, 2017, at 21:51, Matthias Braun via llvm-dev wrote: > >>> Actually I just remember LNT has a page for viewing the log: > >>> > >>> http://lnt.llvm.org/log > >>> > >>> ... > >>> OperationalError: (OperationalError) (2006, 'MySQL server has gone away') > >>> 'SELECT `NT_Run`.`ID` AS `NT_Run_ID`, `NT_Run`.`MachineID` AS > >>> `NT_Run_MachineID`, `NT_Run`.`OrderID` AS `NT_Run_OrderID`, > >>> `NT_Run`.`ImportedFrom` AS `NT_Run_ImportedFrom`, `NT_Run`.`StartTime` AS > >>> `NT_Run_StartTime`, `NT_Run`.`EndTime` AS `NT_Run_EndTime`, > >>> `NT_Run`.`SimpleRunID` AS `NT_Run_SimpleRunID`, `NT_Run`.`Parameters` AS > >>> `NT_Run_Parameters` \nFROM `NT_Run` \nWHERE `NT_Run`.`ID` = %s' (62996L,) > >>> ... > >>> > >>> Not sure what is causing that or what to do about it though. > >>> > >>> - Matthias > >>> > >>>> On Aug 2, 2017, at 12:47 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > >>>> > >>>> It's not super urgent, just wanted to make sure it does not get lost. > >>>> I CC Chris to make sure he has a full inbox when coming back from > >>>> vacation. ;-) > >>>> > >>>> @Chris, hope you have a good and relaxed stay. > >>>> > >>>> Best, > >>>> Tobias > >>>> > >>>> On Wed, Aug 2, 2017, at 21:46, Matthias Braun wrote: > >>>>> Chris is on vacation this week. Not sure if someone else has access to > >>>>> the logs... > >>>>> > >>>>> - Matthias > >>>>> > >>>>>> On Aug 2, 2017, at 12:45 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > >>>>>> > >>>>>> OK, who is in charge? > >>>>>> > >>>>>> Best, > >>>>>> Tobias > >>>>>> > >>>>>> On Wed, Aug 2, 2017, at 21:45, Matthias Braun via llvm-dev wrote: > >>>>>>> > >>>>>>>> On Aug 2, 2017, at 12:43 PM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > >>>>>>>> > >>>>>>>> On Tue, Aug 1, 2017, at 00:33, Matthias Braun via llvm-dev wrote: > >>>>>>>>> The run page problem were triggered by one of my commits (sorry) and > >>>>>>>>> should be mitigated now, see the thread at > >>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html > >>>>>>>>> <http://lists.llvm.org/pipermail/llvm-dev/2017-July/115971.html> > >>>>>>>>> > >>>>>>>>> I don't know about the submission problems, could they just an occasional > >>>>>>>>> network problem or are they a common phenomenon? Chris did some > >>>>>>>>> improvements to LNT to report back problems from the server side rather > >>>>>>>>> than a generic "500 internal server error" in case of errors. This should > >>>>>>>>> become active the next time lnt.llvm.org <http://lnt.llvm.org/> is > >>>>>>>>> updated. > >>>>>>>> > >>>>>>>> Just FYI, I still see such errors on the most recent bui;lds: > >>>>>>> > >>>>>>> Seems it is something else then, someone needs to inspect the server logs > >>>>>>> to find out more about what is going on. > >>>>>>> > >>>>>>> - Matthias > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> LLVM Developers mailing list > >>>>>>> llvm-dev at lists.llvm.org > >>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >>>>> > >>> > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >> _______________________________________________ > >> LLVM Developers mailing list > >> llvm-dev at lists.llvm.org > >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev