webmail.gandi.net
2024-Feb-21 07:01 UTC
[Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
Thank you, Ivan for this investigation. I inspected the R changes file (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing about this. I should inspect the sources too! It could possibly break other Tcl/Tk related stuff. The doc about Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl is used from an external program, Tcl_ServiceAll should be used in its event loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes in that direction). But in the other hand, it is indicated that Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent should be called in this case. I think we spotted one case where Tcl_ServiceAll is not doing its job correctly. There may be others. Since the {tcltk} package was working fine with "while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev?;", unless there is a clear performance enhancement with "while (i-- && Tcl_ServiceAll())", it would perhaps be wise to revert this back. Indeed, when I use this on the server side with R 4.3.2: library(tcltk) cmd <- r"( proc accept {chan addr port} { ;# Make a proc to accept connections puts "$addr:$port says [gets $chan]" ;# Receive a string puts $chan goodbye ;# Send a string close $chan ;# Close the socket (automatically flushes) } ;# socket -server accept 12345 ;# Create a server socket)" .Tcl(cmd) .Tcl("vwait myvar") It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of course, in this case, R is blocked and waits for the `myvar` variable on the Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not catching all Tcl events. All the best, Philippe ..............................................<?}))><........ ) ) ) ) ) ( ( ( ( ( Prof. Philippe Grosjean ) ) ) ) ) ( ( ( ( ( Numerical Ecology ) ) ) ) ) Mons University, Belgium ( ( ( ( ( ..............................................................> Le 20 f?vr. 2024 ? 17:13, Ivan Krylov via R-devel <r-devel at r-project.org> a ?crit : > > ? Tue, 20 Feb 2024 12:27:35 +0100 > "webmail.gandi.net" <phgrosjean at sciviews.org> ?????: > >> When R process #1 is R 4.2.3, it works as expected (whatever version >> of R #2). When R process #1 is R 4.3.2, nothing is sent or received >> through the socket apparently, but no error is issued and process #2 >> seems to be able to connect to the socket. > > The difference is related to the change in > src/library/tcltk/src/tcltk_unix.c. > > In R-4.2.1, the function static void TclSpinLoop(void *data) says: > > int max_ev = 100; > /* Tcl_ServiceAll is not enough here, for reasons that escape me */ > while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; > > In R-devel, the function instead says: > > int i = R_TCL_SPIN_MAX; > while (i-- && Tcl_ServiceAll()) > ; > > Manually calling Tcl_DoOneEvent(0) from the debugger at this point > makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to > be still not enough. I'll try reading Tcl documentation to investigate > this further. > > -- > Best regards, > Ivan > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel[[alternative HTML version deleted]]
Ivan Krylov
2024-Feb-21 12:51 UTC
[Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
? Wed, 21 Feb 2024 08:01:16 +0100 "webmail.gandi.net" <phgrosjean at sciviews.org> ?????:> Since the {tcltk} package was working fine with "while > (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev?;", unless there is > a clear performance enhancement with "while (i-- && > Tcl_ServiceAll())", it would perhaps be wise to revert this back.I forgot to mention the comment in the new version of the function explaining the switch:>> [Tcl_DoOneEvent(TCL_DONT_WAIT)] <...> causes infinite recursion with >> R handlers that have a re-entrancy guard, when TclSpinLoop is >> invoked from such a handler (seen with Rhttp server)The difference between Tcl_ServiceAll() and Tcl_DoOneEvent() is that the latter calls Tcl_WaitForEvent(). The comments say that it is called for the side effect of queuing the events detected by select(). The function can indeed be observed to access the fileHandlers via the thread-specific data pointer, which contain the file descriptors and the instructions saying what to do with them. Without Tcl_WaitForEvent, the only event sources known to Tcl are RTcl_{setup,check}Proc (which only checks file descriptors owned by R), Display{Setup,Check}Proc (which seems to be owned by Tk), and Timer{Setup,Check}Proc (for which there doesn't seem to be any timers by default). As far as I understand the problem, while the function worker_input_handler() from src/modules/internet/Rhttpd.c is running, TclHandler() might be invoked, causing Tcl_DoOneEvent() to call RTcl_checkProc() and therefore trying to run worker_input_handler() again. The Rhttpd handler prevents this and doesn't clear the condition, which causes the event loop to keep calling it. Is that correct? Are there easy ways to reproduce the problem? -- Best regards, Ivan
peter dalgaard
2024-Feb-21 13:10 UTC
[Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
I don't think we're going to fix this before 4.3.3. Given that it has gone unnoticed since June 2022 (yes '22) and that tampering in this area has a history of popping up complications in other areas, I think we should leave it alone until 4.4.0. (I see that Ivan and Tomas has been on the issue since I started writing, but the above probably still holds true.) - Peter D.> On 21 Feb 2024, at 08:01 , webmail.gandi.net <phgrosjean at sciviews.org> wrote: > > Thank you, Ivan for this investigation. I inspected the R changes file (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing about this. I should inspect the sources too! > > It could possibly break other Tcl/Tk related stuff. The doc about Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl is used from an external program, Tcl_ServiceAll should be used in its event loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes in that direction). But in the other hand, it is indicated that Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent should be called in this case. I think we spotted one case where Tcl_ServiceAll is not doing its job correctly. There may be others. > > Since the {tcltk} package was working fine with "while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev?;", unless there is a clear performance enhancement with "while (i-- && Tcl_ServiceAll())", it would perhaps be wise to revert this back. > > Indeed, when I use this on the server side with R 4.3.2: > > library(tcltk) > cmd <- r"( > proc accept {chan addr port} { ;# Make a proc to accept connections > puts "$addr:$port says [gets $chan]" ;# Receive a string > puts $chan goodbye ;# Send a string > close $chan ;# Close the socket (automatically flushes) > } ;# > socket -server accept 12345 ;# Create a server socket)" > .Tcl(cmd) > .Tcl("vwait myvar") > > It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of course, in this case, R is blocked and waits for the `myvar` variable on the Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not catching all Tcl events. > > All the best, > > Philippe > > ..............................................<?}))><........ > ) ) ) ) ) > ( ( ( ( ( Prof. Philippe Grosjean > ) ) ) ) ) > ( ( ( ( ( Numerical Ecology > ) ) ) ) ) Mons University, Belgium > ( ( ( ( ( > .............................................................. > >> Le 20 f?vr. 2024 ? 17:13, Ivan Krylov via R-devel <r-devel at r-project.org> a ?crit : >> >> ? Tue, 20 Feb 2024 12:27:35 +0100 >> "webmail.gandi.net" <phgrosjean at sciviews.org> ?????: >> >>> When R process #1 is R 4.2.3, it works as expected (whatever version >>> of R #2). When R process #1 is R 4.3.2, nothing is sent or received >>> through the socket apparently, but no error is issued and process #2 >>> seems to be able to connect to the socket. >> >> The difference is related to the change in >> src/library/tcltk/src/tcltk_unix.c. >> >> In R-4.2.1, the function static void TclSpinLoop(void *data) says: >> >> int max_ev = 100; >> /* Tcl_ServiceAll is not enough here, for reasons that escape me */ >> while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; >> >> In R-devel, the function instead says: >> >> int i = R_TCL_SPIN_MAX; >> while (i-- && Tcl_ServiceAll()) >> ; >> >> Manually calling Tcl_DoOneEvent(0) from the debugger at this point >> makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to >> be still not enough. I'll try reading Tcl documentation to investigate >> this further. >> >> -- >> Best regards, >> Ivan >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel-- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com
Tomas Kalibera
2024-Feb-21 16:04 UTC
[Rd] Tcl socket server (tcltk) does not work any more on R 4.3.2
On 2/21/24 08:01, webmail.gandi.net wrote:> Thank you, Ivan for this investigation. I inspected the R changes file (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) and found nothing about this. I should inspect the sources too! > > It could possibly break other Tcl/Tk related stuff. The doc about Tcl_ServiceAll and Tcl_DoOneEvent is confusing. On one hand, it says when Tcl is used from an external program, Tcl_ServiceAll should be used in its event loop instead of Tcl_DoOneEvent (and the change in the latest R versions goes in that direction). But in the other hand, it is indicated that Tcl_ServiceAll does not always handle all Tcl events and extra Tcl_DoOneEvent should be called in this case. I think we spotted one case where Tcl_ServiceAll is not doing its job correctly. There may be others. > > Since the {tcltk} package was working fine with "while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev?;", unless there is a clear performance enhancement with "while (i-- && Tcl_ServiceAll())", it would perhaps be wise to revert this back.Yes, for now I've done that in R-devel. The Tcl documentation is really hard to follow, but debugging reveals that Tcl_ServiceAll() doesn't queue new events, as also Ivan reports. Tomas> > Indeed, when I use this on the server side with R 4.3.2: > > library(tcltk) > cmd <- r"( > proc accept {chan addr port} { ;# Make a proc to accept connections > puts "$addr:$port says [gets $chan]" ;# Receive a string > puts $chan goodbye ;# Send a string > close $chan ;# Close the socket (automatically flushes) > } ;# > socket -server accept 12345 ;# Create a server socket)" > .Tcl(cmd) > .Tcl("vwait myvar") > > It works again as expected. And vwait is known to call Tcl_DoOneEvent. Of course, in this case, R is blocked and waits for the `myvar` variable on the Tcl side. Anyway, the problem seems to be really in Tcl_ServiceAll not catching all Tcl events. > > All the best, > > Philippe > > ..............................................<?}))><........ > ) ) ) ) ) > ( ( ( ( ( Prof. Philippe Grosjean > ) ) ) ) ) > ( ( ( ( ( Numerical Ecology > ) ) ) ) ) Mons University, Belgium > ( ( ( ( ( > .............................................................. > >> Le 20 f?vr. 2024 ? 17:13, Ivan Krylov via R-devel <r-devel at r-project.org> a ?crit : >> >> ? Tue, 20 Feb 2024 12:27:35 +0100 >> "webmail.gandi.net" <phgrosjean at sciviews.org> ?????: >> >>> When R process #1 is R 4.2.3, it works as expected (whatever version >>> of R #2). When R process #1 is R 4.3.2, nothing is sent or received >>> through the socket apparently, but no error is issued and process #2 >>> seems to be able to connect to the socket. >> The difference is related to the change in >> src/library/tcltk/src/tcltk_unix.c. >> >> In R-4.2.1, the function static void TclSpinLoop(void *data) says: >> >> int max_ev = 100; >> /* Tcl_ServiceAll is not enough here, for reasons that escape me */ >> while (Tcl_DoOneEvent(TCL_DONT_WAIT) && max_ev) max_ev--; >> >> In R-devel, the function instead says: >> >> int i = R_TCL_SPIN_MAX; >> while (i-- && Tcl_ServiceAll()) >> ; >> >> Manually calling Tcl_DoOneEvent(0) from the debugger at this point >> makes the Tcl code respond to the connection. Tcl_ServiceAll() seems to >> be still not enough. I'll try reading Tcl documentation to investigate >> this further. >> >> -- >> Best regards, >> Ivan >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel