iuke-tier@ey m@iii@g oii uiow@@edu
2024-Apr-24 20:31 UTC
[Rd] [External] Re: Is ALTREP "non-API"?
On Wed, 24 Apr 2024, Hadley Wickham wrote:>> >> >> >>>>> That is not true at all - the presence of header does not constitute >>>> declaration of something as the R API. There are cases where internal >>>> functions are in the headers for historical or other reasons since the >>>> headers are used both for the internal implementation and packages. >> That's >>>> why this is in R-exts under "The R API: entry points for C code": >>>>> >>>>> If I understand your point correctly, does this mean that >>>> Rf_allocVector() is not part of the "official" R API? It does not >> appear to >>>> be documented in the "The R API: entry points for C code" section. >>>>> >>>> >>>> It does, obviously: >>>> https://cran.r-project.org/doc/manuals/R-exts.html#Allocating-storage-1 >>> >>> >>> I'm just trying to understand the precise definition of the official API >>> here. So it's any function mentioned in R-exts, regardless of which >> section >>> it appears in? >>> >>> Does this sentence imply that all functions starting with alloc* are part >>> of the official API? >>> >> >> Again, I can only quote the R-exts (few lines below the previous "The R >> API" quote): >> >> >> We can classify the entry points as >> API >> Entry points which are documented in this manual and declared in an >> installed header file. These can be used in distributed packages and will >> only be changed after deprecation. >> >> >> It says "in this manual" - I don't see anywhere restriction on a >> particular section of the manual, so I really don't see why you would think >> that allocation is not part on the API. >> > > Because you mentioned that section explicitly earlier in the thread. This > obviously seems clear to you, but it's not at all clear to me and I suspect > many of the wider community. It's frustrating because we are trying > our best to do what y'all want us to do, but it feels like we keep getting > the rug pulled out from under us with very little notice, and then have to > spend a large amount of time figuring out workarounds.Please try to keep this discussion non-adversarial.> That is at least > feasible for my team since we have multiple talented folks who are paid > full-time to work on R, but it's a huge struggle for most people who are > generally maintaining packages in their spare time.As you well know, almost all R-core members are also trying to maintain and improve R in their spare time. Good for folks to keep in mind before demanding R-core do X, Y, or Z for you.> For the purposes of this discussion could you please "documented in the > manual" means? For example, this line mentions allocXxx functions: "There > are quite a few allocXxx functions defined in Rinternals.h?you may want to > explore them.". Does that imply that they are documented and free to use?Where we are now in terms of what package authors can use to write R extensions has evolved organically over many years. The current state is certainly not ideal: There are entry points in installed headers that might be available; but to find out if they are in fact available requires reading prose text in the header files and in WRE. Trying to fine-tune wording in WRE, or add a lot of additional entries is not really a good or realistic way forward: WRE is both documentation and tutorial and more legalistic language/more complete coverage would make it less readable and still not guarantee completeness or clarity. We would be better off (in my view, not necessarily shared by others in R-core) if we could get to a point where: all entry points listed in installed header files can be used in packages, at least with some caveats; the caveats are expressed in a standard way that is searchable, e.g. with a standardized comment syntax at the header file or individual declaration level. In principle this is achievable, but getting there from where we are now is a lot of work. There are some 500 entry points in the R shared library that are in the installed headers but not mentioned in WRE. These would need to be reviewed and adjusted. My guess is about a third are fine and intended to be API-stable, another third are not used in packages and don't need to be in public headers. The remainder are things that may be used in current packages but really should not be, for example because they expose internal data in ways that can cause segfaults or they make it difficult to implement performance improvements in the base engine. Sorting through these and working with package authors to find alternate, safer options takes a lot of time (see 'spare time' above) and energy (some package authors are easier to work with than others). Several of us have taken cracks at moving this forward from time to time, but it rarely gets to the top of anyone's priority list.> And in general, I'd urge R Core to make an explicit list of functions that > you consider to be part of the exported API, and then grandfather in > packages that used those functions prior to learning that we weren't > supposed to.Making a list and hoping that it will remain up to date is not realistic. The only way that would work reliably is if the list could be programmatically generated, for example by parsing installed headers for declarations and caveats as above. Which would be possible with changes like the ones listed above. Best, luke> > Hadley > > >-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu/
On Wed, Apr 24, 2024 at 1:32?PM luke-tierney--- via R-devel <r-devel at r-project.org> wrote:> > On Wed, 24 Apr 2024, Hadley Wickham wrote: > > >> > >> > >> > >>>>> That is not true at all - the presence of header does not constitute > >>>> declaration of something as the R API. There are cases where internal > >>>> functions are in the headers for historical or other reasons since the > >>>> headers are used both for the internal implementation and packages. > >> That's > >>>> why this is in R-exts under "The R API: entry points for C code": > >>>>> > >>>>> If I understand your point correctly, does this mean that > >>>> Rf_allocVector() is not part of the "official" R API? It does not > >> appear to > >>>> be documented in the "The R API: entry points for C code" section. > >>>>> > >>>> > >>>> It does, obviously: > >>>> https://cran.r-project.org/doc/manuals/R-exts.html#Allocating-storage-1 > >>> > >>> > >>> I'm just trying to understand the precise definition of the official API > >>> here. So it's any function mentioned in R-exts, regardless of which > >> section > >>> it appears in? > >>> > >>> Does this sentence imply that all functions starting with alloc* are part > >>> of the official API? > >>> > >> > >> Again, I can only quote the R-exts (few lines below the previous "The R > >> API" quote): > >> > >> > >> We can classify the entry points as > >> API > >> Entry points which are documented in this manual and declared in an > >> installed header file. These can be used in distributed packages and will > >> only be changed after deprecation. > >> > >> > >> It says "in this manual" - I don't see anywhere restriction on a > >> particular section of the manual, so I really don't see why you would think > >> that allocation is not part on the API. > >> > > > > Because you mentioned that section explicitly earlier in the thread. This > > obviously seems clear to you, but it's not at all clear to me and I suspect > > many of the wider community. It's frustrating because we are trying > > our best to do what y'all want us to do, but it feels like we keep getting > > the rug pulled out from under us with very little notice, and then have to > > spend a large amount of time figuring out workarounds. > > Please try to keep this discussion non-adversarial. > > > That is at least > > feasible for my team since we have multiple talented folks who are paid > > full-time to work on R, but it's a huge struggle for most people who are > > generally maintaining packages in their spare time. > > As you well know, almost all R-core members are also trying to > maintain and improve R in their spare time. Good for folks to keep in > mind before demanding R-core do X, Y, or Z for you. > > > For the purposes of this discussion could you please "documented in the > > manual" means? For example, this line mentions allocXxx functions: "There > > are quite a few allocXxx functions defined in Rinternals.h?you may want to > > explore them.". Does that imply that they are documented and free to use? > > Where we are now in terms of what package authors can use to write R > extensions has evolved organically over many years. The current state > is certainly not ideal: > > There are entry points in installed headers that might be > available; > > but to find out if they are in fact available requires reading > prose text in the header files and in WRE. > > Trying to fine-tune wording in WRE, or add a lot of additional entries > is not really a good or realistic way forward: WRE is both > documentation and tutorial and more legalistic language/more complete > coverage would make it less readable and still not guarantee > completeness or clarity. > > We would be better off (in my view, not necessarily shared by others > in R-core) if we could get to a point where: > > all entry points listed in installed header files can be used in > packages, at least with some caveats; > > the caveats are expressed in a standard way that is searchable, > e.g. with a standardized comment syntax at the header file or > individual declaration level. > > In principle this is achievable, but getting there from where we are > now is a lot of work. There are some 500 entry points in the R shared > library that are in the installed headers but not mentioned in WRE. > These would need to be reviewed and adjusted. My guess is about a > third are fine and intended to be API-stable, another third are not > used in packages and don't need to be in public headers. The remainder > are things that may be used in current packages but really should not > be, for example because they expose internal data in ways that can > cause segfaults or they make it difficult to implement performance > improvements in the base engine. Sorting through these and working > with package authors to find alternate, safer options takes a lot of > time (see 'spare time' above) and energy (some package authors are > easier to work with than others). Several of us have taken cracks at > moving this forward from time to time, but it rarely gets to the top > of anyone's priority list. > > > And in general, I'd urge R Core to make an explicit list of functions that > > you consider to be part of the exported API, and then grandfather in > > packages that used those functions prior to learning that we weren't > > supposed to. > > Making a list and hoping that it will remain up to date is not > realistic. The only way that would work reliably is if the list could > be programmatically generated, for example by parsing installed > headers for declarations and caveats as above. Which would be possible > with changes like the ones listed above.The R help pages have been proven to work excellently for documenting the R API. Maybe we could leverage that to document the C-level API? That could help to bring more formal structure to the documentation than the "free-text" format that is currently used in RWE. I played around with this idea some years ago, where I created a package 'RNativeAPI' for the sole purpose of prototyping documentation of the C-level API of R, and to see what's possible to do with the Rd format in this sense and to get an idea what the limits are. For example, getAttrib() and setAttrib() are documented in <https://github.com/HenrikBengtsson/RNativeAPI/blob/master/man/getAttrib.Rd>. To see what it looks like, go to <https://henrikbengtsson.github.io/RNativeAPI/reference/getAttrib.html>. If this is feasible, I think there are enough folks out there that are willing to contribute to such documentation. It's obviously a long-term project, but it's doable. /Henrik> > Best, > > luke > > > > > Hadley > > > > > > > > -- > Luke Tierney > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics and Fax: 319-335-3017 > Actuarial Science > 241 Schaeffer Hall email: luke-tierney at uiowa.edu > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu/ > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
On Wed, 24 Apr 2024 15:31:39 -0500 (CDT) luke-tierney--- via R-devel <r-devel at r-project.org> wrote:> We would be better off (in my view, not necessarily shared by others > in R-core) if we could get to a point where: > > all entry points listed in installed header files can be used in > packages, at least with some caveats; > > the caveats are expressed in a standard way that is searchable, > e.g. with a standardized comment syntax at the header file or > individual declaration level.This sounds almost like Doxygen, although the exact syntax used to denote the entry points and the necessary comments is far from the most important detail at this point.> There are some 500 entry points in the R shared library that are in > the installed headers but not mentioned in WRE. These would need to > be reviewed and adjusted.Is there a way for outsiders to help? For example, would it help to produce the linking graph (package P links to entry points X, Y)? I understand that an entry point being unpopular doesn't mean it shouldn't be public (and the other way around), but combined with a list of entry points that are listed in WRE, such a graph could be useful to direct effort or estimate impact from interface changes. -- Best regards, Ivan