tl;dr: S3 lookup no longer works in custom non-namespace environments as of R 3.6.1. Is this a bug? I am implementing S3 dispatch for generic methods in environments that are not packages. I am trying to emulate the R package namespace mechanism by having a ?namespace? environment that defines generics and methods, but only exposes the generics themselves, not the methods. To make S3 lookup work when using the generics, I am using `registerS3method`. While this method itself has no extensive documentation, the documentation of `UseMethod` contains this relevant passage:> Namespaces can register methods for generic functions. To support this, > ?UseMethod? and ?NextMethod? search for methods in two places: in the > environment in which the generic function is called, and in theregistration> data base for the environment in which the generic is defined (typically a > namespace). So methods for a generic function need to be available in the > environment of the call to the generic, or they must be registered. (Itdoes> not matter whether they are visible in the environment in which thegeneric is> defined.) As from R 3.5.0, the registration data base is searched afterthe> top level environment (see ?topenv?) of the calling environment (butbefore> the parents of the top level environment).This used to work but it stopped working in R 3.6.1 and I cannot figure out (a) why, and (b) how to fix it. Unfortunately I am unable to find the relevant information by reading the R source code, even when ?diff?ing what seem to be the only even remotely relevant changes [1]. The R NEWS merely list the following change for R 3.6.0:> * S3method() directives in ?NAMESPACE? can now also be used to performdelayed> S3 method registration. > [?] > * Method dispatch uses more relevant environments when looking up class > definitions.Unfortunately it is not clear to me what exactly this means. Here?s a minimal example code that works under R 3.5.3 but breaks under R 3.6.1 (I don?t know about 3.6.0). ``` # Define ?package namespace?: ns = new.env(parent = .BaseNamespaceEnv) local(envir = ns, { test = function (x) UseMethod('test') test.default = function (x) message('test.default') test.foo = function (x) message('test.foo') .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv) .__S3MethodsTable__.$test.default = test.default .__S3MethodsTable__.$test.foo = test.foo # Or, equivalently: # registerS3method('test', 'default', test.default) # registerS3method('test', 'foo', test.foo) }) # Expose generic publicly: test = ns$test # Usage: test(1) test(structure(1, class = 'foo')) ``` Output in R up to 3.5.3: ``` test.default test.foo ``` Output in R 3.6.1: ``` Error in UseMethod("test") : no applicable method for 'test' applied to an object of class "c('double', 'numeric')" ``` It?s worth noting that the output of `.S3methods` is the same for all R versions, and from my understanding of its output, this *should* indicate that S3 lookup should behave identically, too. Furthermore, lookup via `getS3method` succeeds in all R versions, and (again, in my understanding) the logic of this function should be identical to the logic of R?s internal S3 dispatch: ``` getS3method('test', 'default')(1) getS3method('test', 'foo')(1) ``` Conversely, specialising an existing generic from a loaded package works. E.g.: ``` local(envir = ns, { print.foo = function (x) message('print.foo') registerS3method('print', 'foo', print.foo) }) print(structure(1, class = 'foo')) ``` This prints ?print.foo? in all R versions as expected. So my question is: Why do the `test(?)` calls in R 3.6.1 no longer trigger S3 method lookup in the generic function?s environment? Is this behaviour by design or is it a bug? If it?s by design, why does `getS3method` still use the old behaviour? And, most importantly, how can I fix my definition of `ns` to make S3 dispatch for non-exposed methods work again? ? actually I just found a workaround: ``` ns$.packageName = 'not important' ``` This marks `ns` as a package namespace. To me, the documentation seems to imply that this shouldn?t be necessary (and it previously wasn?t). Furthermore, the code for `registerS3method` explicitly supports non-package namespace environments. Unfortunately this workaround is not satisfactory because pretending that the environment is a package namespace, when it really isn?t, might break other things. [1] See r75273; there?s also r74625, which changes the actual lookup mechanism used by `UseMethod`, but that seems even less relevant, because it is disabled unless a specific environment variable is set. -- Konrad Rudolph [[alternative HTML version deleted]]
On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:> tl;dr: S3 lookup no longer works in custom non-namespace environments as of > R 3.6.1. Is this a bug?I don't know whether this was intentional or not, but a binary search through the svn commits finds that the errors started in this one: ------------------------------------------------------------------------ r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines Changed paths: M /trunk/src/main/objects.c M /trunk/tests/reg-tests-1a.R Have S3 methods lookup by default look for the S3 registry in the topenv of the generic. ------------------------------------------------------------------------ Duncan Murdoch> > I am implementing S3 dispatch for generic methods in environments that are > not > packages. I am trying to emulate the R package namespace mechanism by > having a > ?namespace? environment that defines generics and methods, but only exposes > the > generics themselves, not the methods. > > To make S3 lookup work when using the generics, I am using > `registerS3method`. > While this method itself has no extensive documentation, the documentation > of > `UseMethod` contains this relevant passage: > >> Namespaces can register methods for generic functions. To support this, >> ?UseMethod? and ?NextMethod? search for methods in two places: in the >> environment in which the generic function is called, and in the > registration >> data base for the environment in which the generic is defined (typically a >> namespace). So methods for a generic function need to be available in the >> environment of the call to the generic, or they must be registered. (It > does >> not matter whether they are visible in the environment in which the > generic is >> defined.) As from R 3.5.0, the registration data base is searched after > the >> top level environment (see ?topenv?) of the calling environment (but > before >> the parents of the top level environment). > > This used to work but it stopped working in R 3.6.1 and I cannot figure out > (a) > why, and (b) how to fix it. Unfortunately I am unable to find the relevant > information by reading the R source code, even when ?diff?ing what seem to > be > the only even remotely relevant changes [1]. > > The R NEWS merely list the following change for R 3.6.0: > >> * S3method() directives in ?NAMESPACE? can now also be used to perform > delayed >> S3 method registration. >> [?] >> * Method dispatch uses more relevant environments when looking up class >> definitions. > > Unfortunately it is not clear to me what exactly this means. > > Here?s a minimal example code that works under R 3.5.3 but breaks under > R 3.6.1 > (I don?t know about 3.6.0). > > ``` > # Define ?package namespace?: > ns = new.env(parent = .BaseNamespaceEnv) > local(envir = ns, { > test = function (x) UseMethod('test') > test.default = function (x) message('test.default') > test.foo = function (x) message('test.foo') > > .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv) > .__S3MethodsTable__.$test.default = test.default > .__S3MethodsTable__.$test.foo = test.foo > > # Or, equivalently: > # registerS3method('test', 'default', test.default) > # registerS3method('test', 'foo', test.foo) > }) > > # Expose generic publicly: > test = ns$test > > # Usage: > test(1) > test(structure(1, class = 'foo')) > ``` > > Output in R up to 3.5.3: > > ``` > test.default > test.foo > ``` > > Output in R 3.6.1: > > ``` > Error in UseMethod("test") : > no applicable method for 'test' applied to an object of class > "c('double', 'numeric')" > ``` > > It?s worth noting that the output of `.S3methods` is the same for all R > versions, and from my understanding of its output, this *should* indicate > that > S3 lookup should behave identically, too. Furthermore, lookup via > `getS3method` > succeeds in all R versions, and (again, in my understanding) the logic of > this > function should be identical to the logic of R?s internal S3 dispatch: > > ``` > getS3method('test', 'default')(1) > getS3method('test', 'foo')(1) > ``` > > Conversely, specialising an existing generic from a loaded package works. > E.g.: > > ``` > local(envir = ns, { > print.foo = function (x) message('print.foo') > registerS3method('print', 'foo', print.foo) > }) > > print(structure(1, class = 'foo')) > ``` > > This prints ?print.foo? in all R versions as expected. > > So my question is: Why do the `test(?)` calls in R 3.6.1 no longer trigger > S3 > method lookup in the generic function?s environment? Is this behaviour by > design > or is it a bug? If it?s by design, why does `getS3method` still use the old > behaviour? And, most importantly, how can I fix my definition of `ns` to > make > S3 dispatch for non-exposed methods work again? > > ? actually I just found a workaround: > > ``` > ns$.packageName = 'not important' > ``` > > This marks `ns` as a package namespace. To me, the documentation seems to > imply > that this shouldn?t be necessary (and it previously wasn?t). Furthermore, > the > code for `registerS3method` explicitly supports non-package namespace > environments. Unfortunately this workaround is not satisfactory because > pretending that the environment is a package namespace, when it really > isn?t, > might break other things. > > [1] See r75273; there?s also r74625, which changes the actual lookup > mechanism > used by `UseMethod`, but that seems even less relevant, because it is > disabled unless a specific environment variable is set. >
Oh, I had missed that that code path is now enabled by default. It?s worth noting that the commented-out test in that commit also still succeeds if invoked via `getS3method`. So at the very least there?s now an inconsistency in the lookup performed by R internally (via `UseMethod`) and `getS3method`, which is probably unintentional. I see how the change is beneficial by preventing surprising behaviour in a corner case. Unfortunately it also breaks at least one published package [1], and if I understand correctly it no longer conforms to the documented behaviour (quoted in my initial message), which even explicitly mentions non-namespace environments. [1] https://github.com/klmr/modules/issues/147 On Wed, Oct 9, 2019 at 11:23 PM Duncan Murdoch <murdoch.duncan at gmail.com> wrote:> On 09/10/2019 3:22 p.m., Konrad Rudolph wrote: > > tl;dr: S3 lookup no longer works in custom non-namespace environments as > of > > R 3.6.1. Is this a bug? > > I don't know whether this was intentional or not, but a binary search > through the svn commits finds that the errors started in this one: > > ------------------------------------------------------------------------ > r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines > Changed paths: > M /trunk/src/main/objects.c > M /trunk/tests/reg-tests-1a.R > > Have S3 methods lookup by default look for the S3 registry in the topenv > of the generic. > ------------------------------------------------------------------------ > > Duncan Murdoch > > > > > I am implementing S3 dispatch for generic methods in environments that > are > > not > > packages. I am trying to emulate the R package namespace mechanism by > > having a > > ?namespace? environment that defines generics and methods, but only > exposes > > the > > generics themselves, not the methods. > > > > To make S3 lookup work when using the generics, I am using > > `registerS3method`. > > While this method itself has no extensive documentation, the > documentation > > of > > `UseMethod` contains this relevant passage: > > > >> Namespaces can register methods for generic functions. To support this, > >> ?UseMethod? and ?NextMethod? search for methods in two places: in the > >> environment in which the generic function is called, and in the > > registration > >> data base for the environment in which the generic is defined > (typically a > >> namespace). So methods for a generic function need to be available in > the > >> environment of the call to the generic, or they must be registered. (It > > does > >> not matter whether they are visible in the environment in which the > > generic is > >> defined.) As from R 3.5.0, the registration data base is searched after > > the > >> top level environment (see ?topenv?) of the calling environment (but > > before > >> the parents of the top level environment). > > > > This used to work but it stopped working in R 3.6.1 and I cannot figure > out > > (a) > > why, and (b) how to fix it. Unfortunately I am unable to find the > relevant > > information by reading the R source code, even when ?diff?ing what seem > to > > be > > the only even remotely relevant changes [1]. > > > > The R NEWS merely list the following change for R 3.6.0: > > > >> * S3method() directives in ?NAMESPACE? can now also be used to perform > > delayed > >> S3 method registration. > >> [?] > >> * Method dispatch uses more relevant environments when looking up class > >> definitions. > > > > Unfortunately it is not clear to me what exactly this means. > > > > Here?s a minimal example code that works under R 3.5.3 but breaks under > > R 3.6.1 > > (I don?t know about 3.6.0). > > > > ``` > > # Define ?package namespace?: > > ns = new.env(parent = .BaseNamespaceEnv) > > local(envir = ns, { > > test = function (x) UseMethod('test') > > test.default = function (x) message('test.default') > > test.foo = function (x) message('test.foo') > > > > .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv) > > .__S3MethodsTable__.$test.default = test.default > > .__S3MethodsTable__.$test.foo = test.foo > > > > # Or, equivalently: > > # registerS3method('test', 'default', test.default) > > # registerS3method('test', 'foo', test.foo) > > }) > > > > # Expose generic publicly: > > test = ns$test > > > > # Usage: > > test(1) > > test(structure(1, class = 'foo')) > > ``` > > > > Output in R up to 3.5.3: > > > > ``` > > test.default > > test.foo > > ``` > > > > Output in R 3.6.1: > > > > ``` > > Error in UseMethod("test") : > > no applicable method for 'test' applied to an object of class > > "c('double', 'numeric')" > > ``` > > > > It?s worth noting that the output of `.S3methods` is the same for all R > > versions, and from my understanding of its output, this *should* indicate > > that > > S3 lookup should behave identically, too. Furthermore, lookup via > > `getS3method` > > succeeds in all R versions, and (again, in my understanding) the logic of > > this > > function should be identical to the logic of R?s internal S3 dispatch: > > > > ``` > > getS3method('test', 'default')(1) > > getS3method('test', 'foo')(1) > > ``` > > > > Conversely, specialising an existing generic from a loaded package works. > > E.g.: > > > > ``` > > local(envir = ns, { > > print.foo = function (x) message('print.foo') > > registerS3method('print', 'foo', print.foo) > > }) > > > > print(structure(1, class = 'foo')) > > ``` > > > > This prints ?print.foo? in all R versions as expected. > > > > So my question is: Why do the `test(?)` calls in R 3.6.1 no longer > trigger > > S3 > > method lookup in the generic function?s environment? Is this behaviour by > > design > > or is it a bug? If it?s by design, why does `getS3method` still use the > old > > behaviour? And, most importantly, how can I fix my definition of `ns` to > > make > > S3 dispatch for non-exposed methods work again? > > > > ? actually I just found a workaround: > > > > ``` > > ns$.packageName = 'not important' > > ``` > > > > This marks `ns` as a package namespace. To me, the documentation seems to > > imply > > that this shouldn?t be necessary (and it previously wasn?t). Furthermore, > > the > > code for `registerS3method` explicitly supports non-package namespace > > environments. Unfortunately this workaround is not satisfactory because > > pretending that the environment is a package namespace, when it really > > isn?t, > > might break other things. > > > > [1] See r75273; there?s also r74625, which changes the actual lookup > > mechanism > > used by `UseMethod`, but that seems even less relevant, because it > is > > disabled unless a specific environment variable is set. > > > >-- Konrad Rudolph [[alternative HTML version deleted]]