R devel - Mar 2020 - new bquote feature splice does not address a common LISP @ use case?

Dear R-devel,

There is a new feature in R-devel, which explicitly refers to LISP @
operator for splicing.
> The backquote function bquote() has a new argument splice to enable
splicing a computed list of values into an expression, like ,@ in LISP's
backquote.
Although the most upvoted SO question asking for exactly LISP's @
functionality in R doesn't seems to be addressed by this new feature.

Is it possible to use new splice feature to create `6 - 5 + 4`
expression rather than `6 - (5 + 4)`?

b = quote(5+4)
b
#5 + 4
c = bquote(6-.(b))
c
#6 - (5 + 4)
d = bquote(6-..(b), splice=TRUE)
d
#6 - (5 + 4)

There is corresponding LISP code provided

CL-USER>
(setf b `(5 + 4))
(5 + 4)
CL-USER>
(setf c `(6 - , at b))
(6 - 5 + 4)
CL-USER>
(setf c-non-spliced `(6 - ,b))
(6 - (5 + 4))
CL-USER>

Thanks,
Jan Gorecki

Hi Jan,

In the lisp code you provide the operators are parsed as simple
symbols in a pairlist. In the R snippet, they are parsed as
left-associative binary operators of equal precedence. If you unquote
a call in the right-hand side, you're artificially bypassing the
left-associativity of these operators.

To achieve what you're looking for in a general way, you'll need a
more precise definition of the problem, and a solution that probably
involves rotating the AST accordingly (see
https://github.com/r-lib/rlang/blob/master/src/internal/expr-interp-rotate.c).
Maybe it could be possible to formulate a definition where splicing in
special calls like binary operators produces the same AST as the user
would type by hand. It seems this would make splicing easier to use
for end users, but make the metaprogramming model more complex for
experts. This is an interesting perspective though. It also seems
vaguely connected to the problem of splicing within model formulas.

I see in your example that the new ..() operator in `bquote()` allows
splicing calls, and seems to unquote them instead of splicing. In the
first versions of rlang, splicing with !!! behaved just like this. We
changed this behaviour last year and I would like to share the
motivations behind this decision, as it might be helpful to inform the
semantics of ..() in bquote() in R 4.0.

The bottom line is that calls are now treated like scalars. This is a
slight contortion of the syntax because calls are "language lists",
and so they could be conceived as collections rather than scalars.
However, R is vector-oriented rather than pairlist-oriented, and
treating calls as scalars makes the metaprogramming model simpler.

This is also how `bquote(splice = TRUE)` works. However `bquote()`
and rlang do not treat scalars in the same way. In rlang scalars
cannot be spliced, they must be unquoted.

```
bquote(foo(..(function() NULL)), splice = TRUE)
#> foo(function() NULL)

bquote(foo(..(quote(bar))), splice = TRUE)
#> foo(bar)

expr(foo(!!!function() NULL))
#> Error: Can't splice an object of type `closure` because it is not a
vector.

expr(foo(!!!quote(bar)))
#> foo(bar)
#> Warning message:
#> Unquoting language objects with `!!!` is deprecated as of rlang 0.4.0.
#> Please use `!!` instead.
```

We decided to disallow splicing scalars (and thus calls) in rlang even
though this is a legal operation in many lisps. In lisps, the splicing
operation stands for unquoting in the CDR of a pairlist. By contrast
the unquote operation unquotes in the CAR. For example `(1 , at 3) is
legal in Common Lisp and stands for the cons cell (1 . 3). I think
such semantics are not appealing in a language like R because it is
vector-oriented rather than pairlist oriented. Pairlists are mostly an
implicit data structure that users are not familiar with, and they are
not even fully supported in all implementations of R (for instance
TERR and Renjin do not allow non-NULL terminated pairlists, and while
GNU R has vestigial print() support for these, they cause str() to crash).

In general, it is much more useful to define a splice operation that
also works for vectors:

```
rlang::list2(1, !!!10:11, 3)
#> [[1]]
#> [1] 1
#>
#> [[2]]
#> [1] 10
#>
#> [[3]]
#> [1] 11
#>
#> [[4]]
#> [1] 3
```

Because vectors do not have any notion of CDR, the usual lisp
interpretation of splicing scalars does not apply.

One alternative to make it work is to devolve the splicing operation
into a simple unquote operation, when supplied a scalar. This is how
`bquote(splice = TRUE)` works. However I think this kind of
overloading is more confusing in the long run, and makes it harder for
users to form a correct mental model for programming with these
operations. For this reason it seems preferable to force users to be
explicit about the desired semantics with scalars and calls. In rlang
they must either unquote the call, or explicitly transform it to a
list prior to splicing:

```
x <- quote(bar + baz)

# Unquote instead of splicing
expr(foo(!!x))
#> foo(bar + baz)

# Convert to list and then splice
expr(add(!!!as.list(x[-1])))
#> add(bar, baz)
```

Unquoting could be consistent if all objects were truly vectors in R,
i.e. if they were implicitly wrapped in a list. Then ..(quote(foo))
would be very similar to ..(1). In the former case a list of size 1
would be spliced, in the latter case a vector of size 1 is
spliced. This would explain why .() and ..() have the same behaviour
with scalars. While an interesting thought experiment, this is not
how scalars work in R.

It seems relevant that Clojure is a lisp that does not allow splicing
scalars. Like rlang, Clojure defines the splicing operation in other
contexts than pairlists, such as vectors. I suspect the rationale of
making scalar-splicing an error in Clojure, even in pairlist context,
is to avoid overloading the semantics of this fundamental operation.

Best,
Lionel


On 3/17/20, Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
> Dear R-devel,
>
> There is a new feature in R-devel, which explicitly refers to LISP @
> operator for splicing.
>
>> The backquote function bquote() has a new argument splice to enable
>> splicing a computed list of values into an expression, like ,@ in
LISP's
>> backquote.
>
> Although the most upvoted SO question asking for exactly LISP's @
> functionality in R doesn't seems to be addressed by this new feature.
>
> Is it possible to use new splice feature to create `6 - 5 + 4`
> expression rather than `6 - (5 + 4)`?
>
> b = quote(5+4)
> b
> #5 + 4
> c = bquote(6-.(b))
> c
> #6 - (5 + 4)
> d = bquote(6-..(b), splice=TRUE)
> d
> #6 - (5 + 4)
>
> There is corresponding LISP code provided
>
> CL-USER>
> (setf b `(5 + 4))
> (5 + 4)
> CL-USER>
> (setf c `(6 - , at b))
> (6 - 5 + 4)
> CL-USER>
> (setf c-non-spliced `(6 - ,b))
> (6 - (5 + 4))
> CL-USER>
>
> Thanks,
> Jan Gorecki
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

Thank you Lionel for comprehensive explanation. I think that rotating
AST in base R is not a good way to go, it would probably complicate
the code heavily.

Best,
Jan Gorecki

On Tue, Mar 17, 2020 at 4:49 PM Lionel Henry <lionel at rstudio.com>
wrote:
>
> Hi Jan,
>
> In the lisp code you provide the operators are parsed as simple
> symbols in a pairlist. In the R snippet, they are parsed as
> left-associative binary operators of equal precedence. If you unquote
> a call in the right-hand side, you're artificially bypassing the
> left-associativity of these operators.
>
> To achieve what you're looking for in a general way, you'll need a
> more precise definition of the problem, and a solution that probably
> involves rotating the AST accordingly (see
>
https://github.com/r-lib/rlang/blob/master/src/internal/expr-interp-rotate.c).
> Maybe it could be possible to formulate a definition where splicing in
> special calls like binary operators produces the same AST as the user
> would type by hand. It seems this would make splicing easier to use
> for end users, but make the metaprogramming model more complex for
> experts. This is an interesting perspective though. It also seems
> vaguely connected to the problem of splicing within model formulas.
>
> I see in your example that the new ..() operator in `bquote()` allows
> splicing calls, and seems to unquote them instead of splicing. In the
> first versions of rlang, splicing with !!! behaved just like this. We
> changed this behaviour last year and I would like to share the
> motivations behind this decision, as it might be helpful to inform the
> semantics of ..() in bquote() in R 4.0.
>
> The bottom line is that calls are now treated like scalars. This is a
> slight contortion of the syntax because calls are "language
lists",
> and so they could be conceived as collections rather than scalars.
> However, R is vector-oriented rather than pairlist-oriented, and
> treating calls as scalars makes the metaprogramming model simpler.
>
> This is also how `bquote(splice = TRUE)` works. However `bquote()`
> and rlang do not treat scalars in the same way. In rlang scalars
> cannot be spliced, they must be unquoted.
>
> ```
> bquote(foo(..(function() NULL)), splice = TRUE)
> #> foo(function() NULL)
>
> bquote(foo(..(quote(bar))), splice = TRUE)
> #> foo(bar)
>
> expr(foo(!!!function() NULL))
> #> Error: Can't splice an object of type `closure` because it is not
a vector.
>
> expr(foo(!!!quote(bar)))
> #> foo(bar)
> #> Warning message:
> #> Unquoting language objects with `!!!` is deprecated as of rlang
0.4.0.
> #> Please use `!!` instead.
> ```
>
> We decided to disallow splicing scalars (and thus calls) in rlang even
> though this is a legal operation in many lisps. In lisps, the splicing
> operation stands for unquoting in the CDR of a pairlist. By contrast
> the unquote operation unquotes in the CAR. For example `(1 , at 3) is
> legal in Common Lisp and stands for the cons cell (1 . 3). I think
> such semantics are not appealing in a language like R because it is
> vector-oriented rather than pairlist oriented. Pairlists are mostly an
> implicit data structure that users are not familiar with, and they are
> not even fully supported in all implementations of R (for instance
> TERR and Renjin do not allow non-NULL terminated pairlists, and while
> GNU R has vestigial print() support for these, they cause str() to crash).
>
> In general, it is much more useful to define a splice operation that
> also works for vectors:
>
> ```
> rlang::list2(1, !!!10:11, 3)
> #> [[1]]
> #> [1] 1
> #>
> #> [[2]]
> #> [1] 10
> #>
> #> [[3]]
> #> [1] 11
> #>
> #> [[4]]
> #> [1] 3
> ```
>
> Because vectors do not have any notion of CDR, the usual lisp
> interpretation of splicing scalars does not apply.
>
> One alternative to make it work is to devolve the splicing operation
> into a simple unquote operation, when supplied a scalar. This is how
> `bquote(splice = TRUE)` works. However I think this kind of
> overloading is more confusing in the long run, and makes it harder for
> users to form a correct mental model for programming with these
> operations. For this reason it seems preferable to force users to be
> explicit about the desired semantics with scalars and calls. In rlang
> they must either unquote the call, or explicitly transform it to a
> list prior to splicing:
>
> ```
> x <- quote(bar + baz)
>
> # Unquote instead of splicing
> expr(foo(!!x))
> #> foo(bar + baz)
>
> # Convert to list and then splice
> expr(add(!!!as.list(x[-1])))
> #> add(bar, baz)
> ```
>
> Unquoting could be consistent if all objects were truly vectors in R,
> i.e. if they were implicitly wrapped in a list. Then ..(quote(foo))
> would be very similar to ..(1). In the former case a list of size 1
> would be spliced, in the latter case a vector of size 1 is
> spliced. This would explain why .() and ..() have the same behaviour
> with scalars. While an interesting thought experiment, this is not
> how scalars work in R.
>
> It seems relevant that Clojure is a lisp that does not allow splicing
> scalars. Like rlang, Clojure defines the splicing operation in other
> contexts than pairlists, such as vectors. I suspect the rationale of
> making scalar-splicing an error in Clojure, even in pairlist context,
> is to avoid overloading the semantics of this fundamental operation.
>
> Best,
> Lionel
>
>
> On 3/17/20, Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
> > Dear R-devel,
> >
> > There is a new feature in R-devel, which explicitly refers to LISP @
> > operator for splicing.
> >
> >> The backquote function bquote() has a new argument splice to
enable
> >> splicing a computed list of values into an expression, like ,@ in
LISP's
> >> backquote.
> >
> > Although the most upvoted SO question asking for exactly LISP's @
> > functionality in R doesn't seems to be addressed by this new
feature.
> >
> > Is it possible to use new splice feature to create `6 - 5 + 4`
> > expression rather than `6 - (5 + 4)`?
> >
> > b = quote(5+4)
> > b
> > #5 + 4
> > c = bquote(6-.(b))
> > c
> > #6 - (5 + 4)
> > d = bquote(6-..(b), splice=TRUE)
> > d
> > #6 - (5 + 4)
> >
> > There is corresponding LISP code provided
> >
> > CL-USER>
> > (setf b `(5 + 4))
> > (5 + 4)
> > CL-USER>
> > (setf c `(6 - , at b))
> > (6 - 5 + 4)
> > CL-USER>
> > (setf c-non-spliced `(6 - ,b))
> > (6 - (5 + 4))
> > CL-USER>
> >
> > Thanks,
> > Jan Gorecki
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >

The intent is that ..() only be used with vectors but this isn't
enforced at present. I'll think about signaling an error if
is.vector(mexp) is not true.

Best,

luke

On Tue, 17 Mar 2020, Lionel Henry wrote:
> Hi Jan,
>
> In the lisp code you provide the operators are parsed as simple
> symbols in a pairlist. In the R snippet, they are parsed as
> left-associative binary operators of equal precedence. If you unquote
> a call in the right-hand side, you're artificially bypassing the
> left-associativity of these operators.
>
> To achieve what you're looking for in a general way, you'll need a
> more precise definition of the problem, and a solution that probably
> involves rotating the AST accordingly (see
>
https://github.com/r-lib/rlang/blob/master/src/internal/expr-interp-rotate.c).
> Maybe it could be possible to formulate a definition where splicing in
> special calls like binary operators produces the same AST as the user
> would type by hand. It seems this would make splicing easier to use
> for end users, but make the metaprogramming model more complex for
> experts. This is an interesting perspective though. It also seems
> vaguely connected to the problem of splicing within model formulas.
>
> I see in your example that the new ..() operator in `bquote()` allows
> splicing calls, and seems to unquote them instead of splicing. In the
> first versions of rlang, splicing with !!! behaved just like this. We
> changed this behaviour last year and I would like to share the
> motivations behind this decision, as it might be helpful to inform the
> semantics of ..() in bquote() in R 4.0.
>
> The bottom line is that calls are now treated like scalars. This is a
> slight contortion of the syntax because calls are "language
lists",
> and so they could be conceived as collections rather than scalars.
> However, R is vector-oriented rather than pairlist-oriented, and
> treating calls as scalars makes the metaprogramming model simpler.
>
> This is also how `bquote(splice = TRUE)` works. However `bquote()`
> and rlang do not treat scalars in the same way. In rlang scalars
> cannot be spliced, they must be unquoted.
>
> ```
> bquote(foo(..(function() NULL)), splice = TRUE)
> #> foo(function() NULL)
>
> bquote(foo(..(quote(bar))), splice = TRUE)
> #> foo(bar)
>
> expr(foo(!!!function() NULL))
> #> Error: Can't splice an object of type `closure` because it is not
a vector.
>
> expr(foo(!!!quote(bar)))
> #> foo(bar)
> #> Warning message:
> #> Unquoting language objects with `!!!` is deprecated as of rlang
0.4.0.
> #> Please use `!!` instead.
> ```
>
> We decided to disallow splicing scalars (and thus calls) in rlang even
> though this is a legal operation in many lisps. In lisps, the splicing
> operation stands for unquoting in the CDR of a pairlist. By contrast
> the unquote operation unquotes in the CAR. For example `(1 , at 3) is
> legal in Common Lisp and stands for the cons cell (1 . 3). I think
> such semantics are not appealing in a language like R because it is
> vector-oriented rather than pairlist oriented. Pairlists are mostly an
> implicit data structure that users are not familiar with, and they are
> not even fully supported in all implementations of R (for instance
> TERR and Renjin do not allow non-NULL terminated pairlists, and while
> GNU R has vestigial print() support for these, they cause str() to crash).
>
> In general, it is much more useful to define a splice operation that
> also works for vectors:
>
> ```
> rlang::list2(1, !!!10:11, 3)
> #> [[1]]
> #> [1] 1
> #>
> #> [[2]]
> #> [1] 10
> #>
> #> [[3]]
> #> [1] 11
> #>
> #> [[4]]
> #> [1] 3
> ```
>
> Because vectors do not have any notion of CDR, the usual lisp
> interpretation of splicing scalars does not apply.
>
> One alternative to make it work is to devolve the splicing operation
> into a simple unquote operation, when supplied a scalar. This is how
> `bquote(splice = TRUE)` works. However I think this kind of
> overloading is more confusing in the long run, and makes it harder for
> users to form a correct mental model for programming with these
> operations. For this reason it seems preferable to force users to be
> explicit about the desired semantics with scalars and calls. In rlang
> they must either unquote the call, or explicitly transform it to a
> list prior to splicing:
>
> ```
> x <- quote(bar + baz)
>
> # Unquote instead of splicing
> expr(foo(!!x))
> #> foo(bar + baz)
>
> # Convert to list and then splice
> expr(add(!!!as.list(x[-1])))
> #> add(bar, baz)
> ```
>
> Unquoting could be consistent if all objects were truly vectors in R,
> i.e. if they were implicitly wrapped in a list. Then ..(quote(foo))
> would be very similar to ..(1). In the former case a list of size 1
> would be spliced, in the latter case a vector of size 1 is
> spliced. This would explain why .() and ..() have the same behaviour
> with scalars. While an interesting thought experiment, this is not
> how scalars work in R.
>
> It seems relevant that Clojure is a lisp that does not allow splicing
> scalars. Like rlang, Clojure defines the splicing operation in other
> contexts than pairlists, such as vectors. I suspect the rationale of
> making scalar-splicing an error in Clojure, even in pairlist context,
> is to avoid overloading the semantics of this fundamental operation.
>
> Best,
> Lionel
>
>
> On 3/17/20, Jan Gorecki <j.gorecki at wit.edu.pl> wrote:
>> Dear R-devel,
>>
>> There is a new feature in R-devel, which explicitly refers to LISP @
>> operator for splicing.
>>
>>> The backquote function bquote() has a new argument splice to enable
>>> splicing a computed list of values into an expression, like ,@ in
LISP's
>>> backquote.
>>
>> Although the most upvoted SO question asking for exactly LISP's @
>> functionality in R doesn't seems to be addressed by this new
feature.
>>
>> Is it possible to use new splice feature to create `6 - 5 + 4`
>> expression rather than `6 - (5 + 4)`?
>>
>> b = quote(5+4)
>> b
>> #5 + 4
>> c = bquote(6-.(b))
>> c
>> #6 - (5 + 4)
>> d = bquote(6-..(b), splice=TRUE)
>> d
>> #6 - (5 + 4)
>>
>> There is corresponding LISP code provided
>>
>> CL-USER>
>> (setf b `(5 + 4))
>> (5 + 4)
>> CL-USER>
>> (setf c `(6 - , at b))
>> (6 - 5 + 4)
>> CL-USER>
>> (setf c-non-spliced `(6 - ,b))
>> (6 - (5 + 4))
>> CL-USER>
>>
>> Thanks,
>> Jan Gorecki
>>
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu