R help - Mar 2018 - Using R and the Tidyverse for an economic model

I've been translating an economic model from Python into R, and I thought
members of the list would like to see a presentation I've written about it.
I've blogged this at
http://www.j-paine.org/blog/2018/03/r-taxben-a-microsimulation-economic-model-in-r.html
, and the presentation itself is a slideshow at
http://www.j-paine.org/rtaxben/R/reveal/rtaxben.html . The slideshow is
written as one side of a conversation which reveals R and the Tidyverse a
feature at a time to a colleague not familiar with R. Those who _are_
familar with R might prefer the version at
http://www.j-paine.org/rtaxben/R/reveal/rtaxben_anim.html . Exactly the
same material, but, as explained in my introduction, quicker to read. Read
the blog post first.

Our model, R-Taxben, is a microeconomic model, which simulates at the level
of individual people rather than bulk variables such as unemployment and
inflation. It works, roughly speaking, by reading survey data about actual
households, then applying taxes and benefits to calculate net income and
expenditure from gross. It has four main parts: (1) read and process
parameters which describe the taxes and benefits; (2) read the household
data from CSV files and transform into data frames usable by the model; (3)
apply the taxes and benefits, calculating such things as council tax, VAT,
child benefit, and pensions; (4) display the results.

My slides are mainly about (2) and (4), but do touch on the others. I
suggest, for example, that legible R code for (3) could be used as a
"reference standard" against which to describe the notoriously complex
UK
benefits system. Organisations such as the Child Poverty Action Group have
written handbooks for benefits advisers which try to specify the system
precisely. We'd like to use R for an electronic version of these.

I've said quite a bit about R for probing and plotting data. Not only for
economists, but for students learning about economics, fiscal policy, and
statistics. And after a brief intro to base R, I've concentrated on the
Tidyverse, because of what I see as its advantages. There are lots of small
demos of the Tidyverse scattered around the web, but fewer of big projects
which use lots of different features from it. So my examples here might be
useful.

Reliability and accuracy are vital, which is why I have more slides about
testing than about anything else, with examples of "testthat".

Near the end, I show a web interface, built using Vis.js , which displays
dataflow in the model. The aim is to make it completely scrutable, so that
none of its economic assumptions are a mystery.

We're looking for funding to go beyond this prototype. There are places
where we'll probably need help with such things as efficiency (see the
section on representation-independent selectors), efficiency again
(multiple JOINs), and the best way to overcome lack of static typing. It
would be great to have R experts, even R implementors, who were willing to
advise on this, and even to collaborate on our grant applications.

	[[alternative HTML version deleted]]

Looks like you have made an impressive start and some attractive 
introductions. I have no significant interest in your topic (sorry), but 
it seems that you are re-inventing the wheel a bit in regards to much of 
your documentation and modularization... R packages can help you solve 
these problems in a cross-platform way. You might try starting with [1] 
and referring to [2] as needed.

Regarding your representation-independent selectors... this looks to me 
like yet another representation (I think the term is "network
database"),
and subject to specific advantages and limitations that this 
representation imposes. For most work I do, tidy data frames have an 
excellent balance of speed and adaptability. For other types of analyses, 
multi-dimensional arrays would be better. Nested lists are extremely 
flexible, but not particularly fast (some would say quite slow, but that 
depends on your use case). Sometimes a relational database or the 
data.table package [3] can be used for increased performance, but your 
functional interface would not be compatible with _merging_ the 
information efficiently, while dplyr can theoretically support any data 
store that presents a tabular data interface with data merge capability.

R seems to work best when used in the functional paradigm operating on 
general-purpose objects... functions that transform, analyze, and present 
data. Having more general classes of objects means more re-use and ad-hoc 
analysis can occur. If I make an object of class "myspecial", only 
functions I write will be useful. Making it a subclass of a more general 
class is one way to make it more widely useful, but avoiding making it a 
subclass of the general class at all can be the most flexible design 
principle... which is what "tidy data" aspires to do with data frames.

That is, I think you should not be avoiding $ (or more generally the
"[["
operator)... you should be embracing it and enabling users to use it as 
well. Just don't go all multi-level with it... prefer multi-column indexes 
in data frames in most cases (e.g [4]).

[1] http://r-pkgs.had.co.nz/
[2] https://cran.r-project.org/doc/manuals/R-exts.html
[3]
https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html
[4] https://www.jstatsoft.org/article/view/v040i01/v40i01.pdf

On Mon, 26 Mar 2018, Jocelyn Ireson-Paine via R-help wrote:
> I've been translating an economic model from Python into R, and I
thought
> members of the list would like to see a presentation I've written about
it.
> I've blogged this at
>
http://www.j-paine.org/blog/2018/03/r-taxben-a-microsimulation-economic-model-in-r.html
> , and the presentation itself is a slideshow at
> http://www.j-paine.org/rtaxben/R/reveal/rtaxben.html . The slideshow is
> written as one side of a conversation which reveals R and the Tidyverse a
> feature at a time to a colleague not familiar with R. Those who _are_
> familar with R might prefer the version at
> http://www.j-paine.org/rtaxben/R/reveal/rtaxben_anim.html . Exactly the
> same material, but, as explained in my introduction, quicker to read. Read
> the blog post first.
>
> Our model, R-Taxben, is a microeconomic model, which simulates at the level
> of individual people rather than bulk variables such as unemployment and
> inflation. It works, roughly speaking, by reading survey data about actual
> households, then applying taxes and benefits to calculate net income and
> expenditure from gross. It has four main parts: (1) read and process
> parameters which describe the taxes and benefits; (2) read the household
> data from CSV files and transform into data frames usable by the model; (3)
> apply the taxes and benefits, calculating such things as council tax, VAT,
> child benefit, and pensions; (4) display the results.
>
> My slides are mainly about (2) and (4), but do touch on the others. I
> suggest, for example, that legible R code for (3) could be used as a
> "reference standard" against which to describe the notoriously
complex UK
> benefits system. Organisations such as the Child Poverty Action Group have
> written handbooks for benefits advisers which try to specify the system
> precisely. We'd like to use R for an electronic version of these.
>
> I've said quite a bit about R for probing and plotting data. Not only
for
> economists, but for students learning about economics, fiscal policy, and
> statistics. And after a brief intro to base R, I've concentrated on the
> Tidyverse, because of what I see as its advantages. There are lots of small
> demos of the Tidyverse scattered around the web, but fewer of big projects
> which use lots of different features from it. So my examples here might be
> useful.
>
> Reliability and accuracy are vital, which is why I have more slides about
> testing than about anything else, with examples of "testthat".
>
> Near the end, I show a web interface, built using Vis.js , which displays
> dataflow in the model. The aim is to make it completely scrutable, so that
> none of its economic assumptions are a mystery.
>
> We're looking for funding to go beyond this prototype. There are places
> where we'll probably need help with such things as efficiency (see the
> section on representation-independent selectors), efficiency again
> (multiple JOINs), and the best way to overcome lack of static typing. It
> would be great to have R experts, even R implementors, who were willing to
> advise on this, and even to collaborate on our grant applications.
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
---------------------------------------------------------------------------
Jeff Newmiller                        The     .....       .....  Go Live...
DCN:<jdnewmil at dcn.davis.ca.us>        Basics: ##.#.       ##.#.  Live
Go...
                                       Live:   OO#.. Dead: OO#..  Playing
Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
/Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k

Jeff, thanks for taking the time to read and comment.

About the points you raise:

1) Packages. I do understand the advantages of packages. But I'm still
prototyping, and it seemed too complicated to make every source file a
separate package, when I'm still often moving source code from one to
another. At the moment, one directory full of source files is simpler
to experiment with and test.

2) Documentation. What I'm concerned with here is not so much where to
put the documentation, as what to write. I follow the
abstract-datatype approach (
https://www.geeksforgeeks.org/abstract-data-types/ ), which is that
the internal structure of a value should be hidden. What's important
is its behaviour: the functions one can call on it, and the types of
result they return. The implementor should be free to change
representation as radically as he or she desires, as long as those
calls remain the same. That is, the _interface_ is not affected. (See
the cartoon I drew at
http://www.j-paine.org/dobbs/engineers_honouring_the_uniform_referent_principle_708.png
.)

But $ and [[ are pervasive in R, and as you say, there are times when
I shouldn't avoid them. On the other hand, I know that I'm likely to
make big changes to some structures while experimenting with
efficiency. I want to proof the rest of the program against those
changes, and I want to proof the rest of my _documentation_ against
those changes. So that means I have to document in a
representation-independent way.

There are also some very R-specific things that I need to think about.
For example, I need to say whether every function is vectorised, don't
I? And I need to say whether NA can occur in every value, and if so,
what it means. These are the documentation-related things I've been
thinking about.

3) Classes. I agree with you, and I have not used any of R's object
systems. Most of my data is, in fact, lists (possibly
tree-structured), vectors, or tibbles.

4) Representation-independent selectors. See 2. It's the
abstract-datatype thing again, wanting to proof code and documentation
outside a module against changes to how the data defined by that
module is stored.

Thanks again. I do welcome comment on these points, because there are
lots of trade-offs to consider, and I know I may be overlooking all
kinds of things I should know.

Jocelyn

On Tue, Mar 27, 2018 at 7:20 PM, Jeff Newmiller
<jdnewmil at dcn.davis.ca.us> wrote:
> Looks like you have made an impressive start and some attractive
> introductions. I have no significant interest in your topic (sorry), but it
> seems that you are re-inventing the wheel a bit in regards to much of your
> documentation and modularization... R packages can help you solve these
> problems in a cross-platform way. You might try starting with [1] and
> referring to [2] as needed.
>
> Regarding your representation-independent selectors... this looks to me
like
> yet another representation (I think the term is "network
database"), and
> subject to specific advantages and limitations that this representation
> imposes. For most work I do, tidy data frames have an excellent balance of
> speed and adaptability. For other types of analyses, multi-dimensional
> arrays would be better. Nested lists are extremely flexible, but not
> particularly fast (some would say quite slow, but that depends on your use
> case). Sometimes a relational database or the data.table package [3] can be
> used for increased performance, but your functional interface would not be
> compatible with _merging_ the information efficiently, while dplyr can
> theoretically support any data store that presents a tabular data interface
> with data merge capability.
>
> R seems to work best when used in the functional paradigm operating on
> general-purpose objects... functions that transform, analyze, and present
> data. Having more general classes of objects means more re-use and ad-hoc
> analysis can occur. If I make an object of class "myspecial",
only functions
> I write will be useful. Making it a subclass of a more general class is one
> way to make it more widely useful, but avoiding making it a subclass of the
> general class at all can be the most flexible design principle... which is
> what "tidy data" aspires to do with data frames.
>
> That is, I think you should not be avoiding $ (or more generally the
"[["
> operator)... you should be embracing it and enabling users to use it as
> well. Just don't go all multi-level with it... prefer multi-column
indexes
> in data frames in most cases (e.g [4]).
>
> [1] http://r-pkgs.had.co.nz/
> [2] https://cran.r-project.org/doc/manuals/R-exts.html
> [3]
>
https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html
> [4] https://www.jstatsoft.org/article/view/v040i01/v40i01.pdf
>
> On Mon, 26 Mar 2018, Jocelyn Ireson-Paine via R-help wrote:
>
>> I've been translating an economic model from Python into R, and I
thought
>> members of the list would like to see a presentation I've written
about
>> it.
>> I've blogged this at
>>
>>
http://www.j-paine.org/blog/2018/03/r-taxben-a-microsimulation-economic-model-in-r.html
>> , and the presentation itself is a slideshow at
>> http://www.j-paine.org/rtaxben/R/reveal/rtaxben.html . The slideshow is
>> written as one side of a conversation which reveals R and the Tidyverse
a
>> feature at a time to a colleague not familiar with R. Those who _are_
>> familar with R might prefer the version at
>> http://www.j-paine.org/rtaxben/R/reveal/rtaxben_anim.html . Exactly the
>> same material, but, as explained in my introduction, quicker to read.
Read
>> the blog post first.
>>
>> Our model, R-Taxben, is a microeconomic model, which simulates at the
>> level
>> of individual people rather than bulk variables such as unemployment
and
>> inflation. It works, roughly speaking, by reading survey data about
actual
>> households, then applying taxes and benefits to calculate net income
and
>> expenditure from gross. It has four main parts: (1) read and process
>> parameters which describe the taxes and benefits; (2) read the
household
>> data from CSV files and transform into data frames usable by the model;
>> (3)
>> apply the taxes and benefits, calculating such things as council tax,
VAT,
>> child benefit, and pensions; (4) display the results.
>>
>> My slides are mainly about (2) and (4), but do touch on the others. I
>> suggest, for example, that legible R code for (3) could be used as a
>> "reference standard" against which to describe the
notoriously complex UK
>> benefits system. Organisations such as the Child Poverty Action Group
have
>> written handbooks for benefits advisers which try to specify the system
>> precisely. We'd like to use R for an electronic version of these.
>>
>> I've said quite a bit about R for probing and plotting data. Not
only for
>> economists, but for students learning about economics, fiscal policy,
and
>> statistics. And after a brief intro to base R, I've concentrated on
the
>> Tidyverse, because of what I see as its advantages. There are lots of
>> small
>> demos of the Tidyverse scattered around the web, but fewer of big
projects
>> which use lots of different features from it. So my examples here might
be
>> useful.
>>
>> Reliability and accuracy are vital, which is why I have more slides
about
>> testing than about anything else, with examples of
"testthat".
>>
>> Near the end, I show a web interface, built using Vis.js , which
displays
>> dataflow in the model. The aim is to make it completely scrutable, so
that
>> none of its economic assumptions are a mystery.
>>
>> We're looking for funding to go beyond this prototype. There are
places
>> where we'll probably need help with such things as efficiency (see
the
>> section on representation-independent selectors), efficiency again
>> (multiple JOINs), and the best way to overcome lack of static typing.
It
>> would be great to have R experts, even R implementors, who were willing
to
>> advise on this, and even to collaborate on our grant applications.
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
> ---------------------------------------------------------------------------
> Jeff Newmiller                        The     .....       .....  Go Live...
> DCN:<jdnewmil at dcn.davis.ca.us>        Basics: ##.#.       ##.#. 
Live Go...
>                                       Live:   OO#.. Dead: OO#..  Playing
> Research Engineer (Solar/Batteries            O.O#.       #.O#.  with
> /Software/Embedded Controllers)               .OO#.       .OO#.  rocks...1k
> ---------------------------------------------------------------------------