R devel - Dec 2008 - R in a sandbox/jail

Someone recently suggested building a system for automatically testing
student's R programs. They would upload them to our Virtual Learning
Environment, which would then run the code on some inputs and see if
it got the right output. If it does, the student scores points for
that course.

My first thought was "you want to run unchecked, student-submitted
code on a server that has access to students' grades?".

Can this be done securely? The idea might be to run R in a
chroot-jail, freshly generated for each run. The jail would not be
able to access anything outside of it, and once the R session has
finished the calling process can pick up the output from within the
jail.

Maybe that's overkill. Perhaps if you run the user's code as an
ordinary user and store the answers/results in a directory only root
can read that would work (given no local root exploits). Other
precautions could include limiting the runtime or cputime for the R
session. It might be necessary to limit network access too.

 Anyone done anything like this? Personally I think there are too many
other problems with automated systems like this, particularly that
just because a program produces the correct output that makes it a
good one. Sure, at the production stage that's a requirement, but I'd
rather students learnt to program well than to program correctly -
since correctness follows goodness but goodness does not follow
correctness. But that's an argument for another day!

Barry

I've not tried an automated system like you describe but I have tried to
make
test scripts available that compare the output of their programs to correct 
output.  That way the students can check their progress as they work.  In 
general, I find this approach doesn't work well for grading because in order
to
specify a homework to the point where the students' output matches your
output,
the homework ends up being 10 pages long (not unlike a legal document).

For example, in a number of instances I'd say "the function should
return a
number..." and then I'd compare the returned result to the actual
answer.  But a
student would 'print' the number instead, not knowing the difference
between
'return+autoprint' and 'print'.  I found it difficult to write a
test script
that gives partial credit for getting the answer right but the concept wrong. 
Perhaps that's a problem with the way I grade things, but I found it
difficult
to write an acceptable test script.

-roger

Barry Rowlingson wrote:
> Someone recently suggested building a system for automatically testing
> student's R programs. They would upload them to our Virtual Learning
> Environment, which would then run the code on some inputs and see if
> it got the right output. If it does, the student scores points for
> that course.
> 
> My first thought was "you want to run unchecked, student-submitted
> code on a server that has access to students' grades?".
> 
> Can this be done securely? The idea might be to run R in a
> chroot-jail, freshly generated for each run. The jail would not be
> able to access anything outside of it, and once the R session has
> finished the calling process can pick up the output from within the
> jail.
> 
> Maybe that's overkill. Perhaps if you run the user's code as an
> ordinary user and store the answers/results in a directory only root
> can read that would work (given no local root exploits). Other
> precautions could include limiting the runtime or cputime for the R
> session. It might be necessary to limit network access too.
> 
>  Anyone done anything like this? Personally I think there are too many
> other problems with automated systems like this, particularly that
> just because a program produces the correct output that makes it a
> good one. Sure, at the production stage that's a requirement, but
I'd
> rather students learnt to program well than to program correctly -
> since correctness follows goodness but goodness does not follow
> correctness. But that's an argument for another day!
> 
> Barry
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> .
> 
-- 
Roger D. Peng  |  http://www.biostat.jhsph.edu/~rpeng/

Hi

I set up R in a chroot jail for a course and it worked fine (students
were able to submit R code via a web form, which was
password-protected).  I recall it taking a bit of setting up, but hard
to estimate how much of that was figuring out how to use chroot.  I have
some notes, so might be able to help out if you have questions.

I no longer maintain the R-via-web set up, but I do automate some of the
marking of student R code submissions.  See the R News 8(2) for a
description of how I use the 'compare' package for this sort of thing.
The R code submissions still need to be viewed by eye for style and
sanity, but I find it worthwhile to automate the monotonous task of
checking whether the code produces the right answer.

Paul


Barry Rowlingson wrote:
> Someone recently suggested building a system for automatically testing
> student's R programs. They would upload them to our Virtual Learning
> Environment, which would then run the code on some inputs and see if
> it got the right output. If it does, the student scores points for
> that course.
> 
> My first thought was "you want to run unchecked, student-submitted
> code on a server that has access to students' grades?".
> 
> Can this be done securely? The idea might be to run R in a
> chroot-jail, freshly generated for each run. The jail would not be
> able to access anything outside of it, and once the R session has
> finished the calling process can pick up the output from within the
> jail.
> 
> Maybe that's overkill. Perhaps if you run the user's code as an
> ordinary user and store the answers/results in a directory only root
> can read that would work (given no local root exploits). Other
> precautions could include limiting the runtime or cputime for the R
> session. It might be necessary to limit network access too.
> 
>  Anyone done anything like this? Personally I think there are too many
> other problems with automated systems like this, particularly that
> just because a program produces the correct output that makes it a
> good one. Sure, at the production stage that's a requirement, but
I'd
> rather students learnt to program well than to program correctly -
> since correctness follows goodness but goodness does not follow
> correctness. But that's an argument for another day!
> 
> Barry
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
-- 
Dr Paul Murrell
Department of Statistics
The University of Auckland
Private Bag 92019
Auckland
New Zealand
64 9 3737599 x85392
paul at stat.auckland.ac.nz
http://www.stat.auckland.ac.nz/~paul/