Hi Dr. Snow, & R-helpers,
Thank you for your reply! I hadn't heard of the {microbenchmark}
package & was excited to try it! Thank you for the suggestion! I did
check the reference source for which() beforehand, which included the
statement to remove NAa, and I didn't have any missing values or NAs:
sum(is.na(dat$gender2))
sum(is.na(dat$gender))
sum(is.na(dat$y))
[1] 0
[1] 0
[1] 0
I still had a 10ms difference in the value returned by microbenchmark
between the following methods: one with and one without using which().
The difference is reversed from what I expected, since which() is an
extra step.
microbenchmark(
head(
dat[which(dat$gender2=="other"),],), times=100L)
microbenchmark(
head(
dat[dat$gender2=="other",],), times=100L)
min lq mean
head(dat[which(dat$gender2 == "other"), ], ) 62.93803
74.25939 88.4704
head(dat[dat$gender2 == "other", ], ) 71.8914
87.95844 103.7231
Is which() invoking c-level code by chance, making it slightly faster
on average? The difference likely becomes important on terabytes of
data. The addition of which() still seems superfluous to me, and I'd
like to know whether it's considered best practice to keep it. What is
R inoking when which() isn't called explicitly? Is R invoking which()
eventually anyway?
Cheers!
Keith
> Message: 2
> Date: Mon, 12 Oct 2020 13:01:36 -0600
> From: Greg Snow <538280 at gmail.com>
> To: "1/k^c" <kchamberln at gmail.com>
> Cc: r-help <r-help at r-project.org>
> Subject: Re: [R] which() vs. just logical selection in df
> Message-ID:
> <CAFEqCdyUuHh5TZ7t5NJ8cs_4xB61mNeUgasncekD485eBNRK6Q at
mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> I would suggest using the microbenchmark package to do the time
> comparison. This will run each a bunch of times for a more meaningful
> comparison.
>
> One possible reason for the difference is the number of missing values
> in your data (along with the number of columns). Consider the
> difference in the following results:
>
> > x <- c(1,2,NA)
> > x[x==1]
> [1] 1 NA
> > x[which(x==1)]
> [1] 1
>
>
>
> On Sat, Oct 10, 2020 at 5:25 PM 1/k^c <kchamberln at gmail.com>
wrote:
> >
> > Hi R-helpers,
> >
> > Does anyone know why adding which() makes the select call more
> > efficient than just using logical selection in a dataframe?
Doesn't
> > which() technically add another conversion/function call on top of the
> > logical selection? Here is a reproducible example with a slight
> > difference in timing.
> >
> > # Surrogate data - the timing here isn't interesting
> > urltext <- paste("https://drive.google.com/",
> > "uc?id=1AZ-s1EgZXs4M_XF3YYEaKjjMMvRQ7",
> > "-h8&export=download",
sep="")
> > download.file(url=urltext, destfile="tempfile.csv") #
download file first
> > dat <- read.csv("tempfile.csv", stringsAsFactors = FALSE,
header=TRUE,
> > nrows=2.5e6) # read the file; 'nrows' is a
slight
> > # overestimate
> > dat <- dat[,1:3] # select just the first 3 columns
> > head(dat, 10) # print the first 10 rows
> >
> > # Select using which() as the final step ~ 90ms total time on my
macbook air
> > system.time(
> > head(
> > dat[which(dat$gender2=="other"),],),
> > gcFirst=TRUE)
> >
> > # Select skipping which() ~130ms total time
> > system.time(
> > head(
> > dat[dat$gender2=="other", ]),
> > gcFirst=TRUE)
> >
> > Now I would think that the second one without which() would be more
> > efficient. However, every time I run these, the first version, with
> > which() is more efficient by about 20ms of system time and 20ms of
> > user time. Does anyone know why this is?
> >
> > Cheers!
> > Keith
> >
> > ______________________________________________
> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>
>
> --
> Gregory (Greg) L. Snow Ph.D.
> 538280 at gmail.com
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 12 Oct 2020 08:33:44 +0200 (CEST)
> From: =?UTF-8?Q?Frauke_G=C3=BCnther?= <guenther at leibniz-bips.de>
> To: "r-help at r-project.org" <r-help at r-project.org>
> Cc: William Michels <wjm1 at caa.columbia.edu>, "smm at
posteo.org"
> <smm at posteo.org>
> Subject: Re: [R] Fwd: Help using the exclude option in the neuralnet
> package
> Message-ID: <957726669.124476.1602484424752 at srvmail.bips.eu>
> Content-Type: text/plain; charset="utf-8"
>
> Dear all,
>
> the exclude and constant.weights options are used as follows:
>
> exclude: A matrix with n rows and 3 columns will exclude n weights. The the
first column refers to the layer, the second column to the input neuron and the
third column to the output neuron of the weight.
>
> constant.weights: A vector specifying the values of the weights that are
excluded from the training process and treated as fix.
>
> Please refer to the following example:
>
> Not using exclude and constant.weights (all weights are trained):
>
> > nn <- neuralnet(Species == "setosa" ~ Petal.Length +
Petal.Width, iris, linear.output = FALSE)
> >
> > nn$weights
> [[1]]
> [[1]][[1]]
> [,1]
> [1,] 6.513239
> [2,] -0.815920
> [3,] -5.859802
> [[1]][[2]]
> [,1]
> [1,] -4.597934
> [2,] 9.179436
>
> Using exclude (2 weights are excluded --> NA):
>
> > nn <- neuralnet(Species == "setosa" ~ Petal.Length +
Petal.Width, iris, linear.output = FALSE,
> exclude = matrix(c(1,2,1, 2,2,1),byrow=T, nrow=2))
> > nn$weights
> [[1]]
> [[1]][[1]]
> [,1]
> [1,] -0.2815942
> [2,] NA
> [3,] 0.2481212
> [[1]][[2]]
> [,1]
> [1,] -0.6934932
> [2,] NA
>
> Using exclude and constant.weights (2 weights are excluded and treated as
fix --> 100 and 1000, respectively):
>
> > nn <- neuralnet(Species == "setosa" ~ Petal.Length +
Petal.Width, iris, linear.output = FALSE,
> exclude = matrix(c(1,2,1, 2,2,1),byrow=T, nrow=2),
> constant.weights=c(100,1000))
> > nn$weights
> [[1]]
> [[1]][[1]]
> [,1]
> [1,] 0.554119
> [2,] 100.000000
> [3,] 1.153611
> [[1]][[2]]
> [,1]
> [1,] -0.3962524
> [2,] 1000.0000000
>
> I hope you will find this example helpful.
>
> Sincerely,
> Frauke
>
>
> > William Michels <wjm1 at caa.columbia.edu mailto:wjm1 at
caa.columbia.edu > hat am 10.10.2020 18:16 geschrieben:
> >
> >
> > Forwarding: Question re "neuralnet" package on the
R-Help mailing list:
> >
> > https://stat.ethz.ch/pipermail/r-help/2020-October/469020.html
> >
> > If you are so inclined, please reply to:
> >
> > r-help at r-project.org mailto:r-help at r-project.org <r-help
at r-project.org mailto:r-help at r-project.org >
> >
> > ---------- Forwarded message ---------
> > From: Dan Ryan <Dan.Ryan at unbc.ca mailto:Dan.Ryan at unbc.ca
>
> > Date: Fri, Oct 9, 2020 at 3:52 PM
> > Subject: Re: [R] Help using the exclude option in the neuralnet
package
> > To: r-help at r-project.org mailto:r-help at r-project.org
<r-help at r-project.org mailto:r-help at r-project.org >
> >
> > Good Morning,
> >
> > I am using the neuralnet package in R, and am able to produce some
> > basic neural nets, and use the output.
> >
> > I would like to exclude some of the weights and biases from the
> > iteration process and fix their values.
> >
> > However I do not seem to be able to correctly define the exclude
and
> > constant.weights vectors.
> >
> > Question: Can someone point me to an example where exclude and
> > contant.weights are used. I have search the R help archive, and
> > haven't found any examples which use these on the web.
> >
> > Thank you in advance for any help.
> >
> > Sincerely
> >
> > Dan
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org mailto:R-help at r-project.org mailing
list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Tue, 13 Oct 2020 08:04:32 +0200
> From: Ablaye Ngalaba <ablayengalaba at gmail.com>
> To: R-help at r-project.org
> Subject: [R] package for kernel on R
> Message-ID:
> <CAOkWQv2YoQPpsBUJzV3i4EhAYHNRVZP3vuRXeBA28fLKSUdeqA at
mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
> Please, I want to know which package to install on R when coding the kernel
> functions
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 5
> Date: Tue, 13 Oct 2020 09:09:00 +0200
> From: Ablaye Ngalaba <ablayengalaba at gmail.com>
> To: R-help at r-project.org
> Subject: [R] help for R code
> Message-ID:
> <CAOkWQv0LsgxkHdqpai1=9BpLmp6tAdNwZiqTihA8zrirkf2yFQ at
mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Good morning dear administrators,
> Please help me to code this code in R.
> I use in this file the redescription function ? which by making a scalar
> product gives a . You can also choose instead of the redescription function
> ? a kernel k(x,x).
>
>
>
>
> Sincerely
>
> [[alternative HTML version deleted]]
>
>
>
>
> ------------------------------
>
> Message: 6
> Date: Tue, 13 Oct 2020 11:21:45 +0300
> From: Eric Berger <ericjberger at gmail.com>
> To: Ablaye Ngalaba <ablayengalaba at gmail.com>
> Cc: R mailing list <R-help at r-project.org>
> Subject: Re: [R] help for R code
> Message-ID:
> <CAGgJW74TP-+L6gg0_BLbnayL657Ejw+_fvQ+tScsaDgEj8vQDA at
mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Ablaye,
> The CRAN repository has thousands of available R packages. To help
> people find relevant packages amid such a huge collection, there are
> some 'task view' pages that group packages according to a
particular
> task. I am guessing that you are interested in kernels because of
> their use in machine learning, so you might want to look at the
> Machine Learning task view at:
>
> https://cran.r-project.org/web/views/MachineLearning.html
>
> If you search for 'kernels' on that page you will find
>
> 'Support Vector Machines and Kernel Methods' which mentions a few
> packages that use kernels.
>
> Good luck,
> Eric
>
>
> On Tue, Oct 13, 2020 at 10:09 AM Ablaye Ngalaba <ablayengalaba at
gmail.com> wrote:
> >
> > Good morning dear administrators,
> > Please help me to code this code in R.
> > I use in this file the redescription function ? which by making a
scalar
> > product gives a . You can also choose instead of the redescription
function
> > ? a kernel k(x,x).
> >
> >
> >
> >
> > Sincerely
> >
> > [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
> ------------------------------
>
> End of R-help Digest, Vol 212, Issue 12
> ***************************************