R devel - Dec 2016 - Fwd: Re: RSiteSearch, sos, rdocumentation.org, ...?

On 9/8/2016 5:01 AM, Jonathan Baron wrote:
> OK.  It is sort of fixed and sort of works.
>
> We'll keep it for now, but this is not going to work forever. When
> namazu fails completely I will not have the time to install a new
> search engine.
>
> One option is to use google. For a site like this, I think they will
> want some money, but I'm not sure, and I do not have the time to deal
> with it.
>
> We have over 10,000 packages now. I wonder if searching all help files
> is really helpful anymore.

       The fastest way I know to do a literature search for anything 
statistical uses the sos package as follows:


             1.  docPages <- findFn('search string') or
findFn('{search
string}')


             2.  installPackages(docPages) # this installs packages to 
enable a more complete package summary


             3.  writeFindFn2xls(docPages) # this creates an Excel file 
with 3 sheets:  a package summary, the findFn table, and the call.


             4.  Then I open the Excel file, and review the package 
summary sheet.  I prioritize my search from there based on the number 
and strength of matches, how close it sounds to what I want, the date of 
the last update, whether it has a vignette, and the authors and 
maintainers.


       There may be a better way to do this using Google or something 
else.  I'd be pleased if someone else could enlighten me.  I admit to 
being biased:  I'm the lead author and maintainer of "sos".
However, I
don't want to perpetuate a tool that has outlived its usefulness, and 
I'm too blind to see that!


       Spencer
>
> Jon
>
> On 09/07/16 22:06, Jonathan Baron wrote:
>> Don't do anything yet. I may have found the problem by accident.
>>
>> I tried to use the computer from something else, and it was being
>> drastically slowed down by some leftover processes, which turned out
>> to be xlhtml. That is something that converts Excel files. Apparently,
>> some excel files got into the libraries, and they were causing the
>> indexing to hang completely.
>>
>> I am now running everything again, starting from scratch, and it might
>> work. (I'm doing it wrong, but it is 3/4 done. I will do it right
>> tomorrow, if it works overnight.)
>>
>> Jon
>>
>> On 09/07/16 16:53, Jonathan Baron wrote:
>>> Spencer,
>>>
>>> Thanks for the quick reply.
>>>
>>> I am open to someone who knows Perl getting an account on my site
and
>>> trying to get it working. It will probably involve fixing more than
>>> one thing, as mknmz depends on some perl modules that also generate
>>> errors.
>>>
>>> My main contribution is figuring out how to extract the html help
>>> files and vignettes only, with some help from R developers and
Fedora
>>> maintainers. Here is the trick, for someone who wants to do it:
>>>
>>> m0 <- rownames(installed.packages())
>>> m1 <- m0[which(m0 %in% needed.packages)]
>>> source("http://bioconductor.org/biocLite.R")
>>> update.packages(oldPkgs=m1,repos=biocinstallRepos())
>>>
update.packages(dependencies=FALSE,INSTALL_opts=c("--no-configure","--no-test-load",
>>>
>> "
>>>
--no-R","--no-clean-on-error","--no-libs","--no-data","--no-demo","--no-exec","--htm
>>>
>> l
>>> "),repos=biocinstallRepos(),ask=F)
>>> m3 <- new.packages()
>>>
install.packages(m3,dependencies=FALSE,INSTALL_opts=c("--no-configure","--no-test-lo
>>>
>> a
>>>
d","--no-R","--no-clean-on-error","--no-libs","--no-data","--no-demo","--no-exec","-
>>>
>> -
>>> html"),repos=biocinstallRepos())
>>>
>>> Note 1: The first 4 lines are designed to deal with a list of the
>>> packages that you actually use. These can be eliminated if you
don't
>>> use R on the same machine. The last 3 lines are all you need.
>>>
>>> Note 2: This works on Fedora, but I think that the Fedora
maintainers
>>> of R have set some defaults that are helpful.
>>>
>>> Jon
>>>
>>> On 09/07/16 15:41, Spencer Graves wrote:
>>>> Hello, All:
>>>>
>>>>
>>>>       Jonathan Baron is "giving up" maintaining the
RSiteSearch
>>>> database.
>>>>
>>>>
>>>>       This breaks three things:  (1) The R Site Search web
service
>>>> that Baron has maintained.  (2) The RSiteSearch function in the
>>>> utils package.  (3) The sos package, for which I'm the
maintainer
>>>> and lead author.
>>>>
>>>>
>>>>       Might someone else be willing to take these over?
>>>>
>>>>
>>>>       For me, the "findFn" capability with
"writeFindFn2xls" is the
>>>> fastest literature search for anything statistical. However, I 
>>>> don't have the resources to take over the management of
Baron's R
>>>> Site Search database.
>>>>
>>>>
>>>>       He's provided a great service for the R community for
many
>>>> years.  I hope we can find a way to keep the system maintained.
>>>> Failing that, I could use help in adapting the sos package to 
>>>> another database.
>>>>
>>>>
>>>>       Thanks,
>>>>       Spencer Graves
>>>>
>>>>
>>>> -------- Forwarded Message --------
>>>> Subject:     Re: RSiteSearch, sos, rdocumentation.org, ...?
>>>> Date:     Wed, 7 Sep 2016 16:15:22 -0400
>>>> From:     Jonathan Baron <baron at psych.upenn.edu>
>>>> To:     Spencer Graves <spencer.graves at prodsyse.com>
>>>> CC:     Jonathan Baron <baron at psych.upenn.edu>, 
>>>> chris.is.fun at gmail.com, info at datacamp.com <info at
datacamp.com>,
>>>> Sundar Dorai-Raj <sdorairaj at gmail.com>, webmaster at
www.r-project-org
>>>>
>>>>
>>>>
>>>> R site search has stopped working. The indexing scrip, mknmz,
failed
>>>> to complete. It has been producing more and more errors and
warnings,
>>>> since it has not been updated for 5 yeaers.
>>>>
>>>> I am giving up on this site. I have too many other things to do
aside
>>>> from find bugs in programs written in languages I don't
know (Perl),
>>>> or set up an alternative search engine.
>>>>
>>>> Please inform anyone else who needs to be informed.
>>>>
>>>> I cannot find the email of the www.r-project.org webmaster, so
I'm
>>>> taking a stab. There are several links to this site in those
pages.
>>>>
>>>> Jon
>>>> -- 
>>>> Jonathan Baron, Professor of Psychology, University of
Pennsylvania
>>>> Home page: http://www.sas.upenn.edu/~baron
>>>> Editor: Judgment and Decision Making (http://journal.sjdm.org)
>>>>
>>>
>>> -- 
>>> Jonathan Baron, Professor of Psychology, University of Pennsylvania
>>> Home page: http://www.sas.upenn.edu/~baron
>>> Editor: Judgment and Decision Making (http://journal.sjdm.org)
>>
>> -- 
>> Jonathan Baron, Professor of Psychology, University of Pennsylvania
>> Home page: http://www.sas.upenn.edu/~baron
>> Editor: Judgment and Decision Making (http://journal.sjdm.org)
>

Spencer and others.

I am going to have to take down the server for RSiteSearch, which is
finzi.psych.upenn.edu, for at least a couple of days starting Sunday
morning. It has been hacked. And I have another server that has also
been hacked, which is higher priority (sjdm.org). On Monday, I will
probably have time to rebuild that one, but I may not have time to
rebuild finzi for another week. I will try to get it all done in one
day, but I don't know if I can.

Sorry about this.

I thought that there was an alternative to this site, namely
http://rdocumentation.org/
but, as bad is my site is, that one, I think, is worse.

Jon

Hi, Jonathan:


       Thanks for letting us know.  I can't imagine that the 
unavailability of RSiteSearch would be more than an inconvenience.


       When you get time, I'd like to know more about what you know 
about how it was hacked, the host operating system, any anti-virus / 
Internet protection software that failed and anything you think might 
reduce the risk of a repeat in the future.


       Best Wishes,
       Spencer Graves


On 12/17/2016 2:32 PM, Jonathan Baron wrote:
> Spencer and others.
>
> I am going to have to take down the server for RSiteSearch, which is
> finzi.psych.upenn.edu, for at least a couple of days starting Sunday
> morning. It has been hacked. And I have another server that has also
> been hacked, which is higher priority (sjdm.org). On Monday, I will
> probably have time to rebuild that one, but I may not have time to
> rebuild finzi for another week. I will try to get it all done in one
> day, but I don't know if I can.
>
> Sorry about this.
>
> I thought that there was an alternative to this site, namely
> http://rdocumentation.org/
> but, as bad is my site is, that one, I think, is worse.
>
> Jon

Unfortunately, I am unable to get this search site working again. (The
message below explains why I had to rebuild it.)

The computer worked for the better part of a day downloading and
installing all the help files from all CRAN packages. Somehow it
failed to get the vignettes this time. But I pushed ahead and ran the
part of namazu that makes the search indices: mknmz. And you can see
the results if you go to finzi.psych.upenn.edu. The search works, but
then you can't get the actual help pages.

The problem seems to be that namazu search is ignoring the "Replace"
rules in the configuration file, namazurc. I have tried 4 versions of
that file, in combination with 3 versions of namazu-cgi, and nothing
works. Note that the last version of namzu was about 2009, the result
of some fixes by linux distribution maintainers, not by the original
creators, who gave up a few years before.

If someone wants to replace namazu with a different search engine, let
me know. I do have a server with all the help files.

And I can tell you how I made them if you want to do this somewhere
else. The trick is:

update.packages(dependencies=FALSE,INSTALL_opts=c("--no-configure","--no-test-load","--no-R","--no-clean-on-error","--no-libs","--no-data","--no-demo","--no-exec","--html"),repos=biocinstallRepos(),ask=F)

and

install.packages(m3,dependencies=FALSE,INSTALL_opts=c("--no-configure","--no-test-load","--no-R","--no-clean-on-error","--no-libs","--no-data","--no-demo","--no-exec","--html"),repos=biocinstallRepos())

This works as is in the fedora version of R, which I think is modified
(for my benefit, about 10 years ago) from the distribution version,
and I think I know what the modification is. The first is for packages
I already have. The second can be used to build the site.

(I use the bioconductor repository because most of the time it doesn't
matter, but there were once a few packages that were only there, ones
that I used myself, like multtest.)

Of course, if there is a human being who reads this and wants to
fiddle with namazu, he or she should contact me.

Jon

On 12/17/16 15:32, Jonathan Baron wrote:
>Spencer and others.
>
>I am going to have to take down the server for RSiteSearch, which is
>finzi.psych.upenn.edu, for at least a couple of days starting Sunday
>morning. It has been hacked. And I have another server that has also
>been hacked, which is higher priority (sjdm.org). On Monday, I will
>probably have time to rebuild that one, but I may not have time to
>rebuild finzi for another week. I will try to get it all done in one
>day, but I don't know if I can.
>
>Sorry about this.
>
>I thought that there was an alternative to this site, namely
>http://rdocumentation.org/
>but, as bad is my site is, that one, I think, is worse.
>
>Jon
-- 
Jonathan Baron, Professor of Psychology, University of Pennsylvania
Home page: http://www.sas.upenn.edu/~baron
Editor: Judgment and Decision Making (http://journal.sjdm.org)