Evans, Richard K. (GRC-H000)
2018-May-08 15:51 UTC
[R] help with json data from the web into data frame in R
Right. I'm trying to access a server within my organization which has a cert error that I cannot fix. The example link I provided was to a site on the web that does not have the cert error. From the linux shell I use the "-k" switch with cURL to ignore cert errors.. is there an equivalent in the R world? -Rich -----Original Message----- From: David Winsemius [mailto:dwinsemius at comcast.net] Sent: Tuesday, May 08, 2018 11:48 AM To: Evans, Richard K. (GRC-H000) Cc: Eric Berger; r-help at r-project.org Subject: Re: [R] help with json data from the web into data frame in R> On May 8, 2018, at 8:36 AM, Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov> wrote: > > I?ve been tinkering and discovered that the link I need to read json data from is ?https? and there is a certificate warning that I have to click through from a browser. That might be my issue. Is there any way in the json package to tell it to ignore self-signed cert errors in a url?I didn't have that issue when using the link you offered: library(jsonlite) myJSON <- fromJSON( url("https://www.semantic-mediawiki.org/w/api.php?action=ask&query=%5B%5BCategory:City%5D%5D&format=json") ) # results in a complex list (not trivially reducible to a dataframe: str(myJSON) List of 1 $ query:List of 5 ..$ printrequests:'data.frame': 1 obs. of 5 variables: .. ..$ label : chr "" .. ..$ key : chr "" .. ..$ redi : chr "" .. ..$ typeid: chr "_wpg" .. ..$ mode : int 2 ..$ results :List of 39 .. ..$ File:2166320938 5cfc9ec72a z.jpg :List of 6 .. .. ..$ printouts : list() .. .. ..$ fulltext : chr "File:2166320938 5cfc9ec72a z.jpg" .. .. ..$ fullurl : chr "https://www.semantic-mediawiki.org/wiki/File:2166320938_5cfc9ec72a_z.jpg" #-----trimmed----------- David> > -Rich > > From: Eric Berger [mailto:ericjberger at gmail.com] > Sent: Tuesday, May 08, 2018 11:31 AM > To: Evans, Richard K. (GRC-H000) > Cc: r-help at r-project.org > Subject: Re: [R] help with json data from the web into data frame in R > > Hi Rich, > Take a look at the function fromJSON found in the rjson package. > Note that the Usage in the help page: ?fromJSON names the second > argument 'file' but if you look at the description the argument can be a URL. > > HTH, > Eric > > > On Tue, May 8, 2018 at 6:16 PM, Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov<mailto:richard.k.evans at nasa.gov>> wrote: > Hello > > I am able to construct a url that points to some data online in the JSON format. See an example at [0]. > > I would like to work with this data as a dataframe in R. > > I know that there is a package for handling json data [1] but it assumes the data is in a local file but It is not clear to me how to request the data from the web in an R script and get the json data converted into a data frame in R. > > Can anyone provide a basic example or some guidance please? > > -Rich (revansx) > > [0] > https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Catego > ry:City]]&format=json<https://www.semantic-mediawiki.org/w/api.php?act > ion=ask&query=%5b%5bCategory:City%5d%5d&format=json> > [1] https://www.tutorialspoint.com/r/r_json_files.htm > > ______________________________________________ > R-help at r-project.org<mailto:R-help at r-project.org> mailing list -- To > UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. > > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.David Winsemius Alameda, CA, USA 'Any technology distinguishable from magic is insufficiently advanced.' -Gehm's Corollary to Clarke's Third Law
Evans, Richard K. (GRC-H000)
2018-May-08 16:03 UTC
[R] help with json data from the web into data frame in R
That said, I have two issues to ask for help with:
1) how to ignore cert errors with a fromJSON call
And
2) why the json data from the example link doesn't convert to a data frame.
As seen in the following example
library("rjson")
result <- fromJSON(file =
"https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json")
json_data_frame <- as.data.frame(result)
print(json_data_frame)
which results in:
> library("rjson")
Warning message:
package ?rjson? was built under R version 3.4.4
> result <- fromJSON(file =
"https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json")
> json_data_frame <- as.data.frame(result)
Error in (function (..., row.names = NULL, check.rows = FALSE, check.names =
TRUE, :
arguments imply differing number of rows: 0, 1
> print(json_data_frame)
Error in print(json_data_frame) : object 'json_data_frame' not found
>
-----Original Message-----
From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Evans,
Richard K. (GRC-H000)
Sent: Tuesday, May 08, 2018 11:52 AM
To: David Winsemius
Cc: r-help at r-project.org
Subject: Re: [R] help with json data from the web into data frame in R
Right. I'm trying to access a server within my organization which has a cert
error that I cannot fix.
The example link I provided was to a site on the web that does not have the cert
error.
From the linux shell I use the "-k" switch with cURL to ignore cert
errors.. is there an equivalent in the R world?
-Rich
-----Original Message-----
From: David Winsemius [mailto:dwinsemius at comcast.net]
Sent: Tuesday, May 08, 2018 11:48 AM
To: Evans, Richard K. (GRC-H000)
Cc: Eric Berger; r-help at r-project.org
Subject: Re: [R] help with json data from the web into data frame in R
> On May 8, 2018, at 8:36 AM, Evans, Richard K. (GRC-H000)
<richard.k.evans at nasa.gov> wrote:
>
> I?ve been tinkering and discovered that the link I need to read json data
from is ?https? and there is a certificate warning that I have to click through
from a browser. That might be my issue. Is there any way in the json package to
tell it to ignore self-signed cert errors in a url?
I didn't have that issue when using the link you offered:
library(jsonlite)
myJSON <- fromJSON(
url("https://www.semantic-mediawiki.org/w/api.php?action=ask&query=%5B%5BCategory:City%5D%5D&format=json")
)
# results in a complex list (not trivially reducible to a dataframe:
str(myJSON)
List of 1
$ query:List of 5
..$ printrequests:'data.frame': 1 obs. of 5 variables:
.. ..$ label : chr ""
.. ..$ key : chr ""
.. ..$ redi : chr ""
.. ..$ typeid: chr "_wpg"
.. ..$ mode : int 2
..$ results :List of 39
.. ..$ File:2166320938 5cfc9ec72a z.jpg :List of 6
.. .. ..$ printouts : list()
.. .. ..$ fulltext : chr "File:2166320938 5cfc9ec72a z.jpg"
.. .. ..$ fullurl : chr
"https://www.semantic-mediawiki.org/wiki/File:2166320938_5cfc9ec72a_z.jpg"
#-----trimmed-----------
David
>
> -Rich
>
> From: Eric Berger [mailto:ericjberger at gmail.com]
> Sent: Tuesday, May 08, 2018 11:31 AM
> To: Evans, Richard K. (GRC-H000)
> Cc: r-help at r-project.org
> Subject: Re: [R] help with json data from the web into data frame in R
>
> Hi Rich,
> Take a look at the function fromJSON found in the rjson package.
> Note that the Usage in the help page: ?fromJSON names the second
> argument 'file' but if you look at the description the argument can
be a URL.
>
> HTH,
> Eric
>
>
> On Tue, May 8, 2018 at 6:16 PM, Evans, Richard K. (GRC-H000)
<richard.k.evans at nasa.gov<mailto:richard.k.evans at nasa.gov>>
wrote:
> Hello
>
> I am able to construct a url that points to some data online in the JSON
format. See an example at [0].
>
> I would like to work with this data as a dataframe in R.
>
> I know that there is a package for handling json data [1] but it assumes
the data is in a local file but It is not clear to me how to request the data
from the web in an R script and get the json data converted into a data frame in
R.
>
> Can anyone provide a basic example or some guidance please?
>
> -Rich (revansx)
>
> [0]
> https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Catego
>
ry:City]]&format=json<https://www.semantic-mediawiki.org/w/api.php?act
> ion=ask&query=%5b%5bCategory:City%5d%5d&format=json>
> [1] https://www.tutorialspoint.com/r/r_json_files.htm
>
> ______________________________________________
> R-help at r-project.org<mailto:R-help at r-project.org> mailing list
-- To
> UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius
Alameda, CA, USA
'Any technology distinguishable from magic is insufficiently advanced.'
-Gehm's Corollary to Clarke's Third Law
______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
David Winsemius
2018-May-08 16:25 UTC
[R] help with json data from the web into data frame in R
> On May 8, 2018, at 9:03 AM, Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov> wrote: > > That said, I have two issues to ask for help with: > > 1) how to ignore cert errors with a fromJSON callIf you can do it with curl, then why aren't you doing one of a) a system call, b) installing and loading RCurl, c) installing and loading curl (the R package with that name)?> > And > > 2) why the json data from the example link doesn't convert to a data frame.That was already answered in my earlier response. It's not a tabular result, so it doesn't "fit" into a tabular structure. -- David.> As seen in the following example > > library("rjson") > result <- fromJSON(file = "https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json") > json_data_frame <- as.data.frame(result) > print(json_data_frame) > > which results in: > >> library("rjson") > > Warning message: > package ?rjson? was built under R version 3.4.4 > >> result <- fromJSON(file = "https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json") >> json_data_frame <- as.data.frame(result) > > Error in (function (..., row.names = NULL, check.rows = FALSE, check.names = TRUE, : > arguments imply differing number of rows: 0, 1 > >> print(json_data_frame) > > Error in print(json_data_frame) : object 'json_data_frame' not found > >> > > -----Original Message----- > From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Evans, Richard K. (GRC-H000) > Sent: Tuesday, May 08, 2018 11:52 AM > To: David Winsemius > Cc: r-help at r-project.org > Subject: Re: [R] help with json data from the web into data frame in R > > Right. I'm trying to access a server within my organization which has a cert error that I cannot fix. > > The example link I provided was to a site on the web that does not have the cert error. > > From the linux shell I use the "-k" switch with cURL to ignore cert errors.. is there an equivalent in the R world? > > -Rich > > > -----Original Message----- > From: David Winsemius [mailto:dwinsemius at comcast.net] > Sent: Tuesday, May 08, 2018 11:48 AM > To: Evans, Richard K. (GRC-H000) > Cc: Eric Berger; r-help at r-project.org > Subject: Re: [R] help with json data from the web into data frame in R > > >> On May 8, 2018, at 8:36 AM, Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov> wrote: >> >> I?ve been tinkering and discovered that the link I need to read json data from is ?https? and there is a certificate warning that I have to click through from a browser. That might be my issue. Is there any way in the json package to tell it to ignore self-signed cert errors in a url? > > I didn't have that issue when using the link you offered: > > library(jsonlite) > myJSON <- fromJSON( url("https://www.semantic-mediawiki.org/w/api.php?action=ask&query=%5B%5BCategory:City%5D%5D&format=json") ) > > # results in a complex list (not trivially reducible to a dataframe: > > str(myJSON) > List of 1 > $ query:List of 5 > ..$ printrequests:'data.frame': 1 obs. of 5 variables: > .. ..$ label : chr "" > .. ..$ key : chr "" > .. ..$ redi : chr "" > .. ..$ typeid: chr "_wpg" > .. ..$ mode : int 2 > ..$ results :List of 39 > .. ..$ File:2166320938 5cfc9ec72a z.jpg :List of 6 > .. .. ..$ printouts : list() > .. .. ..$ fulltext : chr "File:2166320938 5cfc9ec72a z.jpg" > .. .. ..$ fullurl : chr "https://www.semantic-mediawiki.org/wiki/File:2166320938_5cfc9ec72a_z.jpg" > #-----trimmed----------- > > David > >> >> -Rich >> >> From: Eric Berger [mailto:ericjberger at gmail.com] >> Sent: Tuesday, May 08, 2018 11:31 AM >> To: Evans, Richard K. (GRC-H000) >> Cc: r-help at r-project.org >> Subject: Re: [R] help with json data from the web into data frame in R >> >> Hi Rich, >> Take a look at the function fromJSON found in the rjson package. >> Note that the Usage in the help page: ?fromJSON names the second >> argument 'file' but if you look at the description the argument can be a URL. >> >> HTH, >> Eric >> >> >> On Tue, May 8, 2018 at 6:16 PM, Evans, Richard K. (GRC-H000) <richard.k.evans at nasa.gov<mailto:richard.k.evans at nasa.gov>> wrote: >> Hello >> >> I am able to construct a url that points to some data online in the JSON format. See an example at [0]. >> >> I would like to work with this data as a dataframe in R. >> >> I know that there is a package for handling json data [1] but it assumes the data is in a local file but It is not clear to me how to request the data from the web in an R script and get the json data converted into a data frame in R. >> >> Can anyone provide a basic example or some guidance please? >> >> -Rich (revansx) >> >> [0] >> https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Catego >> ry:City]]&format=json<https://www.semantic-mediawiki.org/w/api.php?act >> ion=ask&query=%5b%5bCategory:City%5d%5d&format=json> >> [1] https://www.tutorialspoint.com/r/r_json_files.htm >> >> ______________________________________________ >> R-help at r-project.org<mailto:R-help at r-project.org> mailing list -- To >> UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. > > David Winsemius > Alameda, CA, USA > > 'Any technology distinguishable from magic is insufficiently advanced.' -Gehm's Corollary to Clarke's Third Law > > > > > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.David Winsemius Alameda, CA, USA 'Any technology distinguishable from magic is insufficiently advanced.' -Gehm's Corollary to Clarke's Third Law
MacQueen, Don
2018-May-09 15:07 UTC
[R] help with json data from the web into data frame in R
Regarding the question:
From the linux shell I use the "-k" switch with cURL to ignore
cert errors.. is there an equivalent in the R world?
I have (sometimes) had success by creating a .curlrc file and putting the
necessary curl options in it. When R invokes curl, curl picks up the options.
[my context was running install.packages( method='curl' ) pointing to a
local repository]
Whether the .curlrc file goes in the local working directory, or your home
directory, I'm not sure anymore. I've tried both.
-Don
--
Don MacQueen
Lawrence Livermore National Laboratory
7000 East Ave., L-627
Livermore, CA 94550
925-423-1062
Lab cell 925-724-7509
?On 5/8/18, 9:03 AM, "R-help on behalf of Evans, Richard K.
(GRC-H000)" <r-help-bounces at r-project.org on behalf of
richard.k.evans at nasa.gov> wrote:
That said, I have two issues to ask for help with:
1) how to ignore cert errors with a fromJSON call
And
2) why the json data from the example link doesn't convert to a data
frame. As seen in the following example
library("rjson")
result <- fromJSON(file =
"https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json")
json_data_frame <- as.data.frame(result)
print(json_data_frame)
which results in:
> library("rjson")
Warning message:
package ?rjson? was built under R version 3.4.4
> result <- fromJSON(file =
"https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Category:City]]|?Capital%20of|?Has%20area&format=json")
> json_data_frame <- as.data.frame(result)
Error in (function (..., row.names = NULL, check.rows = FALSE, check.names =
TRUE, :
arguments imply differing number of rows: 0, 1
> print(json_data_frame)
Error in print(json_data_frame) : object 'json_data_frame' not found
>
-----Original Message-----
From: R-help [mailto:r-help-bounces at r-project.org] On Behalf Of Evans,
Richard K. (GRC-H000)
Sent: Tuesday, May 08, 2018 11:52 AM
To: David Winsemius
Cc: r-help at r-project.org
Subject: Re: [R] help with json data from the web into data frame in R
Right. I'm trying to access a server within my organization which has a
cert error that I cannot fix.
The example link I provided was to a site on the web that does not have the
cert error.
From the linux shell I use the "-k" switch with cURL to ignore
cert errors.. is there an equivalent in the R world?
-Rich
-----Original Message-----
From: David Winsemius [mailto:dwinsemius at comcast.net]
Sent: Tuesday, May 08, 2018 11:48 AM
To: Evans, Richard K. (GRC-H000)
Cc: Eric Berger; r-help at r-project.org
Subject: Re: [R] help with json data from the web into data frame in R
> On May 8, 2018, at 8:36 AM, Evans, Richard K. (GRC-H000)
<richard.k.evans at nasa.gov> wrote:
>
> I?ve been tinkering and discovered that the link I need to read json
data from is ?https? and there is a certificate warning that I have to click
through from a browser. That might be my issue. Is there any way in the json
package to tell it to ignore self-signed cert errors in a url?
I didn't have that issue when using the link you offered:
library(jsonlite)
myJSON <- fromJSON(
url("https://www.semantic-mediawiki.org/w/api.php?action=ask&query=%5B%5BCategory:City%5D%5D&format=json")
)
# results in a complex list (not trivially reducible to a dataframe:
str(myJSON)
List of 1
$ query:List of 5
..$ printrequests:'data.frame': 1 obs. of 5 variables:
.. ..$ label : chr ""
.. ..$ key : chr ""
.. ..$ redi : chr ""
.. ..$ typeid: chr "_wpg"
.. ..$ mode : int 2
..$ results :List of 39
.. ..$ File:2166320938 5cfc9ec72a z.jpg :List of 6
.. .. ..$ printouts : list()
.. .. ..$ fulltext : chr "File:2166320938 5cfc9ec72a z.jpg"
.. .. ..$ fullurl : chr
"https://www.semantic-mediawiki.org/wiki/File:2166320938_5cfc9ec72a_z.jpg"
#-----trimmed-----------
David
>
> -Rich
>
> From: Eric Berger [mailto:ericjberger at gmail.com]
> Sent: Tuesday, May 08, 2018 11:31 AM
> To: Evans, Richard K. (GRC-H000)
> Cc: r-help at r-project.org
> Subject: Re: [R] help with json data from the web into data frame in R
>
> Hi Rich,
> Take a look at the function fromJSON found in the rjson package.
> Note that the Usage in the help page: ?fromJSON names the second
> argument 'file' but if you look at the description the argument
can be a URL.
>
> HTH,
> Eric
>
>
> On Tue, May 8, 2018 at 6:16 PM, Evans, Richard K. (GRC-H000)
<richard.k.evans at nasa.gov<mailto:richard.k.evans at nasa.gov>>
wrote:
> Hello
>
> I am able to construct a url that points to some data online in the
JSON format. See an example at [0].
>
> I would like to work with this data as a dataframe in R.
>
> I know that there is a package for handling json data [1] but it
assumes the data is in a local file but It is not clear to me how to request the
data from the web in an R script and get the json data converted into a data
frame in R.
>
> Can anyone provide a basic example or some guidance please?
>
> -Rich (revansx)
>
> [0]
>
https://www.semantic-mediawiki.org/w/api.php?action=ask&query=[[Catego
>
ry:City]]&format=json<https://www.semantic-mediawiki.org/w/api.php?act
> ion=ask&query=%5b%5bCategory:City%5d%5d&format=json>
> [1] https://www.tutorialspoint.com/r/r_json_files.htm
>
> ______________________________________________
> R-help at r-project.org<mailto:R-help at r-project.org> mailing
list -- To
> UNSUBSCRIBE and more, see https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius
Alameda, CA, USA
'Any technology distinguishable from magic is insufficiently
advanced.' -Gehm's Corollary to Clarke's Third Law
______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Maybe Matching Threads
- help with json data from the web into data frame in R
- help with json data from the web into data frame in R
- help with json data from the web into data frame in R
- help with json data from the web into data frame in R
- help with json data from the web into data frame in R