R help - Apr 2010 - import file formatted RFC-822

Dear R-list users:

I would like to import a database of web robots,
http://www.robotstxt.org/db/all.txt, it?s formatted RFC-822, ?how can
I do it?

The RFC 822 specification defines a standard format for electronic
messages, which consists of a set of header fields and an optional
body. The headers contain information about the message, such as to
whom it is being sent, from whom it is being sent, when it was sent,
the subject, and so on. The body, if present, is separated from the
header fields by an empty line (\r\n). The following is an example of
a simple message in this format:

From: example at example.com
To: example2 at example.com
Subject: As basic as it gets

This is the plain text body of the message.  Note the blank line
between the header information and the body of the message.

Both the headers and the optional body must consist only of US-ASCII
characters. Each header consists of a name and a value that is
separated by a colon character. The header name must consist of
printable US-ASCII characters. The header value can span one or two
lines (folded or unfolded) and is terminated by a carriage return and
line feed sequence followed by a non-white space character.

The RFC 822 format is the basis for many other more specific message
formats. The following example is a message in USENET article (or
message) format specified by RFC 850, which is based on RFC 822:

From: example at example.com
Subject: As basic as it gets
Newsgroups: comp.microsoft.test

This is the plain text body of the message.  Note the blank line
between the header information and the body of the message.

Thanks,

Sebasti?n.

On Tue, Apr 13, 2010 at 6:26 PM, Sebastian Kruk <residuo.solow at
gmail.com> wrote:
> Dear R-list users:
>
> I would like to import a database of web robots,
> http://www.robotstxt.org/db/all.txt, it?s formatted RFC-822, ?how can
> I do it?
 RFC822 looks very much like R's package DESCRIPTION files, and they
are read in using read.dcf because they are conformant to 'Debian
Control File' format. So I tried read.dcf on it:

 > robots = read.dcf("all.txt")
 > dim(robots)
 [1] 298  38

 so that's a matrix:

 > dimnames(robots)
[[1]]
NULL

[[2]]
 [1] "robot-id"                  "robot-name"
 [3] "robot-cover-url"           "robot-details-url"
 [5] "robot-owner-name"          "robot-owner-url"
 [7] "robot-owner-email"         "robot-status"
 [9] "robot-purpose"             "robot-type"
[11] "robot-platform"            "robot-availability"
[13] "robot-exclusion"           "robot-exclusion-useragent"
[15] "robot-noindex"             "robot-host"
[17] "robot-from"                "robot-useragent"
[19] "robot-language"            "robot-description"
[21] "robot-history"             "robot-environment"
[23] "modified-date"             "modified-by"
[25] "robot-nofollow"            "robot-owner-name2"
[27] "robot-owner-url2"          "robot-owner-email2"
[29] "robot-owner-name3"         "robot-owner-name4"
[31] "robot-environment1"        "robot-environment2"
[33] "robot-purpose1"            "robot-purpose2"
[35] "robot-purpose3"            "robot-platform1"
[37] "robot-description1"        "robot-description2"

 and I guess it pads out the columns so every row has every possible
variable value even if it doesn't exist in the record for that robot.

 Sorted?

Barry

Barry, thank you so much! It's work.