R help - Feb 2010 - Import fixed-format ascii file with mixed record types

I need to import several ascii files in fixed format with two different record
types. The data comes from European Labor Force Surveys, wich is a household
survey. The first record type is for people over 16 years, and the second much
sorter is for people aged 15 or less (this record has a filler with several
blanks to get the same record length).
The files tipically have 160000 records, with 176 characters per record, the
data is numeric, corresponding to 102 variables, mostly integers (seven
variables have two decimals). My opertating system is Windows XP.
My questions:
1. Wich do you think is the best way to import the files into R?
2. Could you give me any references or examples?
Thanking you in advance,
Hug



      
	[[alternative HTML version deleted]]

On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
> I need to import several ascii files in fixed format with two  
> different record types. The data comes from European Labor Force  
> Surveys, wich is a household survey. The first record type is for  
> people over 16 years, and the second much sorter is for people aged  
> 15 or less (this record has a filler with several blanks to get the  
> same record length).
> The files tipically have 160000 records, with 176 characters per  
> record, the data is numeric, corresponding to 102 variables, mostly  
> integers (seven variables have two decimals). My opertating system  
> is Windows XP.
> My questions:
> 1. Wich do you think is the best way to import the files into R?

?read.fwf
> 2. Could you give me any references or examples?
There are examples in the help page.
> Thanking you in advance,
> Hug
>
>
>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius, MD
Heritage Laboratories
West Hartford, CT

Thanks David, but can read.fwf cope with different record types?
For example, if recordtype is the 4th character, I could have:

011125678 ---> This is record Type 1
011136779 ---> This is record Type 1
011124943 ---> This is record Type 1
011286711 ---> This is record Type 2
011234872 ---> This is record Type 2
011135628 ---> This is record Type 1

So, how can I tell read.fwf to take the correct type into account?
Thanks again,
Hug

--- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net> wrote:

From: David Winsemius <dwinsemius at comcast.net>
Subject: Re: [R] Import fixed-format ascii file with mixed record types
To: "trece por ciento" <el13porciento at yahoo.com>
Cc: r-help at r-project.org
Date: Monday, February 1, 2010, 12:01 PM


On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
> I need to import several ascii files in fixed format with two different
record types. The data comes from European Labor Force Surveys, wich is a
household survey. The first record type is for people over 16 years, and the
second much sorter is for people aged 15 or less (this record has a filler with
several blanks to get the same record length).
> The files tipically have 160000 records, with 176 characters per record,
the data is numeric, corresponding to 102 variables, mostly integers (seven
variables have two decimals). My opertating system is Windows XP.
> My questions:
> 1. Wich do you think is the best way to import the files into R?

?read.fwf
> 2. Could you give me any references or examples?
There are examples in the help page.
> Thanking you in advance,
> Hug
> 
> 
> 
> 
> ??? [[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius, MD
Heritage Laboratories
West Hartford, CT

On Feb 1, 2010, at 2:33 PM, trece por ciento wrote:
> Thanks David, but can read.fwf cope with different record types?
I do not see any facility for that goal.
> For example, if recordtype is the 4th character, I could have:
>
> 011125678 ---> This is record Type 1
> 011136779 ---> This is record Type 1
> 011124943 ---> This is record Type 1
> 011286711 ---> This is record Type 2
> 011234872 ---> This is record Type 2
> 011135628 ---> This is record Type 1
>
> So, how can I tell read.fwf to take the correct type into account?
It would be easier on the readers of the list if you would offer a  
more complete description of the problem. Problems which are unfolded  
bit by bit are rather annoying to some. Is there a website or document  
that describes this data resource? Perhaps an alternate strategy that  
used a database interface would be more appropriate?

-- 
David.

> Thanks again,
> Hug
>
> --- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net>
wrote:
>
> From: David Winsemius <dwinsemius at comcast.net>
> Subject: Re: [R] Import fixed-format ascii file with mixed record  
> types
> To: "trece por ciento" <el13porciento at yahoo.com>
> Cc: r-help at r-project.org
> Date: Monday, February 1, 2010, 12:01 PM
>
>
> On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
>
>> I need to import several ascii files in fixed format with two  
>> different record types. The data comes from European Labor Force  
>> Surveys, wich is a household survey. The first record type is for  
>> people over 16 years, and the second much sorter is for people aged  
>> 15 or less (this record has a filler with several blanks to get the  
>> same record length).
>> The files tipically have 160000 records, with 176 characters per  
>> record, the data is numeric, corresponding to 102 variables, mostly  
>> integers (seven variables have two decimals). My opertating system  
>> is Windows XP.
>> My questions:
>> 1. Wich do you think is the best way to import the files into R?
>
>
> ?read.fwf
>
>> 2. Could you give me any references or examples?
>
> There are examples in the help page.
>
>> Thanking you in advance,
>> Hug
>>
>>
>>
>>
>>     [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius, MD
> Heritage Laboratories
> West Hartford, CT
>
>
>
>
>
David Winsemius, MD
Heritage Laboratories
West Hartford, CT

On Feb 1, 2010, at 2:33 PM, trece por ciento wrote:
> Thanks David, but can read.fwf cope with different record types?
> For example, if recordtype is the 4th character, I could have:
>
> 011125678 ---> This is record Type 1
> 011136779 ---> This is record Type 1
> 011124943 ---> This is record Type 1
> 011286711 ---> This is record Type 2
> 011234872 ---> This is record Type 2
> 011135628 ---> This is record Type 1
>
> So, how can I tell read.fwf to take the correct type into account?
You may need to separate the line-types first. If the numbers of lines  
are not too large then this would exemplify a strategy:

 > txt <- "011125678
+ 011136779
+ 011124943
+ 011286711
+ 011234872
+ 011135628"

 > substr(readLines(textConnection(txt)), 4,4)
[1] "1" "1" "1" "2" "2"
"1"
 > file1 <- readLines(textConnection(txt)) 
[substr(readLines(textConnection(txt)), 4,4) == "1"]
 > file2 <- readLines(textConnection(txt)) 
[substr(readLines(textConnection(txt)), 4,4) == "2"]
 > file1
[1] "011125678" "011136779" "011124943"
"011135628"
 > file2
[1] "011286711" "011234872"

Then these text objects could be processed with  
read.fwf(textConnection(file1) and the same for file2.

-- 
David.
> Thanks again,
> Hug
>
> --- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net>
wrote:
>
> From: David Winsemius <dwinsemius at comcast.net>
> Subject: Re: [R] Import fixed-format ascii file with mixed record  
> types
> To: "trece por ciento" <el13porciento at yahoo.com>
> Cc: r-help at r-project.org
> Date: Monday, February 1, 2010, 12:01 PM
>
>
> On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
>
>> I need to import several ascii files in fixed format with two  
>> different record types. The data comes from European Labor Force  
>> Surveys, wich is a household survey. The first record type is for  
>> people over 16 years, and the second much sorter is for people aged  
>> 15 or less (this record has a filler with several blanks to get the  
>> same record length).
>> The files tipically have 160000 records, with 176 characters per  
>> record, the data is numeric, corresponding to 102 variables, mostly  
>> integers (seven variables have two decimals). My opertating system  
>> is Windows XP.
>> My questions:
>> 1. Wich do you think is the best way to import the files into R?
>
>
> ?read.fwf
>
>> 2. Could you give me any references or examples?
>
> There are examples in the help page.
>
>> Thanking you in advance,
>> Hug
>>
>>
>>
>>
>>     [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius, MD
> Heritage Laboratories
> West Hartford, CT
>
>
>
>
>
David Winsemius, MD
Heritage Laboratories
West Hartford, CT

David,
In this link there is an excel file with the data dictionary
ftp://www.ine.es/temas/epa/disereg_epa0509.zip
If I understand the second part of the answer: do you mean to process the files,
for example with MS Access, and then importing the result?
(any examples?)
Regards,
Hug

--- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net> wrote:
> From: David Winsemius <dwinsemius at comcast.net>
> Subject: Re: [R] Import fixed-format ascii file with mixed record types
> To: "trece por ciento" <el13porciento at yahoo.com>
> Cc: r-help at r-project.org
> Date: Monday, February 1, 2010, 1:54 PM
> 
> On Feb 1, 2010, at 2:33 PM, trece por ciento wrote:
> 
> > Thanks David, but can read.fwf cope with different
> record types?
> 
> I do not see any facility for that goal.
> 
> > For example, if recordtype is the 4th character, I
> could have:
> > 
> > 011125678 ---> This is record Type 1
> > 011136779 ---> This is record Type 1
> > 011124943 ---> This is record Type 1
> > 011286711 ---> This is record Type 2
> > 011234872 ---> This is record Type 2
> > 011135628 ---> This is record Type 1
> > 
> > So, how can I tell read.fwf to take the correct type
> into account?
> 
> It would be easier on the readers of the list if you would
> offer a more complete description of the problem. Problems
> which are unfolded bit by bit are rather annoying to some.
> Is there a website or document that describes this data
> resource? Perhaps an alternate strategy that used a database
> interface would be more appropriate?
> 
> --David.
> 
> 
> > Thanks again,
> > Hug
> > 
> > --- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net>
> wrote:
> > 
> > From: David Winsemius <dwinsemius at comcast.net>
> > Subject: Re: [R] Import fixed-format ascii file with
> mixed record types
> > To: "trece por ciento" <el13porciento at yahoo.com>
> > Cc: r-help at r-project.org
> > Date: Monday, February 1, 2010, 12:01 PM
> > 
> > 
> > On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
> > 
> >> I need to import several ascii files in fixed
> format with two different record types. The data comes from
> European Labor Force Surveys, wich is a household survey.
> The first record type is for people over 16 years, and the
> second much sorter is for people aged 15 or less (this
> record has a filler with several blanks to get the same
> record length).
> >> The files tipically have 160000 records, with 176
> characters per record, the data is numeric, corresponding to
> 102 variables, mostly integers (seven variables have two
> decimals). My opertating system is Windows XP.
> >> My questions:
> >> 1. Wich do you think is the best way to import the
> files into R?
> > 
> > 
> > ?read.fwf
> > 
> >> 2. Could you give me any references or examples?
> > 
> > There are examples in the help page.
> > 
> >> Thanking you in advance,
> >> Hug
> >> 
> >> 
> >> 
> >> 
> >>? ???[[alternative HTML version
> deleted]]
> >> 
> >> ______________________________________________
> >> R-help at r-project.org
> mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained,
> reproducible code.
> > 
> > David Winsemius, MD
> > Heritage Laboratories
> > West Hartford, CT
> > 
> > 
> > 
> > 
> > 
> 
> David Winsemius, MD
> Heritage Laboratories
> West Hartford, CT
> 
>

you probably need to split your file into several files according to  
the format. you can use something like perl or do it within R by doing  
readline and then separating the lines. you can use grep, split and  
textConnection to do most of the work.

What is the problem you are trying to solve?

Sent from my iPhone.

On Feb 1, 2010, at 14:33, trece por ciento <el13porciento at yahoo.com>  
wrote:
> Thanks David, but can read.fwf cope with different record types?
> For example, if recordtype is the 4th character, I could have:
>
> 011125678 ---> This is record Type 1
> 011136779 ---> This is record Type 1
> 011124943 ---> This is record Type 1
> 011286711 ---> This is record Type 2
> 011234872 ---> This is record Type 2
> 011135628 ---> This is record Type 1
>
> So, how can I tell read.fwf to take the correct type into account?
> Thanks again,
> Hug
>
> --- On Mon, 2/1/10, David Winsemius <dwinsemius at comcast.net>
wrote:
>
> From: David Winsemius <dwinsemius at comcast.net>
> Subject: Re: [R] Import fixed-format ascii file with mixed record  
> types
> To: "trece por ciento" <el13porciento at yahoo.com>
> Cc: r-help at r-project.org
> Date: Monday, February 1, 2010, 12:01 PM
>
>
> On Feb 1, 2010, at 11:40 AM, trece por ciento wrote:
>
>> I need to import several ascii files in fixed format with two  
>> different record types. The data comes from European Labor Force  
>> Surveys, wich is a household survey. The first record type is for  
>> people over 16 years, and the second much sorter is for people aged  
>> 15 or less (this record has a filler with several blanks to get the  
>> same record length).
>> The files tipically have 160000 records, with 176 characters per  
>> record, the data is numeric, corresponding to 102 variables, mostly  
>> integers (seven variables have two decimals). My opertating system  
>> is Windows XP.
>> My questions:
>> 1. Wich do you think is the best way to import the files into R?
>
>
> ?read.fwf
>
>> 2. Could you give me any references or examples?
>
> There are examples in the help page.
>
>> Thanking you in advance,
>> Hug
>>
>>
>>
>>
>>     [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
> David Winsemius, MD
> Heritage Laboratories
> West Hartford, CT
>
>
>
>
>
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.