R help - Apr 2012 - Finding Instances of a Pattern Throughout Data Set

I have approximately 2.5 million rows from a number of sensor
readings. Having plotted these, I can see a given pattern (say a spike
in the amplitude away from the mean). I would now like to automate
this procedure as we're expecting a great deal more data in the near
future. Is there any package or function that will make this possible?
Many thanks! I suppose, I could do something like:
amplitude <- abs(sensorReadings[sensorReadings[,2] > 1.2 *
median(sensorReadings[,2]),])
Is this the most efficient way to do what I want or not?
-- H

-- 
Sent from my mobile device
Envoyait de mon portable

I strongly suggest you consult with a local statistician. Your
description is far too vague (to me anyway) to make any sense of and
probably requires a good deal of back and forth between you and a
competent data analyst to pin down what the issues and constraints
are. For example, what constitutes "interesting" patterns? --  do the
data need to be analyzed in real time? -- Are visual displays
sufficient or is some kind of numeric indication needed? ... etc. etc.

One tentative word of advice, though: with this much data, do not get
embroiled with P values or other measures of statistical
"significance": anything you can see will be "significant."

-- Bert

On Mon, Apr 2, 2012 at 8:42 PM, Hasan Diwan <hasan.diwan at gmail.com>
wrote:
> I have approximately 2.5 million rows from a number of sensor
> readings. Having plotted these, I can see a given pattern (say a spike
> in the amplitude away from the mean). I would now like to automate
> this procedure as we're expecting a great deal more data in the near
> future. Is there any package or function that will make this possible?
> Many thanks! I suppose, I could do something like:
> amplitude <- abs(sensorReadings[sensorReadings[,2] > 1.2 *
> median(sensorReadings[,2]),])
> Is this the most efficient way to do what I want or not?
> -- H
>
> --
> Sent from my mobile device
> Envoyait de mon portable
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 

Bert Gunter
Genentech Nonclinical Biostatistics

Internal Contact Info:
Phone: 467-7374
Website:
http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm

Mr Gunter,

On 2 April 2012 21:15, Bert Gunter <gunter.berton at gene.com>
wrote:
> I strongly suggest you consult with a local statistician. Your
> description is far too vague (to me anyway) to make any sense of and
> probably requires a good deal of back and forth between you and a
> competent data analyst to pin down what the issues and constraints
> are. For example, what constitutes "interesting" patterns? -- ?do
the
> data need to be analyzed in real time? -- Are visual displays
> sufficient or is some kind of numeric indication needed? ... etc. etc.
> One tentative word of advice, though: with this much data, do not get
> embroiled with P values or other measures of statistical
> "significance": anything you can see will be
"significant."
All I want to show is that I can find instances of any pattern
throughout our data sets and list them. If the pattern I see is when
the accelerometer is 120% of normal and the gyroscope reading is
steady for 3 seconds, how do I find this pattern throughout different
data sets? -- H
-- 
Sent from my mobile device
Envoyait de mon portable