To make a long story short I was doing some in-sample testing in which some
dynamically created regressors would end up either all true or all false
based on the validation portion. In my case a new mainframe configuration
(this is a crappy way to handle a level shift but I do what I can.) So here
is the code snippet that finally let me pre-check my regressors and drop any
of them that were all true or all false.
First the automagic STL outlier grabber that caused part of the problem:
############################################################
# tsSource being my time Series source.
# sh2 is a table of all my regessors that have been previously pulled in
# this has historic and future values in it also, it gets sliced later.
# the EOM is the regessor holding weeks that contain an 'End of Month'
#
# This appends the found IOs to the regressor table. Stepwise tends to
# remove them later on. I needed a programtic way of removing useless
# regressors for model verification since I would not know their names
# if any are found
############################################################
tsSourceDiag <- stl(tsSource,s.window="per", robust=TRUE)
#
tsSourceIO <- which(tsSourceDiag $ weights < 1e-8)
#
# This is how to append run-time regessors
for(z in tsSourceIO) {
tmpname <-paste("PreIO",z,sep="")
#COPY EOM AS A TEMPLATE
sh2[[tmpname]] <- sh2[["EOM"]]
#SET IT ALL TO 0
sh2[[tmpname]][]<-FALSE
#SET The Proper Indice to TRUE
sh2[[tmpname]][z]<- TRUE
}
So to get rid of them (those empty useless regressors) I cooked up this:
###################################################
#Prune Empty Regressors (All false or all true)
# the newmcReg you see is a copy of the sh2 from earlier
# newmcReg = New Model Current Regressors
# sh2 later became cReg.
#
# Yes it makes my eyes bleed. in short we count all the trues
# and all the false and if they happen to be the same number
# as the length we know they are all true or false.
#
# the trick I finally found was that you could in fact -c()
# a list (e.g. ask for everything but the following) but you
# can't apparently do that inline so we just make a list of
# regressors that get shown the door then after hunting
# them down we give em the boot. This mess is soley
# so my in-sample Arima doesn't choke on xreg=newmcReg
# in which one of the newmcReg happen to be all true or false.
#
# God I wish I had taken more then a Trig course. Where was I?
#
# Yes that phantom 'i' you see is that this is all in a big loop
# for 6 possible models
# lm1 = all regressors w/ intercept
# lm2 = lm1 stepwise removal
# lm3 = all regressors wo/ intercept
# lm4 = lm3 stepwise removal
# lm5 = Hand Tuned
# lm6 = lm5 stepwise removal
###################################################
toPurge=c()
for(k in names(newmcReg[[i]])) {
print (paste("check to see if",k,"is a useless regressors for
model",i))
if(sum(newmcReg[[i]][k][,1])==length(newmcReg[[i]][k][,1])) {
print(paste("All of",k,"are TRUE"))
getLost=which(names(newmcReg[[i]])==k)
toPurge=c(toPurge,getLost)
print(paste(k, "has been added to the purge list for model",
i,"!"))
}
if(sum(newmcReg[[i]][k][,1]==FALSE)==length(newmcReg[[i]][k][,1])) {
print(paste("All of",k,"are FALSE"))
getLost=which(names(newmcReg[[i]])==k)
toPurge=c(toPurge,getLost)
print(paste(k, "has been added to the purge list for model",
i,"!"))
}
}
toPurge
# Do this only if there are any or R will beat you senseless and
# steal all your M&Ms!
if(length(toPurge)!=0) {
names(newmcReg[[i]])
names(newmcReg[[i]][-c(toPurge)])
newmcReg[[i]] <- newmcReg[[i]][-c(toPurge)]
newmfReg[[i]] <- newmfReg[[i]][-c(toPurge)]
names(newmcReg[[i]])
}
######################################################
# End Regressor Pruning
######################################################
Big thanks to the help so far. Now about those darn transfer functions...
hmm and pulse detection...
[[alternative HTML version deleted]]
