Bug with syn.collinear

[joepemberton1995]

Assume that two variables of a dataset x,y are collinear, and we want to predict y from x. Let xp be the synthetic version of x

The use of syn.collinear(y, x, xp) when generating non NA values relies on the basis that each xp term is somewhere in x (that is, it is not unique to xp). A term unique to xp will result in an NA index after the line

indexp <- match(xp,x)

At this point the line

yp <- y[indexp]

will result in NA values itself. This is something that is not prepared for and will cause the syn.sampler() method to then pluck out the values of the "columnname.NA" column (in this case the non-NA values are of the form 999,9999 etc).

Below is a text file which can be uploaded as a csv. Use seed 268453891 and apply syn(text table, seed = seed) for guaranteed example.

bugExample.txt

[gillian-raab]

Dear Joe,
Our apologies for taking so long to respond to this. You are quite correct that the collinear method does not work correctly in the presence of missing values. The easiest solution is just to do a simple linear prediction in these cases. A function for this is below. You will see that it works OK on your example.

The purpose of identifying collinearity was to stop users inadvertertently submitting linearly dependent variables, usually by not understanding their data well. This is the most likely cause for large data sets which synthpop is intended for. But it is certainly possible that some versions of such variables might have missing values, so you are quite right to point out this bug. We will correct it in the next version and acknowledge you in the news file.

Thank you very much for your help

gillian ([email protected])

###-----syn.collinear------------------------------------------------------
syn.collinear <- function (y, x, xp, ...) {
x <- x[,1]
xp <- xp[,1]
slope <- sum((x-mean(x))(y-mean(y)))/ sum((x-mean(x))^2)
int <- mean(y) - slopemean(x)
yp <- int + slope*xp
return(list(res = yp, fit = "collinear"))
}