zscore accuracy

[maximsch2]

Some inconsistency in zscore when all values are the same:

@show zscore(zeros(5) + log(1e-5))[1] # NaN
@show zscore(zeros(8) + log(1e-5))[1] # -0.9354143466934853
@show zscore(zeros(10) + log(1e-5))[1] # -0.9486832980505138

I think 0 would have been a better answer in all of those cases, but NaN is certainly better than -0.935

[andreasnoack]

I don't really see how we could do better here. Scaling a degenerate distribution to unit variance is not really possible. In numerical terms that just means you get maximum errors in the result.

[ararslan]

@andreasnoack We could try something like allunique(x) ? Inf : zscore!(...), where Inf could also be NaN or whatever. Dunno if that's the best course of action, just an idea.

[maximsch2]

Even non-unique items can lead to issues. I've run into the same issue before (on the same dataset :() in scikit-learn which lead to: scikit-learn/scikit-learn#4436, scikit-learn/scikit-learn#3725.
I think scikit-learn tries to fix std from 0 to 1, and also does a bit more to ensure stability.

[andreasnoack]

I think scikit-learn tries to fix std from 0 to 1

That's a pretty significant discontinuity to introduce at zero and the result definitely doesn't have unit variance. @maximsch2 can you explain what kind of computations you make after the zscore that would be erroneous for the present version of zscore?

The allunique check should be very cheap so that might be a good way to capture a constant vector. It seems to me that the constant case is the most likely problematic case.

[simonbyrne]

We've hit this issue with mean and var before (I think there's an issue somewhere, though I can't find it now). One solution would be to compute the mean using a superaccumulator algorithm.

[maximsch2]

@andreasnoack I'm processing microarray gene expression data and values for one probe happen to be all equal across sample. Standardizing the data (zero mean, variance one) is a standard pre-processing step to do linear modeling afterwards and, from that point of view, replacing a vector that has all values the same with zeros is OK. I guess you can think of it as just ignoring univariance transform if the variance is exactly zero.

I think regardless of your opinion on magically converting std=0 to std=1 (which I agree is potentially a troublesome thing in the edge cases), I think zscore should return the same outputs for all arrays from my original post, maybe NaN is a correct value (although I would've preferred 0 for practical purposes), but -0.93 certainly doesn't make much sense.

[ararslan]

From a purely mathematical standpoint, a z-score of 0 wouldn't make sense for a constant vector since you're dividing by a standard deviation of 0.

Btw for what it's worth R gives NaN in all cases listed here, both by manually calculating the z-scores and by using scale.

[simonbyrne]

I think NaN is the most appropriate result.

[ararslan]

In terms of implementation would it make more sense to catch a constant vector using allunique or something like isapprox(std(x), 0)?

[simonbyrne]

I don't think we want allunique (as allunique([1,1,2]) == false), but we could just check if all entries are the same as the first.

Computing the mean via a superaccumulator should also fix the problem (as this will give std(x) == 0)

[ararslan]

Seems like adjusting the mean computation would be a good candidate for an enhancement in Base rather than having a competing mean implementation here.

[andreasnoack]

@simonbyrne Wouldn't a fancy mean be much slower? Checking that all the values in a vector are identical (which indeed is not what allunique does) should be really cheap.

[ararslan]

Regarding speed, I was looking at this paper and it indeed seems that precision-preserving summation is much slower.

[simonbyrne]

In that case, why don't we add a pass for mean which checks if all entries are the same, and if so returns that value (we could also do it in the same pass as the summation if that's faster)? This should fix all the downstream problems as well.