Skip to content

Compute unit-count products in fmatch's feasibility check as doubles - #261

Open
nullsatz wants to merge 1 commit into
masterfrom
fmatch-integer-overflow
Open

nullsatz wants to merge 1 commit into
masterfrom
fmatch-integer-overflow

Conversation

@nullsatz

Copy link
Copy Markdown
Collaborator

On a large sparse problem, fullmatch stops with "missing value where TRUE/FALSE needed" rather than matching or reporting that the restrictions cannot be met. Reproducible with simulated data on 0.10.8.9002:

library(optmatch)
set.seed(1)
nt <- 14748L; nc <- 372150L
tn <- paste0("t", seq_len(nt)); cn <- paste0("c", seq_len(nc))
rows <- rep(seq_len(nt), each = 20L)
cols <- sample.int(nc, nt * 20L, replace = TRUE)
keep <- !duplicated(paste(rows, cols))
dist <- optmatch:::makeInfinitySparseMatrix(runif(sum(keep)),
          rows = rows[keep], cols = cols[keep], rownames = tn, colnames = cn)
z <- data.frame(z = c(rep(1, nt), rep(0, nc)), row.names = c(tn, cn))
fullmatch(dist, data = z, max.controls = 4)
#> Warning: In nt * mxc : NAs produced by integer overflow
#> Error: missing value where TRUE/FALSE needed

The cause is the recognizably infeasible check in fmatch. Both nt times mnc and nt times mxc are integer products of unit counts, and here the second is about 5.5 billion, so it overflows to NA and the enclosing if cannot evaluate. This change computes the two products in double precision, which leaves the comparisons unchanged.

With the change the same call returns an unmatched result and the warning the check exists to produce, "Matching failed. (Restrictions impossible to meet?)", which is right: four controls per treated unit cannot absorb 372,150 controls. Adding mean.controls makes the problem feasible and it matches normally, with and without the change.

One related observation I have not fixed here. A few lines further on, the node supply is built with mxc times nt, which overflows the same way and still warns. That value has to remain an integer for the NodeInfo class, so it needs a different treatment, and it only arises on problems that are already infeasible. Happy to take that on in a separate change if you would like.

I did not add a test, since reproducing the overflow needs a problem large enough to be slow in a test suite. Glad to add one if you would prefer it.

Composed with assistance from Claude Opus 5.

On a large problem the products of unit counts in the "recognizably
infeasible" check overflow integer arithmetic: with 14,748 treated units and
372,150 controls, nt * mxc is about 5.5 billion, so it becomes NA and the
enclosing if() stops with "missing value where TRUE/FALSE needed" rather than
reporting that the restrictions cannot be met.

Computing the two products in double precision leaves the comparisons
unchanged and lets the check do its job.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@benthestatistician benthestatistician left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks. I don't think the change necessitates its own test. I'm also happy either to leave in place the other NA integer overflow warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants