[spokodev]

mod(x, y) is floored modulo, but _modBigint returns an out-of-range result when both arguments are negative: mod(-10n, -4n) and mod(-10, -4n) are -6n, while mod(-10, -4) is -2. invmod on bigints inherits this: invmod(2n, -7n) is -10n, while invmod(2, -7) is -3.

_modBigint now adds y only when the remainder and the divisor have different signs.

Three tests in mod.test.js and invmod.test.js cover this. They fail on develop and pass with the fix.

AI use: This PR is AI-assisted. An AI coding assistant found the remaining cases, wrote the change and the tests, and ran them. I reviewed the change and the results, and I decided to open it.

[lbesecker195]

I checked the new bigint branch against x - y * floor(x / y) for 28 sign combinations of dividend and divisor: zero mismatches. This brings bigint in line with the number, BigNumber, and Fraction implementations. LGTM.

[gwhitney]

I believe this was resolved by #3710. If you think there are any remaining difficulties in develop with respect to the issue addressed here, or any tests that should be in place that aren't, please post here. Otherwise please close this PR. Thanks so much.

[spokodev]

#3710 fixed mod(10n, -4n), but develop (2f97b8c) still returns out-of-range results when both arguments are negative, and the new bigint invmod inherits that:

math.mod(-10n, -4n)   // -6n, expected -2n (mod(-10, -4) is -2)
math.mod(-8n, -3n)    // -5n, expected -2n
math.mod(-10, -4n)    // -6n, expected -2n
math.invmod(2n, -7n)  // -10n, expected -3n (invmod(2, -7) is -3)
math.invmod(5n, -13n) // -18n, expected -5n

_modBigint adds y to a non-zero remainder whenever x < 0 || y < 0, but when both are negative % already has the sign of y. For two bigints this predates #3710 (mod(-10n, -4n) was already -6n). The mixed calls are new, because they now go through _modBigint: mod(-10, -4n) returned -2 before #3710.

I merged develop and reduced this PR to that one condition, plus tests in mod.test.js and invmod.test.js. For every integer pair in [-40, 40], bigint mod now matches number, BigNumber and Fraction (6561 pairs, 1442 differ on develop), and bigint invmod matches number (6480 pairs with a non-zero modulus, 959 differ on develop). npm run test:src and npm run lint pass.

If you would rather fix it differently, I am fine closing this.

[gwhitney]

No this is fine. Added a comment for the slightly arcane final formula in _modBigint -- please check that you agree with the comment. Merging.