Simpqs-2.0 full

[hvds]

TLDR: quadratic sieve gets 50% speedup for large and medium inputs, some (smaller) speedup for small inputs; memory use 4x smaller for large inputs, roughly unchanged for medium and small inputs.

Incorporate block Lanczos, partial relations and sieving speedups from simpqs-2.0. There are some additional minor fixes/optimizations along the way, and comments marked '(hv)' where I think there may be additional bugs or infelicities that I was not able to resolve. Results from timing tests below show user CPU time and maxresident from /usr/bin/time on an unloaded machine.

Build A: MPUG master @2389dcbc44, fixed to build standalone
- fails to find factors for ds=88 (asterisked) due to bug fixed in 955bb25
Build B: MPUG with simpqs-2.0, without last commit (midprime sieving)
Build C: MPUG with simpqs-2.0, with last commit

CPU   46    61     70      80      88
A:  0.12  4.03  68.26  542.25 3405.12*
B:  0.09  2.73  37.44  289.59 1730.13
C:  0.09  2.33  30.70  248.16 1482.69

RAM(k) 46     61     70      80      88
A:   5904  17672  37596  587976 1324160*
B:   5600  19720  38036  183036  333624
C:   5672  19684  38140  182900  333560

Test cases:
ds=46: 7520587276150280509023659781989524950620229823
= 342594914346335019487 * 21951835713905522424868129
ds=61: 1000000000000000000000000000045999999999999999999999999999373
= 999999999999999999999999999989 * 1000000000000000000000000000057
ds=70: 9999999999999999999999999999890000000120999999999999999999999999998669
= 999999999999999999999999999989 * 10000000000000000000000000000000000000121
ds=80: 99999999999999999999999999999999999941006299999999999999999999999999999999996283
= 99999999999999999999999999999999999941 * 1000000000000000000000000000000000000000063
ds=88: 9180435894215652020112626451519741807924491872283194331211065824189266475562769247511369
= 1059405511716795159376523757718030895723651 * 8665648604506982829490012915377592720602462019

[hvds]

I note that this occasionally shows:

lanczos error: not all columns used
linear algebra failed; retrying...

This does not seem to cause a problem, and it seems one or two retries is always enough to succeed; however the noise to stdout is a bit irritating, and should probably hide behind a verbosity level. I'll try to get some stats on how often this happens, and whether it depends on the size of the matrix.

(For a factor base of size numPrimes, simpqs looks for numPrimes + 64 relations whereas we require only numPrimes relations. It is conceivable that having some extra relations acts as a buffer in some way that avoids this error; I'll have to experiment with that too.)

[hvds]

It appears that "not all columns used" is relatively common, and "Failed, no rows found (mask == 0)" is rarer but also possible (and should not be fatal). I've added a commit to cope with those more cleanly. I assume the remaining check "lanczos error: dependencies don't work" would represent an actual bug, and have left that as a fatal error.

I ran my code using this for a while with extra instrumentation. There were 1700 numbers that reached _GMP_simpqs, of size 41 to 50 digits. There were 70 that failed at least once (about 4%); number of occasions that failed n times was (56, 12, 1, 0, 0, 0, 1), implying that chances of subsequent failures on a given matrix are higher; to cope with that I've also added a hard limit of 100 retries - that could be made run-time configurable if desired.

Since the evaluation time is typically about 15-20% of the total (slightly larger for the smallest inputs), this implies runtimes about 1% longer on average than with no failures.

There was no obvious correlation with size of target, and seeking numPrimes + 64 relations was no different from seeking numPrimes relations. Number of targets for a given number of digits in the test run, from 41 digits, was (1, 1, 2, 45, 241, 733, 586, 60, 29, 2).

[hvds]

Sigh, I've now found a case where - for a given state of the (QS, not block_lanczos) rng - it fails even after 10000 iterations. Analyzing the input matrix, it appears to have an unluckily high number of linearly-dependent entries: it needs numPrimes + 130 relations before it gets enough linear independence to succeed. So I'm now considering making just 1 or 2 attempts (instead of 100), then finding more relations before trying again.

For the record, the failure occurs when the QS rng starts with randval = 1935087715 when trying to factorize n = 52125806551349591 * 3654328997351298614303.

(I am tempted to reset the rngs for each new factorization to aid reproducibility - I was lucky that this failure occurred just 30 minutes into a run - but I think it is unwise. I'll probably instead combine the two simplistic rngs into one, and capture the initial seed to output in the case of eventual failure instead.)

[hvds]

Ah, now fixed: the source value (X) for relations was stored signed, and the failure case had an unusually high number of (X, -X) duplicates (119 in total) leaving the matrix underspecified. Now changed to store abs(X), which has fixed the problem.

I note that this makes the two largest testcases about 10% slower. I assume they also had a bunch of such duplicates - not enough to leave the matrix underspecified, but enough that removing them causes significant extra work to make up the shortfall. In the future it may be worth experimenting to see if we can systematically demand fewer relations without risking underspecification.

I'll continue running with this (updated) code to keep an eye out for any other problems.

[hvds]

FWIW, here's an updated version of the tables above, adding build D (MPUG with latest commits up to c7c5a4e) and for comparison build E (latest msieve run with -q -t 1 to limit to a single thread). This shows there is still lots of opportunity for further improvement in both speed and memory use.

CPU   46    61     70      80      88
A:  0.12  4.03  68.26  542.25 3405.12*
B:  0.09  2.73  37.44  289.59 1730.13
C:  0.09  2.33  30.70  248.16 1482.69
D:  0.09  2.31  30.58  281.93 1631.79
E:  0.11  2.19  25.89  191.85  764.24

RAM    46     61     70      80      88
A:   5904  17672  37596  587976 1324160*
B:   5600  19720  38036  183036  333624
C:   5672  19684  38140  182900  333560
D:   5692  19832  37988  183260  336012
E:  20516  22008  27600   48924   76348

[hvds]

I've been using this implementation without problems for the last 18 months, so I have high confidence in it. If it is of interest to anybody I can update it for latest master: it needs minor adaptations to rebase.

[hvds]

I've rebased over 39982f8 into a new branch hvds:simpqs-post54, and redone the manual tests. Memory use is unaltered and factorization is successful (suggesting I probably didn't mess up the rebase), but for reasons I don't understand the testcases are almost all 20% slower than before. Load average was around 1.25 during the tests, which I think is comparable with previous trials.

@danaj Likely I've messed something up, but it seems possible some dependency in MPUG has slowed down substantially. Are you aware of any bugfixes since the previous merge-base b995fd3 that knowingly slowed down anything that QS might use? I'll have to investigate more deeply, but I don't have time to do it right now.

Row E has the new results, previous "msieve-latest" tests moved to F:

CPU   46    61     70      80      88
A:  0.12  4.03  68.26  542.25 3405.12*
B:  0.09  2.73  37.44  289.59 1730.13
C:  0.09  2.33  30.70  248.16 1482.69
D:  0.09  2.31  30.58  281.93 1631.79
E:  0.10  2.57  36.14  335.61 1968.08
F:  0.11  2.19  25.89  191.85  764.24

RAM    46     61     70      80      88
A:   5904  17672  37596  587976 1324160*
B:   5600  19720  38036  183036  333624
C:   5672  19684  38140  182900  333560
D:   5692  19832  37988  183260  336012
E:   5700  19952  37960  183320  336012
F:  20516  22008  27600   48924   76348

[danaj]

I'm working on this now. My current plan is to move lanczos to it's own file so it can be shared. I'll put simpqs2 in alongside simpqs1. See if we can get simpqs1 to use the new block lanczos. Then work on creating a new siqs.c and see how that goes.

[hvds]

I'm working on this now. My current plan is to move lanczos to it's own file so it can be shared. I'll put simpqs2 in alongside simpqs1. See if we can get simpqs1 to use the new block lanczos. Then work on creating a new siqs.c and see how that goes.

Let me know what if anything you want from me.

Claude failed to replicate the slowdown I found when comparing the old and new rebase branches, so something must have changed my end, but I don't know what: maybe gcc or gmplib version.

[danaj]

I added the gcd destroy change to master, so all that is left is simpqs2. What I should have done here, I think, is make a feature branch that we work on, then merge that in. But I got involved in it and it's on master with changes. The factor recipe has changed some as well, along with some updates for tinyqs. lanczos is in its own file. There are some minor changes for simpqs2 but I've already switched factor.c to use it. It is noticeably better, and you fixed all my objections to using the verbatim SIMPQS 2.0.

Right now I'm working on some minor details such as how the factors are returned (in a dynamic list rather than pre-allocated).

[danaj]

In acde10f.

Again I apologize for not doing this through the PR itself.

[hvds]

In acde10f.

Again I apologize for not doing this through the PR itself.

Thanks, I'm excited to see it go in. :)

I hope you've had a chance to look through the '(hv)' comments added in 2c17542.

[danaj]

In acde10f.
I hope you've had a chance to look through the '(hv)' comments added in 2c17542.

2c7d1f8 addressed an important one, and got a decent performance gain out of it. Good call to question it. They were assuming slot 1 held p=2 which is true sometimes but not always. It did give a small gain. I tried checking both 0 and 1 for p=2 and using the exponent when appropriate, but that was a wash. The better solution which gave me improved results for all cases I tried was to consistently use the exponent in all slots. So for all p, not just p=2 or slot 1.

124515a was after looking at all the comments.

• the soln2 guard is harmless so I left it in. You're right that for the input range we have, we're a long way from hitting it so it wouldn't hurt to take it out like Flint does.
• your comment about fitting in a ui -- I simplified that code so it should be clearer why it fits.
• for the gcd, that doesn't seem right. The value of temp2-temp (X-Y) could be > 1 and still not give a factor. After gcd it's either 1, n, or a factor. If you think X-Y should always give a factor when it's > 1, then I'll look into it closer and collect examples.
• The screwy parameters for > 91 digit input is straight from Flint. They're plain wrong. Without doing testing / tuning, I decided to make them inherit the 91-digit parameters. That means they're likely going to be too small, but it's a safe configuration. The biggest concern I saw was that we could have a 91 digit input, which is in range, but then the multiplier bumps it into 92 digits and we'd have stupid parameters. Now factor.c was using a 299-bit top end, which meant a factor of 8 or so headroom, but that's not enough, and it was a good change to make.

[hvds]

for the gcd, that doesn't seem right. The value of temp2-temp (X-Y) could be > 1 and still not give a factor. After gcd it's either 1, n, or a factor. If you think X-Y should always give a factor when it's > 1, then I'll look into it closer and collect examples.

Oh, maybe I was forgetting the multiplier. I'm happy to take your word for it in any case, and glad that some of the comments proved useful.