[eendebakpt]

x_mul() checks for signals on every row, x_divrem() on every quotient digit and the decimal string conversion on every digit. PyErr_CheckSignals() costs about as much as a hundred digit operations, so for small operands the checks cost more than the arithmetic.

SIGCHECK() now takes the loop index and checks on every 64th iteration, the same pattern _sre uses.

Benchmark	main	PR
a * b (10 x 10 digits)	159 ns	97.2 ns: 1.64x faster
a * b (60 x 60 digits)	2.61 us	2.22 us: 1.18x faster
a * b (500 x 500 digits)	99.1 us	79.7 us: 1.24x faster
a // b (1000 // 2 digits)	13.8 us	10.3 us: 1.34x faster
a // b (1000 // 500 digits)	204 us	not significant
pow(3, 5000)	10.7 us	8.38 us: 1.28x faster
math.factorial(2000)	97.7 us	81.6 us: 1.20x faster
str(7**1000)	8.07 us	7.34 us: 1.10x faster

Benchmark script

import pyperf

runner = pyperf.Runner()
def ints(na, nb):
    return (f"import random; r = random.Random(1); "
            f"a = r.getrandbits({na*30-1}) | (1 << {na*30-2}); "
            f"b = r.getrandbits({nb*30-1}) | (1 << {nb*30-2})")
runner.timeit("a * b (10 x 10 digits)", "a * b", setup=ints(10, 10))
runner.timeit("a * b (60 x 60 digits)", "a * b", setup=ints(60, 60))
runner.timeit("a * b (500 x 500 digits)", "a * b", setup=ints(500, 500))
runner.timeit("a // b (1000 // 2 digits)", "a // b", setup=ints(1000, 2))
runner.timeit("a // b (1000 // 500 digits)", "a // b", setup=ints(1000, 500))
runner.timeit("pow(3, 5000)", "pow(3, 5000)")
runner.timeit("math.factorial(2000)", "factorial(2000)",
              setup="from math import factorial")
runner.timeit("str(7**1000)", "str(x)", setup="x = 7**1000")

Generated with Claude Code

• Issue: Improve performance of pidigits and int arithmetic #157742

[skirpichev]

Could you test pidigits on this pr?

[eendebakpt]

Could you test pidigits on this pr?

pidigits is in the noise (although on a quiet machine one can measure the improvement I believe). With the same script as above one can see that also the multiplication of 64-bits ints gains.

Benchmark	main (run 1 / run 2)	PR (run 1 / run 2)	Result
a * b (1 x 2 digits)	25.3 / 25.2 ns	23.2 / 23.2 ns	1.09x faster
a * b (2 x 1 digits)	25.5 / 25.7 ns	22.9 / 22.8 ns	1.12x faster
a * b (3 x 1 digits)	25.1 / 25.5 ns	23.4 / 23.3 ns	1.08x faster
pidigits	155 / 155 ms	155 / 155 ms	not significant

[skirpichev]

LGTM

[skirpichev]

This affects also string conversion.

[eendebakpt]

Yes, but the gain is not really worth mentioning. For small number of digits the overhead dominates, for large number of digits the bigint algorithm in _pylong takes over.

[skirpichev]

On your benchmarks: 1.10x speedup, no? Looks too big for noise.

[eendebakpt]

I benchmarked again on another system, not a factor 1.04 so it is not noise. But conversion to string is mostly relevant for small strings and the gain there very small (due to construction of the str object), For very large objects another code path is used (_pylong).

[picnixz]

For systems under duress, how long should we wait when doing a ctrl+c for Python to exit properly (is it visible?)

Note to myself and for others in general but did we audit other parts of the code where we spent long calculations without checking for signals? I never really considered this problem but I wonder if hashlib in general should not check that when doing hashing per large blocks. I never worked with the check signal API so I do not know when this applies as well (if we are not holding the GIL should we check for signals?)

[eendebakpt]

Note to myself and for others in general but did we audit other parts of the code where we spent long calculations without checking for signals? I never really considered this problem but I wonder if hashlib in general should not check that when doing hashing per large blocks. I never worked with the check signal API so I do not know when this applies as well (if we are not holding the GIL should we check for signals?)

The signal checks are not done everywhere consistently.

It is not present in x_add (but does not really need it either, it linear in the input). For math.gcd it is not there, so on my system math.gcd(100k digits) is blocked for 25 seconds (both main and the PR).

For x_mul the situation is more complex: for normal multiplications the number of signal checks is too high (hence this PR). For a 60 x 60 digit multiplication on main we have a check every 70 ns, with the first version of the PR no checks, for the entire computation of 3 us, which seems reasonable. For a larger number of digits the karatsuba path is followed, which does not have signal checks inside k_mul. karatsuba calls multiplications for smaller sizes, but whether they hit the i==63 check in this PR depends on the size. For a = (1 << 6_000_000) - 1 and b = a - 1 we have a computation of 3.4 seconds, with no signal checks from this PR.

I updated the PR to add a signal check to k_mul itself. That helps for karatsuba and some other cases as well. On main pow(a, 2**300000, m), m = 60 digits was blocking for 3 seconds, in the PR for only 2 ms.

A parallel approach is to improve the performance of the signal check itself. That is done in #157748.

[StanFromIreland]

This is probably less impressive after #157748, but it still seems worthwhile. Also, the title is out-of-date.

[eendebakpt]

This is probably less impressive after #157748, but it still seems worthwhile. Also, the title is out-of-date.

Gain is indeed smaller now:

Benchmark	main	PR #157744
mul 1x2	23.0 ns	22.2 ns: 1.04x faster
mul 2x1	23.3 ns	23.3 ns: not significant
mul 1x10	28.2 ns	27.2 ns: 1.04x faster
mul 2x10	35.0 ns	34.2 ns: 1.02x faster
mul 3x3	29.2 ns	28.3 ns: 1.03x faster
mul 10x10	92.4 ns	87.6 ns: 1.05x faster
mul 60x60	2.46 us	2.18 us: 1.13x faster
mul 1x2000	1.26 us	1.25 us: 1.00x faster
mul 500x500	76.6 us	78.5 us: 1.03x slower
square 60	1.22 us	1.19 us: 1.03x faster
div 2//2	41.1 ns	40.4 ns: 1.02x faster
div 3//2	51.0 ns	47.5 ns: 1.07x faster
div 10//2	120 ns	118 ns: 1.02x faster
div 1000//2	9.52 us	9.47 us: 1.01x faster
div 10//9	79.4 ns	78.7 ns: 1.01x faster
div 100//99	358 ns	353 ns: 1.01x faster
div 1000//500	184 us	187 us: 1.02x slower
div 1000//999	3.20 us	3.21 us: 1.00x slower
str 10 digits	143 ns	143 ns: not significant
str 300 digits	81.7 us	81.1 us: 1.01x faster
pow(3, 5000)	8.29 us	8.21 us: 1.01x faster
factorial(2000)	77.8 us	77.2 us: 1.01x faster
pidigits	155 ms	155 ms: 1.00x faster

[StanFromIreland]

Ouch, slightly worse than I anticipated. I'm +0 on this now.