[naaz234]

Description

unique in the toc extension finds a free id by probing base, base_1, base_2 and so on until one is not in ids. TocTreeprocessor.run calls it once per header with the slugified text, so when many headers share the same text (or all slugify to the same thing, such as headers made only of punctuation) the n-th call probes about n ids and the total work grows with the square of the header count. Timings for markdown.markdown('# Header\n' * n, extensions=['toc']) on CPython 3.11:

n=4000   (36 KB):   2.3 s  ->  0.09 s
n=8000   (72 KB):   9.5 s  ->  0.14 s
n=16000  (144 KB):  49 s   ->  0.34 s

unique now takes an optional counters dict which records, per base id, the highest n for which base_1 through base_n are all known to be taken, and the treeprocessor passes one dict for the whole document. A repeated id resumes the probe after that run instead of from the start. The cache is only consulted or extended when the probe run is contiguous with it, which keeps the result identical to the plain probe when used_ids already holds explicit ids from attr_list that leave gaps, or ids such as foo_07 that collide numerically but not as strings. I compared the new function with the old one on about 780k calls over random id sequences and pre-seeded sets, with and without the cache, and the returned ids and final sets matched in every case. Calling unique(id, ids) without the cache behaves as before.

Added a unit test for the cache edge cases and a rendering test with 10,000 identical headers (about 14 s before the change, instant with it), plus a changelog entry. The full suite and flake8 pass.

AI Assistance Disclosure

• No AI tools were used in preparing this PR.

• AI tools were used and I have disclosed which ones, and have fully reviewed and verified their output.

  AI tools were use to assist with the following:

  • Code
  • Code Comments
  • Documentation
  • Changelog Entry
  • Commit Message
  • The Description Above
  • Other: tests, and the timing and equivalence scripts used to confirm the problem and the fix

Checklist

• This PR follows the contribution guidelines.
• The code follows the Code Style Guide.
• The commit message follows the Commit Message Style Guide.
• I have added or updated relevant docs, including release notes if applicable which follow the Documentation Style Guide.
• I have added or updated relevant tests.
• I have not requested, and will not request, an automated AI review for this PR.

[waylan]

As a general observation, I understand the problem being solved and I understand how the provided code solves it, but I feel like there has to be a more elegant solution. I'm just not sure what it is.

[naaz234]

Agreed it's more involved than I'd like. Most of the extra logic is there so the output stays exactly what the plain probe gives when used_ids already has ids from attr_list that leave gaps, or ids like foo_07. If it's acceptable for generated ids to differ from today's in those edge cases, it could shrink to a per-base "last suffix used" dict in TocTreeprocessor.run and drop the contiguity check. I'm happy to rework it either way, or to try a different approach if you have one in mind.

[waylan]

Most of the extra logic is there so the output stays exactly

That is the crux of the issue. If we don't maintain the preexisting behavior, then we break/change users' permalinks. We don't want to do that, so we will leave it as-is.