[gpshead]

On the free-threaded build, the marshal output for code compiled from an AST depended on which code objects had been created earlier in the process:

SRC = "def f(a, b):\n    return g(a=1, b=2)\n"
# optionally first: compile(ast.parse("def other(a, b):\n    return h(a=3, b=4)\n"), ...)
d_src = marshal.dumps(compile(SRC, 'm.py', 'exec'))
d_ast = marshal.dumps(compile(ast.parse(SRC), 'm.py', 'exec'))

free-threaded main	len(d_src)	len(d_ast)
nothing compiled before	230	230
an unrelated AST compiled before	230	237

The default build gives 230 in every case, and so does the free-threaded build with this PR.

Cause and change

The compiler makes a tuple of names (co_names, co_localsplusnames) the same object as an equal constant: keyword names, a from list, __static_attributes__. On the free-threaded build, intern_constants() replaces each constant by the equal immortal object of a per-interpreter table, if there is one, but does not replace the tuple of names. compile() of a source string turns that off with tstate->suppress_co_const_immortalization (gh-118527). The branch that handles AST objects returned before the flag was set.

flowchart TB
  subgraph before["main, free-threaded"]
    direction TB
    b0["compiler: constant ('a', 'b')<br>is the same tuple as co_names"]
    b0 -- "compile(ast)" --> b2["immortalization<br>not suppressed"]
    b0 -- "compile(str)" --> b1["immortalization<br>suppressed"]
    b2 --> b4{"equal tuple already<br>in the immortal table?"}
    b4 -- yes --> b6["constant replaced, co_names not:<br>two tuples, marshal writes both"]
    b4 -- no --> b5["still one tuple"]
    b1 --> b3["still one tuple<br>marshal: object + reference"]
  end
  subgraph after["this PR"]
    direction TB
    a0["compiler: constant ('a', 'b')<br>is the same tuple as co_names"]
    a0 -- "compile(ast)" --> a1["immortalization<br>suppressed"]
    a0 -- "compile(str)" --> a1
    a1 --> a3["still one tuple<br>marshal: object + reference"]
  end

Loading

The change is the same increment/decrement pair around _PyAST_Compile() in builtin_compile_impl() that already surrounds _Py_CompileString().

Effects

• Tools that compile an AST and write a .pyc (assertion rewriters, SourceLoader.source_to_code() given an AST) get reproducible output. Compiling the standard library from ASTs in one process, 179 of 2067 files gave different bytes depending on file order; now none do.
• Code from compile(ast) no longer has immortal constants, as is already the case for compile(str). It gets them when it is later loaded from a .pyc. Tools that compile many ASTs stop growing the immortal table: for 10000 distinct functions, +141286 memory blocks on main, +606 with this PR.
• The GIL enabled build is not affected.

Not covered

exec() of a string, Py_CompileString*(), PyRun_*() and marshal.loads() do not set the flag and keep the same dependence on history. None of them is on a path that writes .pyc files.

Tests

Every TestConstantsIdentity case (added by gh-159193) is now also compiled from an AST. test_constants_not_immortal (free-threaded only) checks that a tuple constant from compile() of a string or an AST is not immortal even after an equal one entered the table.

Authored using Claude, reviewed by me.

• Issue: Free-threaded build: marshal output of compile() of an AST depends on what was compiled before #159188

[miss-islington-app]

Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @gpshead, I could not cleanly backport this to 3.15 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 9a235416c19edb3ab6fd2808a8917f5831e5a5ba 3.15