Skip to content

fix: three parser/graph bugs - module-level async def, entities without an end line, lost import edges - #30

Open
nanami7777777 wants to merge 3 commits into
xnuinside:mainfrom
nanami7777777:fix/async-parser-crash
Open

nanami7777777 wants to merge 3 commits into
xnuinside:mainfrom
nanami7777777:fix/async-parser-crash

Conversation

@nanami7777777

@nanami7777777 nanami7777777 commented Oct 29, 2026 •

Copy link
Copy Markdown

Fixes #31

Three crashes / silent-wrong-result bugs found while running codegraph on a real-world project (326 modules). Two of them are hard crashes, one makes --file-path / --distance always report nothing.


1. AttributeError: 'NoneType' object has no attribute 'file' — module-level async def

create_objects_array()'s async branch assumed the token following async was always def, so async for / async with were parsed as function definitions (in async for x in ... the "name" becomes x). It also pushed None on the scope stack for a module-level async def instead of the AsyncFunction it had just created, so the next nested def / async token called _nest_function(None, ...) → ob.file → AttributeError.

Repro, three lines:

async def foo():
    return [x async for x in bar()]

Fix: validate the token after async (skip if it is not def), and assign cur_func before pushing it on the stack, mirroring the synchronous def branch.

The whole tree is parsed in one pass, so one such file made codegraph <dir> fail for the entire directory — 16 of 326 modules here.

2. TypeError: '<=' not supported between instances of 'int' and 'NoneType' — entities without an end line

Scopes popped by a sibling def / class (a one-line body such as def f(self) -> None: ... never emits INDENT/DEDENT) or still open at EOF never reach the DEDENT handler, so their endno stayed None. usage_graph() then compared entity[0] <= line <= entity[1] with entity[1] is None.

Fix: new _close_scopes() finalizes endno whenever a scope is popped; used in the async / def / class branches and once more after the token loop for anything left open at EOF.

On the project this took entities with endno is None from 74 → 0.

3. usage_graph() silently loses all import edges on a second call

get_imports_and_entities_lines() did code_objects[path].pop(-1) on the Import object, so a second usage_graph() call ran with an empty import set and therefore no cross-module edges. Since the CLI builds the graph (for the visualisation / CSV) before searching it, get_dependencies() always saw a degraded graph and --file-path / --distance printed nothing:

before:  {1: set(), 2: set()}
after:   {1: {llm.py, types.py, events.py, decorator.py, ...},
          2: {engine.py, context.py, records.py, gates.py, ...}}

Fix:

  • keep the Import object on the list and skip it where entities are iterated, instead of popping it;
  • get_dependencies() resolves a relative --file-path (graph keys are absolute, so codegraph src --file-path src/x.py never matched);
  • an edge is followed past distance 1 when its bare "<module>.py" name maps to exactly one file. Ambiguous names are still reported but not traversed — there can be hundreds of __init__.py, and traversing them fans the BFS out to unrelated modules (distance 2 went from 8 meaningful files to ~120 without this guard).

Tests

tests/test_graph_generation.py:

  • TestAsyncParsing — 6 cases (nested sync/async def, async for comprehension, async for statement, async with, module-level async def followed by a top-level def). 5 fail with the AttributeError before the fix.
  • TestEntityEndLines — 4 cases (one-line body, last entity in module, scope closed by a sibling def, module-level async function).
  • TestUsageGraphIdempotency — 2 cases (second usage_graph() equals the first; get_dependencies() still finds edges after usage_graph() was called).

46 tests pass; flake8 codegraph/ tests/ is clean.

Note on 1.3.0

#26 (Release/1.3.0) replaces the tokenizer parser with an AST-based one, which resolves 1 (and probably 2) for the default path — I verified the release/1.3.0 branch parses all 326 modules where 1.2.0 fails on 16. This PR targets the released 1.2.x line for users who cannot move yet; feel free to close it if 1.3.0 ships first. The legacy codegraph/parser.py on that branch still carries defects 1 and 2, so the same patch is worth applying there if that path stays reachable.

Known limitation left alone

Cross-file edges are derived from module.symbol usage plus basename matching, so a package import (from everos.component import embedding) collapses to a single __init__ node shared by every package. That is why distance-1 output contains a bare __init__.py. Fixing it properly needs real import resolution, which is a feature rather than a bug fix — out of scope here.

…ync with

create_objects_array()'s `async` branch assumed the token following `async`
was always `def`, so `async for` and `async with` were parsed as function
definitions. It also pushed `None` on the scope stack for a module-level
`async def` instead of the AsyncFunction it had just created, so the next
nested `def` / `async` token dereferenced None:

    AttributeError: 'NoneType' object has no attribute 'file'

Both are fixed: the token after `async` is now validated before being
treated as a function definition, and the module-level branch assigns
cur_func before pushing it on the stack - mirroring what the synchronous
`def` branch already does.

Adds regression tests; 5 of them fail with the AttributeError above before
the fix and all pass after.
Scopes popped by a sibling `def` / `class` (a one-line body such as
`def f(self) -> None: ...` never emits INDENT/DEDENT) or still open at EOF
never reach the DEDENT handler, so their `endno` stayed `None` and
`CodeGraph.usage_graph()` later raised:

    TypeError: '<=' not supported between instances of 'int' and 'NoneType'
      if entity[0] <= method_usage_line <= entity[1]:

Introduce `_close_scopes()` which finalizes `endno` whenever a scope is
popped, use it in the `async` / `def` / `class` branches, and call it once
more after the token loop to close everything left open at EOF.

On a real project this dropped entities with `endno is None` from 74 to 0.
`get_imports_and_entities_lines()` popped the `Import` object off the parsed
module, so calling `usage_graph()` twice left the second call with no
imports at all - and therefore no cross-module edges. Since the CLI builds
the graph (for the visualisation / CSV) before searching it, `--file-path`
/ `--distance` always reported an empty dependency set.

* keep the `Import` object on the list and skip it where entities are
  iterated, instead of popping it;
* `get_dependencies()` now resolves a relative `--file-path` (graph keys are
  absolute), and follows an edge past distance 1 when its bare
  "<module>.py" name maps to exactly one file - ambiguous names (there can
  be hundreds of `__init__.py`) are reported but not traversed, otherwise
  the BFS fans out to unrelated modules.

Before: `{1: set(), 2: set()}`
After:  `{1: {llm.py, types.py, events.py, ...}, 2: {engine.py, ...}}`
@nanami7777777 nanami7777777 changed the title fix: crash on module-level async def containing nested def / async for / async with fix: three parser/graph bugs - module-level async def, entities without an end line, lost import edges Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AttributeError: 'NoneType' object has no attribute 'file' — module-level async def with nested def / async for / async with

1 participant