Repository navigation
fix: three parser/graph bugs - module-level async def, entities without an end line, lost import edges - #30
Open
nanami7777777 wants to merge 3 commits into
Open
nanami7777777 wants to merge 3 commits into
nanami7777777 wants to merge 3 commits into
Conversation
…ync with
create_objects_array()'s `async` branch assumed the token following `async`
was always `def`, so `async for` and `async with` were parsed as function
definitions. It also pushed `None` on the scope stack for a module-level
`async def` instead of the AsyncFunction it had just created, so the next
nested `def` / `async` token dereferenced None:
AttributeError: 'NoneType' object has no attribute 'file'
Both are fixed: the token after `async` is now validated before being
treated as a function definition, and the module-level branch assigns
cur_func before pushing it on the stack - mirroring what the synchronous
`def` branch already does.
Adds regression tests; 5 of them fail with the AttributeError above before
the fix and all pass after.
Scopes popped by a sibling `def` / `class` (a one-line body such as
`def f(self) -> None: ...` never emits INDENT/DEDENT) or still open at EOF
never reach the DEDENT handler, so their `endno` stayed `None` and
`CodeGraph.usage_graph()` later raised:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
if entity[0] <= method_usage_line <= entity[1]:
Introduce `_close_scopes()` which finalizes `endno` whenever a scope is
popped, use it in the `async` / `def` / `class` branches, and call it once
more after the token loop to close everything left open at EOF.
On a real project this dropped entities with `endno is None` from 74 to 0.
`get_imports_and_entities_lines()` popped the `Import` object off the parsed
module, so calling `usage_graph()` twice left the second call with no
imports at all - and therefore no cross-module edges. Since the CLI builds
the graph (for the visualisation / CSV) before searching it, `--file-path`
/ `--distance` always reported an empty dependency set.
* keep the `Import` object on the list and skip it where entities are
iterated, instead of popping it;
* `get_dependencies()` now resolves a relative `--file-path` (graph keys are
absolute), and follows an edge past distance 1 when its bare
"<module>.py" name maps to exactly one file - ambiguous names (there can
be hundreds of `__init__.py`) are reported but not traversed, otherwise
the BFS fans out to unrelated modules.
Before: `{1: set(), 2: set()}`
After: `{1: {llm.py, types.py, events.py, ...}, 2: {engine.py, ...}}`
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #31
Three crashes / silent-wrong-result bugs found while running codegraph on a real-world project (326 modules). Two of them are hard crashes, one makes
--file-path/--distancealways report nothing.1.
AttributeError: 'NoneType' object has no attribute 'file'— module-levelasync defcreate_objects_array()'sasyncbranch assumed the token followingasyncwas alwaysdef, soasync for/async withwere parsed as function definitions (inasync for x in ...the "name" becomesx). It also pushedNoneon the scope stack for a module-levelasync definstead of theAsyncFunctionit had just created, so the next nesteddef/asynctoken called_nest_function(None, ...)→ob.file→AttributeError.Repro, three lines:
Fix: validate the token after
async(skip if it is notdef), and assigncur_funcbefore pushing it on the stack, mirroring the synchronousdefbranch.The whole tree is parsed in one pass, so one such file made
codegraph <dir>fail for the entire directory — 16 of 326 modules here.2.
TypeError: '<=' not supported between instances of 'int' and 'NoneType'— entities without an end lineScopes popped by a sibling
def/class(a one-line body such asdef f(self) -> None: ...never emits INDENT/DEDENT) or still open at EOF never reach the DEDENT handler, so theirendnostayedNone.usage_graph()then comparedentity[0] <= line <= entity[1]withentity[1] is None.Fix: new
_close_scopes()finalizesendnowhenever a scope is popped; used in theasync/def/classbranches and once more after the token loop for anything left open at EOF.On the project this took entities with
endno is Nonefrom 74 → 0.3.
usage_graph()silently loses all import edges on a second callget_imports_and_entities_lines()didcode_objects[path].pop(-1)on theImportobject, so a secondusage_graph()call ran with an empty import set and therefore no cross-module edges. Since the CLI builds the graph (for the visualisation / CSV) before searching it,get_dependencies()always saw a degraded graph and--file-path/--distanceprinted nothing:Fix:
Importobject on the list and skip it where entities are iterated, instead of popping it;get_dependencies()resolves a relative--file-path(graph keys are absolute, socodegraph src --file-path src/x.pynever matched);"<module>.py"name maps to exactly one file. Ambiguous names are still reported but not traversed — there can be hundreds of__init__.py, and traversing them fans the BFS out to unrelated modules (distance 2 went from 8 meaningful files to ~120 without this guard).Tests
tests/test_graph_generation.py:TestAsyncParsing— 6 cases (nested sync/async def,async forcomprehension,async forstatement,async with, module-level async def followed by a top-level def). 5 fail with theAttributeErrorbefore the fix.TestEntityEndLines— 4 cases (one-line body, last entity in module, scope closed by a sibling def, module-level async function).TestUsageGraphIdempotency— 2 cases (secondusage_graph()equals the first;get_dependencies()still finds edges afterusage_graph()was called).46 tests pass;
flake8 codegraph/ tests/is clean.Note on 1.3.0
#26 (
Release/1.3.0) replaces the tokenizer parser with an AST-based one, which resolves 1 (and probably 2) for the default path — I verified therelease/1.3.0branch parses all 326 modules where 1.2.0 fails on 16. This PR targets the released 1.2.x line for users who cannot move yet; feel free to close it if 1.3.0 ships first. The legacycodegraph/parser.pyon that branch still carries defects 1 and 2, so the same patch is worth applying there if that path stays reachable.Known limitation left alone
Cross-file edges are derived from
module.symbolusage plus basename matching, so a package import (from everos.component import embedding) collapses to a single__init__node shared by every package. That is why distance-1 output contains a bare__init__.py. Fixing it properly needs real import resolution, which is a feature rather than a bug fix — out of scope here.