feat(crawlers): implement `Request.skip_navigation` to skip navigation and run the handler directly by enriquealonso01 · Pull Request #2338 · apify/crawlee-python

enriquealonso01 · 2026-10-16T11:16:21Z

CrawleeRequestData.skip_navigation was declared, persisted and serialized as skipNavigation, but nothing in src/crawlee read it — setting it had no effect.

This implements the flag with the same semantics as JS SDK v4 (apify/crawlee#3107, apify/crawlee#3381), per janbuchar's direction in the issue:

When request.skip_navigation is True, BasicCrawler bypasses the context pipeline (whose middlewares perform the navigation) and invokes the request handler directly.
The navigation-only request.loaded_url member raises the new public NavigationSkippedError instead of returning None.
Since no navigation happens, response/parsed-content members never exist in the context — use BasicCrawler-level helpers (push_data, add_requests, send_request) instead.

Not breaking: the flag's storage format is unchanged, and accessing navigation-only members of a plain (navigated) context behaves exactly as before.

Testing

New end-to-end + serialization tests: 2/2 pass
tests/unit/crawlers: 590 passed, 1 skipped; the only 2 failures (parsel xhtml-as-xml, playwright firefox headless) fail identically on clean master (stash-verified, pre-existing)
tests/unit/test_request.py: 4/4 pass
ruff check + ruff format clean on all touched files

vdusek

This should target v2 branch.

…and run the handler directly Implements the previously unused `skip_navigation` flag (apify#2311): when set, BasicCrawler bypasses the context pipeline (whose middlewares perform the navigation) and invokes the request handler directly. Accessing the navigation-only `request.loaded_url` member raises the new public `NavigationSkippedError`, mirroring the JS SDK behavior (apify/crawlee - add public `NavigationSkippedError` to crawlee.errors - add `Request.skip_navigation` property backed by CrawleeRequestData (persisted/serialized via the existing `skipNavigation` alias) - BasicCrawler: route skip-navigation requests around the context pipeline - regression tests for end-to-end behavior and serialization roundtrip

enriquealonso01 · 2026-10-23T05:04:47Z

Retargeted to v2 as requested, rebased on the current v2 head (51efb37). Happy to adjust if v2 moves.

vdusek · 2026-10-26T15:01:17Z

+        await wait_for(
+            lambda: self.router(guarded_context),
+            timeout=self._request_handler_timeout,
+            timeout_message=f'{self._request_handler_timeout_text}'
+            f' {self._request_handler_timeout.total_seconds()} seconds',
+            logger=self._logger,
+        )


Important: The router runs outside ContextPipeline, so handler errors are not wrapped in RequestHandlerError. A raw exception reaches except Exception as internal_error in __run_task_function and terminates the whole crawl, with no retries and no failed-request handler. I reproduced it with a handler that raises ValueError.

Suggested change

await wait_for(

lambda: self.router(guarded_context),

timeout=self._request_handler_timeout,

timeout_message=f'{self._request_handler_timeout_text}'

f' {self._request_handler_timeout.total_seconds()} seconds',

logger=self._logger,

)

try:

await wait_for(

lambda: self.router(guarded_context),

timeout=self._request_handler_timeout,

timeout_message=f'{self._request_handler_timeout_text}'

f' {self._request_handler_timeout.total_seconds()} seconds',

logger=self._logger,

)

except SessionError:

raise

except Exception as e:

raise RequestHandlerError(e, context) from e

vdusek · 2026-10-26T15:01:18Z

            ),
        )

+    async def _run_request_handler_with_skipped_navigation(self, context: BasicCrawlingContext) -> None:


Important: Bypassing the pipeline means the handler gets a plain BasicCrawlingContext, even when it is typed as ParselCrawlingContext or PlaywrightCrawlingContext. context.page, context.selector or enqueue_links then fail with AttributeError. JS keeps the pipeline and guards only the navigation steps, so a browser crawler still opens the page and skips just the goto, and HTTP crawlers install getters for response, body, parsed content etc. that raise NavigationSkippedError. Could we follow that design and skip the navigation middleware per crawler?

vdusek · 2026-10-26T15:01:18Z



+@docs_group('Errors')
+class NavigationSkippedError(Exception):


Note: In JS, NavigationSkippedError extends NonRetryableError, so the request fails right away. Here it is a plain Exception, so reading loaded_url is retried max_request_retries times and fails the same way every time. Setting request.no_retry (or treating this error as non-retryable) would match JS.

vdusek · 2026-10-26T15:01:18Z

+    """Raised when a navigation-only crawling context member is accessed, but `skip_navigation` was used.
+
+    When a request has `skip_navigation` set to `True`, the crawler does not open the page and the crawling
+    context does not contain the results of the navigation (e.g. `http_response`, `$`, parsed content or
+    `request.loaded_url`). Accessing such properties raises this error at runtime.


Note: Only request.loaded_url raises this error at the moment. http_response, parsed content and the page are simply missing from the context (AttributeError), and $ is a JS-only name. The same goes for the Request.skip_navigation docstring. Either describe the current behavior or implement the placeholders.

vdusek · 2026-10-26T15:01:18Z

+    def __getattr__(self, name: str) -> Any:
+        # Attribute writes (e.g. `request.state = ...` in the router) must land on the wrapped request,
+        # so intercept only reads that the wrapped request cannot resolve itself.
+        if name == 'loaded_url':
+            return self._raise_on_loaded_url()
+        return getattr(self._request, name)


Note: copy.copy() or pickle of context.request recurses forever here: while the copy is being rebuilt, _request is unset, so __setattr__ reads it, __getattr__ runs, reads self._request again, and so on until RecursionError. Raising AttributeError for _request in __getattr__ fixes it. repr(context.request) also shows the proxy object, so a __repr__ forwarding to the request would help in logs. The comment is also inaccurate: __getattr__ runs when the proxy cannot resolve the name, not the wrapped request.

vdusek · 2026-10-26T15:01:18Z

+class _NavigationSkippedRequestProxy:
+    """Wraps a `Request` whose navigation was skipped due to `skip_navigation`.
+
+    All attribute access is forwarded to the wrapped request, except `loaded_url`, which raises a
+    `NavigationSkippedError` — mirroring the JS SDK behavior for skipped navigations.
+    """
+
+    __slots__ = ('_raise_on_loaded_url', '_request')


Nit: raise_on_loaded_url only ever has one value, so __getattr__ could raise NavigationSkippedError directly and the extra slot and callable can go. The docstring also has a non-ASCII em dash and a reference to the JS SDK, which the repo avoids in source.

vdusek · 2026-10-26T15:01:18Z

+        )
+        guarded_context = cast(
+            'TCrawlingContext',
+            dataclass_replace(context, request=guarded_request),  # ty: ignore[invalid-argument-type]


Nit: With the locked ty version this ignore is reported as unused, so the type-check job fails.

Suggested change

dataclass_replace(context, request=guarded_request), # ty: ignore[invalid-argument-type]

dataclass_replace(context, request=guarded_request),

vdusek · 2026-10-26T15:01:18Z

        await cast('Any', crawler)._BasicCrawler__is_finished_function()
+
+
+async def test_skip_navigation_runs_handler_without_navigation() -> None:


Note: BasicCrawler never navigates, so these tests cannot show that navigation is actually skipped. Could you add a test with HttpCrawler (assert http_client.crawl is not called), one for a handler that raises (retried, then the failed-request handler gets the original error), and one for AdaptivePlaywrightCrawler?

vdusek · 2026-10-26T15:01:18Z

+    processed: list[str] = []
+    navigation_members: dict[str, str] = {}


Nit: The contributing guide prefers Mock over hand-rolled lists and flag dicts, e.g. processed = Mock() and processed.call_args_list == [call('plain'), call('skipped')].

github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 16, 2026

janbuchar requested a review from vdusek October 19, 2026 09:38

vdusek requested changes Oct 19, 2026

View reviewed changes

enriquealonso01 changed the base branch from master to v2 October 23, 2026 05:01

enriquealonso01 force-pushed the fix/skip-navigation-2311 branch from cbc5dc3 to 05543bf Compare October 23, 2026 05:04

vdusek requested a review from Pijukatel October 26, 2026 15:43

vdusek reviewed Oct 26, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(crawlers): implement `Request.skip_navigation` to skip navigation and run the handler directly - #2338

feat(crawlers): implement `Request.skip_navigation` to skip navigation and run the handler directly#2338
enriquealonso01 wants to merge 1 commit into
apify:v2from
enriquealonso01:fix/skip-navigation-2311

enriquealonso01 commented Oct 16, 2026

Uh oh!

vdusek left a comment

Uh oh!

enriquealonso01 commented Oct 23, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

vdusek Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		@docs_group('Errors')
		class NavigationSkippedError(Exception):

	dataclass_replace(context, request=guarded_request), # ty: ignore[invalid-argument-type]
	dataclass_replace(context, request=guarded_request),

		await cast('Any', crawler)._BasicCrawler__is_finished_function()


		async def test_skip_navigation_runs_handler_without_navigation() -> None:

		processed: list[str] = []
		navigation_members: dict[str, str] = {}

Conversation

enriquealonso01 commented Oct 16, 2026

Testing

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

enriquealonso01 commented Oct 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants