feat: Add AdaptivePlaywrightCrawler #872
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add method to BasicCrawler to handle just one request.
Add statistics TODO: Make mypy happy about statistics. Wrap existing statistics from init in adaptive statistics. Silent subcrawler statistics and loggers in general. (Set level to error?)
Ignore and create TODO follow up issue for refactoring Statistics class after technical discussion.
Handle use state.
Pre-navigation hooks delegation to sub crawler hooks.
Statistics were marked as generics, but in reality were not. Hardcoding state_model to make it explicit and clear.
WIP KVS handling. Currently it does not go through Result handler.
This reverts commit d345259.
github-actions
bot
added
t-tooling
Fix wrong id for predictor_state persistence.
Add test for pre nav hook Add test for statistics in crawler init
…nternals of sub crawlers) Cleanup commit results.
Pijukatel
force-pushed
the
adaptive-PwCrawler
branch
from
340c53d to
f12f605
Compare
janbuchar
reviewed
Jan 7, 2025
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
Add it in adaptive crawler instead at the cost of accessing many private members.
janbuchar
reviewed
Jan 22, 2025
tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
…ght_crawler.py Co-authored-by: Jan Buchar <[email protected]>
janbuchar
reviewed
Jan 22, 2025
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawling_context.py
Outdated
Show resolved
Hide resolved
tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py
Outdated
Show resolved
Hide resolved
|
I have to integrate properly |
Pijukatel
force-pushed
the
adaptive-PwCrawler
branch
from
baa2052 to
8bce425
Compare
janbuchar
reviewed
Jan 29, 2025
janbuchar
reviewed
Jan 30, 2025
janbuchar
reviewed
Jan 30, 2025
Co-authored-by: Jan Buchar <[email protected]>
janbuchar
approved these changes
Jan 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add AdaptivePlaywrightCrawler, based on JS implementation
Issues
#249
Purpose:
Adaptive crawler can choose to crawl page with either static crawler(like BeautifulSoupCrawler or ParselCrawler) or browser-based PlaywrightCrawler.
This is done to theoretically benefit from both approaches - use faster static crawler where it is possible and fallback to slower browser-based crawler where needed.
Implementation
Python implementation differs from JS implementation mainly in the way adaptive crawler performs the browser crawling or the static crawling.
JS implementation explicitly defines the code that is used for static/browser crawling - this makes it more simple and easier to understand, but it is some code duplication(code that already exists in other crawlers) and also it does drop some features that exist on those crawlers, like pre-navigation hooks.
Python implementation does not define explicitly code for static/browser crawling. It instead delegates this task to "sub crawlers" (instance of implementation of AbstractHttpCrawler for static crawling and instance of PlaywrightCrawler.)
This avoids code duplication and supports more features and control over the static/browser crawling, but it adds complexity of connecting all the components together.
"Sub crawlers" are directly used only in
__init__method of top crawler to create their context pipelines. These are then used by top crawler to perform single request crawls when needed. This approach re-uses existing code from other crawlers and fully preserve their context pipeline handling.Crawling contexts
Since
AdaptivePlaywrightCrawlercan crawl page in two different manners the contexts can also look differently. I selected approach where there is unified context for both crawl types(static/browser) and this unified context contains all the attributes that are found both on static context and browser context, but browser based attributes will raiseAdaptiveContextError(RuntimeError)when accessed during static crawling. It is impossible to know in advance which mode of crawling will be used, so this forces users to usetry-catchpattern when working with these unified contexts.For user defined pre-navigation hooks there is
AdaptivePlaywrightPreNavCrawlingContext(very nice name!) that haspageproperty that can raiseAdaptiveContextError.For user defined request handler there is
AdaptivePlaywrightCrawlingContextthat haspage,infinite_scrollandresponseproperties (all copied from PlaywrightCrawlingContext) that can raiseAdaptiveContextError.Missing in this PR
Documentation
Documentation, examples and template will be added as last step in follow up PRs. The reason is that this feature should be documented only when it is in state that is ready to be used by users. Updating documentation too early would lead to users to try to use incomplete feature.
Rendering type predictor
JS implementation uses statistical regression based predictor to decide what crawl type should be used. This predictor is not implemented in this PR and is replaced by dummy version -
RandomRenderingTypePredictor.Proper predictor will be implemented in follow up PR.
Helper methods on adaptive context
JS implementation adds several helper methods on adaptive context. Since adaptive crawler in JS is not generic with respect to static crawling, it is straight forward to implement it with the only option Cheerio.
If same was to be done in Python it would require more handling of generics as it would basically require to wrap some sort of generic "select" method from whatever generic parser is used and it's return value would have to be another generic parameter.
This change is standalone and it is better to do it separately.
Parity changes to JS code
Some improvements could be ported back to JS implementation. These are improvements and issues are summarized in apify/crawlee#2798