Code

feat: Add AdaptivePlaywrightCrawler#872

Code

Open

Pijukatel wants to merge 68 commits intomasterapify/crawlee-python:masterfrom adaptive-PwCrawlerapify/crawlee-python:adaptive-PwCrawlerCopy head branch name to clipboard

+1,571-63

Commits

Commits on Dec 24, 2024

• WIP

  Pijukatel

  committed
  d8b2438

  Copy full SHA for d8b2438

Commits on Dec 31, 2024

• More feasible version of composition.

  Show description for c12acda
  Pijukatel

  committed
  c12acda

  Copy full SHA for c12acda

• Pass properly all kwargs to subcrawlers

  Pijukatel

  committed
  623d341

  Copy full SHA for 623d341

Commits on Jan 1, 2025

• Add run decision logic from JS version

  Show description for 548349e
  Pijukatel

  committed
  548349e

  Copy full SHA for 548349e

• Statistics class change lead too many ripple changes.

  Show description for 29d510a
  Pijukatel

  committed
  29d510a

  Copy full SHA for 29d510a

• Handle sub crawlers loggers

  Show description for 04eefd9
  Pijukatel

  committed
  04eefd9

  Copy full SHA for 04eefd9

Commits on Jan 2, 2025

• Statistics change to be usable without ignores.

  Show description for 33efed3
  Pijukatel

  committed
  33efed3

  Copy full SHA for 33efed3

• Align use_state with JS implementation.

  Pijukatel

  committed
  202aceb

  Copy full SHA for 202aceb

• Remove "fake generics" from Statistics.

  Show description for d474a94
  Pijukatel

  committed
  d474a94

  Copy full SHA for d474a94

• Align result comparator witrh JS implementation.

  Pijukatel

  committed
  e95ff19

  Copy full SHA for e95ff19

• Add doc strings.

  Show description for 2408d85
  Pijukatel

  committed
  2408d85

  Copy full SHA for 2408d85

Commits on Jan 3, 2025

• use_state through RequestHandlerRunResult

  Pijukatel

  committed
  d345259

  Copy full SHA for d345259

Commits on Jan 6, 2025

• Revert "use_state through RequestHandlerRunResult"

  Show description for bfa9290
  Pijukatel

  committed
  bfa9290

  Copy full SHA for bfa9290

• Add basic delegation test.

  Pijukatel

  committed
  63f278a

  Copy full SHA for 63f278a

• Add context test.

  Pijukatel

  committed
  e190788

  Copy full SHA for e190788

• Add tests for use_state and predictor.

  Pijukatel

  committed
  0ecb137

  Copy full SHA for 0ecb137

• Remove unintended edit.

  Pijukatel

  committed
  b73c702

  Copy full SHA for b73c702

Commits on Jan 7, 2025

• Add tests for statistics.

  Show description for 5c79d0d
  Pijukatel

  committed
  5c79d0d

  Copy full SHA for 5c79d0d

• Add test for error handling adn commiting correct results.

  Show description for 957915a
  Pijukatel

  committed
  957915a

  Copy full SHA for 957915a

• Add crawl_one_required_contexts property. (Alternative to accessing internals of sub crawlers)

  Show description for f12f605
  Pijukatel

  committed
  f12f605

  Copy full SHA for f12f605

• Lint

  Pijukatel

  committed
  5256af2

  Copy full SHA for 5256af2

Commits on Jan 8, 2025

• Remove BasicCrawler modifications.

  Show description for 2fd7aae
  Pijukatel

  committed
  2fd7aae

  Copy full SHA for 2fd7aae

• Make _commit_result consistent with how other result components are handled.

  Pijukatel

  committed
  714b5bd

  Copy full SHA for 714b5bd

• Remove subcrawlers and add _OrphanPipeline

  Pijukatel

  committed
  b38dda1

  Copy full SHA for b38dda1

• Use dummy statistics in subcrawlers.

  Show description for ffb2a78
  Pijukatel

  committed
  ffb2a78

  Copy full SHA for ffb2a78

• Keep predictor related functions on predictor_state

  Pijukatel

  committed
  3b05228

  Copy full SHA for 3b05228

• Unify pre-nav hooks.

  Show description for dc06490
  Pijukatel

  committed
  dc06490

  Copy full SHA for dc06490

Commits on Jan 9, 2025

• Simplify pre-nav hook common context.

  Pijukatel

  committed
  0766b7a

  Copy full SHA for 0766b7a

• Make static crawling part of AdaptiveCrawler generic.

  Show description for 4a63c2c
  Pijukatel

  committed
  4a63c2c

  Copy full SHA for 4a63c2c

• Update tests to remove bs references.

  Show description for bd72a84
  Pijukatel

  committed
  bd72a84

  Copy full SHA for bd72a84

Commits on Jan 10, 2025

• Revert accidental Lint edits to website/*.py

  Pijukatel

  committed
  c964d44

  Copy full SHA for c964d44

Commits on Jan 14, 2025

• Review comments.

  Pijukatel

  committed
  a471395

  Copy full SHA for a471395

Commits on Jan 15, 2025

• Sub crawler timeout handling + test

  Pijukatel

  committed
  dbf6310

  Copy full SHA for dbf6310

• Simplify prenav hooks

  janbuchar

  committed
  8ee8f99

  Copy full SHA for 8ee8f99

• Simplify context manager handling

  janbuchar

  committed
  c291d3e

  Copy full SHA for c291d3e

• Review comments - _run_request_handler + timeouts

  Pijukatel

  committed
  2c23238

  Copy full SHA for 2c23238

• Statistics.

  Show description for b2a29c1
  Pijukatel

  committed
  b2a29c1

  Copy full SHA for b2a29c1

• Make statistics generic again!

  Show description for 0786f87
  Pijukatel

  committed
  0786f87

  Copy full SHA for 0786f87

• Merge remote-tracking branch 'origin/master' into adaptive-PwCrawler

  Pijukatel

  committed
  4f7d7f4

  Copy full SHA for 4f7d7f4

Commits on Jan 16, 2025

• Mock requests in tests.

  Pijukatel

  committed
  1234ea7

  Copy full SHA for 1234ea7

Commits on Jan 17, 2025

• Improve error readability.

  Show description for e85ec9f
  Pijukatel

  committed
  e85ec9f

  Copy full SHA for e85ec9f

• Mock both static and browser requests in tests.

  Pijukatel

  committed
  6e8635a

  Copy full SHA for 6e8635a

• Create proper example code

  Show description for 0e9146a
  Pijukatel

  committed
  0e9146a

  Copy full SHA for 0e9146a

• Merge remote-tracking branch 'origin/master' into adaptive-PwCrawler

  Pijukatel

  committed
  a4eac8f

  Copy full SHA for a4eac8f

Commits on Jan 20, 2025

• Relax timeout in test to avoid flakiness in CI

  Pijukatel

  committed
  44ad898

  Copy full SHA for 44ad898

Commits on Jan 21, 2025

• Remove AdaptivePlaywrightCrawlerStatistics

  Show description for f219453
  Pijukatel

  committed
  f219453

  Copy full SHA for f219453

• WIP

  Pijukatel

  committed
  64d9e54

  Copy full SHA for 64d9e54

• Update options typed dicts

  Show description for 08fc81f
  Pijukatel

  committed
  08fc81f

  Copy full SHA for 08fc81f

• Add docstrings to adaptive context public stuff

  Show description for 9bde9dc
  Pijukatel

  committed
  9bde9dc

  Copy full SHA for 9bde9dc

• Make crawl_one_with private.

  Show description for 9a14569
  Pijukatel

  committed
  9a14569

  Copy full SHA for 9a14569

Commits on Jan 22, 2025

• Update tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py

  Show description for fd8dd82
  

  Pijukatel

  and

  janbuchar

  authored
  fd8dd82

  Copy full SHA for fd8dd82

• Review comments

  Pijukatel

  committed
  4221219

  Copy full SHA for 4221219

• Remove _run_subcrawler_pipeline

  Pijukatel

  committed
  565d36b

  Copy full SHA for 565d36b

• Remove Orphans

  Pijukatel

  committed
  4bd8251

  Copy full SHA for 4bd8251

• Merge remote-tracking branch 'origin/master' into adaptive-PwCrawler

  Pijukatel

  committed
  a3ab2e8

  Copy full SHA for a3ab2e8

• Move SubCrawlerRun to where it is used

  Pijukatel

  committed
  fc95132

  Copy full SHA for fc95132

• Use custom _TestInput dataclass

  Pijukatel

  committed
  4f316b5

  Copy full SHA for 4f316b5

Commits on Jan 23, 2025

• Review comments

  Pijukatel

  committed
  949c4ff

  Copy full SHA for 949c4ff

Commits on Jan 27, 2025

• Add optional argument to pre navigation hook decorator

  Pijukatel

  committed
  56ad33a

  Copy full SHA for 56ad33a

• Remove _push_result_to_context and add result argument/return to _run_request_handler.

  Pijukatel

  committed
  781d5ff

  Copy full SHA for 781d5ff

• Merge remote-tracking branch 'origin/master' into adaptive-PwCrawler

  Pijukatel

  committed
  a93d6a1

  Copy full SHA for a93d6a1

Commits on Jan 28, 2025

• Add block_request to adaptive pre nav context

  Pijukatel

  committed
  b4ba31b

  Copy full SHA for b4ba31b

Commits on Jan 29, 2025

• Use context result map for handling request handler results

  Pijukatel

  committed
  8bce425

  Copy full SHA for 8bce425

Commits on Jan 30, 2025

• Review comments based comments

  Pijukatel

  committed
  5f8c26c

  Copy full SHA for 5f8c26c

• Merge remote-tracking branch 'origin/master' into adaptive-PwCrawler

  Pijukatel

  committed
  e1d0c7e

  Copy full SHA for e1d0c7e

• Integrate RenderingTypePredictor

  Pijukatel

  committed
  aacb90a

  Copy full SHA for aacb90a

• Update src/crawlee/crawlers/_basic/_basic_crawler.py

  Show description for ab1c40e
  

  Pijukatel

  and

  janbuchar

  authored
  ab1c40e

  Copy full SHA for ab1c40e

• Finalize exports and re exports

  Pijukatel

  committed
  2bf43f6

  Copy full SHA for 2bf43f6