Code

News and wat wet compatibilities#32

Code

Open

jt55401 wants to merge 10 commits intomaincommoncrawl/cc-index-table:mainfrom news-and-wat-wet-compatibilitiescommoncrawl/cc-index-table:news-and-wat-wet-compatibilitiesCopy head branch name to clipboard

+216-14

Commits

Commits on May 31, 2024

• Added devcontainer for Java 11

  jt55401

  committed
  1e07009

  Copy full SHA for 1e07009

• Seperated logic for filename parsing, and added a test for it.

  jt55401

  committed
  dc08484

  Copy full SHA for dc08484

• Modified CCIndex2Table to use CCIndex2FilenameParser

  jt55401

  committed
  99acc5a

  Copy full SHA for 99acc5a

• Added Dockerfile to build and run convert_url_index.sh

  jt55401

  committed
  ab34b57

  Copy full SHA for ab34b57

• Modified convert_url_index.sh to prefer user classes over those included in spark

  jt55401

  committed
  fc800a6

  Copy full SHA for fc800a6

Commits on Aug 27, 2024

• Merge remote-tracking branch 'origin/main' into news-and-wat-wet-compatibilities

  jt55401

  committed
  e737fba

  Copy full SHA for e737fba

Commits on Sep 10, 2024

• fix issue with build not working due to upgraded crawl-common which requires java 11 now.

  jt55401

  committed
  96a0ca6

  Copy full SHA for 96a0ca6

• upgrade spark to 3.5.1 to match the rest of the project

  jt55401

  committed
  d6ed799

  Copy full SHA for d6ed799

• add error handling & logging for issues with hostname parsing of uri field, and default crawl/segment/subset to "unknown" if they can't be found

  jt55401

  committed
  cdf11e2

  Copy full SHA for cdf11e2

• filter out null lines/entries

  jt55401

  committed
  100e186

  Copy full SHA for 100e186