Skip to content

NUTCH-3110 Upgrade to Tika 3.1.0 #850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

sebastian-nagel
Copy link
Contributor

@sebastian-nagel sebastian-nagel commented Mar 28, 2025

  • Upgrade to shaded Tika packages 3.1.0.0 provided by Tim Allison.
    The shaded packages are required to avoid version conflicts when running in distributed mode caused by incompatible versions of the commons-io jar shipped with Hadoop and required by Tika, cf. NUTCH-2959.
  • Add "text/javascript" as MIME type supported by "parse-js". Note: This fixes the parse-js unit tests. Tika 3.1.0 identifies the Javascript test document as "text/javascript" instead of "application/javascript".

Todo:

  • fix unit test o.a.n.parse.tika.TestDOMContentUtils : duplicated outlinks
  • fix unit test o.a.n.parse.tika.TestHtmlParser : parsing a UTF-16 encoded HTML fails partially (no title, no keywords). Note: it might be that there are two BOMs in the test document.
  • regression: ensure that NUTCH-3108 is solved (the initial solution would introduce it again)

Upgrade to shaded Tika packages 3.1.0.0 provided by Tim Allison.
The shaded packages are required to avoid version conflicts when
running in distributed mode caused by incompatible versions of
the commons-io jar shipped with Hadoop and required by Tika,
cf. NUTCH-2959.
Add "text/javascript" as MIME type supported by "parse-js".
Note: fixes parse-js unit tests. Tika 3.1.0 identifies
the Javascript test document as "text/javascript" instead of
"application/javascript".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant