tokenizer function for NLP
#104
Comments
phodal
added a commit
that referenced
this issue
Sep 26, 2024
phodal
added a commit
that referenced
this issue
Sep 30, 2024
…est #104 This commit introduces variable substitution in the tokenizer function. It also includes a new test case for the tokenizer in the regression test suite. The tokenizer function now accepts variables and replaces them with their corresponding values before tokenization.
phodal
added a commit
that referenced
this issue
Sep 30, 2024
The tokenizer function description and example in PatternActionFuncType.kt have been updated. The function now tokenizes text using traditional NLP methods and supports `word`, `naming`, and `stopwords` types. The example has been updated to reflect these changes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: