Skip to content

add possibility to add extra user-defined rules on startup #47

@kosloot

Description

@kosloot

recently an --add-tokens option was introduced to Ucto to add extra 'TOKENS' to the configuration.
We might consider extending this, so a user could add extra, non-default rules/items to the tokenizer.

Some caveats to consider:

  • are the extra rules additional, of do they override?
  • make it possible to disable a certain rule
  • are the additions language specific? How to express that

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions