FAQ

Why can't it translate from English to Toki Pona?

The translator is rule-based: Meaning before translating, it recognizes the patterns and construction of the Toki Pona text through traditional programming instead of neural networks. This is a lot of effort but it's doable due to the relatively simple rules of Toki Pona.

If we instead start at English and translate it into Toki Pona, now we're dealing with the rules of English, which is very huge and inconsistent.

I personally think instead of rule-based approach, using neural networks would be preferable for English (or any language) to Toki Pona machine translation. That is, if there's enough data set available. Otherwise, the translation would be inaccurate. There isn't much data set available in Toki Pona because the language is still young. There are already existing machine learning translators nonetheless.

Why not use AI/neural networks/machine learning

AI is a confusing term, at least for me, so I use the terms neural networks or machine learning if possible instead.

When using machine learning, we need huge amount of data set, like a lot. Otherwise, the translation would be very inaccurate. There isn't much data set in Toki Pona because the language is still young. There are already existing machine learning translators nonetheless.

For the time being, making use of traditional programming to parse the patterns and constructions of the text is preferable.

Why not translate multiple sentences?

The translator is limited to translating at most 2 sentences. There are two reasons:

1. The translator considers many ways the original Toki Pona text could be interpreted and outputs multiple results. If it accepts multiple sentences, the output would grow exponentially as there are many sentences.
2. When the original Toki Pona text spans multiple sentences, it's important to take note of context between sentences, which the translator cannot do.

You can still give it complicated sentences like giving the word so many modifiers or give the sentence so many predicates. It may however give up if you give it complicated enough texts.

Why is it called "ilo Token"

"Token" is derived from the ISO codes of Toki Pona and English: "tok" and "en". "Token" also happens to be a jargon in parsing, which is fitting.