FAQ
jan Koko loved the idea of automated Toki Pona analysis or parsers that takes account of many possible grammatical interpretation of the input Toki Pona texts. That "tawa" could be a preposition or a modifier, such tool will output both. Then another idea clicked: a translator that also takes account of its many possible semantic interpretation. Now we're dealing with the meaning of the word itself, that "sike" could mean a circle or a cycle. jan Koko loved this idea that they have to build it.
The translator is rule-based: Meaning before translating, it recognizes the patterns and construction of the Toki Pona text through traditional programming instead of machine learning. This is a lot of effort but it's doable due to the relatively simple rules of Toki Pona.
If we instead start at English and translate it into Toki Pona, now we're dealing with the rules of English, which is very huge and inconsistent.
jan Koko personally think instead of rule-based approach, using machine learning would be preferable for English (or any language) to Toki Pona machine translation. That is, if there's enough data set available. Otherwise, the translation would be inaccurate. There isn't much data set available in Toki Pona because the language is still young. There are already existing machine learning translators nonetheless.
When using machine learning, we need huge amount of data set, like a lot. Otherwise, the translation would be very inaccurate. There isn't much data set in Toki Pona because the language is still young. There are already existing machine learning translators nonetheless.
For the time being, making use of traditional programming to parse the patterns and constructions of the text is preferable.
The translator is limited to translating at most 2 sentences. There are two reasons:
- The translator considers many ways the original Toki Pona text could be interpreted and outputs multiple results. If it accepts multiple sentences, the output would grow exponentially as there are many sentences.
- When the original Toki Pona text spans multiple sentences, it's important to take note of context between sentences, which the translator cannot do. This is the main reason we disallow it.
By allowing 2 sentences, we could translate sentences that uses "ni:", we could also translate sentences in form "X o, Y" which are technically 2 sentences according to ilo Token.
You can still give it complicated sentences like giving the word so many modifiers or give the sentence so many predicates. We found no reason to limit that. However, ilo Token may "give up" on complex sentences as it runs out of possible translation methods. Be careful not to crash the browser as well.
"Token" is derived from the ISO codes of Toki Pona and English: "tok" and "en". "Token" also happens to be a jargon in parsing, which is fitting.
It is a simplified representation of a flowchart. ilo Token's processes can be described with a bunch of flowcharts. This highlights ilo Token is rule-based as opposed to using machine learning.
Upon testing, it appears Chromium based browsers performs really well. Chromium based browsers includes Chrome, Edge, Brave, and Opera. Firefox appears to be a little slow for ilo Token. We haven't tested it on Safari.
Programming is always about weighing the pros and cons and potentially compromising. We've chosen good maintainability, speedy development, and better translation quality over optimization. Over time we'll develop ilo Token further and it may become faster.
It's theoretically possible to do this by having compound to word dictionary. However, we didn't do it for two reasons: First, we want to discourage lexicalization. Second, we don't want to compete with Sonja's Toki Pona Dictionary. If you want compound to word translations, buy the book.