Releases: Protostarship/Dyk-Knyh-engine
Release list
✨ ILTE v3-Alpha.3 | ILTE-ATI Release ✨
🚀 Indigenous Language Translator Engine (ILTE) v3.0.0-Alpha.3 Release 🎉
🌟 Introducing ILTE-ATI v3.0.0-Alpha.3 – A Next-Gen Iterative Translation Engine
The Indigenous Language Translator Engine (ILTE) v3.0.0-Alpha.3 introduces ILTE-ATI (Iterative Translation Engine), the most advanced version yet, with attention-based iterative translation, multi-stage candidate refinement, and dynamic normalization.
This release includes:
- 🌱 ILTE-ALT v2.1.0-Beta.2 – Lightweight dictionary-based translation engine.
- 🧐 ILTE-ADV v2.1.1-Alpha.2 – AI-driven, context-aware translation engine.
- 🧠 ILTE-ZS v2.1.2-Beta.3 – Hybrid, resource-efficient translation engine with RBMT, FST, semantic matching, and zero-shot capabilities.
- ✨ ILTE-ATI v3.0.0-Alpha.3 (NEW!) – The most advanced iterative and attention-based translation engine with multi-pass processing and enhanced contextual awareness.
🚨 This is an Alpha release. Further refinements and optimizations will follow. 🚨
🔄 What's New in v3.0.0?
✨ ILTE-ATI v3.0.0-Alpha.3 (Iterative Translation Engine with Attention Mechanism)
- 🏆 Attention-Based Context Window
- Wider sentence analysis ensures better context comprehension.
- Dynamically adapts to sentence structure for more accurate translations.
- 🌀 Iterative Multi-Pass Translation for Refinement
- Each translation cycle improves on the last for enhanced accuracy.
- Prevents misinterpretations by checking and refining in multiple iterations.
- 🔍 Enhanced Word Candidate Selection
- Uses similarity-based ranking to determine best word choices.
- Dynamically selects alternative phrases when necessary.
- ⚡ Optimized Performance & Resource Handling
- Improved multi-threading & multi-processing for faster execution.
- Better cache management to avoid redundant processing.
- 📊 Detailed Report Generation
- DOCX report now includes:
- Confidence scores, refinement passes, and method breakdowns.
- Preserved document formatting (paragraphs, indentations, and spacing).
- DOCX report now includes:
📊 ILTE-ATI vs. ILTE-ZS vs. ILTE-ALT vs. ILTE-ADV
| Feature | ILTE-ALT 🌱 | ILTE-ZS 🧠 | ILTE-ADV 🧐 | ILTE-ATI ✨ |
|---|---|---|---|---|
| Translation Approach | Dictionary-Based | Dictionary + RBMT + FST + Semantic + Zero-Shot | Dictionary + Semantic + Context-Based AI | Attention-Based + Iterative Multi-Pass |
| Processing Speed | Fast | Optimized | Slower | Balanced, Optimized for Accuracy |
| Handling Large Files | Struggles | Efficient Chunking | Slower | Optimized Chunking & Iterative Processing |
| Memory Usage | Low | Moderate | High | Balanced, Adaptive Resource Usage |
| Context Awareness | None | Partial | Strong | 🏆 Very Strong (Multi-Pass Context Analysis) |
| Idiomatic Expressions | Limited | Rule-Based (RBMT) | AI-Based | AI-Based + Refinement Mechanism |
| Parallelization | Minimal | Thread + Process Pool | DataLoader-Based | Multi-Process + Thread Optimization |
| Zero-Shot Capability | No | Yes | Yes | Yes (Adaptive Candidate Selection) |
| Best Use Case | Fast, simple translation | Large text processing | High-accuracy, AI-based translation | High-Precision, Context-Aware, Adaptive Translation |
🎯 ILTE-ATI v3.0.0 Translation Flow Breakdown
1️⃣ Advanced Preprocessing
- Text normalization using hierarchical rules (Unicode cleanup, stemming, and regex-based tokenization).
- Sentence restructuring to improve translation context.
- Broad context windowing for improved sentence understanding.
2️⃣ Multi-Stage Translation Pipeline
- Dictionary Lookup → RBMT & FST Transformation → Contextual Semantic Matching → Adaptive Candidate Selection.
- Iterative refinement cycles for self-correcting translation passes.
- Parallel execution for optimal speed and efficiency.
3️⃣ Confidence Scoring & Adaptive Candidate Refinement
- Assigns dynamic confidence scores based on:
- Translation method (Dictionary, Semantic, RBMT, FST, Attention-Based, etc.).
- Word position within a sentence and its surrounding context.
- Iterates until an optimal translation is reached.
4️⃣ Final Output & Report Generation
- Outputs structured translation results with full formatting preservation.
- DOCX report includes:
- Translation accuracy scores, processing time, and method logs.
- Iterative refinement breakdown to track improvement over cycles.
🌐 Installation & Usage
Installing Dependencies
pip install -r requirements.txtRunning ILTE-ATI v3.0.0 (Iterative Attention Engine)
python engine_ATI.pySelecting Input Type
👉 Manual Input: Enter text directly.
👉 File Upload: Provide a document for batch processing.
🛠️ System Requirements
ILTE-ATI v3.0.0 (Iterative Attention Engine)
- CPU: Intel Core i7 / AMD Ryzen 7 or higher.
- RAM: 16GB+ (32GB Recommended for Large Translations).
- Storage: 2GB+ free space.
- Python: 3.10+.
- GPU Acceleration Highly Recommended (Supports CUDA for PyTorch models).
🌟 Future Plans & Improvements
- ✅ Further Optimizing Attention Mechanisms for better handling of complex sentences.
- ✅ Fine-Tuning Multi-Pass Translation Logic for iterative refinements.
- ✅ Expanding contextual understanding through ML-driven linguistic models.
- ✅ Lowering computation time without reducing accuracy.
📚 License & Contribution
- Licensed under GPL v3 – Open Source & Community Driven! 🌍
- Contribute & Improve – Join us in preserving indigenous languages! 🌱
🏆 Developed by XI TJKT 2 Development Team 🧑💻
🎉 v2.1.2 BETA-3 Release - ILTE-ZS 🎉
🚀 Indigenous Language Translator Engine (ILTE) v2.1.2 Release 🎉
🌟 Introducing ILTE-ZS - A Hybrid, Efficient, and Scalable Translation Engine
The Indigenous Language Translator Engine (ILTE) v2.1.2 marks the release of ILTE-ZS, a new version designed for high-performance translation with batch multi-processing and hybrid methodologies.
This release now includes:
- 🌱 ILTE-ALT v2.1.0-Beta.2 – Lightweight dictionary-based translation engine.
- 🧐 ILTE-ADV v2.1.1-Alpha.2 – AI-driven, context-aware translation engine.
- 🧠 ILTE-ZS v2.1.2-Beta.3 – Hybrid, resource-efficient translation engine with RBMT, FST, semantic matching, and zero-shot capabilities.
🚨 This is a BETA release. Further optimizations and refinements will follow. 🚨
🔄 What's New in v2.1.2?
🧠 ILTE-ZS v2.1.2-Beta.3 (Hybrid & High-Performance Processing)
- ⚡ Optimized for Large-Scale Text Handling using chunking & batch multi-processing.
- 🛠️ Hybrid Translation Pipeline combining:
- Dictionary-based lookup
- Rule-Based Machine Translation (RBMT) & Finite State Transducer (FST)
- Semantic similarity matching (SentenceTransformers)
- Zero-Shot Translation with MBart-50
- 🔋 Improved Resource Management:
- Efficient caching with expiration control.
- ThreadPoolExecutor + ProcessPoolExecutor for parallelization.
- GPU memory cleanup after processing.
- 🌍 Expanded File Support:
- Supports
.txt,.docx, with auto-encoding detection & Unicode normalization. - Enhanced batch file processing with real-time monitoring.
- Supports
- ⚖️ Context-Based Translation Scoring:
- Evaluates word confidence based on translation method (Dictionary, Semantic, RBMT, Zero-Shot, etc.).
- Sentence position, grammar structure, and surrounding words are considered.
📊 ILTE-ZS vs. ILTE-ALT vs. ILTE-ADV
| Feature | ILTE-ALT 🌱 | ILTE-ZS 🧠 | ILTE-ADV 🧐 |
|---|---|---|---|
| Translation Approach | Dictionary-Based | Dictionary + RBMT + FST + Semantic + Zero-Shot | Dictionary + Semantic + Context-Based AI |
| Processing Speed | Fast | Optimized (Batch Multi-Processing) | Slower (Context-Aware AI) |
| Handling Large Files | Struggles | 🔄 Efficient Chunking & Batch Processing | Slower |
| Memory Usage | Low | Moderate | High |
| Context Awareness | None | Partial (RBMT & FST) | ✅ Strong (BERT + Semantic Matching) |
| Idiomatic Expressions | Limited | Rule-Based (RBMT) | AI-Based |
| Parallelization | Minimal | ✅ Thread + Process Pool | DataLoader-Based |
| Zero-Shot Capability | No | Yes | Yes |
| Best Use Case | Fast, simple translation | Handling large files, scalable processing | High-accuracy, AI-based translation |
🎯 ILTE-ZS Translation Flow Breakdown
1️⃣ Preprocessing
- Text normalization: Unicode cleanup & encoding detection.
- Sentence splitting using FST & RBMT rules.
2️⃣ Translation Pipeline
- Dictionary Lookup → RBMT & FST Transformation → Semantic Matching → Zero-Shot Translation (if needed).
- Hybrid scoring mechanism determines best method.
- Parallel execution ensures high-speed processing.
3️⃣ Confidence Scoring & Caching
- Evaluates word confidence based on:
- Translation method (Dictionary, Semantic, RBMT, etc.)
- Sentence structure & position.
- Stores results in JSON-based cache for reuse.
4️⃣ Final Output & Report Generation
- Outputs structured translation results.
- DOCX report includes confidence levels, processing time, and translation method logs.
🌐 Installation & Usage
Installing Dependencies
pip install -r requirements.txtRunning ILTE-ZS (Hybrid Mode)
python engine_ZS.pySelecting Input Type
👉 Manual Input: Enter text directly.
👉 File Upload: Provide a document for batch processing.
🛠️ System Requirements
ILTE-ZS (Hybrid Mode)
- CPU: Intel Core i5 / AMD Ryzen 5
- RAM: 8GB+ (16GB Recommended)
- Storage: 1GB+ free space
- Python: 3.10+
- GPU Acceleration Recommended
🌟 Future Plans & Improvements
- ✅ Enhancing FST & RBMT Rules for better morphological transformations.
- ✅ Fine-tuning Zero-Shot Translation performance.
- ✅ Expanding dictionary coverage for more languages.
- ✅ Further optimization for lower-end machines.
📚 License & Contribution
- Licensed under GPL v3 – Open Source & Community Driven! 🌍
- Contribute & Improve – Join us in preserving indigenous languages! 🌱
🏆 Developed by XI TJKT 2 Development Team 🧑💻
🎉 v2.1 BETA-2 and ALPHA-2 Release 🎉
🚀 Indigenous Language Translator Engine (ILTE) v2.1 Release 🎉
🔄 A More Refined & Efficient Translation Engine
The Indigenous Language Translator Engine (ILTE) v2.1 introduces significant refinements and optimizations, delivering improved speed, accuracy, and resource efficiency. This release includes two updated versions:
- 🌱 ILTE-ALT v2.1.0-Beta.2 – A lightweight, optimized dictionary-based translation engine.
- 🧠 ILTE-ADV v2.1.1-Alpha.2 – An enhanced AI-driven translation engine with improved semantic understanding.
🔍 What's New in v2.1?
🌱 ILTE-ALT v2.1.0-Beta.2 (Optimized & Faster)
- ✅ Optimized Dictionary Lookup for Faster Performance.
- ✅ Improved Stemming for Better Word Normalization.
- ✅ Levenshtein Distance Matching Enhancement for higher accuracy in fuzzy matches.
- ✅ Improved Confidence Scoring to prevent overestimated confidence percentages.
- ✅ DOCX Report Refinements with structured formatting.
- ✅ Lower GPU/CPU Usage, making it ideal for low-resource environments.
🧠 ILTE-ADV v2.1.1-Alpha.2 (AI-Powered, Smarter & More Context-Aware)
- 🧠 IndoBERT Contextual Translation Optimization for improved sentence handling.
- 🔍 Enhanced Zero-Shot Learning for better handling of unknown words.
- 🔄 Sentence Transformer (MiniLM-L12) Tweaks for better semantic matching.
- 📖 Refined Pattern-Based Learning & Context Awareness.
- ⚡ Increased GPU Utilization Efficiency, making translations faster and smarter.
- 📚 Updated Confidence Calculation to Prevent Values >100%.
- 🔧 Improved Batch Processing for large document translation.
🛠 ILTE-ALT vs. ILTE-ADV: Which One to Use?
| Feature | ILTE-ALT v2.1.0 (Beta) | ILTE-ADV v2.1.1 (Alpha) |
|---|---|---|
| Translation Approach | Dictionary-based | AI-driven, context-aware |
| Processing Speed | ✅ Faster | ⌛ Slower (but smarter) |
| Context Awareness | ❌ None | ✅ Uses IndoBERT embeddings |
| Semantic Matching | ❌ No | ✅ Uses SentenceTransformer |
| Zero-Shot Learning | ❌ No | ✅ Supports DistilBERT MNLI |
| GPU Utilization | ✅ Low | ⚡ High |
| Best For | Quick translations, lightweight usage | Complex, AI-powered translation |
🔄 Translation Flow Breakdown
1️⃣ Preprocessing
- ALT: Tokenization, stemming, direct lookup.
- ADV: Tokenization, embedding extraction, contextual matching.
2️⃣ Translation Processing
ILTE-ALT
- Direct dictionary lookup with fuzzy matching fallback.
- Fallback translation flow: EN → ID → DYK.
- Quick word replacement and confidence scoring.
ILTE-ADV
- Context-Aware Processing with IndoBERT.
- Semantic Matching for Closest Translations.
- Zero-Shot Learning for unknown word prediction.
- Translation Refinement using pattern-based learning.
3️⃣ Confidence Calculation (Fix in v2.1.1)
- ALT: Prevents confidence scores exceeding 100%.
- ADV: Uses averaging instead of multiplicative confidence scores.
4️⃣ DOCX Report Generation
- Both versions generate structured DOCX reports.
- ADV includes memory of translation patterns for future accuracy improvements.
📁 Installation & Usage
Installing Dependencies
pip install -r requirements.txtRunning ILTE-ALT (Fast Mode)
python engine_ALT.pyRunning ILTE-ADV (AI Mode)
python engine_ADV.pySelecting Input Type
👉 Manual Input: Enter text directly.
👉 File Upload: Provide a document for batch processing.
💻 System Requirements
ILTE-ALT (Lightweight Mode)
- CPU: Intel Core i3 / AMD Ryzen 3
- RAM: 4GB+
- Storage: 500MB free space
- Python: 3.8+
ILTE-ADV (AI-Powered Mode)
- CPU: Intel Core i5 / AMD Ryzen 5
- RAM: 8GB+ (16GB Recommended)
- Storage: 1GB+ free space
- Python: 3.10+
- GPU Acceleration (Recommended)
🛠 Known Issues & Future Plans
💡 Current Issues
- ALT: Lacks deep semantic understanding.
- ADV: Higher VRAM usage—may need manual tuning for lower-end GPUs.
🔄 Planned Improvements
- 🔹 ALT: Further dictionary optimization & expanded word support.
- 🔹 ADV: Improved AI accuracy & zero-shot performance.
- 🔹 Potential GUI Version for easier usage.
📚 License & Contribution
- Licensed under GPL v3 – Open Source Forever! 🛡
- Contribute & Improve – Join us in preserving indigenous languages! 🌍
🏆 Developed by XI TJKT 2 Development Team 🤝
ALPHA Release is UP! 🎉
🚀 Indigenous Language Translator Engine (ILTE) v2.0.0 - Alpha Release 🎉
🌍 Expanding Language Preservation with Dual Versions 🌿
The Indigenous Language Translator Engine (ILTE) v2.0.0 is here! This release marks a significant leap forward, offering two distinct versions to cater to a broad range of users—from casual learners to AI enthusiasts.
🆕 What's New in v2.0.0 (Alpha)
🌱 ILTE-ALT (Beta) - Lightweight & Efficient
✅ Basic Stemming for ID & EN (Sastrawi for ID, Snowball for EN)
✅ Dictionary-Based Lookup for direct and closest matches
✅ Levenshtein Distance for fuzzy matching
✅ Automated Confidence Scoring
✅ Structured DOCX Report Generation
✅ Low Memory Usage – Ideal for lower-end machines
✅ Minimal System Requirements for faster performance
🧠 ILTE-ADV (Alpha) - AI-Powered & Context-Aware
🧠 Contextual Translation with IndoBERT & Sentence Transformers
🔎 Zero-Shot Learning for Unknown Words
📖 Pattern-Based Learning – Adapts to text structures
🔄 Semantic Similarity Matching – Finds best translations beyond dictionary
📊 Enhanced Translation Confidence Metrics
⚡ Requires Higher Memory & Processing Power – Ideal for researchers and professionals
🎯 Who Should Use Each Version?
- 🏫 Educators & Students – Use ILTE-ALT (Beta) for quick, easy translations with minimal processing requirements.
- 🔬 Researchers & Professionals – Use ILTE-ADV (Alpha) for AI-powered, context-aware translations that improve over time.
- 🌍 Cultural Enthusiasts & Linguists – Choose between ALT for lightweight usage or ADV for deeper language learning insights.
📜 How to Use
Install the requirements.txt
pip -r install requirements.txtRunning ILTE-ALT (Beta) – Fast & Simple
python engine_ALT.pyRunning ILTE-ADV (Alpha) – AI-Enhanced
python engine_ADV.pySelecting Input Type
🔹 Manual Input: Type text directly into the CLI.
🔹 File Upload: Provide a file path for bulk translation.
🖥️ System Requirements
ILTE-ALT (Beta) - Minimal System Impact
- CPU: Intel Core i3 / AMD Ryzen 3
- RAM: 4GB+
- Storage: 500MB free space
- Python: 3.8+
ILTE-ADV (Alpha) - AI-Powered, Requires More Resources
- CPU: Intel Core i5 / AMD Ryzen 5
- RAM: 8GB+ (16GB Recommended)
- Storage: 1GB+ free space
- Python: 3.10+
- GPU Acceleration (Optional, but Recommended)
⚠️ Important Notes & Warnings 🚨
This tool is released under the GNU General Public License v3.0 (GPL v3), meaning you are free to use, modify, and distribute it as long as it remains open-source. Any use of this tool for personal financial gain or commercial exploitation is strictly prohibited.
💡 ILTE was built to support language preservation efforts, not to be monetized or misused. Let’s keep knowledge free and accessible for all!
🔮 What's Next for ILTE?
- 🔹 Further Optimization for ILTE-ALT – More lightweight improvements.
- 🔹 Enhanced AI for ILTE-ADV – Better zero-shot classification and pattern learning.
- 🔹 Potential GUI Version – A more user-friendly interface in future updates.
- 🔹 Expanded Dictionary Support – More words, more accuracy, better translations!
🤝 Get Involved!
As an open-source project, we welcome contributions from developers, linguists, and cultural enthusiasts. Want to add new features, improve translations, or expand the dictionary? Fork the repo and submit a pull request! 🚀
📜 Together, we can preserve indigenous languages for future generations. Let’s build a more connected and inclusive world. 🌎💡
📌 Developed for Indigenous Language Preservation ✊💬
🔗 License: GPL v3 | Open Source Forever 🛡️
Our regards, XI TJKT 2 Development Team. 🤝
Official BETA Release
🚀 Indigenous Language Translator Engine (ILTE) v1.0.0 - Beta Release 🎉
🌍 Breaking Language Barriers, Preserving Indigenous Heritage 🌿
The Indigenous Language Translator Engine (ILTE) is here! Designed to bridge the gap between Indonesian (ID), English (EN), and Dayak Kenyah (DYK), this open-source translation engine empowers linguists, educators, and communities to translate, document, and preserve indigenous languages with precision and efficiency.
🆕 What's New in v1.0.0 (Beta)
✅ Optimized Multi-Step Translation Flow – Efficiently translates between ID ↔ EN ↔ DYK using intelligent dictionary-based lookups.
✅ Advanced Preprocessing – Includes stemming, tokenization, and confidence-based word matching.
✅ Automated Confidence Scoring – Ensures translations are reliable with calculated match rates.
✅ DOCX Report Generation – Every translation session generates a structured report with performance metrics.
✅ Multi-Input Support – Accepts both manual text input and file-based translations.
✅ Levenshtein-Based Closest Matching – Finds near-exact word matches for improved translation accuracy.
🎯 Who is This For?
- 🏫 Linguists & Researchers – Preserve and document indigenous languages with accuracy.
- 📖 Educators & Students – Learn and teach Dayak Kenyah in a structured way.
- 🌍 Cultural Enthusiasts – Engage with indigenous languages and aid their survival in the modern world.
⚠️ Important Notes & Warnings 🚨
This tool is released under the GNU General Public License v3.0 (GPL v3), meaning you are free to use, modify, and distribute it as long as it remains open-source. Any use of this tool for personal financial gain or commercial exploitation is strictly prohibited.
💡 ILTE was built to support language preservation efforts, not to be monetized or misused. Let’s keep knowledge free and accessible for all!
🔮 What's Next?
- 🔹 Context-Aware Translations – Improve accuracy with phrase-based recognition.
- 🔹 Graphical User Interface (GUI) – A user-friendly interface for non-technical users.
- 🔹 Expanded Dictionary Support – More words, more accuracy, better translations!
🛠 How to Get Started
1️⃣ Clone the repository:
git clone https://github.com/YourRepo/ILTE.git2️⃣ Install dependencies:
pip install -r requirements.txt3️⃣ Run the translator:
python engine_alt.py🤝 Get Involved!
As an open-source project, we welcome contributions from developers, linguists, and cultural enthusiasts. Want to add new features, improve translations, or expand the dictionary? Fork the repo and submit a pull request! 🚀
📜 Together, we can preserve indigenous languages for future generations. Let’s build a more connected and inclusive world. 🌎💡
📌 Developed for Indigenous Language Preservation ✊💬
🔗 License: GPL v3 | Open Source Forever 🛡️
Our regards, XI TJKT 2 Development Team 🤝