Skip to content

taingph2502/MangaTrans-CLI

BranchesTags

Repository files navigation

MangaTrans CLI

A local comic translation pipeline. Reads an image, detects and recognizes text in multiple languages (Japanese, Chinese, Korean, English, Vietnamese, and more), translates it via a local SLM (Qwen 3.5) with Ollama, and renders the translated text back onto the page with proper inpainting and font styling.

Bonus: Fine-Tuning PaddleOCR-VL on the Manga109 dataset for Japanese manga text recognition. Placed in the SFT-PaddleOCR-VL/ folder.

1. How It Works

The pipeline is organized into three distinct phases: Pre-Translation (image preprocessing), Translation (OCR + LLM), and Post-Translation (inpainting + rendering). Each phase is fully asynchronous and GPU-accelerated.

pipeline

Pipeline Details

1. Pre-Translation Pipeline:

PreTranslationPipeline.run() runs sequentially: optional colorization → optional upscaling → image loading → detection (DBNet or CTD) → OCR (48px or MangaOCR) → textline merging → pre-dictionary substitution.

2. Translation:

Operates per-text-region in parallel batches. The LLM receives either a single-page prompt or a multi-page batch prompt. A ContextHistoryManager accumulates translations from previous pages (up to 25,000 tokens) to give the LLM conversational continuity. Built-in quality gates detect repetition hallucinations (via repeating_sequence), wrong-language output (via py3langid), and source-text echoing. Up to 3 automatic retries with progressively stricter system prompts handle each failure class.

3. Post-Translation Pipeline:

Refines the text mask, inpaints the original image using the mask, then renders translated text with FreeType using the detected font colors, stroke width, balloon-aware sizing, and spiral-collision resolution.

2. Demo

Original Translated to Vietnamese
English comic sample English comic translated to Vietnamese
Japanese manga sample Japanese manga translated to Vietnamese

Reproduce the demo outputs:

manga-trans local -i demo/raw-en.png -o demo/raw-en-vi.png --target-lang VIN --content-type comic --use-gpu --overwrite
manga-trans local -i demo/raw-jp.png -o demo/raw-jp-vi.png --target-lang VIN --content-type manga --use-gpu --overwrite

3. Technology Stack

Component Technology Version Notes
Runtime Python 3.13+ uv-managed virtual environment
Deep Learning PyTorch (CUDA) 2.11.0+ CUDA 12.6 build via pytorch-cuda index
OCR manga-ocr 0.1.14+ End-to-end manga text recognition
OCR (alt) Model 48px (custom) — ConvNeXt + Transformer, ocr_ar_48px.ckpt
Text Detection DBNet — ResNet34/101 backbone, detect-*.ckpt
Text Detection (alt) CTD (Comic Text Detector) — YOLOv5-based, for manga/manhwa/manhua
Inpainting LaMa Large — FFCRESNETGenerator, lama_large_512px.ckpt
Inpainting (alt) AOT-GAN — Attention-based inpainting
Upscaling ESRGAN — Real-ESRGAN
Upscaling (alt) 4x-UltraSharp — High-quality upscaling
Colorization Manga Colorization V2 (MC2) — denoiser + U-Net
Translation Ollama + qwen3.5:4b — Local SLM, /api/chat endpoint
Text Rendering FreeType + PIL — Balloon-aware, collision-resolution

4. Installation & Setup

Prerequisites

  • Python 3.13 or newer
  • NVIDIA GPU with CUDA support
  • Ollama running locally (for the default translation engine)
  • uv package manager

Model Setup

Models are downloaded automatically on first use. The default storage location is ./models in the project root (customizable via --model-dir).

Ollama (Primary Translation Engine)

Download model:

# Pull the default translation model
ollama pull qwen3.5:4b

# Run the model (--think=false disables reasoning blocks that interfere with parsing)
ollama run qwen3.5:4b --think=false

Configure the API base and model name via environment variables:

export OLLAMA_API_BASE=http://localhost:11434   # default
export OLLAMA_MODEL=qwen3.5:4b                  # default
export OLLAMA_KEEP_ALIVE=10m                    # default; keeps model warm between pages
export OLLAMA_FLASH_ATTENTION=1                # enable Flash Attention 2

5. Usage

Verify your environment before running translations:

manga-trans doctor

Translate to Vietnamese with a content-type preset:

manga-trans local -i manga_page.png --target-lang VIN --content-type manga

By default, translated images are saved to result/final.png. Use -o to specify a custom output path or directory for batch processing.

6. CLI Reference

Global Flags (all modes)

These flags apply to both local and doctor modes:

Flag Type Default Description
-v, --verbose flag False Print debug info and save intermediate images (bboxes.png, mask_raw.png, etc.) in the result folder
--attempts int 0 Number of retry attempts on encountered error. -1 means infinite retries.
--ignore-errors flag False Skip the current image and continue when an error is encountered
--model-dir path ./models Directory where model weights are stored
--use-gpu flag False Enable GPU acceleration (auto-selects between MPS and CUDA)
--use-gpu-limited flag False Enable GPU with reduced VRAM usage for translation-related models
--font-path path auto Path to a custom TTF font file
--pre-dict path none Path to pre-translation dictionary (dict/pre_dict.txt format)
--post-dict path none Path to post-translation dictionary (dict/post_dict.txt format)
--kernel-size int 3 Convolution kernel size for text erasure (complete cleanup of text residues)
--context-size int 0 Number of previous pages to include as context for LLM translation (0 = no context)
--batch-size int 1 Number of images per translation batch (1 = no batching)
--batch-concurrent flag False Process each image in its own translation request instead of merging batches. Prevents output truncation and hallucination.
--content-type choice none Comic type preset: manga, comic, manhwa, manhua. Auto-selects detector + OCR preset.
--target-lang choice ENG Target language for translation: ENG (English) or VIN (Vietnamese)
--runtime-profile choice auto Runtime profile affecting detection/inpainting sizes: auto, low, medium, high
--detector choice auto Text detector model: dbnet (default for comic), ctd (for manga/manhwa/manhua)
--upscaler choice auto Image upscaler: esrgan (fast), 4xultrasharp (high quality)
--colorizer choice none Colorization model: none, mc2 (Manga Colorization V2)
--ocr choice auto OCR model: 48px (default for comic/manhwa/manhua), mocr (MangaOCR, for manga)
--inpainter choice lama_large Inpainting model: lama_large, aot (AOT-GAN)

Subcommand: local

Flag Type Default Description
-i, --input path+ required Path to an image file or directory containing images
-o, --dest str result/final.png Destination directory or file path for translated images
-f, --format choice auto Output format: png, jpg, jpeg, webp
--overwrite flag False Overwrite existing translated images
--skip-no-text flag False Skip and do not save images that contain no detected text
--save-text flag False Save extracted text and translations to a .txt file alongside the image
--load-text flag False Load pre-saved text and translations instead of running OCR + translation
--save-text-file str none Like --save-text but saves to a specific file path
--save-quality int 100 JPEG output quality (0–100)
--config-file path none Path to a JSON/TOML config file for full pipeline configuration
--chapter-max-retries int 2 Maximum retries per page in chapter/folder processing
--chapter-no-resume flag False Disable manifest-based resume for chapter/folder translation
--chapter-max-pages int 0 Limit number of chapter pages to process (0 = all pages)

Subcommand: doctor

Flag Type Default Description
--doctor-timeout float 5.0 Timeout in seconds for Ollama runtime reachability checks
--translator choice all Runtime to check: all or ollama

Subcommand: config-help

Prints the full Config JSON schema to stdout. Pipe to jq or a file for inspection:

manga-trans config-help | jq .

7. Usage Examples

Translate a Single Image (English Output)

manga-trans local -i manga_page.png -o translated.png

Translate a Folder (Batch Mode)

manga-trans local -i ./chapter_01/ -o ./chapter_01_translated/

Translate to Vietnamese with Manga Preset

Uses CTD detector + MangaOCR automatically:

manga-trans local \
    -i ./manga_pages/ \
    --target-lang VIN \
    --content-type manga \
    --batch-concurrent

Debug Mode with Verbose Output

Saves intermediate images (detection bboxes, masks, inpainted canvas) to the result folder:

manga-trans local -i input.png -v
# Produces: result/bboxes_unfiltered.png, result/mask_raw.png, result/bboxes.png

Custom Config File

manga-trans local -i input.png --config-file ./config.json

GPU with Low VRAM Profile

Auto-detects VRAM and selects appropriate detection/inpainting sizes:

manga-trans local -i input.png \
    --runtime-profile low \
    --use-gpu-limited \
    --detector dbnet \
    --inpainter lama_large

Multi-Page Context for Better Continuity

Include 2 previous pages as context so the LLM maintains dialogue consistency:

manga-trans local -i ./pages/ -o ./translated/ \
    --context-size 2 \
    --batch-concurrent

Custom Font and Dictionary

manga-trans local -i input.png \
    --font-path ./my_font.ttf \
    --pre-dict ./dict/pre_dict.txt \
    --post-dict ./dict/post_dict.txt

8. Fine-Tuning PaddleOCR-VL

The SFT-PaddleOCR-VL/ directory contains scripts for fine-tuning PaddleOCR-VL on the Manga109 dataset for Japanese manga text recognition.

This is a separate research/fine-tuning workspace. The production CLI currently uses MangaOCR for the manga preset and does not load PaddleOCR-VL at runtime.

Results

Fine-tuned on an RTX 6000 Ada (48 GB VRAM) with BF16 mixed precision:

Metric Base Model Fine-Tuned Improvement
Exact Match 9.2% 65.9% +56.7 points
Character Error Rate (CER) 55.4% 9.1% ~83.6% reduction

The base PaddleOCR-VL model struggles with vertical Japanese manga reading order. After fine-tuning, the model better handles manga-specific text layouts.

Dataset Preparation

  1. Download the dataset from Manga109-s on Hugging Face.
  2. Extract it to the project directory and set MANGA109_ROOT.
  3. Prepare cropped text regions:
cd SFT-PaddleOCR-VL
uv pip install -r requirements.txt
python prepare_manga109_data.py

Training

cd SFT-PaddleOCR-VL
bash train.sh

Evaluation

# Evaluate fine-tuned model
uv run python evaluate_model.py --model_path ./sft_output --num_samples 500

# Compare with base model
uv run python evaluate_model.py --model_path PaddlePaddle/PaddleOCR-VL --num_samples 500

9. Configuration

Config File Format

Full pipeline configuration can be saved as JSON (Pydantic v2 schema):

manga-trans config-help > config_schema.json

A minimal config file (config.json) example:

{
  "render": {
    "alignment": "auto",
    "disable_font_border": false,
    "font_size_offset": 0,
    "font_size_minimum": -1,
    "direction": "auto",
    "no_hyphenation": true,
    "line_spacing": null,
    "font_path": null,
    "min_hyphen_word_len": 0,
    "rtl": true,
    "layout_overflow_tolerance": 0.02,
    "layout_max_shrink_steps": 8,
    "layout_min_font_size": 6,
    "stroke_width_min": 0.06,
    "stroke_width_max": 0.12
  },
  "upscale": {
    "upscaler": "esrgan",
    "revert_upscaling": false,
    "upscale_ratio": null
  },
  "translator": {
    "translator": "ollama",
    "target_lang": "VIN",
    "use_visual_context": false,
    "enable_post_translation_check": true,
    "post_check_max_retry_attempts": 3,
    "post_check_repetition_threshold": 20,
    "post_check_target_lang_threshold": 0.5
  },
  "detector": {
    "detector": "ctd",
    "detection_size": 2048,
    "text_threshold": 0.5,
    "box_threshold": 0.7,
    "unclip_ratio": 2.3,
    "det_rotate": false,
    "det_auto_rotate": false,
    "det_invert": false,
    "det_gamma_correct": false
  },
  "colorizer": {
    "colorization_size": 576,
    "denoise_sigma": 30,
    "colorizer": "none"
  },
  "inpainter": {
    "inpainter": "lama_large",
    "inpainting_size": 2048,
    "inpainting_precision": "bf16",
    "use_mpe": false
  },
  "ocr": {
    "ocr": "mocr",
    "min_text_length": 0,
    "ignore_bubble": 0,
    "prob": null,
    "low_confidence_threshold": 0.6,
    "drop_low_confidence_noise": true
  },
  "force_simple_sort": false,
  "kernel_size": 3,
  "mask_dilation_offset": 20,
  "filter_text": null
}

Content Type Presets

Setting --content-type auto-configures detector + OCR. Font and hyphenation are selected by target language: ENG uses anime_ace_3.ttf with an 8-character minimum for hyphenation; VIN uses NotoSans-Regular.ttf with hyphenation disabled.

Content Type Detector OCR Model
manga (Japanese) CTD MangaOCR
comic (Western) DBNet 48px
manhwa (Korean) CTD 48px
manhua (Chinese) CTD 48px

Runtime Profiles

Affects detection_size and inpainting_size. In auto mode, the profile is inferred from GPU VRAM:

Profile Detection Size Inpainting Size VRAM Threshold
low 1024 1024 < 6 GB
medium 1536 1536 6–10 GB
high 2048 2048 10+ GB

VRAM guard thresholds: warns and downscales at < 1 GiB free; forces CPU fallback at < 0.5 GiB free.

Pre/Post Translation Dictionaries

Both dictionaries use tab-separated format (PATTERN<tab>REPLACEMENT), with comment lines starting with # or //.

Pre-dictionary (dict/pre_dict.txt) — applied to OCR output before translation:

  • OCR error corrections (fullwidth digits ０１２３ → 0123)
  • Chapter/episode marker normalization
  • SFX standardization

Post-dictionary (dict/post_dict.txt) — applied to LLM translation output:

  • Markdown artifact removal (**bold** → bold)
  • Punctuation normalization (excessive dots ..... → ...)
  • Character name standardization
  • Whitespace cleanup

About

Local AI comic translation tool that detects, translates, erases, and redraws text across manga, manhua, manhwa, and comics.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages