GAJMA 2.0 — Anime Generation

Case study di Computer Vision (UniBA, A.A. 2025/26). Genera brevi video di transizione tra pannelli manga consecutivi.

Pipeline: Manga109 -> balloon removal (LaMa) -> preprocessing (LAB) -> ToonCrafter 16 frame -> RIFE 4x.

Struttura

progetto-animeGen/
  src/                # moduli core (preprocessing, balloon_removal, metrics, extract_panels)
  scripts/            # script CLI (run_rife, evaluate_videos, aggregate_metrics, ...)
  notebooks/
    kaggle_tooncrafter_sharded.ipynb   # inferenza ToonCrafter multi-GPU T4x2
  data/
    manga109/         # dataset (non in git, vedi sotto)
    pairs/            # coppie consecutive 512x320 (non in git)
    metadata/         # CSV con pair selection
  outputs/
    metrics/16f/      # CSV per-clip + ablation table 2x2x2
    user_study/       # 20 video side-by-side, mapping, analisi risposte
    submission/       # submission.csv + samples.zip per Challenge Hub
  report/             # main.tex + slides.tex

Setup

/opt/homebrew/bin/python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Per Manga109: registrazione su hal-utokyo/Manga109 (gated). Una volta scaricato, mettere sotto data/manga109/.

Per RIFE: clonare Practical-RIFE in external/Practical-RIFE/ e scaricare i pesi v4.25.

Riproduzione (in ordine)

# 1. estrazione pannelli + balloon removal + coppie 512x320
python scripts/build_pairs.py --books LoveHina_vol01 ARMS SaladDays_vol01 Akuhamu PrismHeart --method lama

# 2. preprocessing
python scripts/apply_preprocessing.py --in data/panels/clean_lama --out data/panels/clean_lama_pp

# 3. selezione 40 coppie buone via LPIPS
python scripts/select_pairs.py --per-book 8 --device mps

# 4. inferenza ToonCrafter (su Kaggle T4x2)
#    vedi notebooks/kaggle_tooncrafter_sharded.ipynb

# 5. RIFE 4x in locale
python scripts/run_rife.py --in outputs/videos/videos_clean_lama_pp_16f --out outputs/videos/videos_clean_lama_pp_16f_rife --multi 4

# 6. metriche (per ogni variant)
python scripts/evaluate_videos.py --videos outputs/videos/videos_clean_lama_pp_16f_rife --out outputs/metrics/16f/clean_lama_pp_rife.csv --device mps --label clean_lama_pp_rife

# 7. ablation 2x2x2
python scripts/aggregate_metrics.py outputs/metrics/16f/*.csv --out outputs/metrics/16f/ablation_table_2x2x2.csv

# 8. submission Challenge Hub
python scripts/prepare_submission.py

User study (Google Form, 2-AFC pairwise)

# genera 20 video affiancati (4 pair x 5 confronti)
python scripts/prepare_user_study.py

# dopo aver caricato i video su Drive e incollato i link in mapping.csv:
python scripts/fill_form_links.py

# dopo aver scaricato il CSV con le risposte:
python scripts/analyze_user_study.py --responses outputs/user_study/responses.csv

Report

cd report
pdflatex main.tex && bibtex main && pdflatex main.tex && pdflatex main.tex

Template: ufficiale del corso (arxiv.sty).