Subtitle system for streaming audo/video

[PTwr]

A simple subtitle system to allow for translating games without any riivolution patches or romhacking.

It started as my pet project, R79JAF translation, to allow translation of voice files that had no accompanying on-screen text.
After boasting about it on discord some translators expressed interest in it so I upgraded it to be more universal.

Target games are ones with a lot of audio/video files streamed on-demand from disc. It won't work on pre-fetched stuff. So stuff like "visual novels" would be prime target.

Basically, it takes json file like this

[
  {
    "FileName": "sound/stream/evz136.brstm",
    "Translation": "This time it's a bit of an irregular job.\nKlaus, did you get that thing?",
    "Miliseconds": 5000,
    "Color": "Green",
    "Enabled": true,
    "AllowDuplicate": false,
    "Scale": 2
  }
]

And does stuff like this:

Under the hood it works same as FileMonitor log, Simple hook in DVDThread and lookup for filename in dictionary.
As such it can be used for any file read from disc.

Subtitles are (re)loaded when game starts.

Display is done wtih ImGui through OSD, which I upgraded to handle multiple "zones".

Fancy functions:

• file location is same as other stuff from Load directory.
• support for multiple subtitle files per game, including nested directories. Quite useful when you have six thousands voice files to translate.
• support for placing subtitles in long-playing audio/video stream through relative offset and timestamp (measured in real time, not emulated)
• html colors
• ugly text size, Dolphin appears to not be prepared for scalable fonts in ImGui.

Possible issues:

• memory management, pointer/reference mixup, and other standard problems C# developer have in C++ :)
• do we want to keep my changes to OSD, or should subtitles be done separately?
• some automatic generator of subtitle file for translator could be useful for non-techie translators, and maybe nicer file format

[sepalani]

You'll find below a cursory review. You might want to check the project coding style and the lint bot result to address some of the issues.

I'd really like Dolphin to have a translation/subtitle system/API. However, I'm unsure which form it should take and if your current proposal is the best way to implement one. As you said, you're hooking file loads and it won't work if the file is used later.

A simple subtitle system to allow for translating games without any riivolution patches or romhacking.

I can understand the rational as it can make translation easier. However, this approach prevents the translation project from being used on real hardware (which is kinda sad for a translation project).

Some games might bundle several texts, voices, videos into a single file. I've rarely seen games using one file (audio, video or text) for a single message which makes your approach very situational.

Other ways that might be used to extract texts or its address/properties of Dolphin (from an external process):

• Using Dolphin Memory Engine
• Using File/IO logs and checking Dolphin logs
• Using log breakpoints and checking Dolphin logs

Possible alternatives to extract texts from Dolphin (to be implemented):

• Custom HLE hooks
• Custom breakpoint log messages (to fetch text from specific address/register)
• Custom File/IO/DVD hooks

For reference, common methods to translate games:

• Create a game translation patch, using Wiimm ISO Patcher (based on WIT) and/or Riivolution.
• IIRC, libretro can (try to) translate using a screenshot, send it to an OCR and send its results to a translator.
• Text extracting tool can be used (Textractor, for instance) and its results can be sent to a translator.

[PTwr]

You'll find below a cursory review. You might want to check the project coding style and the lint bot result to address some of the issues.

I'd really like Dolphin to have a translation/subtitle system/API. However, I'm unsure which form it should take and if your current proposal is the best way to implement one. As you said, you're hooking file loads and it won't work if the file is used later.

A simple subtitle system to allow for translating games without any riivolution patches or romhacking.

I can understand the rational as it can make translation easier. However, this approach prevents the translation project from being used on real hardware (which is kinda sad for a translation project).

Some games might bundle several texts, voices, videos into a single file. I've rarely seen games using one file (audio, video or text) for a single message which makes your approach very situational.

Other ways that might be used to extract texts or its address/properties of Dolphin (from an external process):

* Using Dolphin Memory Engine

* Using File/IO logs and checking Dolphin logs

* Using log breakpoints and checking Dolphin logs

Possible alternatives to extract texts from Dolphin (to be implemented):

* Custom HLE hooks

* Custom breakpoint log messages (to fetch text from specific address/register)

* Custom File/IO/DVD hooks

For reference, common methods to translate games:

* Create a game translation patch, using Wiimm ISO Patcher _(based on WIT)_ and/or Riivolution.

* IIRC, libretro can _(try to)_ translate using a screenshot, send it to an OCR and send its results to a translator.

* Text extracting tool can be used _(Textractor, for instance)_ and its results can be sent to a translator.

This little fork came as a result of my translation effort of R79JAF, as a companion to riivolution patch with text and textures.
I might have picked wrong screenshot to explain this situation, but game has 6 thousands voice files, played all around the game, including in-battle, without any accompanying text field on screen.

In example I provided at start I do not use subtitles, as I am in process of patching all on-screen text.
Subtitles are using in places like this:

or like this:

Subtitles on actual hardware are, obviously, out of scope.

[Anuskuss]

I've been waiting for something like this. Great job!
Although I was wondering, could this be expanded to also support swapping tracks with your own audio files, preferrably with audio formats that the Wii can't even play (e.g. 48kHz FLAC)?

[PTwr]

I've been waiting for something like this. Great job! Although I was wondering, could this be expanded to also support swapping tracks with your own audio files, preferrably with audio formats that the Wii can't even play (e.g. 48kHz FLAC)?

TLDR: Yes, but...

There was a discuss about something like that on modding Discord. Idea being that if we can't really make emulated (or real) Wii to play super duper 4k video, we can just make Dolphin play it.
Basically, if we can monitor IO to streaming file, we can just play custom audio/video in popup for as long as game is streaming original file.

There are some issues to be resolved to get such system to work

• getting some good audio/video player that can be controlled programmatically
• syncing real and emulated time or frames
• mixing replaced sound with background music
• stopping playback when player skips it

Vdeo:
Simplest way for video replacement, would be to just pause the game, play the video until player hits skip, resume game, send virtual skip keypress to game. No muxing issues, no time sync problems.
You could monitor for continuos reads and sync custom video with real... but if its full screen video, why bother.
It would be quite easy to do, and rather problem free. But it will only work for cutscenes in exclusive playback.

However, there are more complicated video playbacks. Some are looped, some are mixed into gameplay scene.
TLDR: Cutscenes/intro/outro yes, anything more complex no.

Sound:
Unlike video, sounds are bound to be played multiple at a time, so just muting game won't work, and being played in-game makes them prone to desync, and unsuitable for pause+overlay trick.
It's still doable, but you'd need to somehow mute replaced file, simplest way would be to provide silent stream of same length.
Issue here would be prefetched sounds. in R79JAF, and some other voice-heavy games I looked at, voices and background music are streamed directly from disc, but sound effects are pre-fetched thus undetectable for FileMonitor.