web: Implement basic IME #19896
Open
web: Implement basic IME #19896
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
|
This isn't working on Android when I try to type in an EditText with the virtual keyboard. |
|
A virtual keyboard like the one on Android |
|
To explain the current logic, which I guess may need comments.
|
|
I'm guessing the IME logic can be used to support this more properly, but landing this PR without IME support would regress the virtual keyboard. |
|
Maybe we can do exactly this but also keydown and keyup the |
kjarosh
force-pushed
the
web-basic-ime
branch
2 times, most recently
from
a00d15d to
2af747a
Compare
|
@danielhjacobs can you check if the current code works properly? It does for me |
|
Tried with GBoard and it worked perfectly. With Samsung Keyboard it's unfortunately a different story, see recording. Screen_Recording_20250329_171917_Chrome.online-video-cutter.com.mp4 |
|
It seems that this keyboard uses IME for inputting all text. Without implementing IME on web we cannot have both IME preview and IME input working :/ This PR breaks IME preview, but fixes IME input. |
kjarosh
changed the title
|
Okay, as IME on web is a mess, I've decided to implement basic IME mechanics (including preediting) on web. @danielhjacobs @n0samu You can test it out here: https://kjarosh.github.io/ruffle/pr19896/ |
kjarosh
added
A-input
This patch implements IME preediting and committing on web. It does not implement moving the cursor and proper positioning yet.
|
As stated on Discord, but putting here for future people, this is now working with Samsung Keyboard. |
|
@jmousy Could you check if it works properly? It's available at https://kjarosh.github.io/ruffle/pr19896/ |
|
This is a summary of the chat on Discord.
|
|
Below is the output from the link below, as requested by Daniel Jacobs on Discord: Input text: '가나다' (rkskek) iOS 18.4 (default keyboard)Android 14 (samsung keyboard)Android (GBoard)Windows 11Ubuntu 24.04macOS 15.4 |
|
Explanation of issues: iOS default keyboardBecause we clear the input for each character typed, the characters do not combine, as the only input event that fires is an insertText for each individual character (all six) and there are no composition events or advanced text events. We'd need to properly handle the advanced text events anyway.iOS GBoardUnknown issue as I don't know what events it fires. Probably relates at least partially to clearing the input for each character typed.Android Samsung KeyboardNo issue, every input event occurs during composition and the data at compositionend is correct, containing all the entered text.Android GBoardFor the initially mentioned input, all input events happen during composition and the data at compositionend is correct. I'd need to see the events that fire when entering 1 + 2 + 3 + ㄱ + ㅏ + ㄴ + ㅏ + ㄷ + ㅏ to know the issue there.Windows default keyboardBecause we clear the input for each character typed, the deleteContentBackward event that is supposed to fire before a final input event duplicates the last combined character does not fire. Even if it did, we don't handle deleteContentBackward.macOS default keyboardUnsure, based on the listed events I would expect everything to work correctly. If you ignore the order, macOS seems to be typing everything twice.Linux default keyboardNo issue, there is a non-composing insertText input event containing 가 that occurs following a compositionend event with no data, a non-composing insertText input event containing 나 that occurs following a compositionend event with no data, and a compositionend event at the end containing 다 as data. |
danielhjacobs
approved these changes
Apr 24, 2025
|
I'm about 99.9% sure we need to stop clearing the hidden input for each character typed in the future. |
|
I have a theory for what Mac is doing wrong, and if I'm right I wonder if this would help: this.virtualKeyboard.addEventListener("keydown", this.ignoreComposingKeyEvents.bind(this));
this.virtualKeyboard.addEventListener("keyup", this.ignoreComposingKeyEvents.bind(this));
ignoreComposingKeyEvents(event: KeyboardEvent) {
if (event.isComposing) {
event.preventDefault();
event.stopPropagation();
return;
}
}My theory is maybe Mac attempts to fire events for composing key events, causing the keyup/keydown to write text too. |
Me too. Moreover, I'm 97% sure we should just synchronize the text field with the HTML input and issue Flash events based on changes. But that's a separate issue from IME, as IME in Flash behaves differently to normal input (and it gets underlined). Even with full text synchronization we should translate IME events to Flash. |
danielhjacobs
approved these changes
May 21, 2025
There was a problem hiding this comment.
I don't have perfect knowledge of Rust but I'm comfortable enough to approve this PR after it's been sitting for weeks. The web-side code seems fine. As mentioned, it's not perfect but it doesn't make things any worse and it can't be perfect unless Rust and the web start talking to each other a lot more when it comes to EditTexts.
|
Note, the tsx files weren't being linted so it's entirely possible that when this gets rebased it ends up with lint failures. |
|
BTW I'm still trying to write a test for it, but the process is tedious. I'll try my best to include it here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch implements IME preediting and committing on web. It does not
implement moving the cursor and proper positioning yet.