Skip to content

Commit f31934c

Browse files
khaledalamkhaledalam
authored
Merge pull request #7 from mahendra189/main
feat: add LipSyncAvatoon component and Android reference
2 parents 7bb51e5 + 5333c59 commit f31934c

7 files changed

Lines changed: 501 additions & 81 deletions

File tree

‎android_reference/IMPLEMENTATION_GUIDE.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Android 3D Avatar Lip Sync Implementation Guide
2+
3+
This guide explains how to implement the 3D Avatar with "Lip Sync Only" mode (random mouth movement, no audio) and locked camera panning in a native Android application using Kotlin and [Sceneview](https://github.com/Sceneview/sceneview).
4+
5+
## 1. Project Setup
6+
7+
### Dependencies (build.gradle.kts)
8+
Ensure you have the Sceneview dependency in your module-level `build.gradle.kts`:
9+
10+
```kotlin
11+
dependencies {
12+
implementation("io.github.sceneview.android:sceneview:2.0.3") // Check for latest version
13+
}
14+
```
15+
16+
## 2. Helper Class
17+
We have created a helper class `LipSyncManager` that handles the morph target manipulation.
18+
Copy the file `LipSyncManager.kt` into your source set (e.g., `app/src/main/java/com/example/yourapp/LipSyncManager.kt`).
19+
20+
**Key Features of LipSyncManager:**
21+
- **Randomized Visemes**: Picks random mouth shapes to simulate talking.
22+
- **Smooth Interpolation**: Uses `lerp` to smoothly transition between mouth shapes so it looks natural, not robotic.
23+
- **Toggle Control**: `setTalking(true/false)` to start/stop.
24+
25+
## 3. Implementation Steps
26+
27+
1. **Layout XML**: Add a `io.github.sceneview.SceneView` and a `Button` to your layout.
28+
2. **Activity/Fragment**:
29+
- Load your `.glb` model into a `ModelNode`.
30+
- Add the node to the `SceneView`.
31+
- Initialize `LipSyncManager` with the node.
32+
- Set up the Button `OnClickListener` to toggle `lipSyncManager.setTalking(isActive)`.
33+
- **Disable Panning**: Configure the camera manipulator to disallow panning/movement if essentially "locking" the view is desired, or simply don't attach a manipulator that allows it.
34+
35+
## 4. AI Prompt for Integration
36+
37+
If you need to generate the specific Activity/Fragment code for your existing Android project, use the prompt below. Copy and paste this into your AI coding assistant (like Android Studio Bot, ChatGPT, or Cursor).
38+
39+
---
40+
### 📋 Copy This Prompt:
41+
42+
```text
43+
I have a Kotlin Android project using Sceneview.
44+
I need to implement a 3D Avatar viewer with the following specific requirements:
45+
46+
1. **Load Avatar**: Load a GLB model (e.g., "avatar.glb") into the SceneView.
47+
2. **Lock Camera**: The user should NOT be able to pan or move the camera. The camera should be fixed on the avatar's head/upper body.
48+
3. **Lip Sync Feature**:
49+
- Use the provided `LipSyncManager` class (I will provide this class definition).
50+
- I need a toggle Button on the UI (overlaying the 3D view).
51+
- When clicked, it should start the random lip-sync animation (no audio) using `lipSyncManager.setTalking(true)`.
52+
- When clicked again, it should stop it.
53+
4. **UI Layout**: Please provide the XML layout with a SceneView and a styled floating button at the bottom.
54+
55+
Here is the helper class I have:
56+
57+
class LipSyncManager(private val avatarNode: ModelNode, private val scope: CoroutineScope) {
58+
// ... (Your AI agent will infer the methods setTalking) ...
59+
// It uses standard morph target setting on the node.
60+
}
61+
62+
Please write the MainActivity.kt and activity_main.xml code.
63+
```
64+
---
65+
66+
## 5. Sample Usage Code
67+
68+
Here is a quick preview of how the `MainActivity` might look:
69+
70+
```kotlin
71+
class MainActivity : AppCompatActivity() {
72+
73+
private lateinit var sceneView: SceneView
74+
private lateinit var lipSyncManager: LipSyncManager
75+
private var isTalking = false
76+
private val modelUrl = "models/avatar.glb" // In assets folder
77+
78+
override fun onCreate(savedInstanceState: Bundle?) {
79+
super.onCreate(savedInstanceState)
80+
setContentView(R.layout.activity_main)
81+
82+
sceneView = findViewById(R.id.sceneView)
83+
val toggleButton = findViewById<Button>(R.id.btnToggleTalk)
84+
85+
lifecycleScope.launchWhenCreated {
86+
val modelInstance = sceneView.modelLoader.loadModelInstance(modelUrl) ?: return@launchWhenCreated
87+
val modelNode = ModelNode(
88+
modelInstance = modelInstance,
89+
scaleToUnits = 1.0f
90+
).apply {
91+
// Position avatar so head is visible
92+
position = Position(y = -1.0f)
93+
}
94+
95+
sceneView.addChild(modelNode)
96+
97+
// Initialize Manager
98+
lipSyncManager = LipSyncManager(modelNode, lifecycleScope)
99+
100+
// Lock Camera (Disable manipulation)
101+
sceneView.cameraNode.manipulator = null
102+
}
103+
104+
toggleButton.setOnClickListener {
105+
isTalking = !isTalking
106+
lipSyncManager.setTalking(isTalking)
107+
toggleButton.text = if (isTalking) "Stop Talking" else "Start Talking"
108+
}
109+
}
110+
}
111+
```

‎android_reference/LipSyncManager.kt‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
package com.example.avatoon
2+
3+
import android.animation.ValueAnimator
4+
import android.util.Log
5+
import io.github.sceneview.SceneView
6+
import io.github.sceneview.math.MathUtils
7+
import io.github.sceneview.node.ModelNode
8+
import kotlinx.coroutines.*
9+
import kotlin.random.Random
10+
11+
/**
12+
* A helper class to manage Lip Sync animation for a 3D Avatar using Sceneview.
13+
*
14+
* Usage:
15+
* 1. Initialize with your ModelNode containing the avatar.
16+
* 2. Call `setTalking(true)` to start random lip movement.
17+
* 3. Call `setTalking(false)` to stop.
18+
*
19+
* Dependencies: io.github.sceneview.android:sceneview:2.0.3 (or later)
20+
*/
21+
class LipSyncManager(
22+
private val avatarNode: ModelNode,
23+
private val scope: CoroutineScope = CoroutineScope(Dispatchers.Main)
24+
) {
25+
26+
private var isTalking = false
27+
private var updateJob: Job? = null
28+
29+
// Map of logical phonemes to the actual Morph Target names in your GLB
30+
// Ensure these match the morph target names exported in your 3D model
31+
private val visemeMap = mapOf(
32+
"A" to "viseme_aa",
33+
"B" to "viseme_PP",
34+
"C" to "viseme_CH",
35+
"D" to "viseme_DD",
36+
"E" to "viseme_E",
37+
"F" to "viseme_FF",
38+
"I" to "viseme_I",
39+
"O" to "viseme_oo",
40+
"R" to "viseme_RR"
41+
)
42+
43+
private val activeVisemes = visemeMap.values.toList()
44+
45+
// State for smooth animation
46+
private var currentViseme: String? = null
47+
private var currentInfluence = 0f
48+
private var targetInfluence = 0f
49+
50+
// Animation loop speed
51+
private val updateIntervalMs = 50L
52+
53+
fun setTalking(talking: Boolean) {
54+
if (isTalking == talking) return
55+
isTalking = talking
56+
57+
if (talking) {
58+
startLoop()
59+
} else {
60+
stopLoop()
61+
}
62+
}
63+
64+
private fun startLoop() {
65+
updateJob?.cancel()
66+
updateJob = scope.launch {
67+
var nextChangeTime = System.currentTimeMillis()
68+
69+
while (isActive) {
70+
val now = System.currentTimeMillis()
71+
72+
// Pick a new random viseme every ~100-200ms
73+
if (now >= nextChangeTime) {
74+
currentViseme = activeVisemes.random()
75+
targetInfluence = if (Random.nextBoolean()) {
76+
Random.nextFloat() * 0.7f + 0.3f // 0.3 to 1.0
77+
} else {
78+
0f // Occasionally pause/silence
79+
}
80+
81+
// Schedule next change
82+
nextChangeTime = now + Random.nextLong(50, 200)
83+
}
84+
85+
// Interpolate (Lerp) towards target
86+
// In a real game loop, you'd use delta time. Here we approximate with fixed delay.
87+
currentInfluence = MathUtils.lerp(currentInfluence, targetInfluence, 0.2f)
88+
89+
// Apply to model
90+
updateMorphTargets()
91+
92+
delay(updateIntervalMs)
93+
}
94+
}
95+
}
96+
97+
private fun stopLoop() {
98+
updateJob?.cancel()
99+
// Smoothly close mouth
100+
scope.launch {
101+
targetInfluence = 0f
102+
while (currentInfluence > 0.01f) {
103+
currentInfluence = MathUtils.lerp(currentInfluence, 0f, 0.2f)
104+
updateMorphTargets()
105+
delay(updateIntervalMs)
106+
}
107+
currentInfluence = 0f
108+
updateMorphTargets()
109+
}
110+
}
111+
112+
private fun updateMorphTargets() {
113+
// Reset all known visemes to 0 first (or handle blending if your engine supports it)
114+
activeVisemes.forEach { visemeName ->
115+
avatarNode.setMorphTargetWeight(visemeName, 0f)
116+
}
117+
118+
// Apply current active viseme weight
119+
currentViseme?.let { visemeName ->
120+
avatarNode.setMorphTargetWeight(visemeName, currentInfluence)
121+
}
122+
}
123+
}

‎bundle-report.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4929,7 +4929,7 @@
49294929
</script>
49304930
<script>
49314931
/*<!--*/
4932-
const data = {"version":2,"tree":{"name":"root","children":[{"name":"Avatoon.umd.js","children":[{"name":"src","children":[{"name":"constants/phonemeToViseme.ts","uid":"8ae8bf35-1"},{"name":"components","children":[{"uid":"8ae8bf35-3","name":"AvatoonModel.tsx"},{"uid":"8ae8bf35-5","name":"CameraFovAnimator.tsx"},{"uid":"8ae8bf35-7","name":"Avatoon.tsx"}]},{"uid":"8ae8bf35-9","name":"index.ts"}]}]}],"isRoot":true},"nodeParts":{"8ae8bf35-1":{"renderedLength":539,"gzipLength":0,"brotliLength":0,"metaUid":"8ae8bf35-0"},"8ae8bf35-3":{"renderedLength":7045,"gzipLength":0,"brotliLength":0,"metaUid":"8ae8bf35-2"},"8ae8bf35-5":{"renderedLength":485,"gzipLength":0,"brotliLength":0,"metaUid":"8ae8bf35-4"},"8ae8bf35-7":{"renderedLength":1779,"gzipLength":0,"brotliLength":0,"metaUid":"8ae8bf35-6"},"8ae8bf35-9":{"renderedLength":0,"gzipLength":0,"brotliLength":0,"metaUid":"8ae8bf35-8"}},"nodeMetas":{"8ae8bf35-0":{"id":"/src/constants/phonemeToViseme.ts","moduleParts":{"Avatoon.umd.js":"8ae8bf35-1"},"imported":[],"importedBy":[{"uid":"8ae8bf35-2"}]},"8ae8bf35-2":{"id":"/src/components/AvatoonModel.tsx","moduleParts":{"Avatoon.umd.js":"8ae8bf35-3"},"imported":[{"uid":"8ae8bf35-10"},{"uid":"8ae8bf35-11"},{"uid":"8ae8bf35-12"},{"uid":"8ae8bf35-13"},{"uid":"8ae8bf35-14"},{"uid":"8ae8bf35-0"}],"importedBy":[{"uid":"8ae8bf35-6"}]},"8ae8bf35-4":{"id":"/src/components/CameraFovAnimator.tsx","moduleParts":{"Avatoon.umd.js":"8ae8bf35-5"},"imported":[{"uid":"8ae8bf35-12"},{"uid":"8ae8bf35-11"},{"uid":"8ae8bf35-14"}],"importedBy":[{"uid":"8ae8bf35-6"}]},"8ae8bf35-6":{"id":"/src/components/Avatoon.tsx","moduleParts":{"Avatoon.umd.js":"8ae8bf35-7"},"imported":[{"uid":"8ae8bf35-10"},{"uid":"8ae8bf35-11"},{"uid":"8ae8bf35-12"},{"uid":"8ae8bf35-13"},{"uid":"8ae8bf35-2"},{"uid":"8ae8bf35-4"}],"importedBy":[{"uid":"8ae8bf35-8"}]},"8ae8bf35-8":{"id":"/src/index.ts","moduleParts":{"Avatoon.umd.js":"8ae8bf35-9"},"imported":[{"uid":"8ae8bf35-6"}],"importedBy":[],"isEntry":true},"8ae8bf35-10":{"id":"react/jsx-runtime","moduleParts":{},"imported":[],"importedBy":[{"uid":"8ae8bf35-6"},{"uid":"8ae8bf35-2"}],"isExternal":true},"8ae8bf35-11":{"id":"react","moduleParts":{},"imported":[],"importedBy":[{"uid":"8ae8bf35-6"},{"uid":"8ae8bf35-2"},{"uid":"8ae8bf35-4"}],"isExternal":true},"8ae8bf35-12":{"id":"@react-three/fiber","moduleParts":{},"imported":[],"importedBy":[{"uid":"8ae8bf35-6"},{"uid":"8ae8bf35-2"},{"uid":"8ae8bf35-4"}],"isExternal":true},"8ae8bf35-13":{"id":"@react-three/drei","moduleParts":{},"imported":[],"importedBy":[{"uid":"8ae8bf35-6"},{"uid":"8ae8bf35-2"}],"isExternal":true},"8ae8bf35-14":{"id":"three","moduleParts":{},"imported":[],"importedBy":[{"uid":"8ae8bf35-2"},{"uid":"8ae8bf35-4"}],"isExternal":true}},"env":{"rollup":"4.40.2"},"options":{"gzip":false,"brotli":false,"sourcemap":false}};
4932+
const data = {"version":2,"tree":{"name":"root","children":[{"name":"Avatoon.umd.js","children":[{"name":"src","children":[{"name":"constants/phonemeToViseme.ts","uid":"8a07c42f-1"},{"name":"components","children":[{"uid":"8a07c42f-3","name":"AvatoonModel.tsx"},{"uid":"8a07c42f-5","name":"CameraFovAnimator.tsx"},{"uid":"8a07c42f-7","name":"Avatoon.tsx"},{"uid":"8a07c42f-9","name":"LipSyncAvatoon.tsx"}]},{"uid":"8a07c42f-11","name":"index.ts"}]}]}],"isRoot":true},"nodeParts":{"8a07c42f-1":{"renderedLength":539,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-0"},"8a07c42f-3":{"renderedLength":7045,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-2"},"8a07c42f-5":{"renderedLength":485,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-4"},"8a07c42f-7":{"renderedLength":1779,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-6"},"8a07c42f-9":{"renderedLength":4941,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-8"},"8a07c42f-11":{"renderedLength":0,"gzipLength":0,"brotliLength":0,"metaUid":"8a07c42f-10"}},"nodeMetas":{"8a07c42f-0":{"id":"/src/constants/phonemeToViseme.ts","moduleParts":{"Avatoon.umd.js":"8a07c42f-1"},"imported":[],"importedBy":[{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"}]},"8a07c42f-2":{"id":"/src/components/AvatoonModel.tsx","moduleParts":{"Avatoon.umd.js":"8a07c42f-3"},"imported":[{"uid":"8a07c42f-12"},{"uid":"8a07c42f-13"},{"uid":"8a07c42f-14"},{"uid":"8a07c42f-15"},{"uid":"8a07c42f-16"},{"uid":"8a07c42f-0"}],"importedBy":[{"uid":"8a07c42f-6"}]},"8a07c42f-4":{"id":"/src/components/CameraFovAnimator.tsx","moduleParts":{"Avatoon.umd.js":"8a07c42f-5"},"imported":[{"uid":"8a07c42f-14"},{"uid":"8a07c42f-13"},{"uid":"8a07c42f-16"}],"importedBy":[{"uid":"8a07c42f-6"}]},"8a07c42f-6":{"id":"/src/components/Avatoon.tsx","moduleParts":{"Avatoon.umd.js":"8a07c42f-7"},"imported":[{"uid":"8a07c42f-12"},{"uid":"8a07c42f-13"},{"uid":"8a07c42f-14"},{"uid":"8a07c42f-15"},{"uid":"8a07c42f-2"},{"uid":"8a07c42f-4"}],"importedBy":[{"uid":"8a07c42f-10"}]},"8a07c42f-8":{"id":"/src/components/LipSyncAvatoon.tsx","moduleParts":{"Avatoon.umd.js":"8a07c42f-9"},"imported":[{"uid":"8a07c42f-12"},{"uid":"8a07c42f-13"},{"uid":"8a07c42f-14"},{"uid":"8a07c42f-15"},{"uid":"8a07c42f-16"},{"uid":"8a07c42f-0"}],"importedBy":[{"uid":"8a07c42f-10"}]},"8a07c42f-10":{"id":"/src/index.ts","moduleParts":{"Avatoon.umd.js":"8a07c42f-11"},"imported":[{"uid":"8a07c42f-6"},{"uid":"8a07c42f-8"}],"importedBy":[],"isEntry":true},"8a07c42f-12":{"id":"react/jsx-runtime","moduleParts":{},"imported":[],"importedBy":[{"uid":"8a07c42f-6"},{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"}],"isExternal":true},"8a07c42f-13":{"id":"react","moduleParts":{},"imported":[],"importedBy":[{"uid":"8a07c42f-6"},{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"},{"uid":"8a07c42f-4"}],"isExternal":true},"8a07c42f-14":{"id":"@react-three/fiber","moduleParts":{},"imported":[],"importedBy":[{"uid":"8a07c42f-6"},{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"},{"uid":"8a07c42f-4"}],"isExternal":true},"8a07c42f-15":{"id":"@react-three/drei","moduleParts":{},"imported":[],"importedBy":[{"uid":"8a07c42f-6"},{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"}],"isExternal":true},"8a07c42f-16":{"id":"three","moduleParts":{},"imported":[],"importedBy":[{"uid":"8a07c42f-8"},{"uid":"8a07c42f-2"},{"uid":"8a07c42f-4"}],"isExternal":true}},"env":{"rollup":"4.40.2"},"options":{"gzip":false,"brotli":false,"sourcemap":false}};
49334933

49344934
const run = () => {
49354935
const width = window.innerWidth;

0 commit comments

Comments
 (0)