GestureVerse

Choose-your-own-adventure Game using Hand Gesture Classification and Detection

Keynote and Demo · Report Bug · Request Feature

Table of Contents

1. Wireframes
2. About The Project
   • Features
   • Built With
3. Getting Started
   • Prerequisites
   • Installation
4. Works Cited
5. Contact

Disclaimer

I do not own the stories included in this project--this is the gesture-driven version of the original Twine/hyperlink-based stories. The sources for the original stories are included in the reference section at the end of the stories.

Wireframes

Link to the wireframes for the website.

About The Project

This is Next.js application that guides users through a choose-your-own-adventure game using Windrift. During the game, instead of inputting text or clicking on buttons to proceed like usual, users can interact using hand movements through the devices’ web camera to answer questions prompted on the screen. For example, a thumbs-up gesture would mean “yes”, and a thumbs-down gesture would mean “no”. Handedness (i.e. left or right hand) can also be used to provide input to questions, depending on the story chosen.

Included in this project are 2 stories: a beach walk and procrastinate. Two stories are rather text-based (though a beach walk includes more visual materials than the other).

Through this project, I also want to introduce additional advanced features compared to the original Windrift JS framework:

• ChoiceBlock component that involves making choice based on user's gesture

  • Navigate based on Google MediaPipe model's classification classes (gestures are accepted only if the prediction confidence score is > 0.6):
    • "None": "No action",
    • "Closed_Fist": "Close your fist ✊",
    • "Open_Palm": "Show your palm 🖐️",
    • "Pointing_Up": "Point your (index) finger upwards ☝️",
    • "Thumb_Down": "Put your thumb down 👎",
    • "Thumb_Up": "Put your thumb up 👍",
    • "Victory": "Make a V sign ✌️",
    • "ILoveYou": "Do the I Love You sign 🤟 (you can try doing 🫶 to see what happens)"
  • Navigate based on handedness
  • Display category, confidence score, and handedness in real-time
  • Adjust the number of hands the model can predict for.

• Additional configuration to the original Choice, Nav classes to allow custom handling, add custom styling, ...

• A variable management system that allows keeping track of variables within the story.

  • Score keeping feature
  • Update/access the variables any time during the story

• Audio/music playlist for stories (play, pause, next, previous control)

(back to top)

Progress

• Wireframes
• Set up game in Windrift
• Integrate MediaPipe
  • Set up MediaPipe component to listen to events of the story
  • Refactor instructions and camera into a component (called game block)
  • Set up game blocks to listen to each other
  • Enhance the UI
    • Add animations
    • Add gradient background config setup
• Finish story (edit UI, texts, citations, ...)
  • A beach walk (View demo)
  • Procrastinate (View demo)
  • Congee (View demo)
• Add background music widget

See the open issues for a full list of proposed features (and known issues).

(back to top)

Built With

• NodeJS
• React v18.3.1
• MediaPipe (Tasks-vision: v0.10.14)
  • Gesture Recognition
  • Hand landmark detection
• Next.js v13.5.6
• React Bootstrap v2.10.4
• Ionic React v8.2.6

For complete list, please view the package tree in requirements.txt.

(back to top)

Prerequisites

• Node 18.20.3
• [email protected]
• [email protected]
• @mediapipe/[email protected]
• [email protected]
• [email protected]
• [email protected]
• [email protected]
• @ionic/[email protected]
• @ionic/[email protected]

For complete list, please view the package tree in requirements.txt.

(back to top)

Installation

1. Clone the repo

git clone https://github.com/LynnHaDo/GestureVerse.git

According to the Windrift repo:

2. Install a fully-compaticble version of Node:

nvm install 18
nvm use 18

3. Install packages

npm install

4. Run the local development environment

npm run dev

(back to top)

Works cited

1. interstellar-bird (n.d.). Beach Walk by interstellar-bird. [online] itch.io. Available at: https://interstellar-bird.itch.io/beach-walk [Accessed 16 Jul. 2024].
2. j-mo (2024). Procrastinate. [online] itch.io. Available at: https://j-mo.itch.io/procrastinate [Accessed 6 Aug. 2024].
3. Becci (2020). Congee. [online] itch.io. Available at: https://becciness.itch.io/congee [Accessed 12 Aug. 2024].

(back to top)

Contact

Email: [email protected]