Human Pose Detection Demo with Expo Modules

A modern React Native app demonstrating real-time human pose detection using custom Expo modules, MediaPipe, and React Native Vision Camera. This project showcases how to build cross-platform mobile applications with advanced computer vision capabilities for Android, iOS, and Web.

Features

Real-time Pose Detection: Live human pose estimation with 33 keypoints using MediaPipe Pose Landmarker
Custom Expo Module: Native pose detection module with TypeScript support
Camera Integration: React Native Vision Camera with frame processing capabilities
Visual Overlay: Real-time skeleton rendering with customizable line and circle drawing
Cross-platform: Works on Android, iOS, and Web platforms
Authentication Flow: Login, logout, and user state management with Redux Toolkit
Navigation: Bottom tab navigation with Home and Profile screens using React Navigation
Custom Theming: Light and dark themes powered by Unistyles
Localization: Built-in support for English and Spanish (i18n-js, expo-localization)
Reusable Components: Button, TextField, FullScreenLoader, Toast notifications, and more
API Integration: Axios-based API client with RTK Query for authentication and user endpoints
Persistent State: Redux state persistence using redux-persist and MMKV storage
Custom Fonts: Open Sans font family included and pre-configured
Responsive Design: Breakpoints and scaling utilities for adaptive layouts
Expo EAS Ready: Pre-configured for EAS build and deployment

Pose Detection Capabilities

33 Keypoints: Full body pose detection including face, arms, legs, and torso
Real-time Processing: Frame-by-frame pose estimation with minimal latency
Visual Feedback: Toggle-able skeleton overlay with customizable colors and styles
Camera Controls: Front/back camera switching and permission handling
Performance Optimized: Efficient frame processing using worklets and shared values

Reference Implementation

This project is inspired by the excellent work in the Movement repository by Ailson Freire, which demonstrates advanced pose detection techniques and visual overlays.

Project Structure

reactnativetemplateapp/
  ├── android/           # Native Android project
  ├── ios/               # Native iOS project
  ├── modules/           # Custom Expo modules
  │   └── expo-pose-detection/  # Pose detection module
  │       ├── android/   # Android native implementation
  │       ├── ios/       # iOS native implementation
  │       ├── src/       # TypeScript module interface
  │       └── expo-module.config.json
  ├── src/
  │   ├── assets/        # Fonts and images
  │   ├── components/    # Reusable UI components
  │   ├── constants/     # App-wide constants
  │   ├── localization/  # i18n setup and translations
  │   ├── navigation/    # Navigation stacks and tabs
  │   ├── redux/         # Redux store, slices, services
  │   ├── screens/       # App screens (Home with pose detection, Login, Profile)
  │   │   └── Home/      # Pose detection implementation
  │   │       ├── Home.js           # Main camera and pose display
  │   │       └── usePosedetection.ts # Pose detection hook
  │   ├── storage/       # Storage utilities
  │   ├── styles/        # Theming and breakpoints
  │   ├── theme/         # Font and text styles
  │   └── utils/         # Helper functions
  ├── App.js             # App entry point
  ├── app.json           # Expo app config
  ├── package.json       # Project dependencies and scripts
  └── ...

Pose Detection Module Architecture

The expo-pose-detection module provides a bridge between React Native and native pose detection capabilities:

Native Implementation: Android (Kotlin) and iOS (Swift) modules using MediaPipe
TypeScript Interface: Type-safe API for pose landmark detection
Event-driven: Real-time pose data through event listeners
Cross-platform: Unified API across Android, iOS, and Web platforms

Getting Started

Prerequisites

Node.js (v18 or newer recommended)
Yarn or npm
Expo CLI

Installation

Clone the repository:

git clone <your-repo-url>
cd reactnativetemplateapp

Install dependencies:
```
yarn install
# or
npm install
```
Start the development server:
```
yarn start
# or
npm run start
```

Running on Devices

Android:
```
yarn android
```
iOS:
```
yarn ios
```
Web:
```
yarn web
```

Build Scripts

yarn build:android:staging / yarn build:android:development / yarn build:android:production
yarn build:ios:staging / yarn build:ios:development / yarn build:ios:production

See package.json for all available scripts.

Theming & Customization

Uses react-native-unistyles for adaptive theming (light/dark).
Edit src/styles/themes.ts to customize colors.
Responsive breakpoints in src/styles/breakpoints.ts.

Localization

English and Spanish translations in src/localization/translations/.
Add more languages by extending the translation files and updating src/localization/i18n.js.

API & State Management

API base URL is set via environment variables (EXPO_PUBLIC_BASE_URL).
Authentication endpoints: /auth/login, /auth/register, /auth/logout.
Redux Toolkit Query for API calls, with persistent state using MMKV.

Custom Components

Button: Primary, secondary, and disabled styles, loading state, icons.
TextField: Customizable input with optional icons.
FullScreenLoader: Modal loading indicator.
ToastAlert: Success and error toast notifications.
ScreenWrapper: Handles safe area, scroll, and loading overlay.

Fonts

Open Sans font family included in src/assets/fonts/ and loaded via src/theme/fonts.js.

Pose Detection Usage

Basic Implementation

import {
  addPoseLandmarksListener,
  addPoseStatusListener,
  addPoseErrorListener,
  type PoseLandmark,
} from "../modules/expo-pose-detection";

// Subscribe to pose detection events
const landmarksSubscription = addPoseLandmarksListener((event) => {
  const landmarks = event.landmarks[0]; // 33 keypoints array
  // Process pose data
});

const statusSubscription = addPoseStatusListener((event) => {
  console.log("Pose detection status:", event.status);
});

const errorSubscription = addPoseErrorListener((event) => {
  console.error("Pose detection error:", event.error);
});

// Cleanup listeners
landmarksSubscription.remove();
statusSubscription.remove();
errorSubscription.remove();

Custom Hook Usage

import { usePoseDetection } from "./usePosedetection";

function PoseDetectionScreen() {
  const { landmarks, status, error, poseCount } = usePoseDetection();

  return (
    <View>
      <Text>Status: {status}</Text>
      <Text>Pose Count: {poseCount}</Text>
      {error && <Text>Error: {error}</Text>}
    </View>
  );
}

Technologies Used

Core Technologies

Expo - Development platform and tools
React Native - Cross-platform mobile framework
Expo Modules - Custom native modules
MediaPipe - Machine learning framework for pose detection

Camera & Vision

React Native Vision Camera - Camera library with frame processing
Shopify Skia - 2D graphics library for pose visualization
React Native Worklets Core - High-performance frame processing

State Management & Navigation

Redux Toolkit - State management
React Navigation - Navigation library
React Redux - React bindings for Redux

UI & Styling

Unistyles - Adaptive theming system
i18n-js - Internationalization
MMKV Storage - Fast key-value storage

API & Networking

Axios - HTTP client
RTK Query - Data fetching and caching

Environment Variables

Configure API endpoints in .env.* files (see eas.json for examples):
- EXPO_PUBLIC_BASE_URL
- EXPO_PUBLIC_SOCKET_URL

Pose Detection Features

Keypoint Detection

The app detects 33 keypoints representing different body parts:

Face: 0-10 (nose, eyes, ears, mouth)
Upper Body: 11-16 (shoulders, elbows, wrists)
Lower Body: 23-32 (hips, knees, ankles, feet)

Visual Controls

Toggle Lines: Show/hide skeleton connections between keypoints
Toggle Circles: Show/hide keypoint markers
Camera Switch: Toggle between front and back cameras
Real-time Overlay: Live pose visualization on camera feed

Performance Features

Frame Processing: Efficient worklet-based frame processing
Shared Values: Optimized data sharing between JS and native threads
Event-driven Architecture: Real-time pose updates through event listeners

Development Notes

Camera Permissions

The app requires camera permissions to function. Make sure to grant camera access when prompted.

Platform Differences

iOS: Uses RGB pixel format for front camera, YUV for back camera
Android: Uses YUV pixel format by default
Performance: Frame processing optimized for each platform

Troubleshooting

Ensure camera permissions are granted
Check that the pose detection model is properly loaded
Verify MediaPipe dependencies are correctly installed

Build & Deployment

GitHub Actions Workflows

This project includes an automated GitHub Actions workflow for building and releasing APK files:

Available Workflow

build-android-apk.yml - Build & Auto Release Android APK

Manual dispatch only with build type selection (development, staging, production)
Automatic version tagging and GitHub release creation
APK artifacts uploaded to GitHub releases
Build logs and changelog generation
Environment-specific builds with proper naming

Features:

Supports development, staging, and production builds
Automatic version increment based on git tags
GitHub release creation with changelog
APK artifacts available for 30 days
Build logs uploaded for troubleshooting

Usage

Manual: Go to Actions tab → Select "Build & Auto Release Android APK" → Run workflow
Build Types: Choose from development, staging, or production
Download: APK available in GitHub releases section

See .github/workflows/README.md for detailed workflow documentation.

Local APK Building

For local APK builds, you can use the standard React Native build process:

# Install dependencies
yarn install

# Setup Expo
npx expo prebuild --platform android

# Build APK
cd android
./gradlew assembleRelease

EAS Build

Pre-configured for Expo Application Services (EAS).
See eas.json for build profiles.
Custom modules require development builds (not Expo Go)

License

This project is provided as a template and does not include a license by default. Add your own license as needed.

Acknowledgments

Special thanks to Ailson Freire for the inspiring Movement project that served as a reference for this implementation.

Name		Name	Last commit message	Last commit date
Latest commit History 48 Commits
.github/workflows		.github/workflows
android		android
assets		assets
ios		ios
modules/expo-pose-detection		modules/expo-pose-detection
patches		patches
plugins		plugins
src		src
.env.development		.env.development
.env.production		.env.production
.env.staging		.env.staging
.gitignore		.gitignore
App.js		App.js
README.md		README.md
app.json		app.json
babel.config.js		babel.config.js
eas.json		eas.json
index.js		index.js
jsconfig.json		jsconfig.json
metro.config.js		metro.config.js
package.json		package.json
tsconfig.json		tsconfig.json
yarn.lock		yarn.lock

mantu-bit/Expo-React-native-pose-detection-demo

Folders and files

Latest commit

History

Repository files navigation