Reflo
Motion capture pilates personal training with real-time form correction and AI voice cues.
Best experienced on a mobile browser, open in Safari or Chrome on your phone and allow camera access. The live workout screen has a skip button to jump to the last 15 seconds for demo purposes.
Overview
Reflo is a computer vision fitness app that tracks your body in real time and coaches your form through AI voice feedback. It uses MediaPipe Pose to detect joint angles through your device's camera and fires corrections the moment your form breaks. No trainer, no lag, no staring at your screen.
Right now, the phone is the entry point, as it's the most accessible camera most people already own. But the interaction model Reflo is built on (persistent body tracking, spatial overlays, real-time audio feedback tied to movement) is the same foundation that powers AR glasses, smart mirrors, and movement-aware wearables.
This is my first step in building that system accessibly. The goal was to prove the core loop works: track the body, understand the movement, respond with coaching that feels real.
User Flow
How It Works
Body tracking
The browser's getUserMedia API streams the camera feed. MediaPipe Pose detects 33 body landmarks every frame, computing joint positions and angles entirely on-device with no server round-trip.
Spatial overlay
A canvas layer composited over the live feed draws the detected skeleton in real time, joints are highlighted in red, connecting lines showing the body's structure. The visual layer responds to movement as it happens.
Contextual coaching
When a joint angle crosses a threshold, a correction is queued and fires during the next blank gap in the instructor audio track. Positive feedback plays when form is consistently clean. The coaching feels like a trainer in the room.
Detection Logic
Three exercises are tracked in this MVP. Joint angles are computed every frame using dot product math against known landmark positions. A 5-frame rolling average smooths jitter. Corrections queue continuously and fire only during pre-mapped silent windows in the audio track.
Long Stretch
Hip pike > 195°
Drop hips down, squeeze the glutes
Long Stretch
Hip sag < 160°
Squeeze glutes and lift hips back up
Long Stretch
Head drop detected
Curl higher, chin away from chest
The Hundred
Head drop detected
Curl higher, chin away from chest
The Hundred
Arms sinking below hip
Lift arms up, hovering not resting
Footwork Toes
Heels dropping
Keep heels lifted through every rep
Footwork Toes
Knees caving inward
Push knees out over second toe
Wireframes
Every screen was designed before implementation. The full user flow, onboarding through post-workout report, was mapped and reviewed before a single line of code was written.
Tech Stack
Expo SDK 56
App framework + web build
TypeScript
Language
React Navigation
Stack + bottom tabs
getUserMedia API
Browser camera access
MediaPipe Pose
Real-time joint detection
TensorFlow.js
ML runtime
ElevenLabs
Pre-recorded voice corrections
Firebase
Auth + Firestore
Work in progress
Reflo is an active build. The core tracking pipeline, auth flow, and full screen set are complete, but several features are still being refined or are intentionally scoped for a future version.
- —
Firebase auth, Google SSO and email/password
- —
3-step profile onboarding saved to Firestore
- —
10 screens end to end with full navigation
- —
MediaPipe Pose detection with joint angle thresholds
- —
Correction window system, audio queued to blank gaps
- —
Live workout HUD with skeleton overlay and dashed border
- —
5-minute instructor audio track with music and voice
- —
Post workout report with per-exercise rating flow
- —
Weekly plan calendar screen
- —
Vercel web deployment with mobile viewport constraint
- —
Color-coded skeleton overlay so that joints turn red on error, green on clean form
- —
ElevenLabs live API integration, currently testing with pre-recorded clips
- —
Real session data flowing into progress and calendar screens
- —
iOS development build via EAS
- —
Branding polish pass, full font and spacing consistency
- —
AI routine generation on the calendar page
- —
Firestore security rules tightened for production
- —
Performance optimisation for MediaPipe on lower-end devices
- —
WebXR exploration for true spatial overlay on supported devices
Challenges & Learnings
Running ML on-device in the browser
Getting TensorFlow.js and MediaPipe Pose to run reliably required a 5-frame rolling average to smooth landmark jitter, a 4-second cooldown per correction type to prevent spam, and a 0.6 visibility threshold to skip unreliable low-confidence detections entirely.
Timing corrections with the audio track
The hardest product problem was making AI corrections feel natural alongside a real instructor audio track. The solution was a correction window system where MediaPipe assesses continuously but corrections only play during pre-mapped blank gaps, so they never talk over the instructor.
Voice generation strategy
Real-time ElevenLabs API generation is the intended approach as it's the only way to scale to the full range of exercises, users, and corrections without pre-recording thousands of combinations. The current build uses pre-recorded clips while the integration is being explored and refined for the best user experience.
Constraining to side-view only
Rather than trying to support multiple camera angles, the app is designed around a single side-view constraint. This made the joint angle thresholds far more reliable and reduced false positives significantly.