The Challenge

Prompt from Johnson & Johnson

“How may we enhance the recovery experience for patients after orthopedic surgery?”

Context summary

• ACL recovery is quite long(6-12 months).

• Many patients lose motivation and skip exercises.

• Physical therapists lack real-time insight into at-home performance.

  
  

Research Methodology:

Methods

Expert Interviews: Quick 15-minute calls with a physical therapist and Orthopedic surgeon (Dr.

Patient Stories: Analyzed online ACL recovery forums for emotional pain points (motivation loss, lack of feedback, fear of reinjury).

Insights

• Patients need visible progress, feedback, and motivation - not just instructions.

• Clinicians would benefit from exercise data automation.

Daniela Lopez: Suffered ACL injury during Soccer Match

Our Solution

Core Concept:

A computer-vision driven recovery assistant that empowers patients and clinicians through visual feedback, measurable milestones, and rewards.

Key Performance Indicators:

Exercise Adherence Rate: % of prescribed sessions completed (target ≥ 85%)

• Tracked automatically via camera based rep detection.

Recovery Milestone Achievement: % of clinical goals met on schedule (target ≥ 90%)

• Based on ROM, gait quality, and pain-free movement metrics.

Patient Motivation Index: Derived from app engagement, consistency streaks, and daily surveys.

Features

1. Real-Time Exercise Tracking

2. Range of Motion Analytics

• Computer vision detects key joints and tracks every repetition, building consistent form habits.

• Progress visualized over time to show measurable gains and recovery confidence.

3. 3D Pose Review for Clinicians

• Clinicians inspect specific frames and angles to adjust plans remotely.

• AI flags regressions or unsafe deviations in real time.

4. Motivational Milestones & Rewards

Badges:

• Get Rid of your Crutches: (Week 4) patient can let go of crutches

• Drive Again: (Week 6) Approval Required

• Plank Master: (Week 12) Approval Requested

  
  

Brand tie-ins: (CAVA, Smoothie King).

• Behavioral design principles encourage consistency through (micro-rewards.)

Technical Architecture

Contribution:

• I helped integrate the computer vision backend into Unreal Engine through a python server, created data smoothing and threshold logic with Cursor, then transferred analysis through VA Rest Plugin.

• Ui creation / Integration into Unreal Engine, Presentation.

Impact & Results

Quantitative Impact:

• 85% of test users reported feeling more “motivated” and “in control” of their recovery.

• Clinicians noted faster progress documentation and fewer manual check-ins. (Paul, PT) (Dr. Sutherland)

Qualitative Impact:

• “Seeing how my knee was improving real time kept me consistent.” - Krithin R (Test Participant)

My Role

Role: XR/AI Developer + Experience Designer

Team: 4 members (UX, developer, biomedical researcher, product manager)

Tools: Unity, Python (FastAPI), OpenCV, MoveNet, Figma

Contributions:

• Co CV pipeline (camera - cv model - python - Unreal Engine).

• Designed the UX for milestone tracking + motivational system.

• Researched user needs, recovery pipeline, milestones, Expert Interviews

• Final pitch & demo presentation to Johnson & Johnson judges.

Team

Annushka Thakur, Krithin Rajesh, Aidan Dougan, Lars Lagenbach, Zayne Loy, Anna Morris, Valentina Rosas

Reflection

This project reinforced how powerful computer vision can be when paired with human-centered design. I learned to combine emerging technology while keeping patient empathy as a priority, turning abstract metrics into visible progress. Working on a healthcare challenge for Johnson & Johnson has ignited my curiosity and passion for designing tools that heal through experience design, not just technology.

Hack IX 2025 - 1st Place Winner Team
Sponsored by Johnson & Johnson • Awarded for “Best Future of Healthcare”