Handpose detection for VR Application

Project Description

The project was a 2-day hack for MLDA@EEE Deep Learning Week where we had to hack an interesting Deep Learning application. We took inspiration from the recent interest in AR/VR technology (this around 1 year since Meta rebranding). We believed that hand-free operation in VR could be useful and thus, we tried to make a prototype to validate the idea.

We integrated

• An on-arm camera using ESP32 camera to upload the video stream via wifi network
• A laptop to get the video stream and running Blender
• A python script to detect hand pose and import that information to Blender to map the fingers keypoints to a human rig.

Final video

Github

Check out our Github Repository