Open Access||Research ArticleMagic Learn-DrawInAir: Redefining Creativity, Problem Solving, Building Worlds with AI-Powered Gesture Learning
Department of Computer Engineering, Thakur College of Engineering and Technology, Mumbai, Maharashtra, 400056, India
Abstract
Magic Learn-DrawInAir is an AI-powered educational tool that enables users to draw, solve equations, control presentations, stream drawings via virtual camera, and interact through a real-time 3D avatar using only hand gestures and facial tracking, eliminating the need for physical input devices. The system integrates MediaPipe for real-time hand tracking, OpenCV for virtual canvas rendering, and Streamlit for a user-friendly web interface. A unique aspect is the use of Google Gemini API, which analyzes gesture-based drawings to solve mathematical expressions or describe creative visuals. The platform also supports gesture-based navigation of PowerPoint or PDF slides, making it highly suitable for virtual teaching and learning environments. The platform supports gesture-based navigation and annotation of PowerPoint or PDF slides, virtual camera output for drawing and erasing in OBS Studio, Google Meet, and Zoom, and a 3D avatar using MediaPipe FaceMesh for immersive interaction. Designed to be hardware-independent and cost-effective, the system enhances accessibility and creativity in education. It offers a futuristic learning experience through intuitive gesture control, facial tracking, and AI-enhanced understanding.
Graphical Abstract
Novelty Statement
The Magic Learn-DrawInAir system successfully demonstrates the potential of gesture-based, AI-powered educational tools that operate without specialized hardware.