Journal of Computers, Mechanical and Management

S. Sarkar, D. Saliya, H. Patel, D. Shrimal, Magic Learn-DrawInAir: redefining creativity, problem solving, building worlds with AI- powered gesture learning, Journal of Information and Communications Technology: Algorithms, Systems and Applications, 2025, 1(2), 25310, doi: https://doi.org/10.64189/ict.25310.

Rights and permissions

© The Author(s) 2025.

Open Access

This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits the non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as appropriate credit to the original author(s) and the source is given by providing a link to the Creative Commons License and changes need to be indicated if there are any. The images or other third-party material in this article are included in the article's Creative Commons License, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons License and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this License, visit: httpshttps://creativecommons.org/licenses/by-nc/4.0/

Abstract

Magic Learn-DrawInAir is an AI-powered educational tool that enables users to draw, solve equations, control presentations, stream drawings via virtual camera, and interact through a real-time 3D avatar using only hand gestures and facial tracking, eliminating the need for physical input devices. The system integrates MediaPipe for real-time hand tracking, OpenCV for virtual canvas rendering, and Streamlit for a user-friendly web interface. A unique aspect is the use of Google Gemini API, which analyzes gesture-based drawings to solve mathematical expressions or describe creative visuals. The platform also supports gesture-based navigation of PowerPoint or PDF slides, making it highly suitable for virtual teaching and learning environments. The platform supports gesture-based navigation and annotation of PowerPoint or PDF slides, virtual camera output for drawing and erasing in OBS Studio, Google Meet, and Zoom, and a 3D avatar using MediaPipe FaceMesh for immersive interaction. Designed to be hardware-independent and cost-effective, the system enhances accessibility and creativity in education. It offers a futuristic learning experience through intuitive gesture control, facial tracking, and AI-enhanced understanding. Initial testing confirms the system's efficiency in gesture recognition, drawing responsiveness, and AI analysis, making it a valuable contribution to smart education and human-computer interaction.

Graphical Abstract

Novelty statement

The Magic Learn-DrawInAir system successfully demonstrates the potential of gesturebased, AI- powered educational tools that operate without specialized hardware.