| Journal of Information and Communications Technology:
Algorithms, Systems and Applications
Received: 18 July 2025; Revised: 10 September 2025; Accepted: 16 September 2025; Published Online: 18 September 2025.
J. Inf. Commun. Technol. Algorithms Syst. Appl., 2025, 1(2), 25310 | Volume 1 Issue 2 (September 2025) | DOI: https://doi.org/10.64189/ict.25310
© The Author(s) 2025
This article is licensed under Creative Commons Attribution NonCommercial 4.0 International (CC-BY-NC 4.0)
Magic Learn-DrawInAir: Redefining Creativity,
Problem Solving, Building Worlds with AI- Powered
Gesture Learning
Sudeep Sarkar, Deval Saliya, Hiya Patel and Drashti Shrimal*
Department of Computer Engineering, Thakur College of Engineering and Technology, Mumbai, Maharashtra, 400101, India
*Email: drashti.shrimal@thakureducation.org (Drashti Shrimal)
Abstract
Magic Learn-DrawInAir is an AI-powered educational tool that enables users to draw, solve equations, control
presentations, stream drawings via virtual camera, and interact through a real-time 3D avatar using only hand
gestures and facial tracking, eliminating the need for physical input devices. The system integrates MediaPipe
for real-time hand tracking, OpenCV for virtual canvas rendering, and Streamlit for a user-friendly web interface.
A unique aspect is the use of Google Gemini API, which analyzes gesture-based drawings to solve mathematical
expressions or describe creative visuals. The platform also supports gesture-based navigation of PowerPoint or
PDF slides, making it highly suitable for virtual teaching and learning environments. The platform supports
gesture-based navigation and annotation of PowerPoint or PDF slides, virtual camera output for drawing and
erasing in OBS Studio, Google Meet, and Zoom, and a 3D avatar using MediaPipe FaceMesh for immersive
interaction. Designed to be hardware-independent and cost-effective, the system enhances accessibility and
creativity in education. It offers a futuristic learning experience through intuitive gesture control, facial tracking,
and AI-enhanced understanding. Initial testing confirms the system's efficiency in gesture recognition, drawing
responsiveness, and AI analysis, making it a valuable contribution to smart education and human-computer
interaction.
Keywords: AI in education; Gesture recognition; MediaPipe; OpenCV; Google Gemini; Streamlit; Touchless interaction.
1. Introduction
et al.
et al
Xiao et al.
[17]
reported utilization
wearable rings and wrist sensors to track finger movements with high precision. While innovative, the approach
depends on specialized wearable devices, which may not be practical for widespread adoption due to cost and
accessibility issues.et al
et al
et al
Dupréet
al
et al
et al.
et al
Magic Learn – DrawInAir
•
•
•
•
•
• Slides are automatically converted to high- resolution images using LibreOffice or PyMuPDF.
• Navigation is enabled through pointing gestures on- screen arrows.
• Users can annotate directly on the slide using draw/erase gestures, maintaining interactivity during
presentations.
•
•
•
•
•
5. Conclusion
The Magic LearnDrawInAir system successfully demonstrates the potential of gesture-based, AI-powered
educational tools that operate without the need for specialized hardware. By integrating MediaPipe for real-
time hand tracking, OpenCV for virtual drawing and erasing, and the Google Gemini API for intelligent
interpretation of visual content, the system enables users to draw, erase, analyze, and navigate presentations
using only hand gestures and a standard webcam. Quantitative evaluation confirms the syst
achieving an average gesture recognition accuracy of 85.3%, an average latency of 142 milliseconds, and an
overall user satisfaction score of 4.6/5 across pilot tests with 30 participants (including educators and
students). These metrics v -time educational
applications. The study effectively addresses key gaps in accessibility, cost-effectiveness, and interactivity in
modern EdTech by eliminating the dependence on hardware such as styluses or smartboards. It delivers a
hardware-independent, hands-free learning environment ideally suited for remote education, digital
classrooms, and assistive learning contexts. The integration of AI-based real-time content analysis further
enriches the learning experience, allowing users to engage with educational materials in an intelligent and
intuitive way. Empirical results highlight that the system not only simplifies interaction with digital content but
also enhances engagement and learning efficiency by approximately 30% compared to traditional input
methods. This innovative approach paves the way for broader applications in AR/VR-based education, creative
design, and inclusive technology. Moving forward, the system can be enhanced through voice command support,
customizable gestures, and multi-user collaboration to create an even more immersive, adaptive, and intelligent
learning platform.
Funding Declaration
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-
profit sectors.
Data Availability Statement
The datasets generated and/or analyzed during the current study that support the findings are available from
the corresponding author upon reasonable request.
Conflict of Interest
The authors received no specific financial support for the research, authorship, or publication of this work.
Artificial Intelligence (AI) Use Disclosure
The authors declare that artificial intelligence (AI)-assisted tools were used only for language refinement,
grammar improvement, and manuscript structuring purposes during the preparation of this work. All technical
content, experimental implementation, results, and interpretations were independently developed and verified
by the authors.
References
[1]
A. Haleem, M. Javaid, M. S. Qadri, R. Suman, Understanding the role of digital technologies in education:
A review, Sustainable Operations and Computers, 2022, 30, 275-285, doi: 10.1016/j.susoc.2022.05.004.
[2]
D. Mhlanga, Digital transformation of education, the limitations and prospects of introducing the fourth
industrial revolution asynchronous online learning in emerging markets, Discover Education, 2024, 3,
32, doi: 10.1007/s44217-024-00115-9.
[3]
idea that digital remote learning can happen anytime, anywhere
in forced online teacher education is a myth, Technology, Knowledge and Learning, 2023, 28, 1461
1484, doi: 10.1007/s10758-023-09685-3.
[4]
A. Forkosh-Baruch, J. Voogt, G. Knezek, Moving forward to new educational realities in the digital era:
an international perspective, Technology, Knowledge and Learning, 2024, 29, 16851691, doi:
10.1007/s10758-024-09785-8.
[5]
A. Haleem, M. Javaid, M. Asim Qadri, R. Suman, Understanding the role of digital technologies in
education: A review, Sustainable Operations and Computers, 2022, 3, 275-285, doi:
10.1016/j.susoc.2022.05.004.
[6]
E. Dritsas, M. Trigka Methodological and technological advancements in e-learning, Information, 2025,
16, 56, doi: 10.3390/info16010056.
[7]
O. Ali, P. A. Murray, M. Momin, Y. K. Dwivedi, T. Malik, The effects of artificial intelligence applications in
educational settings: Challenges and strategies, Technological Forecasting and Social Change, 2024,
199, 123076, doi: 10.1016/j.techfore.2023.123076.
[8]
M. Herczeg, The role of digital technologies and Human-Computer Interaction for the future of
education, i-com, 2024, 23, 239-247, doi: 10.1515/icom-2024-0008.
[9]
W. Strielkowski, V. Grebennikova, A. Lisovskiy, G. Rakhimova, T. Vasileva, AI-driven adaptive learning
for sustainable educational transformation, Sustainable Development, 2025, 33, 1921-1947, doi:
10.1002/sd.3221.
[10]
S. Zhao, Exploring how interactive technology enhances gesture-based expression and engagement: a
design study, Multimodal Technologies and Interaction, 2019, 3, 13, doi: 10.3390/mti3010013.
[11]
W. Chang, A. Aberash, Embodied learning in the digital age: assessing the impact of gesture-based
educational technology on working memory capacity, spatial reasoning, and engagement, BMC
Psychology, 2026, 14(1), 331, doi: 10.1186/s40359-026-04089-1.
[12]
R. P. Sharma, G. K. Verma, Human computer interaction using hand gesture, Procedia Computer Science,
2015, 54, 721-727, doi: 10.1016/j.procs.2015.06.085.
[13]
Yaseen, Real-time face gesture-based robot control using ghostnet in a unity simulation environment,
Sensors, 2025, 25, 6090, doi: 10.3390/s25196090.
[14]
J. Qi, L. Ma, Z. Cui, Y. Yu, Computer vision-based hand gesture recognition for human-robot interaction:
a review, Complex & Intelligent Systems, 2024, 10, 15811606, doi: 10.1007/s40747-023-01173-6.
[15]
Bertolasi J, Garcia-Hernandez NV, Memeo M, Guarischi M, Gori M. Evaluation of HoloLens 2 for Hand
Tracking and Kinematic Features Assessment. Virtual Worlds, 2025, 4, 31, doi:
10.3390/virtualworlds4030031.
[16]
D, M. Mulla, N. Majoni, P. M. Tilley, P. J. Keir, Two cameras can be as good as four for markerless hand
tracking during simple finger movements, Journal of Biomechanics, 2025, 181, 112534, doi:
10.1016/j.jbiomech.2025.112534.
[17]
Y. Xiao, Z. Huang, Y. Gao, From wrist to finger: hand pose tracking using ring-watch wearables. in
proceedings of the extended abstracts of the CHI Conference on Human Factors in Computing Systems
(CHI EA '25). Association for Computing Machinery, New York, NY, USA, 2025, 294, 17, doi:
10.1145/3706599.3720220.
[18]
T. R. Gadekallu, G. Srivastava, M. Liyanage, Iyapparaja M., C. L. Chowdhary, S. Koppu, P. K. Reddy
Maddikunta, Hand gesture recognition based on a Harris Hawks optimized convolution neural
network, Computers and Electrical Engineering, 2022, 100, 107836, doi:
10.1016/j.compeleceng.2022.107836.
[19]
A. Sen, T. K. Mishra, R. Dash, A novel hand gesture detection and recognition system based on ensemble-
based convolutional neural network, Multimedia Tools and Applications, 2022, 8, 4004340066, doi:
10.1007/s11042-022-11909-0.
[20]
A. S. Mohamed, N. F. Hassan, A. S. Jamil, Real-time hand gesture recognition: a comprehensive review
of techniques, applications, and challenges, Cybernetics and Information Technologies, 2024, 24, 163
181, doi: 10.2478/cait-2024-0031.
[21]
C. Dupré, C.Appert, S.Rey, H. Saidi, E. Pietriga, TriPad: Touch input in AR on ordinary surfaces with hand
tracking only, In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
(CHI '24). Association for Computing Machinery, New York, NY, USA, 2024, 754, 118, doi:
10.1145/3613904.3642323.
[22]
Z. Hao, Z. Sun, F. Li, R. Wang, J. Peng, Millimeter wave gesture recognition using multi-feature fusion
models in complex scenes, Scientific Reports, 2024, 14, 13758, doi: 10.1038/s41598-024-64576-6.
[23]
M. Jonsson, J. Tholander, Cracking the code: Co-coding with AI in creative programming education. In
Proceedings of the 14th Conference on Creativity and Cognition (C&C '22). Association for Computing
Machinery, New York, NY, USA, 2022, 514, doi: 10.1145/3527927.3532801.
[24]
Y. Lei, Y. Deng, L. Dong, X. Li, X. Li X, Z. Su, A novel sensor fusion approach for precise hand tracking in
virtual reality-based humancomputer interaction, Biomimetics, 2023, 8, 326, doi:
10.3390/biomimetics8030326.
[25]
Y. Zhang, J. Wang, X. Wang, H. Jing, Z. Sun, Y. Cai, Static hand gesture recognition method based on the
Vision Transformer, Multimedia Tools and Applications, 2023, 82, 3130931328, doi: 10.1007/s11042-
023-14732-3.
Publisher Note: The views, statements, and data in all publications solely belong to the authors and
contributors. GR Scholastic is not responsible for any injury resulting from the ideas, methods, or products
mentioned. GR Scholastic remains neutral regarding jurisdictional claims in published maps and institutional
affiliations.
Open Access
This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which
permits the non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format,
as long as appropriate credit to the original author(s) and the source is given by providing a link to the Creative
Commons License and changes need to be indicated if there are any. The images or other third-party material
in this article are included in the article's Creative Commons License, unless indicated otherwise in a credit line
to the material. If material is not included in the article's Creative Commons License and your intended use is
not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly
from the copyright holder. To view a copy of this License, visit: https://creativecommons.org/licenses/by-
nc/4.0/
© The Author(s) 2025
