Received: 28 July 2026; Revised: 11 September 2026; Accepted: 17 September 2026; Published Online: 22 September 2026.
J. Biomed. Syst. Eng., 2026, 1(1), 26803 | Volume 1 Issue 1 (September 2026) | DOI: https://doi.org/10.64189/bse.26803
© The Author(s) 2026
This article is licensed under Creative Commons Attribution NonCommercial 4.0 International (CC-BY-NC 4.0)
Real-Time ISL to Text Interpretation and Parkinson’s
Tremor Correction with Augmented Reality Support
Farhana Siddiqui,
Iqra Ansari,
Nusaybah Kazi*
and Jamil Khatri
Department of Computer Engineering, Mohammed Haji Saboo Siddik College of Engineering, Mumbai, Maharashtra, 400008, India
*Email: nusaybah.231215.co@mhssce.ac.in (Nusaybah Kazi)
Abstract
Communication between Indian Sign Language (ISL) users and people who do not understand sign language
can be difficult, whereas involuntary hand movements can make gesture-based interactions more challenging.
This work presents a real-time assistive system that combines ISL alphabet recognition, landmark-based
temporal stabilization, and augmented-reality (AR)-style visual feedback. A webcam provides the input stream,
and MediaPipe Hands obtains 21 three-dimensional hand landmarks from each detected hand. The landmark
representation is converted into 128 × 128 skeleton-based samples and classified using a convolutional neural
network implemented with TensorFlow. The project dataset contains 500 images for each of the 26 ISL alphabet
classes, yielding 13,000 images in total. For the reproducible quantitative evaluation reported here, a stratified
80:20 train test split with random_state = 42 was used, yielding 10,400 training and 2,600 held-out test images.
The CNN achieved 99.27% accuracy, with a macro precision, macro recall, and macro F1 score equal to 99.27%.
Live processing was benchmarked over 100 webcam frames at 108.40 ms per frame, corresponding to
approximately 9.23 FPS. MediaPipe landmark processing accounted for 38.30 ms per frame, and CNN inference
accounted for 69.73 ms per frame. The runtime implementation uses a confidence threshold of 0.85 and a
temporal stability rule requiring 20 consecutive identical predictions before a letter is appended. Shaky-motion
testing was qualitative, and AR-style output was demonstrated in the working prototype. The system is
presented as an assistive recognition prototype and not as a medical treatment or diagnostic system.
Keywords: Indian Sign Language (ISL); Tremor support; Augmented reality (AR); Real-time gesture recognition;
Computer vision; Assistive technology; Hand landmarks.
1. Introduction
Indian sign language (ISL) is a structured visual language used for communication by sign-language users in
India.
[1-3].
Automatic recognition can help reduce communication barriers by converting visible signs into text
and, where needed, speech.
[1,3,4]
Vision-based recognition is attractive because a conventional camera can
capture hand configurations without the use of wearable sensors. Recent work has investigated skeleton and
landmark representations because they preserve hand structure while reducing the dependence on raw RGB
appearance, background clutter, illumination, and other irrelevant visual variations.
[5-8]
This representation is
therefore appropriate for a lightweight real-time pipeline in which the classifier should focus on hand geometry
rather than the complete camera image. Real-time recognition remains challenging because hand position,
signing speed, occlusion, background conditions, and differences between users can affect recognition
performance.
[1,5,9,10]
Involuntary hand movement can introduce short-term fluctuations into the observed
landmark sequence. Computer vision studies have demonstrated that camera-derived motion information can
be used to characterize hand movement and tremor.
[11,12]
In the present work, tremor-related movement is
treated as a recognition-support concern rather than as a clinical measurement or treatment. The implemented
runtime uses temporal stabilization based on 20 consecutive identical predictions; a moving-average filter can
be implemented in the future for detecting tremor movements, as it aligns with the lightweight, real-time and
low-latency requirements of the system. The proposed system integrates hand-landmark extraction, skeleton-
based CNN classification, temporal stabilization, and AR-style visual feedback in one live workflow. The
implementation uses a conventional webcam and a lightweight local processing pipeline so that the system can
be demonstrated on a standard laptop.
1.1 Main contributions