AI Powered Gesture Control for Public Touchscreens

Abstract

Public touchscreen devices such as information kiosks, ticketing machines, and digital terminals have become a common part of everyday life, but their frequent use by different people also raises concerns about hygiene, accessibility, and the need for physical contact. Although touchless interaction can address these concerns, creating a gesture-based system that works reliably in a public environment is challenging because natural hand movements, changes in lighting, background distractions, and unintended gestures can lead to incorrect actions. To address this problem, this work presents an AI-powered, vision-based gesture control system designed to provide a touchless alternative to conventional public touchscreen interaction using only a standard webcam. The system uses Google's MediaPipe Hand Landmarker to track 21 hand landmarks in real time and converts simple hand gestures into actions such as cursor movement, left and right clicking, scrolling, dragging, and user-defined commands. To make the interaction more stable and dependable, the system combines OneEuro filtering to reduce hand jitter, confidence-based intent detection and velocity gating to prevent accidental actions, and a physics-based cursor model to produce smoother and more natural pointer movement. A gesture learning module also allows users to define personalized gestures without retraining the underlying model, while context-aware actions enable the same gesture to perform different functions depending on the active application. The developed system demonstrates that a software-only approach using a regular webcam can provide responsive and practical touchless interaction without requiring specialized sensors or modifications to existing terminal hardware. The results show its potential as a more hygienic, accessible, and user-friendly interaction method for public-facing computing environments.

Keywords

Graphical Abstract

Novelty Statement

A webcam-based AI gesture system enables hygienic touchless public-terminal control using adaptive learning, filtering, confidence detection, and context-aware actions.