Multimodal Approach to Sign Language Recognition and Translation Across Indian Languages

Abstract

In this paper, I would like to propose a Simple and how contextually multimodal deep learning for real-time sign language recognition and translation shall affect Indian regional languages, the challenges of linguistic diversity, and social inclusion. Engrained with Multimodal Integration Theory, Cognitive Linguistics, Universal Grammar, and Social Inclusion principles, the study integrates visual, kinematic, and linguistic cues within a deep neural network architecture to enhance recognition accuracy and translation efficacy. In this Paper, I have tried to reuse well-studied knowledge systems of sign languages, transfer learning, and cross-lingual mapping techniques in order to adjust to low-resource sign language variants such as Indian Sign Language (ISL), Tamil Sign Language (TSL), and Kannada Sign Language (KSL). A systematic literature review and bibliometric analysis reveal the evolution of sign language technologies, highlighting the shift from unimodal to multimodal approaches and the emergence of transformer-based models for capturing spatial-temporal dynamics. The proposed system is evaluated using key performance metrics, including accuracy, frame rate, latency, and translation fluency, to ensure real-world applicability and user-centric design. The study aims to bridge the communication gap for India's diverse deaf and hard-of-hearing community, promoting digital accessibility and social inclusion across linguistic and socio-economic boundaries. Future research directions emphasize the integration of edge computing, lightweight architectures, and open-access datasets to enable scalable, low-resource deployment and foster inclusive innovation in assistive technologies.