sign language translated into natural speech in real time
Custom-built dual-hand smart gloves equipped with EMG, flex, and IMU sensors that capture sign language gestures with high fidelity—processed by a machine learning pipeline with AI grammar correction to produce natural, coherent sentences in near real-time.
Specifications
Sensors Per Glove
7 Flex + EMG + IMUTotal Sensor Channels
14 Dual-Hand InputClassification
ML Scikit-Learn PipelineCompanion App
Android Control & CalibrateEnd-to-End Pipeline
Gesture Capture
Flex sensors detect finger bend angles, EMG sensors read muscle activation, and IMUs track hand orientation—all sampled at high frequency across both hands simultaneously.
Signal Processing
Raw sensor data is filtered, normalized, and converted into feature vectors. The C++ firmware on the microcontroller handles real-time signal conditioning before transmission.
ML Classification
A Scikit-Learn model classifies gesture feature vectors into ASL signs. The model was trained on a custom-collected dataset of dual-hand gesture sequences.
Grammar & Speech
An AI-powered grammar correction module converts raw sign sequences into natural, coherent English sentences, then synthesizes speech output in near real-time.
Real-World Impact
Deaf & Hard-of-Hearing Communication
Enables seamless, real-time communication between sign language users and non-signers without an interpreter, breaking down one of the biggest daily barriers.
Education & Classrooms
Allows deaf students to participate in mainstream classrooms with live speech translation of their signing, promoting inclusive education.
Healthcare Access
Empowers deaf patients to communicate directly with healthcare providers during emergencies and consultations without relying on third-party interpreters.
Assistive Technology Research
Advances the field of wearable AI by demonstrating a full end-to-end system—from sensor capture to natural language output—published as peer-reviewed research.