Applying Swin Architecture to diverse Sign Language Datasets
Click the Poster to View Full Screen, Right click to save image
Grant: Students Partnering with Faculty (SpF)
Annaliese Watson
Co-PIs:
Chin-Chien Lin
College:
The Dorothy and George Hennings College of Science, Mathematics, and Technology
Major:
Computational Science & Engineering
Faculty Research Advisor(s):
Yulia Kumar, Kuan Huang, J. Jenny Li
Abstract:
In the era of Artificial Intelligence (AI), the ability to seamlessly comprehend and respond to non-verbal communication is paramount. This research broadens the scope of AI in reducing the communication gap, contributing significantly to both American Sign Language (ASL) and Taiwan Sign Language (TSL) communities. Central to this study is the application of various AI models, with a primary focus on the Hierarchical Vision Transformer using Shifted Windows (Swin) models, to recognize diverse sign language datasets. The study evaluates the Swin architecture's adaptability to the unique aspects of different sign languages, aiming to establish a universal platform for 'Unvoiced' communities. By leveraging deep learning and transformer technologies, we developed a mobile application prototype that enables ASL-to-English translations and vice versa. The goal is to extend this capability to multiple sign languages. Trained on datasets of varying sizes, the Swin models demonstrate notable accuracy, showcasing their flexibility and effectiveness. This research highlights significant technological advancements in sign language recognition and reaffirms our commitment to inclusive communication in the digital age. Future efforts will concentrate on refining these models and expanding their application to encompass a broader range of sign languages, thus promoting global inclusivity.