SignLink: Machine Learning Model to Detect Sign Language
Mar 1, 2024 | Written By Sophia Mai
Transcript:
Sophia: Communications should be available to everyone, not the select view. Signlink aims to address one crucial issue regarding accessibility for the deaf and hard-of-being community through the use of AI.
My project is a computer vision and machine learning model to detect sign language and bridge the gap in communication between first responders and the deaf and hard-of-hearing community in emergencies.
The inability to communicate effectively in emergencies is a harsh reality for the deaf and hard-of-being community. Access to sign language interpreters and communication accommodations for these individuals can vary depending on the location and may be limited.
The inaccessibility and hesitance to receive help from medical professionals lower feelings of security and standard of living. Clear communication is necessary to ensure the correct treatment and services are timely.
Without access to sign language interpreters or communication accommodations, individuals who are deaf may not be able to understand instructions or communicate their needs which can lead to delay or inadequate medical attention, resulting in serious health consequences and even death.
ASL is a primary language for deaf and hard-of-hearing people to communicate. Though most deaf and hard-of-hearing people know ASL, not many people outside of the disability sector are familiar with this language.
Through the development of a sign language model, there would be a significant step made towards addressing the communication gap between first responders and the deaf and hard-of-hearing community during emergencies.
A machine learning model that utilizes computer vision can be used to detect sign language, improve response times, and ensure that those who communicate primarily through sign language have equal access to emergency services.
Using the camera of any device and computer vision, inputs from the camera are processed through neural networks that have developed an understanding of the patterns and gestures and provide real-time text translations for first responders.
By providing real-time text translations of emergency signs, first responders can quickly understand and respond to the situation, reducing the risk of miscommunication and delay. Here's my prototype to address the disparity in communication between first responders and individuals using sign language.
Some of the tools that were used for this prototype were TensorFlow, Jupyter Notebook, Protocol Buffers, TensorBoard, Anaconda, and LabelImage. The training of the Sign Language Detection Model began with importing and downloading necessary dependencies and programming tools such as Jupyter Notebook, which is an open-source web application that allows people to integrate live code and be used for visualizations and data.
LabelImage, which creates XML files, labels the images, and stores necessary metadata such as bounding boxes of the poses on images that were then given to the machine learning model. TensorFlow, which is an open-source software library for machine learning and artificial intelligence, and Anaconda, which is a distribution of Python used to organize data and machine learning applications in large-scale data processing.
The impact that a sign language detection system provides individuals is a way to effectively communicate ideas and concerns. This can improve relations between the deaf and hard-of-hearing community and the larger population and normalize differences.
The potential impact of this solution can be significant on a global scale as ASL is a primary language for communication among the deaf and hard-of-hearing communities in many countries. A sign language model can humanize interactions, prevent miscommunication, and provide a method of efficient and effective communication.
In addition, a sign language detection system can also promote inclusion and accessibility for the deaf and hard-of-hearing community as it can facilitate their participation in various aspects of society.
The use of AI for language and object detection can have a positive impact on various fields such as healthcare, law enforcement, and transportation, where efficient and effective communication is critical for success.
The development of an AI model represents the opportunity to leverage technologies for social good and promote a more inclusive and connected world. Getting the packages to work together was a struggle.
Some aspects of my code require newer packages while some are depreciated and because some packages were updated, other packages had older files that were no longer compatible with newer packages. This is by far the most challenging part of creating the machine learning model because so many packages were needed and as a result, I needed to find a version of each package that would not result in any errors or conflict with each other.
Creating a virtual environment in the command prompt was something I discovered later on which would help me tweak the package versions without damaging the current packages that are installed on my computer.
What’s next is getting the machine learning model to detect all signs and getting a type of translation from English to sign language.
An AI model can be trained to detect signs from other sign language families around the world by collecting data and training the model on a diverse set of sign languages ultimately allowing for the global recognition and understanding of various sign languages by gathering diverse sign language data and training an AI model on them.
It could be possible to extend sign language detection to other sign language families worldwide such as British and French sign language. Consequently, it will enable people to recognize and comprehend different sign languages at a global level.
Translation services that cover both spoken and written languages can expand the reach of translation services beyond sign language enabling translation between any sign language group and any spoken or written language such as French sign language to Vietnamese.
Additionally through the implementation of a customizable interface for those with visual impairments or other disabilities can be accommodated.
Features like adjustable font sizes customizable user interfaces or compatibility with assistive technologies can further the scope of this technology to support all sign language users.
The model also has the potential to facilitate communication in other settings such as medical appointments or public safety announcements further improving the accessibility for the deaf and hard-of-hearing community.
The inspiration behind this project was my grandparents. With their decline in health and loss of hearing, it has been difficult for them to communicate their needs consequently I felt that there must be a system or software that would help combat this issue to aid those who use sign language to communicate with emergency services.
It is one of my greatest fears that my grandparents cannot communicate their needs to emergency services and I want to find a solution to combat this issue for all families and individuals to use sign language.
I hope you enjoyed this presentation and thank you for listening.
Thank you for taking time out of your day to read this blog on STEMATA, and I hope you found value in the content! Feel free to comment on this blog, share with others, and follow me on Instagram @stemata.learning for updates on when I come out with new blogs, videos, and more!
Consider donating by clicking the donate button or dropdown in the top right corner to support one of our initiatives to provide low-income students with STEM experiment boxes and school supplies and to help them explore their passions in science fields. Thank you for your consideration and support!