AI-assisted computer vision research aims to improve accessibility for deaf, hard of hearing

Teresa Donnellan — Mon, 27 Feb 2023 14:55:35 +0000

AI-assisted computer vision research aims to improve accessibility for deaf, hard of hearing Teresa Donnellan Mon, 02/27/2023 - 09:55

In This Story

Jana Košecká

Digital assistants like Amazon’s Alexa aren’t currently useful for, say, the hard of hearing and deaf community. ��ӽ紫ý researchers led by Jana Košecká are making the Internet of Things more inclusive and accessible to those for whom it has not been designed. For the next year, her work to improve "seeing" computer systems to translate continuous American Sign Language into English will be funded by Amazon’s Fairness in AI Research Program.

Jana Košecká. Photo by Ron Aira/Creative Services

Having worked at Mason for more than 20 years, Košecká began studying computer vision as it applies to American Sign Language in 2019 with colleagues Huzefa Rangwala and Parth Pathak in collaboration with Gallaudet University. Their work resulted in three academic publications on the topic in 2020. The team’s initial work focused on computer vision recognizing American Sign Language at the word level.

Košecká describes her current work as a continuation of earlier work, but now, especially with the help of AI, she’s tackling more complex ASL content, such as sentence-level communication, facial expressions, and very specific hand gesticulation.

“The challenge of extending some of these ideas [of computer translation] to American Sign Language translation is the input is video as opposed to text; it's continuous, and you have a lot of challenges, because you have a lot of variations about how people sign,” says Košecká.

The project is accordingly multifaceted. “We are focusing on better hand modeling, focusing on incorporating the facial features and extending to continuous sign language, so you can have short phrases the model can translate to English,” Košecká explains. “We are basically trying to capture continuous sign language and not just individual words."

To accomplish this goal,Košecká is using weakly supervised learning machine learning methods that provide mechanisms to teach the system without excessive human labelling effort.

“Weakly supervised learning techniques don't need perfect alignment of video sequences that contain multiple words,” she says. “In the word-level recognition, the model is presented with examples of a video snippet of a signed word and the word text, so it has perfect supervision. Given many examples of the sign ‘apple’ as a video snippet, the system will learn to recognize the word 'apple.' ”

“There are some techniques which can discover patterns without this need of direct supervision. If you just give the model a lot of examples, the model will figure out repeating patterns of certain words occurring in certain contexts,” she says. “So we are applying these machine-learning techniques to the setting of American Sign Language.”

Relating her work to AI-powered chatbots like chatGPT, Košecká says, “There has been a lot of headway made in this space for written and spoken languages, and we would like to make a little bit of headway for American Sign Language, using some of these insights and ideas.”

Košecká envisions her research helping improve the interface between hard of hearing people and their environment, whether that be when they’re communicating with Amazon’s Alexa or ordering at a restaurant counter. No doubt her work will help improve inclusivity and accessibility for the deaf and hard of hearing both at Mason and beyond.

Topics

American Sign Language

Computer science; computing; Amazon

Inclusion

Artifical Intelligence

Research

Mason partnering with Amazon to provide computer science education to teachers

Colleen Rich — Tue, 26 Jan 2021 20:55:53 +0000

Mason partnering with Amazon to provide computer science education to teachers Colleen Rich Tue, 01/26/2021 - 15:55

��ӽ紫ý is collaborating with Amazon and the Maryland Chamber Foundation to provide local teachers with computer science externships and education in a pilot program to begin this summer. The program will provide current and potential computer science teachers in Virginia, Maryland and the District of Columbia with an opportunity to shadow Amazon technology experts and take a companion three-credit graduate-level class at Mason.

Erdogan Kaya, an assistant professor of computer science education at Mason’s College of Education and Human Development (CEHD), will teach the eight-week, project-based companion class. In addition, participating teachers will also get an opportunity throughout the fall to meet with each other and exchange best practices. Virginia teachers can also participate in a free Praxis workshop in the fall to prepare to take the exam to become licensed to teach computer science.

“We believe that through this opportunity, teachers can develop meaningful learning experiences that will help them in how they teach computer science to children,” said Kaya, a former K-12 computer science teacher. “Computer science is a dynamic field, and we want to engage with teachers to increase their understanding of the workforce skills and knowledge their students will need.”

The program is currently accepting applications. Preference will be given to local middle school and high school teachers who are interested in teaching computer science or who want to integrate computer science into their instruction. Teachers who work in underserved communities will be prioritized.

The Amazon externship will allow teachers to shadow Amazon professionals during the summer, said Shanika Hope, head of Amazon Future Engineer, U.S., giving them exposure to technology professionals in a substantive, practical way. The pilot program will accept 20 participants.

Mason has been partnering with Amazon on other projects since it established a second headquarters in Northern Virginia. Amy Hutchison, associate professor and director of CEHD’s Division of Elementary, Literacy and Secondary Education, said that collaborating with Amazon has been seamless.

“They are very open to our ideas,” said Hutchison. “This has been very exciting for us to bring together industry and education.”

The Maryland Chamber Foundation is in its third year of offering teaching externships to Maryland educators. The foundation works to bridge the gaps between businesses and educators by pairing teachers with top local businesses.

If successful, the pilot program could be offered again and expanded. In addition, CEHD researchers will study the effectiveness of the new program in changing how computer science is taught in the classroom.

Hope said the genesis of the program was a recognition of the gaps between the skills taught in schools and those needed for the job market.

“Teachers inspire, support and expose students to new opportunities, so their proficiencies and expertise have a direct impact on the paths that students pursue,” said Hope. “We want to invest in teachers in a way that will affect students in the long term.”

Topics

Campus News

College of Education and Human Development

Computer science; computing; Amazon

Amazon

Mason Momentum