GA Tech Google Glass App Does Captioning

  • By Dian Schaffhauser
  • 10/06/14

Could Google Glass help people who are deaf or hard of hearing participate in everyday conversations? A group at the Georgia Institute of Technology College of Computing has developed speech-to-text "Glassware" that does just that. The person who is hard of hearing wears Glass while another person speaks directly into the smartphone. That speech is converted into text that shows up on Glass' display.
Captioning on Glass allows a person to speak into a phone. The text then shows up on Google Glass. Image courtesy of Georgia Tech.

Captioning on Glass is currently available to install from Glassware, Google's site for Glass add-on software.

The idea came when a member of that college said he was having trouble hearing and suggested that Glass could help him.

"This system allows wearers like me to focus on the speaker's lips and facial gestures," said School of Interactive Computing Professor Jim Foley. "If hard-of-hearing people understand the speech, the conversation can continue immediately without waiting for the caption. However, if I miss a word, I can glance at the transcription, get the word or two I need and get back into the conversation."
A hard-of-hearing person wears Glass while a second person speaks into a smartphone, which converts the speech to text and sends it for display on Glass. Image courtesy of Georgia Tech.

Professor Thad Starner led the development effort through the Contextual Computing Group, which is working on a number of learning initiatives, including CopyCat, a game that helps deaf children develop language skills and working memory as they play it.

According to Starner, the phone-to-Glass approach encourages speakers to speak more clearly, avoiding fillers such as "uhs" and "ums." If captioning errors occur, the app lets the speaker edit mistakes and send revised text to the person wearing Glass.

"The smartphone uses an Android transcription API to convert the audio to text," said Jay Zuerndorfer, a computer science graduate student who developed the software. "The text is then streamed to Glass in real time."

"If hard-of-hearing people understand the speech, the conversation can continue immediately without waiting for the caption," added Foley. "However, if I miss a word, I can glance at the transcription, get the word or two I need and get back into the conversation."

The research team is working with the Atlanta chapter of the Association of Late Deafened Adults to improve the program.

The same team is also developing "Translation on Glass," which uses the smartphone-Glass connection to handle foreign language translation on the fly. The sentences are spoken into the smartphone, translated to another language, and sent to Glass. After reading the translation, the Glass wearer can reply, and that response will be translated into the original language on the smartphone. Two-way translations are currently in the works for English, Spanish, French, Russian, Korean and Japanese.

For both applications, the person wearing Glass has to hand his or her smartphone to another person to begin the conversation, explained Starner. "It's not ideal for strangers, but we designed the program to be used among friends, trusted acquaintances or while making purchases."

About the Author

Dian Schaffhauser is a former senior contributing editor for 1105 Media's education publications THE Journal, Campus Technology and Spaces4Learning.

Featured

  • close-up illustration of a hand signing a legislative document

    California Passes AI Safety Legislation, Awaits Governor's Signature

    California lawmakers have overwhelmingly approved a bill that would impose new restrictions on AI technologies, potentially setting a national precedent for regulating the rapidly evolving field. The legislation, known as S.B. 1047, now heads to Governor Gavin Newsom's desk. He has until the end of September to decide whether to sign it into law.

  • Copilot Propels Microsoft to Lead Position in Analytics/BI Market

    A new Gartner report on the analytics/business intelligence market places Microsoft in the lead position of the field. The Redmond cloud giant stands apart and alone atop the axes for both the ability to execute and completeness of vision in Gartner's latest "Magic Quadrant for Analytics and Business Intelligence Platforms."

  • a stylized magnifying glass and a neural network pattern with interconnected nodes, symbolizing search and AI processes

    OpenAI Unveils SearchGPT AI-Powered Search Engine

    OpenAI has introduced SearchGPT, a new AI-powered search engine designed to access information from across the internet in real time. The much-anticipated prototype will provide more organized and meaningful search results by summarizing and contextualizing information rather than returning lists of links.

  • UMGC Officially Adopts InScribe's Student Community Platform

    The University of Maryland Global Campus (UMGC), an offshoot of the University System of Maryland that focuses on hybrid and virtual courses for adult and military students, is officially committing to a university-wide rollout of InScribe's student networking platform.