Skip to main content
Add as a preferred source on Google


MIT Machine Learning Model Learns from Audio Descriptions

  • By Becky Nagel
  • 09/18/18
abstract depiction of brain with audio signal

Computer scientists at the Massachusetts Institute of Technology have invented a new machine learning model for object recognition that incorporates audio descriptions (versus transcripts of audio) along with images.

"The model doesn't require manual transcriptions and annotations of the example [speech] it's trained on," the official announcement explained of the new method. "Instead, it learns words directly from recorded speech clips and objects in raw images, and associates them with one another."

Typically, most machine learning models that incorporate audio require transcriptions of that audio versus using the audio itself. While this current system only recognizes "several hundred words and object types," the researchers who developed it have high hopes for its future.

"We wanted to do speech recognition in a way that's more natural, leveraging additional signals and information that humans have the benefit of using, but that machine learning algorithms don't typically have access to," commented David Harwath, a researcher in MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Spoken Language Systems Group. 

"There's potential there for a Babel Fish-type of mechanism," he continued.

This specific experiment is built upon a 2016 project, but with more images and data added, and with a new approach to the training. Details on how the model was trained can be found in the official announcement of the new project here.

About the Author

Becky Nagel serves as vice president of AI for 1105 Media specializing in developing media, events and training for companies around AI and generative AI technology. She also regularly writes and reports on AI news for PureAI.com, a site she founded, among others. She's the author of "ChatGPT Prompt 101 Guide for Business Users" and other popular AI resources with a real-world business perspective. She regularly speaks, writes and develops content around AI, generative AI and other business tech. Find her on X/Twitter @beckynagel.

Featured

  • Hootsuite

    Beyond the Feed: How Higher Ed Turns Social Media Into a Measurable Enrollment Engine

    More than half of prospective students research programs on social media before visiting a university website, and nearly half will skip applying to a school that looks inactive online. Hootsuite, the all-in-one social OS platform trusted by more than 2,000 higher education institutions, helps campus teams turn everyday posts into measurable outcomes, from applications to alumni giving.

  • closeup of hands using smart phone

    Learning Continuity Built into the LMS Withstands Cloud or Cybersecurity Interruptions

    When your institution's administrative or instructional capabilities face disruption from natural, technical, or malicious events, what measures does your LMS offer to provide a "business as usual" operating and learning environment? Instructure's Ryan Lufkin comments on the LMS and learning continuity.

  • glowing human profile icons in grid formation

    Cisco Introduces Privileged Infrastructure Access for People and AI Agents

    Cisco has unveiled Privileged Infrastructure Access, an approach to infrastructure administration that gives people and artificial intelligence agents temporary, task-specific permissions instead of leaving standing privileges available after their work is done.

  • broken digital lock and glowing chain

    Cybersecurity Researchers: AI Has Crossed into the Live Attack Chain

    AI is moving beyond assisting cybercriminals and beginning to operate within live attacks, according to a new report from Check Point Research. The technology is making sophisticated capabilities faster and more accessible while creating new security risks for businesses deploying their own AI systems.