U Michigan's Sirius Sees Things Siri Is Blind to

• By Dian Schaffhauser
• 03/12/15

Virtual intelligent personal assistants (IPA) — think Siri, Cortana or Now — are about to go open source. A team of researchers at the University of Michigan has developed an open standalone speech- and vision-based IPA service similar to what Apple, Microsoft and Google have introduced into their products.

Sirius, as it's named, incorporates speech recognition, image matching, natural language processing and a question-and-answer system. It was developed by the university's Clarity Lab and is the subject of a paper that will be formaly published in a few days.at the 2015 International Conference on Architectural Support for Programming Languages and Operating Systems in Istanbul.

"Now the core technology is out of the bag, and we all have access to it," said Jason Mars, assistant professor of computer science and engineering and co-director of the lab. "Instead of making an app to run on the Apple Watch, for example, maybe I could make my own watch. We're very excited to see what the world comes together to build and learn with Sirius as a starting point."

In its initial release, users can enter their query solely by talking to a device or by using speech and capturing an image to pose a question such as, "When does this place close?" According to the researchers that kind of functionality isn't offered currently in commercial systems.

"What we've done with Sirius is pushed the limits of the traditional intelligent personal assistant," said Johann Hauswald, a doctoral student in the lab. "Not only can you interact with your voice but you can also ask questions about what you're seeing, which is a new way to interact with this type of device."

A demo version of the technology is loaded with a static version of Wikipedia, allowing users to ask questions that would be found there. However, the team noted, that knowledge base could be switched out with other types of information. For example, focused assistants could be built for specific fields — cooking, auto repair or health. Right now, the university is working with IBM to create an assistant that would be useful for academic advising.

Sirius was built by "stitching together" several other open source projects that the researchers said tap into techniques and algorithms that resemble those in commercial offerings. Those include:

• Speech recognition from Carnegie Mellon University's Sphinx, Microsoft Research's Kaldi and the RWTH Aachen University "RASR" project;
• The question-and-answer framework from OpenEphyra; and
• Image recognition from the SURF computer-vision algorithm Kooaba, which was recently acquired by Qualcomm.

Aside from interest in the IPA itself, Mars said he anticipates that organizations will use Sirius to study how computing infrastructure will stand up under voice-enabled queries, which are a major part of wearable technologies. The work of converting questions into code that can be understood by the program is done in the cloud. The research team has found that the process can be a hundred times more computationally intensive than a straightforward text query. "We have to think of new ways to redesign our cloud platforms to support this type of workload," Mars said.