Students Get Immersive AI Boost to Learn Mandarin

• By Dian Schaffhauser
• 08/29/18

RPI researchers are using simulated experiences to learn how an immersive environment and AI can help students practice foreign language skills. (Photo: Rensselaer Polytechnic Institute)

Imagine the process of going into a restaurant and ordering food. Simultaneously, you could be glancing through the menu while also listening to and speaking with the waiter or your companions. When you're in a place where people are speaking a different language, the complexity of those activities increases multifold. A project taking place at Rensselaer Polytechnic Institute (RPI) hopes to understand how the use of an immersive environment and artificial intelligence can help students practice foreign language skills and increase their confidence when speaking. The researchers are using simulated experiences to test out their ideas.

The Cognitive Immersive Room, or "situation room," allows students to feel as though they're in restaurant in China, a garden or a Tai Chi class. While immersed in the environment, they practice speaking Mandarin with an AI chat agent powered by IBM Watson. The immersive classroom was developed by the Cognitive and Immersive Systems Lab, a research collaboration between IBM Research and RPI. The space taps several technologies: speech-to-text, natural language understanding and computer vision. The room uses a 360-degree panoramic display system, an audio system, multiple cameras, Microsoft Kinect devices and microphones, as well as computers.

According to a blog post by IBM, when students use the room, they converse with virtual conversation partners to practice their vocabulary and pronunciation without fear of being judged by native speakers. The difference between this experience and others that use VR or AR is that participants can walk around the room without wearing specialized gear.

The system was initially tested in a Chinese 1 class on the institute's campus in fall 2017. Students entered the restaurant, greeted the waiter (a digital representation of a panda) and tried to order a drink and food. They could interact with the AI through gestures, such as pointing to an item on the menu and asking the waiter what it was. The system also included help functions to assist the student when he or she was stuck and posed a question in English. Another feature not covered in the test is a tool under development for analyzing pitch tone (Mandarin has multiple tones for syllables), to help the student recognize when tonal pronunciation is wrong and learn how to improve it.

The test was set up to gather data on the system's performance, with a focus on spots where the learner had difficulty. That information will be used to help the AI system "understand and compensate," explained a campus article on the initiative.

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at the university. (Photo: Rensselaer Polytechnic Institute)

This "Mandarin Project," as it's called, is a rerun of another project kicked off in 2012 that combined AR, VR, game design and immersion of the student in a "narrative" to teach Chinese. That project, funded by a seed grant, used the theme of a mysterious book that had been missing for centuries and transported students to several locations in Beijing. Students interacted with live actors playing specific roles while physically moving around in the environment, called a "super cave," which was outfitted with actual tables, chairs and other props. According to the researchers, the students picked up a semester's worth of language skills in a much shorter eight weeks. Plus, every student passed the final exam, which involved an interrogation by a Mandarin speaker representing a police officer.

According to researcher Helen Zhou, an assistant professor in Communication and Media who taught the class used in the latest test, the goal of tackling a foreign language is to be able to "communicate with people of that language in a grammatically correct and culturally appropriate way." As she noted, "Research has shown that task-based, communicative language practices lead to the best results for that purpose." The restaurant, for example, "provides a near real-life environment for RPI students to practice and test their abilities to complete a communicative task, i.e. ordering Chinese food with the AI waiter whose speech is generated from native speaker accents."

Zhou said she was "thrilled" as a foreign language educator to see how novice learners can practice in a simulated restaurant, be motivated to complete the communicative task, and increase their level of confidence regarding speaking."

This fall, the use of AI will expand to two more advanced courses as well, with the goal of merging all three classes into a single course by summer 2019.