Students Get Immersive AI Boost to Learn Mandarin

RPI researchers are using simulated experiences to learn how an immersive environment and AI can help students practice foreign language skills.

RPI researchers are using simulated experiences to learn how an immersive environment and AI can help students practice foreign language skills. (Photo: Rensselaer Polytechnic Institute)

Imagine the process of going into a restaurant and ordering food. Simultaneously, you could be glancing through the menu while also listening to and speaking with the waiter or your companions. When you're in a place where people are speaking a different language, the complexity of those activities increases multifold. A project taking place at Rensselaer Polytechnic Institute (RPI) hopes to understand how the use of an immersive environment and artificial intelligence can help students practice foreign language skills and increase their confidence when speaking. The researchers are using simulated experiences to test out their ideas.

The Cognitive Immersive Room, or "situation room," allows students to feel as though they're in restaurant in China, a garden or a Tai Chi class. While immersed in the environment, they practice speaking Mandarin with an AI chat agent powered by IBM Watson. The immersive classroom was developed by the Cognitive and Immersive Systems Lab, a research collaboration between IBM Research and RPI. The space taps several technologies: speech-to-text, natural language understanding and computer vision. The room uses a 360-degree panoramic display system, an audio system, multiple cameras, Microsoft Kinect devices and microphones, as well as computers.

According to a blog post by IBM, when students use the room, they converse with virtual conversation partners to practice their vocabulary and pronunciation without fear of being judged by native speakers. The difference between this experience and others that use VR or AR is that participants can walk around the room without wearing specialized gear.

The system was initially tested in a Chinese 1 class on the institute's campus in fall 2017. Students entered the restaurant, greeted the waiter (a digital representation of a panda) and tried to order a drink and food. They could interact with the AI through gestures, such as pointing to an item on the menu and asking the waiter what it was. The system also included help functions to assist the student when he or she was stuck and posed a question in English. Another feature not covered in the test is a tool under development for analyzing pitch tone (Mandarin has multiple tones for syllables), to help the student recognize when tonal pronunciation is wrong and learn how to improve it.

The test was set up to gather data on the system's performance, with a focus on spots where the learner had difficulty. That information will be used to help the AI system "understand and compensate," explained a campus article on the initiative.

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at the university.

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at the university. (Photo: Rensselaer Polytechnic Institute)

This "Mandarin Project," as it's called, is a rerun of another project kicked off in 2012 that combined AR, VR, game design and immersion of the student in a "narrative" to teach Chinese. That project, funded by a seed grant, used the theme of a mysterious book that had been missing for centuries and transported students to several locations in Beijing. Students interacted with live actors playing specific roles while physically moving around in the environment, called a "super cave," which was outfitted with actual tables, chairs and other props. According to the researchers, the students picked up a semester's worth of language skills in a much shorter eight weeks. Plus, every student passed the final exam, which involved an interrogation by a Mandarin speaker representing a police officer.

According to researcher Helen Zhou, an assistant professor in Communication and Media who taught the class used in the latest test, the goal of tackling a foreign language is to be able to "communicate with people of that language in a grammatically correct and culturally appropriate way." As she noted, "Research has shown that task-based, communicative language practices lead to the best results for that purpose." The restaurant, for example, "provides a near real-life environment for RPI students to practice and test their abilities to complete a communicative task, i.e. ordering Chinese food with the AI waiter whose speech is generated from native speaker accents."

Zhou said she was "thrilled" as a foreign language educator to see how novice learners can practice in a simulated restaurant, be motivated to complete the communicative task, and increase their level of confidence regarding speaking."

This fall, the use of AI will expand to two more advanced courses as well, with the goal of merging all three classes into a single course by summer 2019.

Featured

  • illustration of a futuristic building labeled "AI & Innovation," featuring circuit board patterns and an AI brain motif, surrounded by geometric trees and a simplified sky

    Cal Poly Pomona Launches AI and Innovation Center

    In an effort to advance AI innovation, foster community engagement, and prepare students for careers in STEM fields and business, California State Polytechnic University, Pomona has teamed up with AI, cloud, and advisory services provider Avanade to launch a new Avanade AI & Innovation Center.

  •  black graduation cap with a glowing blue AI brain circuit symbol on top

    Report: AI Is a Must for Modern Learners

    A new report from VitalSource identifies a growing demand among learners for AI tools, declaring that "AI isn't just a nice-to-have; it's a must."

  • glowing shield hovers above a digital cloud platform with abstract data streams and cloud icons in the background

    Google to Acquire Cloud Security Firm Wiz

    Google has announced it will acquire cloud security startup Wiz. If completed, the acquisition — an all-cash deal valued at $32 billion — would mark the largest in Google's history.

  • digital dashboard featuring a shield icon, graphs, a world map, and network nodes

    IBM Introduces Agentic AI Governance and Security Platform

    IBM has launched a new software stack for enterprise IT teams tasked with managing the complex governance and security challenges posed by autonomous AI systems.