Students Get Immersive AI Boost to Learn Mandarin

RPI researchers are using simulated experiences to learn how an immersive environment and AI can help students practice foreign language skills.

RPI researchers are using simulated experiences to learn how an immersive environment and AI can help students practice foreign language skills. (Photo: Rensselaer Polytechnic Institute)

Imagine the process of going into a restaurant and ordering food. Simultaneously, you could be glancing through the menu while also listening to and speaking with the waiter or your companions. When you're in a place where people are speaking a different language, the complexity of those activities increases multifold. A project taking place at Rensselaer Polytechnic Institute (RPI) hopes to understand how the use of an immersive environment and artificial intelligence can help students practice foreign language skills and increase their confidence when speaking. The researchers are using simulated experiences to test out their ideas.

The Cognitive Immersive Room, or "situation room," allows students to feel as though they're in restaurant in China, a garden or a Tai Chi class. While immersed in the environment, they practice speaking Mandarin with an AI chat agent powered by IBM Watson. The immersive classroom was developed by the Cognitive and Immersive Systems Lab, a research collaboration between IBM Research and RPI. The space taps several technologies: speech-to-text, natural language understanding and computer vision. The room uses a 360-degree panoramic display system, an audio system, multiple cameras, Microsoft Kinect devices and microphones, as well as computers.

According to a blog post by IBM, when students use the room, they converse with virtual conversation partners to practice their vocabulary and pronunciation without fear of being judged by native speakers. The difference between this experience and others that use VR or AR is that participants can walk around the room without wearing specialized gear.

The system was initially tested in a Chinese 1 class on the institute's campus in fall 2017. Students entered the restaurant, greeted the waiter (a digital representation of a panda) and tried to order a drink and food. They could interact with the AI through gestures, such as pointing to an item on the menu and asking the waiter what it was. The system also included help functions to assist the student when he or she was stuck and posed a question in English. Another feature not covered in the test is a tool under development for analyzing pitch tone (Mandarin has multiple tones for syllables), to help the student recognize when tonal pronunciation is wrong and learn how to improve it.

The test was set up to gather data on the system's performance, with a focus on spots where the learner had difficulty. That information will be used to help the AI system "understand and compensate," explained a campus article on the initiative.

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at the university.

Rahul Divekar, a Rensselaer computer science graduate student, demonstrates an AI-assisted Mandarin Chinese language learning aid under development at the university. (Photo: Rensselaer Polytechnic Institute)

This "Mandarin Project," as it's called, is a rerun of another project kicked off in 2012 that combined AR, VR, game design and immersion of the student in a "narrative" to teach Chinese. That project, funded by a seed grant, used the theme of a mysterious book that had been missing for centuries and transported students to several locations in Beijing. Students interacted with live actors playing specific roles while physically moving around in the environment, called a "super cave," which was outfitted with actual tables, chairs and other props. According to the researchers, the students picked up a semester's worth of language skills in a much shorter eight weeks. Plus, every student passed the final exam, which involved an interrogation by a Mandarin speaker representing a police officer.

According to researcher Helen Zhou, an assistant professor in Communication and Media who taught the class used in the latest test, the goal of tackling a foreign language is to be able to "communicate with people of that language in a grammatically correct and culturally appropriate way." As she noted, "Research has shown that task-based, communicative language practices lead to the best results for that purpose." The restaurant, for example, "provides a near real-life environment for RPI students to practice and test their abilities to complete a communicative task, i.e. ordering Chinese food with the AI waiter whose speech is generated from native speaker accents."

Zhou said she was "thrilled" as a foreign language educator to see how novice learners can practice in a simulated restaurant, be motivated to complete the communicative task, and increase their level of confidence regarding speaking."

This fall, the use of AI will expand to two more advanced courses as well, with the goal of merging all three classes into a single course by summer 2019.

Featured

  • digital illustration of a glowing padlock at the center, surrounded by abstract icons of books and graduation caps

    2025 Cybersecurity Predictions for K-20 Education

    What should K-12 and higher education institutions expect on the cybersecurity front in the coming year? Here's what the experts told us.

  • The AI Show

    Register for Free to Attend the World's Greatest Show for All Things AI in EDU

    The AI Show @ ASU+GSV, held April 5–7, 2025, at the San Diego Convention Center, is a free event designed to help educators, students, and parents navigate AI's role in education. Featuring hands-on workshops, AI-powered networking, live demos from 125+ EdTech exhibitors, and keynote speakers like Colin Kaepernick and Stevie Van Zandt, the event offers practical insights into AI-driven teaching, learning, and career opportunities. Attendees will gain actionable strategies to integrate AI into classrooms while exploring innovations that promote equity, accessibility, and student success.

  • smartphone with a glowing lock and shield icon at its center, surrounded by floating security symbols like a fingerprint, key, and authentication checkmark

    Jamf to Acquire Identity Automation, Combining Identity and Device Management in One Platform

    Apple mobile device management company Jamf has announced the intent to acquire Identity Automation, a provider of identity and access management (IAM) solutions for K-12 and higher education.

  • two abstract humanoid figures made of interconnected lines and polygons, glowing slightly against a dark gradient background

    Microsoft Introduces Copilot Chat Agents for Education

    Microsoft recently announced Microsoft 365 Copilot Chat, a new pay-as-you-go offering that adds AI agents to its existing free chat tool for Microsoft 365 education customers.