U.S. College Seniors Beat Peers from China, India, Russia on CS Exam

• By Dian Schaffhauser
• 04/18/19

Undergraduate seniors in the United States showed much higher levels of skill in computer science than their peers in China, India and Russia. A research project led by Stanford University's Prashant Loyalka in the Graduate School of Education made the discovery after testing students on a two-hour standardized exam given in the same languages their classes were delivered in.

The results were published in a new paper in Proceedings of the National Academy of Sciences (PNAS).

"There is this narrative that higher education in the United States is much stronger than in other countries, and we wanted to test whether that's true," said Loyalka, in an article on the project. "Our results suggest that the [United States] is doing a great job at least in terms of computer science education compared to these three other major countries."

In total, 678 students in China, 364 students in India and 551 students in Russia — from both elite and non-elite schools — took the test, which was developed by Educational Testing Service. In the United States, the researchers used assessment data on 6,847 seniors.

The exam, a 66-question multiple-choice test, assessed how well the seniors understand CS-related concepts, principles and knowledge. It didn't assume any particular programming language knowledge and, in fact, used pseudocode that was meant to be easily understood by CS students regardless of program or country. Content areas covered discrete structures, programming, algorithms and complexity, systems, software engineering and information management, among other topics.

Mean estimates for computer science skills among undergraduate seniors in China, India and Russia are statistically lower than those for the United States. Source: "Computer science skills across China, India, Russia, and the United States" from the Proceedings of the National Academy of Sciences.

The tests from those students who didn't answer at least three-quarters of the exam questions were kicked out of the analysis, and the remaining test scores were "scaled to be comparable" across the countries that participated.

Researchers found that the average U.S. CS student ranked higher than about 80 percent of students tested in China, India and Russia. The differences in scores among the students in those other countries "were small and statistically insignificant," the results noted.

In a comparison of students from top-ranking institutions in each country, researchers found that the average student in a top U.S. computer science program also ranked higher than about 80 percent of students from top programs in China, India and Russia. But among regular institutions the top Chinese, Indian and Russian students scored comparably with the U.S. students.

Another question the researchers looked at was whether America's high-flying results were due to a large number of top-scoring international students represented in the U.S. results. The researchers distinguished international students by their language skills. Of all sampled U.S. students, 89 percent reported that their best language was only English, which the researchers translated to mean they were domestic U.S. students.

"There is this sense in the public that the high quality of STEM programs in the United States is driven by its international students," Loyalka explained. "Our data show that's not the case. The results hold if we only consider domestic students in the U.S."

The researchers also found that male students scored just "moderately" higher than female students in each of the four countries; but U.S. female students scored higher than male students in China, India and Russia.

Loyalka and his research team will soon publish another paper, this one examining the skills of students in STEM fields in those same four countries. And he said that additional research beyond that will consider the relationship between skills developed in college and labor market outcomes as well as what's driving the differences in performance among the countries.

"We're looking at different aspects of the college experience, including faculty behavior, instruction and student interactions," Loyalka said. "One of our major goals is to see what types of college experiences could contribute to better student performance."

The full study is openly available on the PNAS website.