Research Project Figures Out How to Crowdsource Predictive Models

crowdsourcing

An MIT research project has come up with a way to crowdsource development of features for use in machine learning. Groups of data scientists contribute their ideas for this "feature engineering" into a collaboration tool named "FeatureHub." The idea, according to lead researcher Micah Smith, is to enable contributors to spend a few hours reviewing a modeling problem and proposing various features. Then the software builds the models with those features and determines which ones are the most useful for a particular predictive task.

As described in "FeatureHub: Towards collaborative data science," the platform lets multiple users write scripts for feature extraction and then request an evaluation of their proposed features. The platform aggregates features from multiple users and automatically builds a machine learning model for the problem at hand. The name was inspired by GitHub, a repository for programming projects, some of which have drawn numerous contributors.

To test the platform, the researchers recruited 41 freelance analysts with data science experience, who spent five hours each with the system, familiarizing themselves with it and using it to propose candidate features for each of two data science problems. In one problem, the participants were given data about users of the home rental site Airbnb and their activity on the site. They were asked to predict, for a given user, the country in which the user would book his or her first rental. In the other problem the workers were given data provided by Sberbank, a Russian bank, on apartment sale transactions and economic conditions in Russia. The test subjects were given the job of predicting for a given transaction the apartment's final selling price. Of the 41 workers who logged into the platform, 32 successfully submitted at least one feature. In total, the project collected 1,952 features.

The predictive models produced with FeatureHub were then compared against the ones submitted to Kaggle, a data-science competition service that uses manual effort for its results. The Kaggle entries had been scored on a 100-point scale, and the FeatureHub models fell within three and five points of the winning entries for the two problems. Importantly, however, while the Kaggle entries took weeks or months of work, the FeatureHub entries were produced in days.

Smith is hopeful for the use of the platform. "I do hope that we can facilitate having thousands of people working on a single solution for predicting where traffic accidents are most likely to strike in New York City or predicting which patients in a hospital are most likely to require some medical intervention," he said, in an MIT article about the project. "I think that the concept of massive and open data science can be really leveraged for areas where there's a strong social impact but not necessarily a single profit-making or government organization that is coordinating responses."

A paper on the project was recently presented at the IEEE International Conference on Data Science and Advanced Analytics in Tokyo. Co-authors included Smith's thesis advisor, Kalyan Veeramachaneni, a principal research scientist at MIT's Laboratory for Information and Decision Systems, and Roy Wedge, a former MIT undergraduate who is now a software engineer at Feature Labs, a data science company based on the group's work.

The project was partially funded through a National Science Foundation grant focused on creating a community software infrastructure, called LearnSphere, that supports sharing, analysis and collaboration across the wide variety of educational data.

About the Author

Dian Schaffhauser is a former senior contributing editor for 1105 Media's education publications THE Journal, Campus Technology and Spaces4Learning.

Featured

  • networked node grid featuring illuminated connectors

    Infrastructure Accounts for More than Half of Worldwide AI Spending

    Gartner forecasts worldwide AI spending will reach $2.67 trillion in 2026, up 49.5% from 2025, with AI infrastructure accounting for nearly $1.5 trillion, or about 56% of the total. For every dollar Gartner expects to be spent on generative AI models, more than $52 will be spent on infrastructure.

  • CIS Sandbox Tutor Eli Blouin at Bentley University imagines a future technology learning partner that supports curiosity, critical thinking, feedback, and personalized learning.

    Student Voices: Technology as a Future Learning Partner

    A data analytics and marketing major at Bentley University and a distinguished lecturer at the university explore the future of "technology learning partners" — technologies that participate in the learning process by asking questions and giving feedback, not just information search results.

  • young man hand working on a laptop, bokeh blurred at night

    Redesigned Copilot Will Build Apps and Work on Its Own

    Microsoft recently announced that it's redesigning Copilot, the company's all-encompassing AI assistant, so users can hand off complex tasks, build tools by describing what they need and assign work to an agent that keeps going after they log off.

  • Abstract glowing network nodes forming conversation bubbles

    OpenAI Introduces Dots, AI Agents that Continue Working Between User Conversations

    OpenAI has announced Dots, GPT-6 Astra-powered agents that run on their own cloud computers, using connected applications to carry on with assignments between conversations and incorporating users' feedback as they go.