Research

NYU-Poly Researchers Awarded DARPA Contract to Explore the Deep Web

• By Rhea Kelly
• 02/25/15

Researchers at the NYU Polytechnic School of Engineering are developing methods to explore hard-to-find information on the Web — both on the surface and on the deep Web, the realm not indexed by standard commercial search engines.

Juliana Freire, a professor in the Department of Computer Science and Engineering, has been awarded $3.6 million by the U.S. Defense Advanced Research Projects Agency as part of its Memex program, a "three-year research effort to develop software that will enable domain-specific indexing of open, public Web content and domain-specific search capabilities." DARPA's initial goal with Memex is to fight human trafficking, by tracking criminals' use of the Web to attract customers.

Freire and her colleagues, including Ari Juels from Cornell Tech and Torsten Suel, a professor in the NYU School of Engineering Department of Computer Science and Engineering, are "working on the design scalable techniques to address the shortcomings of traditional search engines for specific information needs," according to a press release. "Their research aims to make focused crawlers more usable and able to efficiently handle a wide range of search tasks, including adversarial crawling, which is needed for tasks that involve tracking criminal activities. Another important goal of their research is to enable the integration of crawling with search and data analysis in a transparent and reproducible fashion."