DARPA Pursues Deep Web Search Tools - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Government // Leadership

DARPA Pursues Deep Web Search Tools

Defense Advanced Research Projects Agency project Memex will shine light on parts of the Web where commercial engines don't search.

6 Cool Apps From Uncle Sam
6 Cool Apps From Uncle Sam
(Click image for larger view and slideshow.)

The Defense Advanced Research Projects Agency (DARPA) wants to develop the next generation of search technologies, particularly those that can help the military and government agencies find and organize publicly available information on the Internet.

To address the challenges of a one-size-fits-all approach used by today's search engines, DARPA has kicked off a program called Memex, short for memory and index. The program aims to "revolutionize the discovery, organization, and presentation of search results" and "extend the reach of current search capabilities," according to DARPA. Memex will focus on three technical areas: domain-specific indexing, domain-specific search, and applications that pertain to the Department of Defense (DOD).

"We're envisioning a new paradigm for search that would tailor indexed content, search results, and interface tools to individual users and specific subject areas, and not the other way around," DARPA program manager Chris White said in a written statement.

[Memex is just one of DARPA's data development projects. Read DARPA Opens Software, Data To Public.]

DARPA believes current search capabilities miss information in the "deep Web" -- the parts of the Internet that aren't indexed by commercial search engines. They also overlook shared content across Web pages. The agency envisions using commodity hardware and an open-source architecture to build this advanced search technology that would be capable of cross-referencing information more quickly and efficiently.

Memex was inspired by a hypothetical device described in "As We May Think," a 1945 article for The Atlantic Monthly written by Vannevar Bush, a World War II-era director of the Office of Scientific Research and Development (OSRD). DARPA said it wants to improve the ideas described in that article, with the creation of domain-specific Web content indexing and search.

(Image: Montage courtesy of DARPA)
(Image: Montage courtesy of DARPA)

Initially, the program will focus on fighting human trafficking, a problem that touches various parts of government, including the military, law enforcement, and intelligence agencies. The commercial sex trade in particular has garnered significant Web presence via forums, chats, advertisements, and job postings. "An index curated for the counter-trafficking domain, along with configurable interfaces for search and analysis, would enable new opportunities to uncover and defeat trafficking enterprises," DARPA said.

Earlier this month, DARPA issued a solicitation for research on the subject. Research proposals, due on April 8, must address approaches that "enable revolutionary advances in science, devices, or systems," the agency said. Procurement contracts and cooperative agreements, but not grants, will be awarded to the winners. DARPA will hold a conference on February 18 to discuss the technical details of Memex with program participants.

Solid state alone can't solve your volume and performance problem. Think scale-out, virtualization, and cloud. Find out more about the 2014 State of Enterprise Storage Survey results in the new issue of InformationWeek Tech Digest.

Elena Malykhina began her career at The Wall Street Journal, and her writing has appeared in various news media outlets, including Scientific American, Newsday, and the Associated Press. For several years, she was the online editor at Brandweek and later Adweek, where she ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Author
2/12/2014 | 10:32:10 PM
Catching up
Given DARPA's annual budget is dwarded by Google's and Microsoft's resources, it should be interesting to see what comes of DARPA's efforts to develop a deeper search tool.

Thomas Claburn
Thomas Claburn,
User Rank: Author
2/12/2014 | 7:14:55 PM
the dark web
If DARPA is missing too much of the deep web, someone could always ask the NSA...
2021 Outlook: Tackling Cloud Transformation Choices
Joao-Pierre S. Ruth, Senior Writer,  1/4/2021
Enterprise IT Leaders Face Two Paths to AI
Jessica Davis, Senior Editor, Enterprise Apps,  12/23/2020
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll