The future of search: Do you ask Google or the gaggle?

Late last month, media giant Google launched an online featured called SearchWiki, which allows users to rate, annotate, and store results they’ve found particularly useful. The notes have no direct bearing on public rankings, although individual comments are visible to all users.

On the company’s blog, Google said SearchWiki moved search one more step toward a “dynamic” search experience – one in which a community will be able to shape, refine, and organize the raw matter of the World Wide Web.

The concept has a considerable amount of ballast in Silicon Valley, where developers have long predicted that the future of search lies not in proprietary algorithms, such as Yahoo or Google, but in the power of the hive mind.

Over the past few years, a score of so-called “people-powered” search tools have entered the fray, including Stumpedia, Mahalo, Sproose, and Gravee. Most of these sites couple the raw processing power of an algorithmic engine with the functionality of Digg, the community-controlled news aggregator.

“There are a lot of smart people who have looked at Google and Yahoo and said the fundamental way of searching has not changed in nine or 10 years,” says Bob Pack, a founder and CEO of Sproose, which allows users to influence search results with a simple voting mechanism. “You’ve got algorithmic search results, organized into a set of blue links going down the page. Search needs to become richer and more intuitive.”

This community-based approach to search will likely never replace traditional engines when it comes to simple searches, such as checking sports scores or the state of the stock market. But more complex tasks are still handled more effectively by a human.

“Let’s say you’re looking for a hotel room. What you find on Google is endless realms of affiliate marketing sites, selling you the same rooms,” says Jimmy Wales, founder of Wikipedia and Wikia Search, an “open-sourced” search tool launched earlier this year. (Unlike Google’s SearchWiki, which is a feature within Google’s traditional search function, Wikia Search is a stand-alone site.)

“It’s hard to get the hotel site itself,” he says.

A people-powered engine, on the other hand, would take into account the experiences of other users and steer you directly to the best-rated hotels in the area.

Battle of philosophies: Democratic or efficient?
Proponents of community-assisted search also point to the essentially undemocratic nature of the top three traditional portals: Google, Yahoo, and Microsoft.

Each company zealously guards its internal algorithms, and search results can be subject to internal editorial control. In an industry with huge profit margins – search is a multibillion-dollar industry – the lack of transparency can be unsettling, Mr. Wales argues.

“I like to make the analogy between search and journalism,” he says. “Search in a certain sense is reporting on the world. Now, for most high-quality papers, there’s a certain amount of transparency. You understand that advertising might influence the paper. But you also have a reporter’s byline, for instance. Because search is so secretive, and so propriety, there are fewer checks and balances.”

Furthermore, Wales continues, most users rarely stray from the top 10 results coughed up by their favorite search engine.

Since Yahoo, Microsoft, and Google respond to each query in similar ways, there remains a vast swath of relatively pristine Internet wilderness waiting to be discovered. A search tool that effectively utilized human input could open up the Web in a startling way, and allow users easy access to information that once languished on the 475th page of Google results.

Major obstacles remain before users will switch away from their current portal, says Brad Bostic, founder and CEO of ChaCha, a community search tool for mobile phones. Google, for instance, has already won the allegiance of a majority of Web users, and the company name is synonymous in mainstream culture with online search.

Bob Pack, of Sproose, likens the situation to the battle between Coca-Cola and Pepsi.

“Google is Coke,” he says, “and Yahoo is Pepsi, and then you’ve got all the other flavors. Google’s so well-branded.”

It’s a Catch-22, Mr. Pack adds: If people-powered sites could attract millions of users, the quality of rankings could be more dynamic and democratic than anything yielded by a traditional algorithmic engine. Many Google aficionados, however, would be loath to jump to another engine unless the results were already top-notch.

More practically, there is a concern among some developers that handing over control to a community could engender a flood of spam, or devolve into a mess of internecine backbiting among users. (This is the case on the Yahoo Answers site, where queries are answered haphazardly, or with a string of pejoratives, or – less frequently – not at all.)

Finally, the “richness of information” on a people-powered site is often much more unwieldy than the data provided by Google, says James Segil, the president of EdgeCast Networks, an Internet infrastructure company that helps power Mahalo. A major challenge for people-powered search companies is how to juggle the flow of user input, and keep results consistently updated.

There are signs that the search paradigm is already shifting. Google’s SearchWiki has received some good notices – and some bad – from tech reviewers, and brought the community-search debate to center stage.

Some progress for smaller sites
ChaCha, which uses an army of more than 55,000 amateur “guides” to help relay information via SMS text, has enjoyed what Mr. Bostic calls an extraordinary leap in growth. According to information provided by ChaCha, the company has logged more than 56 million queries to date, up 53 percent from January to November of 2008. Thousands of new users experiment with the service every day.

“There’s a feeling of information pollution out there,” says Melek Pulatkonak, president and chief operating officer of a search site called Hakia. “Millions and millions of results [on Google] terrorize the searcher, preventing them from moving forward. People want something different.”

Hakia is not exactly a people-powered engine. Instead, it uses a homegrown algorithm to analyze the wording of a query, thus delivering results more contextually accurate that than of Google. (If a user were looking to “treat” a cold, for example, Hakia would understand that the word “treat” did not refer to candy.)

The site, now in public beta testing, simultaneously maintains a growing network of “credible” sites, vetted by a network of librarians at hospitals and schools around the country.

This combination of algorithmic and human input, Ms. Pulatkonak says, is very attractive to the average user. She does not anticipate Hakia would ever replace Google, but hopes instead that it will be used alongside Google, as a supplementary tool.

“To me the question is not whether humans should be making decisions,” Wales says, pointing out that Google’s internal editorial control is people-power of a kind. “It’s how they should be making the decisions. Should search be democratic? Or should it continue to be a top-down system?”

[Editor's note: The original version of this article misstated Ms. Pulatkonak's first name.]