Hunch, Or the Turing test for the Web

Quick post via Tech Crunch:

Earlier today Flickr co-founder Caterina Fake announced the release of her latest startup, Hunch, in private beta. The site revolves around helping users make decisions spanning a wide array of topics. To help users make their decisions, Hunch presents them with a brief series of questions that have been submitted by other members, using their responses to help them make their ultimate decision. It’s a great idea that combines the crowd-sourced nature of Wikipedia with services like Yahoo Answers. But does it work? We’ve managed to get our hands on an invite to the service, and have put it to the test.

Sounds kind of nuts, right?  Hunch basically will be the closest thing to web-based artificial intelligence. This is kinda/sorta reminiscent of the Turing test, one of the earliest experiments in Artificial Intelligence.

I’m not sure how to read the politics of this yet. It’s easy to say that by outsourcing decision-making to an algorithm is the most sophisticated form of control. And on some level, that may be right. But what are some conceptual openings provided by something like Hunch? Will it lead us to finally put to the grave the idea of “critical reason” as most famously imagined by Kant and then virulently defended by the Frankfurt school? And lastly, is Hunch the signalling of a new trend in web-based computing which is still so rooted in what I’ve been calling on this blog the dominant “culture of search” that has such a strong hold on Web practices?

Let’s see…

The Past, Present and Future of “Infra-Search”

The NY Times has a nice piece by Ashlee Vance in the Business Computing section highlighting a software and a company–Hadoop and Cloudera, respectively–that are in the business of, what I am calling, “infra-search.” (Truth to be told, it is quite simply “data analysis” but that is too broad and also not specifically geared towards what I feel is the “monoculture of Search” of the Web today. What is meant by infra? Here’s the dictionary.com definition:

in⋅fra

–adverb

below, esp. when used in referring to parts of a text.

Compare supra.

---

Origin:
1730–40; < L infrā; cf. under

So what is “below” search you ask? Well, a lot. You need to index and manage data before it is able to be searchable. It is the crucial process before data becomes searchable that I am calling “infra-search.” Here’s the pre-history of the “infra-search.”

By 2003, Google found it increasingly difficult to ingest and index the entire Internet on a regular basis. Adding to these woes, Google lacked a relatively easy to use means of analyzing its vast stores of information to figure out the quality of search results and how people behaved across its numerous online services.

So a couple of cats at Google came up with something called MapReduce to help them manage the data. So why is this important?

MapReduce represented a couple of breakthroughs. The technology has allowed Google’s search software to run faster on cheaper, less-reliable computers, which means lower capital costs. In addition, it makes manipulating the data Google collects so much easier that more engineers can hunt for secrets about how people use the company’s technology instead of worrying about keeping computers up and running.

….

The MapReduce technology helps do grunt work, too. For example, it grabs huge quantities of images — like satellite photos — from many sources and assembles that information into one picture. The result is improved versions of products like Google Maps and Google Earth.

MapReduce allowed Google to focus on search, which is the bread and butter of Google, although it makes money really by selling advertising (how traditional, right?).

So how do we get from MapReduce to Hadoop and Cloudera?

Well this MapReduce thing has been the kind of trade-secret that everyone–business folks and Internet studies people alike–have been wondering about. How does it work? And more significantly, how does it work so well? Well Google has published some papers on MapReduce, which was enough for others to make their own version, including Hadoop, which Yahoo is behind.

Now a lot of folks are jumping on the bandwagon because of how well Hadoop analyzes data. Facebook uses Hadoop to manage all the photos, more specifically photo-tagging, as it allows for Facebook to determine how “close” the connection is between two people. Even the notoriously anti-open source (Hadoop is open-source like any reasonable software should be) Microsoft has been down with it too.

Now what does this all mean for regular people that just use the Web and aren’t fascinated with with the intricacies of search?

“What if Google decided to sell the ability to do amazing things with data instead of selling advertising?” Mr. Hammerbacher [co-founder of Cloudera]asked.

That proposition is pretty significant, no?

We suck at the Internet too

The Bits Blog at the Times reveals that American “broadband penetration” (That’s what they call it folks. Haha. Paging Dr. Freud!) is sadder than Rick Ross’ rap career. Compared to Europe and East Asia, we are in the Dark Ages on two fronts–cost and speed. Other countries out-Internet us for a couple reasons:

Other countries have lower costs for the same reasons their DSL service is faster. Dense urban areas reduce some of the cost of building networks. In addition, governments in some countries subsidized fiber networks.

But, according to Saul Hansell, there are also significant factors that are self-imposed by American business that have made the country’s broadband situation so “primitive.”

“Now hold on there,” you might say to me. Since I wrote that many countries don’t have cable systems and the bulk of broadband is run by way of DSL through existing phone wires, how can there be competition? Aren’t those owned by monopoly phone companies?

True enough. But most big countries have devised a system to create competition by forcing the phone companies to share their lines and facilities with rival Internet providers.

There’s the rub. Now, the US attempted to do this when laws dictated that companies allow other providers to use their infrastructure, in order to stimulate competition. But this was abandoned in 2003 by the FCC. This forced competition or what is called “line-sharing” may be making a comeback soon.

Restoring some form of line sharing is one of the biggest issues facing the F.C.C. Without it, or some other way to increase competition, the oligopolistic nature of the market in the United States may well keep broadband prices well above the rates for similar service in the rest of the world. At the same time, the commission is looking to expand broadband access to rural areas and speed the deployment of higher speeds, so it may not want to slash telco profits if it will also slow investment.

The issue at hand is how and why will companies invest in infrastructure if they are forced to share the lines (and thus profit). This will most likely in the American case (as it was in Japan) be rectified by a nice tax credit to whichever company actually lays the lines down. Either way, the question of technological justice (which in this instance is bringing broadband to rural areas) is something that the US has not yet really reckoned with. As the popular conception of American poverty shifts from urban centers to rural areas, the question of technological access, along with economic growth and job growth, will have to be addressed. We’ll see if the FCC will be able to stand up to the telcos. If they don’t, then the federal governments should force them, like Britain and Japan have. But that probably won’t happen. I have a feeling that bringing broadband to rural areas will inevitably aid the process of oligopolistic business practices among telcos. Sad but true…

Twitter is Twitter–neither Google nor Facebook

Farhad Manjoo of Slate has done it again by laying out the reasons why the media hype that has recently been surrounding Twitter is all-too-simple. And let’s get this right, people associated with Twitter, including Evan Williams, who recently appeared on the Charlie Rose Show, have fanned the fumes. See his interview below:

Naive questions abound such as:

1. Will it be a Google-killer?
2. Will it be a Facebook-killer?

Manjoo deals with each pretty handily, beginning with the Twitter vs. Google notion. Many, including Twitter people, have been suggesting that Twitter-search is a better “search” than Google because it allows you to search in real-time. Hence, when that plane landed on the Hudson, whereas Google did not have much in the way of links on that particular event since it was unraveling so quickly.

In a recent blog post, John Borthwick, one of Twitter’s investors, explained the value of such a service. Imagine that you’re waiting in line for coffee, and you overhear some people talking about a plane landing on the Hudson. When you go back to your desk, you search Google—and come up with nothing. “The day of the incident there was nothing on the topic to be found on Google,” Borthwick writes. “The same holds for any topical issues—lipstick on a pig?—for real-time questions, real-time branding analysis, tracking a new product launch. On pretty much any subject if you want to know what’s happening now, search.twitter.com will come up with a superior result set.”

Here’s Manjoo’s critique:

…Twitter’s search engine, at the moment, is extremely crude. It sorts results only chronologically, so the best answers are easily swamped by silly ones and people repeating silly ones. Of course, Twitter could improve its search engine. But so could Google. Twitter, remember, is a public database. Everything on Twitter is on the Web and can be analyzed by any other firm—in fact, Twitter’s own search engine began as a startup unaffiliated with Twitter. (Twitter bought that company, Summize, last summer.)

Touche. In other words, Twitter search sucks and Google also indexes tweets from Twitter. But additionally, he asks a question about the very culture of Web search.

How often does anyone need “real time” information on the Web? Yes, in the first few minutes after a plane has landed in the Hudson, Twitter might be the best place to find news. But its advantage is short-lived. By the time TV helicopters begin circling overhead, Twitter becomes decidedly slower and noisier than other media—now you’re much better off going to CNN, where, in addition to a live picture, you’ll find comments from authorities, structured interviews with eyewitnesses, and other old-fashioned journalistic information.

So what about Facebook? We have all seen the recent attempt by Facebook to change their home page UI. But again, Manjoo raises skepticism towards the value of real-time data, which Twitter doesn’t do. It provides real-time sociality. These two are rather different, I think.

In a sign that it recognizes the appeal of Twitter, Facebook announced that it will soon change its home page to allow for immediate updates. You’ll be able to see what your friends are doing right now, just as you can on Twitter. It’s a good move, but probably not revolutionary. Again, I think people overestimate the appeal of “real time” data. There are some instances in which real-time interactions will improve Facebook—when all of your friends are watching the State of the Union address, for example, you might want to watch everyone’s status update change as the speech progresses. But will we be much more satisfied to get our friends’ status updates immediately rather than have to wait 10 minutes? Some people may be overjoyed; most likely won’t notice the difference.

…It’s possible that Twitter could do very well—but probably not as a social network, and probably not as a search engine.

I agree! Can we stop trying to frame Twitter within a paradigm that already exists? And I’m talking to you too Twitter investors and creators. Ya’ll need to quit this Twitter as search business because it’s clearly…what’s the word…desperate?