Predicting The Future: Fantasy Or A Good Algorithm?

Predicting The Future: Fantasy Or A Good Algorithm?
by Dina Temple-Raston
Morning Edition - NPR
October 8, 2012

After failing to predict the Arab Spring, intelligence officials are now exploring whether Big Data, the combing of billions of pieces of disparate electronic information, can help them identify hot spots before they explode. The intelligence community has always been in the business of forecasting the future. The question is whether tapping into publicly available data — Twitter and news feeds and blogs among other things — can help them do that faster and more precisely.

Enter a Swedish-American start-up company called Recorded Future. The company has developed algorithms that chew through huge volumes of information to find relationships between people and organizations. Then its visualization software spits out that information in the form of a giant searchable timeline.

"What we're trying to do here at Recorded Future is figure out a cool way that we can observe the world," says co-founder Christopher Ahlberg. "We're trying to find new ways of generating data that tell us what's going on in the world ... what did happen, what will happen. We're not going to get 100 percent in terms of outcome, but we can pull things together in a way that no one else can."

The idea is to give users an ability to see events or relationships in sequence to make it easier to find patterns and relationships that traditional Big Data programs might miss. Hedge funds already use Recorded Future to invest. The intelligence community could use it, among other things, to help predict seminal events before they happen.

'Time Is Often A Forgotten Dimension'

Ahlberg is a former member of the Swedish Special Forces, and his first company, Spotfire, did something similar with business information. It allowed companies to visualize and analyze information in their internal databases. He sold the company for $195 million and started Recorded Future with the intention of making the entire web just as searchable and friendly.

"So, what we are trying to do is figure out how we can take large portions of the web and extract what we call signals of activity that relate to people and places and associate them with events and time," Ahlberg says. "Time is often a forgotten dimension in analysis, and we think it is key."

As Ahlberg sees it, there are hints about the future everywhere. Governments release economic projections; newspapers report on upcoming events; Twitter can provide a pretty good idea of what people are talking about.

In Egypt last year, organizers used Twitter and social media to rally protestors. If intelligence analysts had had a systematic way to track those posts, it might have helped them forecast what was to come.

This isn't a new idea. There have already been efforts to try to tap into what is simmering below the surface by tracking things like Google searches. Researchers at West Point's Combating Terrorism Center and Princeton University tracked Google searches in Egypt starting in January 2011 and found that there were more searches about events in Tunisia and its protests than for Egyptian pop stars. Recorded Future builds on that kind of public intelligence.

"An event we'd track might be people traveling from A to B ... people talking to each other ... a government guy making a statement, a country doing a military maneuver," Ahlberg says. "We capture those activities ... they can be small scale or large scale — and then time is associated with that. What we're doing is organizing the data in a way so you can ask the right questions of it."

Making Smart People Smarter

Steven Skiena is a computer scientist at Stony Brook University in New York who has started his own Big Data company to track consumer sentiment. He says forecasting the future isn't easy.

"I guess it was Yogi Berra who said, predicting is hard, especially about the future," he says. "The Recorded Future idea seems to be if you can get ideas about what other people will be saying is going to be happening in the future for certain kinds of events, it is probably very good information."

In January 2010, Recorded Future predicted in a roster of blog posts that Yemen was headed for disaster. It predicted that a combination of floods, famine and Islamic terrorists were conspiring to wreak havoc. The company used Twitter feeds, blogs, U.N. food program data and news sources to come up with their forecast.

"Yemen took five months longer than we predicted, but if you go back and look at our earliest blog posts, it's all there," Ahlberg says, laughing. "That said, we didn't predict the Arab Spring."

When it comes to computers predicting the future, there are plenty of skeptics. Gary King, a professor from Harvard University and the director of the Institute for Quantitative Social Science, is one of them. "We need to know if you are getting the right answer randomly or you're getting the right answer because of something you're doing," he says.

Recorded Future's Yemen prediction is a start, he says. But it isn't enough.

"It would be a miracle if you got it right all the time," King says. "Picking one prediction out of a hundred and showing that it was consistent with the future is essentially irrelevant ... and 50 percent of the time sounds like luck."

King says there are one billion new social media posts every two and a half days. And while that's an enormous amount of information to mine, it's easy to draw the wrong lessons from all that data, he says.

For example: After the attack on the U.S. consulate in Libya, Ahlberg of Recorded Future sent me a prediction that the attack would eventually be traced to al-Qaida's arm in Yemen, known as al-Qaida in the Arabian Peninsula. That's the same al-Qaida group that was behind the Christmas Day attack on a U.S. airliner in 2009.

Recorded Future came to that conclusion because its algorithm linked a militant group in Libya, Ansar al-Sharia, with an al-Qaida group in Yemen with the same name.

The problem: They share the name but have no other connection. Ahlberg says it happens. He declines to put a figure on the company's prediction track record. "When we do our blogging, it is not meant to be NPR and it is not meant to be academic," he says. "We use that to show highlights of things."

The people who use Recorded Future, Ahlberg says, are experts, so they would catch that kind of mistake.

"The unfortunate aspect of being called Recorded Future is that people expect to push a button and find the future of everything," he says. "We're just trying to visualize data in a way that makes smart people smarter."

Ahlberg has about 100 subscribers and at least two very important financial backers: the CIA's investment arm, called In-Q-Tel, and Google Ventures. They have reportedly poured millions into the company, and maybe they see something about the future that the rest of us don't.