The February 27, 2010 issue of The Economist contains a special feature on the data deluge assembled by its Tokyo correspondent, Kenneth Cukier, who observes, "Information has gone from scarce to superabundant. That brings huge new benefits but also big headaches." Among the many fascinating bits of information to be found among the articles the series comprises is the finding by the University of California in San Diego that the average American household is bombarded with 3.6 zettabytes of information (or 34 gigabytes per person per day). The biggest data hogs were video games and television while, in terms of bytes, written words are insignificant, amounting to less than 0.1% of the total. Data, The Economist reports, is accumulating at a rate that exceeds the capability to store it.

Cukier makes an interesting point, writing that, epistemologically speaking, information is made up of a collection of data and knowledge is made up of different strands of information. ... Given enough raw data, today’s algorithms and powerful computers can reveal new insights that would previously have remained hidden.

He goes on to note that the amount of digital information increases tenfold every five years. Moore’s law, which the computer industry now takes for granted, says that the processing power and storage capacity of computer chips double or their prices halve roughly every 18 months. The software programs are getting better too. Edward Felten, a computer scientist at Princeton University, reckons that the improvements in the algorithms driving computer applications have played as important a part as Moore’s law for decades.

What, then, are the implications of these developments, these "advances" if you will? The series suggests that the shift from information scarcity to surfeit has broad effects. “What we are seeing is the ability to have economies form around the data—and that to me is the big change at a societal and even macroeconomic level,” says Craig Mundie, head of research and strategy at Microsoft. Data are becoming the new raw material of business: an economic input almost on a par with capital and labor. “Every day I wake up and ask, ‘how can I flow data better, manage data better, analyze data better?” says Rollin Ford, the CIO of Wal-Mart.

And if it becomes possible to deduce more and more information from this wealth of data, the implications for an improved state of knowledge are enormous. As an example, Cukier cites the experience of both Craig Mundie of Microsoft and Eric Schmidt, the boss of Google, who sit on a presidential task force to reform American health care. “Early on in this process Eric and I both said: ‘Look, if you really want to transform health care, you basically build a sort of health-care economy around the data that relate to people’,” Mr Mundie explains. “You would not just think of data as the ‘exhaust’ of providing health services, but rather they become a central asset in trying to figure out how you would improve every aspect of health care. It’s a bit of an inversion.”

Failing to comprehend all of this data and to reduce it to useful information can have dire consequences, however. The world is still reeling from the crisis of data mismanagement that beset the banking and finance industry that erroneously relied on models which, although they required a vast amount of information to be fed in, failed to reflect financial risk in the real world. As Cukier writes, "This was the first crisis to be sparked by big data—and there will be more."

Articles in the series

All too much: Monstrous amounts of data

A different game: Information is transforming traditional business

Clicking for gold: How internet companies profit from data on the web

The open society: Governments are letting in the light

Show me: New ways of visualizing data

Needle in a haystack: The uses of information about information

New rules for big data: Regulators are having to rethink their brief

Handling the cornucopia: The best way to deal with all that information is to use machines. But they need watching [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter