The expansion of the mobile marketplace over the last decade has brought companies access to exponentially more data than they previously had on consumer behaviors and desires.  Too much data, as it turns out.  Collecting meaningful data has become a major challenge.  The old tools simply may not work. In a new paper, George Day,  Co-Director of the Mack Center for Technological Innovation and Professor of Marketing at the Wharton School, says there is a dangerous gap between the potential value of all the data and the capacity of companies to adequately analyze that data:

The hypothesis that organizations are not keeping pace with market velocity and complexity is more difficult to test. Suggestive evidence comes from several sources. The first is the vast literature on information overload, which describes how an excess of information has resulted in the loss of the ability to make decisions, process information, and prioritize tasks (Eppler and Mengis 2004; Klingberg 2009; Meyer 1998). The second is the equally large literature on organizational adaptation in the face of environmental change (ranging from Miles and Snow [1978] to Hamel [2007]).

Still, there is no longitudinal measure of the size of the gap. Some evidence comes from recent estimates that the amount of data available expanded at an exponential rate from 100 billion gigabytes in 2005 to 1000 billion gigabytes in 2010 (IDC 2007). This suggests an even greater rate of growth than Davenport and Harris’s (2007) claim that unique information per person is growing at 50% per year. In contrast, they estimate that information consumption per person is only growing at 2% a year. Taken together, a reasonable case can be made that the deluge of data has run up against the barrier of the limited ability of people and organizations to process it. The evidence sug- gests that the volume of inbound data and the proliferation of channels is going to continue for the foreseeable future. Absent any breakthroughs in human beings’ ability to process data, unless new tools and approaches are adopted, the gap will continue to grow.

There are other reasons to suggest that the gap is growing and that new approaches are needed to begin closing it. During periods of technological disruption, most organiza- tions have trouble keeping pace. This is true of the effect of the Internet and cheap, ubiquitous communication technologies on the habits and behaviors of consumers and the creation of new business models for reaching these markets. The tendencies toward inertia and sclerotic decision making are fed by lag effects and organizational rigidities.

Read the full paper here.

And watch Day and colleague David Reibstein discuss the "data deluge" problem in this Knowledge@Wharton interview: