September 21, 2014
If you’ve been at all plugged into the technology discussions of the last couple years, you’ve undoubtedly encountered the buzz around “big data.” In an age where everything is super-sized, we’re sold on the idea that “more is better”. We have wall-sized TV’s and Pop Tart-sized cell phones. Everywhere we turn, something is being promoted as maximized, extended, increased or extreme. In this echo chamber of overkill, it would be easy to make the mistake of lumping big data in with the rest. Big data is changing the way that business is done, that goods and services are sold, that strategy is formed. Big data presents big challenges and even bigger opportunities. A little knowledge goes a long way in solving the former and positioning you to take advantage of the latter.
So, What is Big Data?
In today’s technological world, almost everything we do generates some sort of data that is collected. Cash registers, cell phones, email, websites, video games, virtually every electronic touchpoint in our lives generates collectible data. As a result, our ability to amass information has far outpaced our ability to process and act on it.
According to a 2011 article in the journal Science, the world’s technological per-capita capacity to store information has roughly doubled every 40 months since the 1980s. And in 2001, Gartner analyst Doug Laney pioneered the 3Vs description of big data: volume, velocity and variety. Since then, many have added veracity, a fourth V, to the equation. So, by this definition big data refers to large amounts of complex, diverse, accurate data flowing at tremendous speed—typically impossible to process using traditional data analysis tools.
To grasp the scope and scale of our information overload, here are two examples of the sort of growth that technology is driving. In 2000, the Sloan Digital Sky Survey began collecting astronomical data. Within a few weeks, it accumulated more data than had been collected in the entire history of astronomy up to that point. When the next generation survey telescope goes online in 2016, it is expected to duplicate the total data collected by Sloan EVERY FIVE DAYS! Closer to home, Walmart conducts more than a million customer transactions every hour, and has amassed a store of data that exceeds the total information collected in the Library of Congress 167 times over.
Years ago, former Google CEO Eric Schmidt was quoted as saying, “Every two days, we create as much information as we did from the dawn of civilization up until 2003.” And the pace has only escalated since then. The World Economic Forum has recognized the power of big data and now categorizes it as an economic asset, potentially as valuable as natural resources like oil or gold. Data is the currency of the information age, and we are multiplying our data wealth exponentially every day. But the true value of big data will be determined by our capacity to process and use it.
What Are The Challenges of Big Data?
In a 2012 Oracle survey of more than 300 C-level executives in the U.S. and Canada, 94% said that their organization was collecting more business information than they had in the past. We aren’t talking about a little increase, either. On average, they reported data volumes up 86% since 2010. More importantly, 93% believed that their organization was losing significant revenue because they were failing to make use of data they collect, and a whopping 97% said that their organization would have to make significant changes over the next two years to better leverage the information they collect. That two years brings us to today.
A recent Wall Street Journal blog post does a nice job breaking down the challenges into six primary categories:
- Identifying the right data and determining how to use it
- Hiring or training a workforce capable of working with the new technology and interpreting the data
- Achieving appropriate access and connectivity to be able to aggregate and manage the data
- Finding a technology partner to create an IT architecture that can efficiently adapt to the ever-changing landscape
- Developing the organizational dexterity and leadership (Like a Chief Digital Officer) to leverage data collaboratively across functions of IT, marketing, engineering, finance, etc.
- Maintaining informational security
Each of these points could be expanded upon at great length, but for our purposes here, let’s use them as a basis for establishing the questions your company should be asking and answering.
- What data do you currently collect?
- What business function(s) do you want your data to accomplish?
- How can you use your data increase the value of what you do?
- Is collecting data from your own transactions enough, or should you be accessing data sources external to your company as well?
- Which areas of your business are most likely to benefit from the information being collected?
- Are there security or discriminatory concerns around personal information contained within your data?
- Which departments within your organization have to work together to process data and use it to its best effect?
- Is your technology sufficient for the current task? Will it still be five years from now?
- Is your size and knowledge base of your leadership and staff sufficient for the current task? Will it still be five years from now?
- Can your new data be integrated into your existing business processes, or is a new model required?
- Are there ways in which improved data would reduce costs?
- Are there ways in which improved data would save time or other resources?
- Are there ways in which current decision-making suffers from a lack of actionable data?
- Are there new products or services that better data might inform?
Where Does Big Data Go From Here?
Big data is being used by governments, health care organizations, venture capitalists, banks, and businesses of all sizes, and toward very diverse ends: predicting traffic congestion, maximizing energy availability, fighting cancer, anticipating consumer behavior, identifying business trends, optimizing warehouse usage, fighting crime, fraud detection, improving customer satisfaction, and of course a myriad of advertising and marketing uses.
Regardless of your industry, big data is a tool that can be used to great advantage, but that must be used carefully and responsibly. And if it can’t be efficiently processed, too much information can be just as useless as no information.
Integration into your business strategy and structure will inevitably be a challenging process, one that will take most companies well beyond their established comfort zones. The challenges faced by today’s CIOs, CDOs, CMOs and data scientists are reshaping how information is gathered, processed and valued. For the companies that rise to the occasion, big data represents an opportunity to disrupt established markets and alter the competitive landscape forever. How will you leverage big data to rise above your competition?