While big data has become a trendy catchphrase, the good news is that there is real substance to it. With a little effort, even nontechnical people can understand that substance and start putting it to work for their companies.
Part of demystifying the trendy catchphrase “big data” is understanding that you’re analyzing your business using techniques of statistical analysis, some of which have been around for 50 years or more.
What is fundamentally different about the 21st-century phenomenon of “big data” is the computing power we can bring to bear. Advances in the sensors that collect data, the drives that store it, and the software and hardware to analyze it mean that we can efficiently analyze far more material than was feasible in earlier centuries.
It’s no longer hard to create and store gigabytes of data—the challenge is to find something meaningful in all of that material. What makes analyzing the data such a rich source of business insights?
Big data is good at finding correlations but not at causality
A great place to start is with the distinction between “what you like” and “why you like it”—or what is technically called the difference between correlation and causality. These algorithms don’t know why you like what you like. But they have learned what you will like based on what you’ve purchased before.
From a business perspective, that’s OK—what matters far more than why. Knowing what you will like drives clicks and sales. Skilled data scientists have a host of statistical techniques—some new, some old—for analyzing information. Before you start working with a data scientist, however, there’s an important question you need to ask first.
What’s the type of dataset you want to learn more about?
If you don’t ask this all-important question, you could get overwhelmed with raw data. Many executives feel pressure to just do something with big data, so they begin collecting without a clear goal in mind.
If you do “track everything,” you’ll still have to go through that data again once you figure out what you’re trying to do. And in the meantime, you’ll be racking up software, hardware, and personnel costs.
A key takeaway? Don’t just rush in and start tracking everything. The best way to get started is to look at the types of problems people have successfully attacked with big data in order to see what you might accomplish in your business. Here are a few examples:
- Branding: Look at mentions of a product on Twitter in order to derive an analysis of “customer sentiment.” By collecting mentions of your brand from Twitter, data scientists not only can tell how customers feel about it but also how strongly they feel about it. Data scientists can also then help you automate your responses: re-tweeting of positive comments, and prompt, private messages to unhappy customers.
- Market research: Analyze your past sales records to segment your customer base so that you can find and target like-minded clusters of people with carefully customized marketing campaigns.
- Operations: Analyze the geolocation data of your delivery drivers to optimize the most efficient routes in terms of gasoline usage and time. Data scientists can compare up-to-the-minute data about where your vans are on the road with historical data about what routes are congested with vehicles or require time-consuming left-hand turns across traffic.
- Production optimization: A large beverage company used data to find the optimal blend of different kinds of oranges, which have different costs, astringency, sweetness, and tartness, in order to maximize profit while maintaining quality standards.
- Research: A large hedge fund hired researchers to keep track of real-time news on 200 companies at a time. The team was spending so much time seeking data, like looking for company press releases, regulatory sites, SEC filings, and updates to company websites, that they couldn’t keep up with all of the changes. Data consultancy BrightPlanet put together an algorithm to search the Internet and compile information automatically, freeing up the team to focus on analyzing the findings.
Tips for analyzing big data
There are some unusual features of massive datasets that you should keep in mind.
1. The “messiness” of big data
You may be surprised by how much time your consultants are using on a stage of the project called “data preparation.” Don’t be. Because computers, databases, and algorithms have gotten so fast, getting large datasets, often disorganized and drawn from multiple sources, in a position to be analyzed is quite challenging. “
Data scientists unabashedly describe their datasets as “messy.” (That’s really the technical term for it.) Imagine, for example, you tell a web-crawling algorithm to compile massive amounts of press releases, tweets, news reports, and government filings from different websites and in different formats. The results from the web-crawling algorithm are not going to consist of neat, well-organized rows in a spreadsheet or fields in a database.
This “unstructured” data will need to be “cleaned” or made uniform in a way that algorithms can analyze. That’s why “data preparation” often takes so much time.
2. You don’t need to sample
Unlike the analog days of statistics, when you might have given a survey to 1,100 people to stand in for your entire customer base, computing power today means you can look at all the data. And using all the data instead of a sample can make an enormous difference.
3. “Datafication“
Viktor Mayer-Schönberger and Kenneth Cukier coined the term “datafication,” meaning that inexpensive sensors, hardware, and data storage have made it possible to collect certain types of data that were impractical to track previously.
4. Data exhaust
Because storage and collection has gotten cheap, you can save the equivalent of data “junk” and perhaps find ways to use it. For example, Google receives a large amount of search queries with typos or misspelled words each day. The company has taken this “exhaust” from its lucrative search engine business in order to not only improve search (“Did you mean ornithologist?”) but also to build a powerful spell-checker.