During the Second World War, there was a key issue that researchers looked to data to solve. During their flights over Germany, bombers were getting shot down. The U.S. Center for Naval Analysis decided to start looking at the data coming back from these flights to assess what was the cause of the chink in the armor that was causing planes to go down. They scrutinized the data that was coming back from each mission. The below image is a summary of the data reviewed.
The solution to the problem looks clear. The army needed to strengthen the body and the wings of the aircraft to reduce the number of future missions falling to the same catastrophic fate. Just before the planes were due to be strengthened, a statistician by the name of Abraham Wald took a closer look at the information that had been collected. He found a major flaw.
The researchers had made the mistake of reviewing data from pilots and aircrafts that had returned from their mission. Not a single fallen plane was included in the data. In short, the data had inadvertently summarized the strengths in the planes and not the weaknesses. The bullet holes reviewed and recorded were the aircrafts' strength. It was actually the black areas in the image above that needed to be strengthened. Those shot in the white areas were managing to return from their mission, indicating that shots to the dark areas caused the fatalities. Data is great, but are you looking at it correctly?
Terms like data and big data are terms we hear thrown around a lot. It’s something that we perceive many big companies like Google to benefit from consistently. When it comes to smaller businesses and startups, data is undoubtedly part of the strategy for growth. We integrate it into our ad set up processes by building buyer personas. We use it to make better products. We create ideal customer profiles and we leverage the detailed audience descriptions and options we have in Facebook and Google to grow and scale our businesses online. So what’s the problem? Our assumptions are often wrong. There’s a myriad of reasons for the failures businesses have when leveraging data online. The most common being how our data is collected to begin with. We often make incorrect assumptions when deciding which data is correct and incorrect for our businesses.
When it comes to the SME sector, one of the most common ways incorrect data is collected in my experience is from internal teams. The decision is made to create an ideal customer profile/buyer persona and a key individual from each team is brought together into a meeting. Someone from sales because they speak to the customer day in day out, someone from product development to find out if they can add additional features for the buyer persona. The list goes on. The problem becomes the same as the WWII aircrafts. The fallen aircrafts – aka the customer – are missing from the data collection. The data may not be incorrect, it has just become skewed. Even in the case of the sales person. Yes, they talk to the customer everyday, but they are probably inadvertently biased due to targets they need to meet or other factors that impact their perception of the customer.
We see issues even when it comes to the purveyors of big data themselves. In 2008, Google started what would end in a spectacular failure, a project attempting to predict outbreaks of the flu virus. During the years that followed, Google began creating autosuggestions for search terms. This changed the behavior of users navigating Google. It made some search terms more prevalent, particularly in the health space, creating a skewed data collection process. In 2013, it inaccurately predicted the flu season by 130%. It turned the project from being a key example of how big data could be used for the greater good into the poster child of the impact of inaccurate data collection. Correlation does not always tie itself to reason. Just because there was an increase in the search data coming in for Google didn’t actually mean the flu season was about to happen.
This isn’t to say that data is bad or that using data for your business will end in failed results. Data is vast and even getting a portion of it correct can be profitable for many businesses. The key to data is to collect, manage, and monitor it from as many perceptions as possible to mitigate bias or inaccuracy. Here are some key ways to create and maintain reliable data.
Continuously maintain your CRM and data sources
Data management can be a pain, but it is important when it comes to pulling out data that is no longer relevant or potentially impacting your business. Verify your data and ensure duplication has not occurred on your email or CRM systems, for example. Take a close look at how you acquire and verify data. Is your data being input by a human or technology? Are there issues with errors in either of those systems? In my experience, it’s best practice to prioritize technology for data input as the error rate is lower. Are you running competitions for your business for lead generation? If so that data is not technically correct either. People want the prize, not your service. Take a step back and look holistically at where your data comes from, if you’re creating inaccuracies in the data collection process and once verified is your data being stored correctly for proper utilization?
Reduce the risk of bias in your feedback pathways
Getting feedback from customers or clients is a key for any business to grow and better themselves in a valuable way. But whether you’re undertaking market research or navigating product improvement, eliminating as much bias as possible from the information you’re looking for is key. One key issue is response bias. Here’s an example. You own a coffee shop and you send out a survey to people with the question “Did you like the coffee”. Due to human nature many people will say yes, even if they didn’t, just to be nice. A great way to mitigate response bias is to use open ended questions and ask for people’s opinion. “How would you describe what you like or dislike about our coffee?” or “In your opinion, what are improvements we could make to our product?” This is always going to be better than “What makes our service awesome?” or “What’s the best thing about our coffee?"
Include the customers in your conversations when building ideal customer profiles or buyer personas.
Coming together with the internal team is a great way to build the framework for your ideal customer profile. Just don’t forget to include the customer in your research – as many of them as possible. For some reason, many of us are afraid to pick up the phone and get into the weeds of research. If you call 50 customers, you slowly start hearing the same issues crop up time and time again. Those are your pain points and the problems you need to solve. It is a vital aspect of creating customer personas that resonate when it comes to the ad creation process for customer acquisition. When interviewing and researching customers, do a little research into the world of response bias before putting together your question sheets and try to eliminate as much possible bias or inaccuracies as possible.
Getting data that is 100% accurate is always going to be difficult and a near impossible task. There will likely always be something small overlooked. Creating data that is as accurate as possible is the goal and it can be done by making small changes to your data collection, maintenance, and usage.
In short, when you’re looking at your data, ask yourself one question: Are you looking at the data of the survivors or the full picture?
About the Author: Jen Bryan is the Director Of Growth in Growth University. They provide unparalleled courses on growth strategies, paid acquisition and more. They are quickly becoming the go-to resource for founders who are looking to build their playbook for growth. Inside readers are offered an exclusive 20% off their first month of Launch Growth University membership by using code INSIDE20