Photo by Victoriano Izquierdo on Unsplash

Make “Evidence-based” Decisions, Not “Data-based” Decisions

The shift in language leads to a shift in context, thinking and action, all for the better.

TL;DR — Thinking about “evidence” vs. “data” when making decisions, expands the thinking, actions and discussion. It allows for a broader set of information to be considered. It also leads to a more structured approach to evaluating the information you have to make your decisions which can lead to better decisions overall.

Contents

1. Introduction
2. Language Matters
3. Evidence vs. Data
4. Use Evidence, Not Data
5. Types of Evidence
6. Strong vs. Weak Evidence

I recently posted the following on LinkedIn and some healthy discussion followed. I thought it was worth elaborating it further for those who see this as helpful.

I now use the word EVIDENCE instead of DATA when asking Product Managers about their decisions. e.g.

“What EVIDENCE did you use to come to that decision?”

This simple reframing changes the activity & discussion immensely.

Data implies….data. Numbers, analytics and other similar things.

Evidence implies a broader range of information, including data, but also other relevant information.

It’s a subtle shift, and some people may not appreciate the difference, but in actual use, it opens different avenues of thinking and thus action and discussion.

Evidence >> Data

Try it.

We’ve all read or heard the following quote, or something similar.

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

— Jim Barksdale, Former CEO of Netscape

To be honest, until I looked it up, I thought Andy Grove, former CEO of Intel, had said it.

Regardless, the idea of Data >> Opinion is absolutely true when making business decisions. And certainly there’s been a lot of discussion in recent years for Product Managers to be data-driven, data-informed, data-aware etc. This is a good trend. i.e. the move away from opinions and towards more fact-based decision making.

Language Matters

But language matters, and the language we use confers meaning and intent and ultimately influences and impacts our actions.

Remember “Move fast and break things”?

Yeah, well, people broke things; sometimes important things that had implications. A few years later, Mark Zuckerberg, who proudly touted that first mantra, changed it.

It became “Move fast with stable infra”.

It’s not as catchy as the first line, but certainly better advice given that breaking things, especially for a public company with a billion+ users, is not a great idea.

Getting back to “data”, when someone uses the term data, what comes to mind?

Numbers….analytics, perhaps something in a database or spreadsheet or table. There are phrases like “supporting data”, when thinking of questions like “What supporting data did you use to make your decision?” but the word data has implications and to a certain level, baggage.

Evidence vs. Data

Let’s take a look at these two words, data and evidence.

The dictionary defintions are:

DATA
noun: facts and statistics collected together for reference or analysis.

EVIDENCE
noun: the available body of facts or information indicating whether a belief or proposition is true or valid

Data is somewhat neutral. It’s just a collection of facts/statistics for some purpose.

Evidence is more specific. It’s a collection of facts/information aligned with the truth of some proposition (or hypothesis).

To me, this distinction, even if you’re not familiar with the exact dictionary definitions, is connoted when you use these words. That why, for example, one may often say “supporting data” used in a decision. i.e. the word data itself is not sufficient.

Use Evidence, Not Data

I now try to avoid the word data when evidence is the right word. I use data when appropriate, but when asking about the “supporting data” for decisions, I use the word evidence. e.g.

Can you share the evidence you used to come to that decision?

What I’ve found is that this change — from “data” to “evidence” — changes the nature of the thinking, discussion and actions of people. It opens up their thinking as well as the discussion to lots of different forms of evidence.

Yes, this was always possible before, but for whatever psychological or other reasons, it wasn’t happening as much before as after the switch.

Think about these two questions that might be asked at the end of a short meeting:

Does everyone understand what we have to do?

Does anyone have any questions about what we have to do?

The intent of the two questions is the same— to ensure comprehension of the task at hand — but the implications of the questions are very different. The second will most likely elicit far more response than the first.

The Evidence vs. Data distinction is similar. The change in wording changes the response.

Evidence can be numerical/analytic data, but evidence often is more qualitative than quantitative. Evidence can include stories and anecdotes related to the decision, expert opinions, discussions with customers, partners or prospects, what-if scenarios or anything else that can support the action or decision at hand. In the best of cases, evidence includes the findings of large-scale, directed research into the topics or questions of interest.

Photo by Patrick Perkins on Unsplash

Types of Evidence

In law, which is where we often hear about evidence, there are many different types of evidence: physical evidence, circumstantial evidence, demonstrative evidence, exculpatory evidence etc.

It’s not necessary to understand what all those are, but to understand that evidence comes in different forms with different intents and purposes.

For Product Managers, for simplicity, we can think of evidence in the following ways:

Primary (Direct) Evidence

This is foundational evidence, attained through discovery, experiments or other targeted work that relates directly to our hypothesis or decision. In law, this might be called “Real Evidence”.

Supporting Evidence

This is evidence that is not sufficient on its own to be primary evidence for a hypothesis or decision, but helps support the primary evidence that exists.

This could be anecdotal information, 3rd party research, data points from customer support or other sources. In law, this might be called “Corroborating Evidence”.

NOTE: These evidence definitions are neutral. They don’t imply that either the Primary or Supporting evidence is confirming or disconfirming. i.e. it may support or contradict our hypothesis or proposed decision.

Supporting (or Confirming) Evidence

This is what we think of most when we hear the word evidence. i.e. some kind of information that supports what we are looking to achieve. e.g. a decision we want to make.

But it’s important to keep in mind that we need to be neutral in our work. e.g. our goal should not be to validate or prove a hypothesis correct, but to investigate it and conclude based on the evidence if it is or isn’t correct. So we also need to search for and consider disconfirming evidence.

Disconfirming Evidence

This is information that does NOT support our thesis or runs counter to what we believe or want to enact. It may be tough to accept, but it’s important to explicitly think about (in fact seek out) disconfirming evidence, lest we fall prey to confirmation bias, by only looking for ways to support what we believe.

So in a discussion, we can ask questions like:

• What is the primary evidence to consider?
• How well does the primary evidence support our thesis?
• What supporting evidence have we found?
• Is there any disconfirming evidence? How did we find it?

The language matters. The distinction and separation of the primary and supporting evidence keeps all the information from getting blurred together and focus on what’s truly important and helpful.

The explicit call for disconfirming evidence is needed to make sure a more holistic discussion happens and decisions are made without obvious blindspots.

e.g. in a SWOT chart, we look at Strengths AND Weaknesses, Opportunites AND Threats. If we only looked a Strengths and Opportunities, the world may look great, but we’d quickly find out it is anything but.

Strong Vs. Weak Evidence

While there are different types of evidence, there are also different levels of evidence. We can think about it as the level of quality of the evidence or the strength/weakness of that evidence.

If we think about evidence quality, we can think about it in several dimensions.

• Amount/Quantity
• Completetness
• Consistency
• Relevance/Appropriateness
• Age
• etc.

I won’t go through all of these, but think about them in terms of the information you’d consider when making decisions, then factors such as:

• Amount (interviews with 3 customers vs 10 customers, survey with 20 responses vs. 200 responses etc.),
• Completeness (does the evidence paint a full picture of the situation or only provide parts or snippets of the picture)
• Age (a market study from 5 years ago vs. one from last year)

All factors can be assessed as part of the discussion, and an assessment of how strong or weak the evidence is can be made.

Now any assessment will be somewhat subjective, but we can decompose that a bit more and come up with a scale to decide on level of strength or weakness.

The following is an example of a way to measure the strength of evidence in a discussion or decision. You can (and should) define your own breakout suited to your own needs:

Levels of evidence

Level 0 is the lowest level, and denote Opinion. Our opinions do form based on our experience, but opinion without any clear supporting evidence is the lowest form of evidence. It’s basically a guess.

There is a progression of more rigour, completeness, consistency etc. as you go from Levels 1 to 5 with 5 (in this example) being the strongest evidence.

Let me repeat that this is just AN EXAMPLE of a breakout of evidence with higher level implying stronger evidence for a given context. It’s not necessarily fully comprehensive and may not match your environment, and that’s fine. View it more as instructional in nature vs. representative of all cases.

So there you have it. Think Evidence instead of Data and consider the types of evidence and strength of evidence as you make your decisions.

A little feedback please

If you’ve read this far, thank you. I’d like some feedback on the article to make it better. It should take just 1 minute, but will really be valuable to me. Thanks in advance.

===> Click Here <===