Addressing the Challenges of Product Discovery — Q&A Edition
A followup to my recent article Product Discovery
Two weeks ago, I published this article on Addressing The Challenges of Product Discovery. I received a lot of feedback from people about it and also received a number of questions about discovery. So I decided to answer them here as they are very relevant to the topic.
Q1: How much time should be spent on understanding buyers vs. users?
Q2: How much time should be spent on discovery?
These two questions are related and have similar answers so I’ve grouped them together.
The short answer for both questions is: as much time as necessary.
Yes, I know that’s not a great answer, but it’s really the only correct answer. As I mentioned in the original article, discovery is not a deterministic task. It’s difficult to predict what you will learn or how long it will take.
Asking how much time should be spent on discovery is akin to asking how much time to spend studying for a test. It depends on the type of test and generally you study until you’re ready.
It’s important to think of discovery as an ongoing task, where you’re learning and leveraging knowledge in whatever ways you need.
Decision Making
Let’s keep in mind that the goal of discovery/research is to gain knowledge and understanding to make informed decisions. It’s always the case that we make decisions based on incomplete information. If we wait until we have ALL the information we need, then it’s no longer a decision, but a calculation.
I sometimes talk about what I call the “Decision Zone”. This is a range of knowledge and understanding somewhere between a calculation on the right, and a “wild-ass” guess on the left. It’s somewhere between 50%-80% certainty in my opinion.
In his book Blink, Malcolm Gladwell says:
The key to good decision making is not knowledge. It is understanding. We are swimming in the former. We are desperately lacking in the latter.
Reversible vs. Irreversible Decisions
I’m drifting a bit from the discovery topic, but this an important adjacent topic. There are many good decision frameworks, but one that is relevant and applicable here is Reversible vs. Irreversible Decisions.
In short, all decisions should be informed, but the ones that are low risk or reversible — i.e. that won’t cause much harm if they’re incorrect — should be made quickly, monitored and reversed if needed. As the article says:
Reversible decisions are not an excuse to act reckless or be ill-informed, but rather are a belief that we should adapt the frameworks of our decisions to the types of decisions we are making. Reversible decisions don’t need to be made the same way as irreversible decisions.
But those that are high-risk and NOT reversible — e.g. making a major new investment or an acquisition or turning down a potentially large opportunity — should be made with more information, knowledge and understanding. It is better to wait, gather evidence and minimize the risk in making an incorrect irreversible decision.
Getting Back to Discovery
So from a discovery perspective, how much time you spend really depends on the types of decisions you need to make. If you’re researching a well-understood problem for an upcoming release, it’s OK to time box it, get to a reasonable level of understanding and execute.
But if you’re looking at a whole new market to venture into, or a new product that will require significant investment, spend extra time and effort up front, understand market dynamics, customer use cases, competitive forces, financial and other risk factors, and then make the decisions you need to make.
Q3: We regularly get into internal cost/benefit discussions with our management. e.g. if we spend 6 weeks on discovery at a cost of about $75,000 (time, salaries, other expenses), can we show we will get a much greater return? This usually shuts down discovery work because obviously we can’t guarantee that.
The context here was that the company DOES do evaluative discovery work to get more details on use cases or specific features they plan to build, but the challenge of doing generative (exploratory) discovery is where the friction arises.
Discovery should be a core part of the basic “operating system” of a product company. CEOs and leaders need to understand that discovery is an investment in the future success of the company.
There is no way to predict what kind of return one can get from discovery work. It could be huge or it could small. But you can ABSOLUTELY predict what return you’ll get if you don’t do it; and that is ZERO.
In fact, if you’re not doing good discovery work and your competitors are, your return is actually negative in the context of market opportunity.
What is your “Discovery Mix”
Discovery should not be seen as some one-off or isolated activity that is done “when needed”. Yes, there ARE times when specific research is done on specific topics with specific intent. But that is only one form of discovery.
Consider discovery in the same way that you consider marketing. Marketing is a holistic effort by the company to raise awareness and education in the market, and create interest and demand for your products.
There are many types of marketing activities, from large product launches, customer conferences, roundtables, advertising and media placements, all the way down to SEO, PPC ads, webinars, social media presence etc.
Not every company will do every activity all the time, but there is clear value in a healthy mix of big/small, broad/targeted marketing campaigns. And you know that without all of it, your business will be hampered at best, and fail at worst.
Just like you have a “marketing mix” of different sized/focused marketing activities, you should also have a “discovery mix” of different sized/focused discovery activities.
The same mindset and culture should be understood about discovery. It’s an intrinsic part of business and a driver of growth. It should be looked at holistically as well.
Yes, there are (big bang) discovery projects that need to be done from time to time, but also a lot can be learned from ongoing smaller activities. Just like companies have a “marketing mix” of different activities/impacts, product companies need to consider a “discovery mix” with the same intent.
This means planning (and budgeting for) proactive (generative) research/discovery, and also smaller regular customer studies, competitive research, ongoing evaluative research that may (or may not) be specifically tied to features you are building etc.
As an example — and let me be clear there is no “one size fits all” formula here — when I was working as a Product Manager, I would have a mix that looked something like the following:
- Regular contact with customers/partners for discovery (average 1–2 hours/week)
- Focused discovery projects (mostly evaluative) as needed (several times) per quarter
- Large scale discovery projects with extended team (generative and evaluative) at least once per year (tied to large strategic initiatives)
- Other customer facing activities such as surveys, user groups, user conference discussions, webinars, sales/support call participation,
- Regular (and ad hoc) meetings with internal customer facing teams to understand their contexts, findings, feedback etc.
These were the main activities, but If I were to add it all up, it was probably about 20%-25% of my total working time.
The regular customer contact (calls/emails) and contact with internal teams was about gaining signal on the realities faced by users, buyers etc.
Over time, patterns would emerge and I’d use other avenues (surveys, user groups, etc.) to further understand those signals and then define focused discovery projects based on whatever we deemed important to understand.
The large scale discovery project with the extended team (Eng, UX and others) was tied to larger, longer term strategic areas of business or product. e.g. one year it was on data migration (a big industry focus at the time), another year it was workflow capabilities, and yet another year it was large scale automation capabilities for my product.
So the “discovery mix” was layered with the lower effort, higher frequency activities helping feed into the larger ones.
Q4: We do a lot of discovery work, but often the feedback is that our findings are not really new or are not very helpful to the rest of the business. What are we doing wrong?
It’s hard for me to diagnose this specifically, but the general questions I have are:
- What are the objectives of each of the discovery projects?
- Are you meeting the objectives on each project?
- If yes, then where are these missed expectations coming from?
- If not, what is missing? i.e. where are you lacking in the findings based on objectives?
Define Clear Objectives
Whenever you start a discovery project, have clear (ideally measurable in some way) objectives defined that you can focus your efforts towards. The objectives should not be output based (e.g. to interview 15 people etc.), nor should they be overly specific outcomes. (e.g. to identify a new market opportunity that will generate X amount of revenue…).
In all of my generative research projects, the objectives were qualitative ones (to start), such as: “to get a deeper understanding of the data migration market and identify potential opportunities for the product roadmap”.
This was based on a strategic direction set out by our executives to expand our market presence into other adjacent market categories.
In this case, after spending the better part of 3 months on this effort, we identified a number of possible areas of opportunity (effectively new hypotheses) and in subsequent quarters, continued that work to drill down into specific scenarios, use cases, etc.
So, if the objectives aren’t clear from the start — i.e. to focus on something that is new or meaningful to the company — start there.
Learning Agenda
The next step would be to make sure you’ve identified the right questions you need to answer in your research. Sometimes this is called the “learning agenda”, but it’s really important to have team alignment that the learning agenda — what you need answered — is clear and specific.
I find that after objectives, really defining a clear learning agenda for the discovery work is the most important thing you must do.
It’s easy to create a “rough” set of questions, but the more time you focus on this, discuss/debate/edit etc. those questions, the more likely you’ll end up with really useful responses and insights.
Research Methods
There are MANY different techniques that can be used in discovery. The most common seem to be interviews and surveys. But keep in mind that there are many other types of research, from observational studies, diary (longitudinal) studies, card sorting, simulations, participatory exercises, prototyping, discussion groups, various forms of experimentation etc.
Be creative in how you do your research. Perhaps different approaches are needed to elicit new or different types of data and feedback. It can be very surprising to how different approaches and exercises can surface very different types of responses from people.
Audience and Analysis Process
If you’re doing all of these and you’re still not getting useful information, the problem may lie in who you’re speaking with or your ability to analyze the information, and pull out new, valuable insights from the data. That’s a muscle in and of itself, so that may be where the issue lies.
There are many possibilities, but if you’re explicit in each step (Objectives, Learning Agenda, Research Process, Research Subjects, Analysis) you will increate your odds of success.
I’ll stop here. As always, I’d love your feedack and questions or comments about this post. Please give feedback in the survey link below.
A little feedback on the article
If you’ve read this far, thank you. I’d like some feedback on the article to make it better. Just 3 questions. Should take 1 minute, but will really be valuable to me. Thanks in advance.
===> Click here. <===
Saeed
