Don’t Dunning-Kruger Your Pivot
There is nothing harder than knowing what you don’t know. As Kathryn Schulz explained, being wrong feels exactly the same as being right. It’s realizing that you are wrong that feels confusing, embarrassing, and downright bad. The uncomfortable emotions that wash over us when we admit we are wrong about something or that we don’t know something that we thought we understood lead a lot of us to slip down the Dunning-Kruger slope.
The Dunning-Kruger effect is when a person who is not competent at a specific task or who is lacking a specific skill grossly overestimates their ability to do something requiring that competence. That usually means that they also lack the skill to notice the mistakes they make and underestimate the importance of expertise. I like to say that the Dunning-Kruger effect comes for all of us–we each have our blind spots–so we should be generous when we encounter it with others.
There are so many ways to fall into a Dunning-Kruger trap, where you remain blithely ignorant of your ignorance! Let me name the ways:
- You mistake your assumptions for facts.
- You believe that you know your customer or end user because they are just like you or because you’ve had so many sales conversations.
- You actually did understand the problem and the best ways to solve it, but the world changed and your hard-won lessons have expired.
We do not do this because it is easy. We do it because we thought it would be easy.
In this article I will tell some stories about how this problem manifests itself in situations like pivots and the creation of new products and services, so we can all avoid making the same mistakes.
When Your Expertise Comes Back to Bite You
One of the most annoying ways to be wrong is when you have really learned something deeply and then something about the world changes, or the context you are in changes, and your hard-won knowledge is suddenly outdated.
At Philips, I helped build an app that connected to a blood pressure monitor, a scale, a smartwatch full of sensors, and a doctor-led health program. This was a B2B2C product. The end users were regular people but the paying customers were insurance companies. The app, devices, and content were all FDA-cleared. Philips knows how to make FDA-cleared hardware and software. They have been doing it for decades. All we had to do was to apply the same system and processes to this new product.
Raise your hand if you think it’s a good idea to apply the same quality control processes to a consumer-quality mobile app as you would to the operating system of an MRI machine. I never worked on a project that had so many quality assurance processes that contributed little to nothing to the end user’s pleasure in using the product! We sat in quasi-legalistic meetings and defined all the ways that the product could be misused and mitigations for those cases while manual testers struggled to write a comprehensible bug report and UX testing was something we had to do guerrilla style.
I quickly understood that there was a culture and context problem that I needed to overcome if I wanted to build a great product. I was in a hardware culture where software was considered a kind of supporting add-on. In this culture, products could be finished. That’s a foreign idea to anyone working in software. Even a product that isn’t being developed anymore needs maintenance. You can sunset a product, but it’s never finished.
All of the people at Philips were incredibly smart. They knew their stuff. They simply hadn’t realized that this was not the stuff they thought it was. They were there to solve a problem that they had tackled hundreds of times and they had the tools to do it, except the same problem in a new context is a new problem. We shifted resources away from strictly following the established quality assurance processes toward developing new structures that moved the needle where it mattered–user experience, performance of the apps, and design quality.
We Don’t Know What We Don’t Know
OWI was an AI company that had been using machine learning to automate customer service for massive enterprises in the private and public sectors for a decade. I was brought on to help them with their pivot from enterprise customers to e-commerce SMEs. The CEO had a simple plan: 1. Build modules for e-commerce owners to plug the tools into their customer service systems. 2. Profit.
Of course, I am being flippant. The OWI team was well aware that they needed to learn about their new customers and that they didn’t know much about them at all. Where they underestimated the problem was in seeing how that missing knowledge would touch every part of their business.
OWI knew everything about machine learning and they had customized their tool for much more complex sectors like banking and government. How hard could it be to make the system work for a sector like e-commerce, where the use cases were well understood (returns, where’s my package, sizing questions, etc)? It turns out that when you tackle the same problem for a different customer, it might not be the technical side that’s difficult.
Get Tracy Rolling’s stories in your inbox
Join Medium for free to get updates from this writer.
OWI didn’t know how to talk to these customers. Their website was an austere blue and orange screen with impressive but irrelevant logos. An entrepreneurial dropshipper would be very confused if they came there looking for an automation solution for their customer support needs.
More importantly, OWI did not know what problems these new customers were trying to solve. Of course the tool automates customer service messages, so it saves you time in answering repeat issues, but the reasons a person running an online shop wants help with their customer service are very different from the reasons a government office does. E-commerce shop owners need their customer support tools to help them get into direct contact with customers instead of being dependent on Facebook and Instagram. They need help answering questions in languages they don’t speak so that they can expand their business to other EU countries. They need to develop customer loyalty. All of these needs affect how you position the product as well as how you develop the product for the new target user.
A lot of companies underestimate the challenges of “opening a new vertical.” Context matters. A lot. Any change to it is likely to have more far-reaching effects than you expect.
The first thing that I did for OWI was to prototype a new sub-brand and website that spoke to their new target users. In collaboration with their digital marketing expert, I worked on potential new value propositions based on market research that I conducted, and narrowed the target audience even farther to only specific e-commerce sectors.
Next, I worked with their Sales Lead, training her how to talk to potential customers more as a researcher than as a pitch person. We developed a user interview plan and began recruiting shop owners to talk to.
Dunning-Kruger Comes for Us All
Nobody is immune to overestimating their own competence at one time or another, missing a vital bit of context, or mistaking new territory for old. One of the first times I caught myself in this trap was when I led a team as PM/PO to build a maps and navigation application for Windows tablets, at Nokia. I began my career in a startup called Platial, putting user-generated content on maps. By the time I got to this Windows tablet project I had built probably 7 maps apps for phones.
Before landing in Product I had been a UX designer, so I sat down and whipped up the first several sprints of backlog based on my deep knowledge of what a Maps app does and what it looks like. After only a few weeks of working hands on with the designers and engineers, we realized that many of my assumptions needed to be challenged. The tablet form factor was going to be more for planning than navigating. The Windows design system didn’t allow for some of the interactions I had imagined. The bigger screen meant that people expected to see a lot more information when tapping on a place marker than that we could offer on the phone.
I was wrong about a lot of things and the way out was collaboration with the team and lots of research. We ran prototyping and usability testing once a month. Engineers, testers, and designers all observed user testing. In the end, I think it might have been one of the best products I ever delivered.
Talk to the Humans
Sales calls are great for understanding customer needs and expectations but they aren’t sufficient. You really need to do legit user research or at the very least get real people using the software. Ideally these should be people who you are not currently selling to.
One company I advised spent a lot of time building a platform and showing the platform to experts in the CEO’s network. When the feedback was warm, the team took heart. When the feedback was critical it was rejected because the person was “not the target audience.” The target audience was a nebulous and fuzzy figure.
In this case, I helped the CEO narrow the target user down so that we could first remove half-baked and half-finished features before updating production for the first time in almost a year and properly onboarding some beta testers. I personally onboarded each company, keeping meticulous notes in Confluence and giving brutally honest reports to the team.
The minute I understood that a person was testing the tool as a friend and not as a real potential customer, I knew to take all of their feedback with a massive grain of salt. It is easy to say that a product is “easy to use” when you aren’t really trying to use it. Of seven customers onboarded, two were honestly looking for a solution like ours and we could concentrate our efforts on solving their real problems. Only by doing research with people who are legitimately part of your target user base can you get to the crux of their problems and either solve them or admit that you can’t.
That is not to say that sales calls are not also a treasure trove of information or that you should not listen to what your sales team is telling you. It’s just that the sales context is very different than the research context. People don’t open up in the same way when they are being sold to.
AI in all of this
LLMs love to repeat your assumptions back to you and tell you how smart you are. It’s baked into the LLM training, which is reinforced by human preference. It’s also baked into the business model. Attention and retention are how they make money and if you don’t like what the model is telling you, you might not come back.
I have been doing a lot of prototyping with vibe-coding tools (you can try my game). As I become more and more of a vibe-coding fool I am careful to remember that I am a real-coding genuine fool, lest I Dunning-Kruger. I assume my prototypes are just that and don’t expect them to be robust or ready to scale. Everyone thinks that AI is amazing at doing things they don’t know how to do. Be cautious when you are using LLMs to work outside your expertise. They do an excellent job of giving you something that looks or sounds like a plausible result, but these often fall apart if you scratch the surface.
That definitely does not mean you should not use AI tools! LLMs are great for ping ponging ideas and they can really accelerate desk research, as long as you make sure to get sources and verify bold statements on your own. I have had a few instances where the LLM helping me with desk research came up with a really intriguing hypothesis that was in no way supported by evidence, once I looked at the sources. LLMs will not help you understand what your own blind spots are unless you very explicitly ask them to. It just isn’t what they are built for.
You can use these tools to get your own skeptical juices flowing by asking your favorite LLM, “What can go wrong with this?” or “Help me understand everything that is different about this new context.” Once you have some hypotheses you can do the work to validate or invalidate them. You can also get into the habit of just asking yourself these questions.
Speaking of hypotheses, one great trick to learn is this: if you believe something to your core, it is still ok to call it an “assumption.” Just because it is an assumption doesn’t mean it’s wrong. Just because you are sure you’re right doesn’t mean that you are.
