Planning Under Uncertainty

Planning Under Uncertainty

Help the user navigate product planning when the future is unclear using adaptive planning frameworks from 44 product leaders.

How to Help

When the user asks for help with planning under uncertainty:

1. Understand the uncertainty type - Ask what's driving the ambiguity: technical unknowns, market volatility, AI/ML unpredictability, or organizational change 2. Assess planning horizon - Determine if they need short-term execution tactics or long-term strategic flexibility 3. Match framework to context - Recommend appropriate planning approaches based on their uncertainty profile 4. Build in adaptation mechanisms - Help them create checkpoints and decision criteria for pivoting

Core Principles

Embrace optionality over prediction

Amjad Masad: "Being agile, not being stuck with roadmaps, being able to just say, oh, we're just going to switch priorities right away, is going to be super important." In rapidly changing environments like AI, maintain flexibility to pivot when new capabilities emerge rather than committing to rigid long-term plans.

Build buffers for chaos

Upasna Gautam: "Any time we're planning we build in buffers for all of that chaos that's happening on a daily basis." In chaotic environments, planning must include explicit time buffers and contingency plans ranging from days to months depending on scope.

Use data as compass, not GPS

Shaun Clowes: "Data is more like a compass than a GPS. If you look at data as a way of giving you the answer, you're always wrong." Use data to validate or invalidate intuition rather than waiting for it to tell you exactly what to do.

Value learning over winning

Ramesh Johari: "Experimentation was never historically in science about winners and losers... Experimentation is always very hypothesis driven. It's about, what are you learning?" A healthy experimentation culture values learning from "failed" risky bets more than safe, incremental "wins."

Develop reproducible testing processes

Nikita Bier: "Develop a reproducible testing process, and that will actually influence the probability of your success more than anything." Success in uncertain markets is driven by the quality and speed of the testing process rather than the initial idea.

Diagnose before acting in crisis

Alex Hardimen: "There's this incredible humility that was needed to really understand and first diagnose what was actually happening on the platform." Managing through a crisis requires "wartime" humility to accurately diagnose problems before attempting solutions.

Create decision triggers, not fixed plans

Eric Ries: "Give yourself a fixed period of time to take some decisive action and see if it feels better." Build checkpoints into plans where you'll reassess based on what you've learned, not just calendar dates.

Distinguish reversible from irreversible decisions

Claire Hughes Johnson: "Type one, type two decisions. Is it high impact? Is it irreversible? Is it not?" Spend more time on one-way doors and move quickly on reversible decisions that can be adjusted later.

Questions to Help Users

• "What would need to be true for your current plan to work? Which of those assumptions are you least confident about?"
• "If this takes twice as long as expected, what would you do differently? What if it takes half as long?"
• "What's the smallest thing you could ship to learn whether your core hypothesis is correct?"
• "Is this a one-way door or a two-way door decision?"
• "What signals would tell you to pivot or kill this initiative?"
• "How much buffer have you built in for unexpected chaos?"

Common Mistakes to Flag

• Over-planning - Creating detailed long-term roadmaps that create false confidence and resist necessary pivots
• Analysis paralysis - Waiting for perfect information instead of making decisions with 70% confidence
• Ignoring leading indicators - Not tracking intermediate signals that could tell you earlier if you're on track
• Judging experiments by outcomes alone - Not valuing the learning from "failed" experiments that tested important hypotheses
• Planning theater - Spending excessive time on documents and processes that don't reduce actual uncertainty

Deep Dive

For all 52 insights from 44 guests, see references/guest-insights.md

Related Skills

• prioritizing-roadmap
• running-decision-processes
• scoping-cutting
• problem-definition

Planning Under Uncertainty - All Guest Insights

44 guests, 52 mentions

---

Alex Hardimen

Alex Hardimen

"I think about wartime product management, right? You're coming in, and I think there was... this incredible humility that was needed to really understand and first diagnose what was actually happening on the platform."

Insight: Managing through a crisis requires 'wartime' humility to accurately diagnose problems before attempting solutions.

Tactical advice:

• Prioritize diagnosis over immediate action in a crisis
• Approach unknown platform issues with humility

Timestamp: 00:09:28

---

"These are core product skills that we look for in terms of leadership and grit, and the ability to drive through really, really tough problems that there's no playbook for, nobody has ever really done before."

Insight: The most valuable PMs are those who can navigate problems that lack a pre-existing playbook.

Tactical advice:

• Focus on grit and leadership when there is no playbook
• Develop the ability to drive through unprecedented problems

Timestamp: 00:11:21

Ben Horowitz

Ben Horowitz

"no credit will be given for predicting rain, only credit for building an ark... You have to build the ark. It doesn't matter if you predict you're going to fail, you've still failed. It gets you nothing. So what you have to do is figure your way out of it and spend all your time on that."

Insight: In uncertain or failing conditions, the only valuable action is building a solution (the 'ark') rather than accurately forecasting the disaster.

Tactical advice:

• Shift focus from 'predicting failure' to 'figuring a way out'
• Spend 100% of energy on the solution path once a threat is identified

Timestamp: 01:24:05

Brian Balfour

Brian Balfour

"If you're a late-stage startup... you can afford the luxury to place multiple bets and spread your chips and wait it out a little bit to see who the winner is... The key question for startups is totally different. You don't have the luxury to spread your chips. You have to go all in. You have to choose one and go all in."

Insight: Strategic betting depends on company stage: late-stage companies should diversify bets, while early-stage startups must commit to a single, focused bet.

Tactical advice:

• Assess your resource constraints before deciding on a multi-platform vs. single-platform strategy.
• For early-stage startups, pick the platform with the best retention and commit fully to avoid resource dilution.

Timestamp: 00:48:37

Brian Tolkin

Brian Tolkin

"If you're not going to get significance, if there's no other techniques at your disposal, then sometimes you just got to trust your intuition and ship it. And if that's what you believe, then that's what you believed and you shouldn't spend time trying to get false precision."

Insight: When data volume is too low for statistical significance, rely on intuition and judgment rather than seeking 'false precision' through flawed tests.

Tactical advice:

• Use power analysis to determine if an A/B test is even viable before starting
• When data is lacking, increase conviction by talking to customers or using observational data (diff-in-diff)

Timestamp: 00:40:29

Camille Hearst

Camille Hearst

"One of my favorite takeaways from that is that from this way of working around this dual track agile de-risking your riskiest ideas first approach is a concept of taking the things in the top, the biggest swing and actually prioritizing those first in terms of product discovery and figuring out what can you do to start de-risking because if you constantly put those off in favor of the lower risk or more predictable smaller swings, how are you ever going to truly innovate and get to the next level."

Insight: To innovate, teams must prioritize the discovery and de-risking of high-risk 'big swings' rather than defaulting to safe, incremental tasks.

Tactical advice:

• Prioritize discovery for the riskiest assumptions first
• Give the team permission to fail during the de-risking phase

Timestamp: 00:54:04

Chris Hutchins

Chris Hutchins

"Andy, he always talks about slugging average, not batting average. He's like, 'I don't care if you hit the ball every time. If one in 10 times you hit a home run that's better than someone who hits it every three out of 10 times but gets out a lot.'"

Insight: In high-risk innovation, the magnitude of a single success (slugging average) is more important than the frequency of small wins.

Tactical advice:

• Optimize for outsized impact rather than a high success rate of minor features
• Balance iterative improvements with big, company-altering bets

Timestamp: 00:20:19

Chip Conley

Chip Conley

"Anxiety equals uncertainty times powerlessness... 98% of anxiety comes from two sources. One is what you don't know, and number two is what you can't control or influence."

Insight: Anxiety in uncertain environments can be managed by breaking it down into what is known vs. unknown and controllable vs. uncontrollable.

Tactical advice:

• Create an 'anxiety balance sheet' with four columns: What I know, What I don't know, What I can control, What I can't control

Timestamp: 01:12:32

Crystal W

Crystal W

"Even if you have a sample size of 30, the data you get back, generally, does not change but its precision will. So mathematically speaking, you're going to get the same level of trends, but the precision at which you understand those trends will become more deep if you have more data. But the underlying information that you're getting out of that won't be very different at larger scales."

Insight: Small sample sizes (as low as 30) are sufficient to identify directional trends and validate early hypotheses.

Tactical advice:

• Don't wait for large-scale data to run experiments
• Focus on directional trends rather than high precision in early-stage testing

Timestamp: 00:17:32

Donna Lichaw

Donna Lichaw

"You don't leave a session with me without having tried a little experiment first. The analogy there is we would call it an in the room experiment versus then get out of the building and do an experiment... anything you think is true or you want to do, it's a hypothesis until you test it. And you go out, get data, and then you can do a bigger version."

Insight: Apply lean product methodology to personal growth by treating new behaviors or career goals as hypotheses that require small-scale experimentation.

Tactical advice:

• Run 'in the room' role-play experiments before attempting a new leadership behavior in a high-stakes environment.
• Use 'get out of the building' experiments to test career interests or leadership styles on a small scale to gather data.

Timestamp: 01:03:41

Elena Verna

Elena Verna 3.0

"If every single one of your initiatives that you're doing on growth is an experiment, that's a problem... It's almost like a disease, like a paralyzing disease, that slows down progress... I think that people should trust their intuition a little bit more."

Insight: Over-reliance on scientific experimentation can paralyze growth teams; professional intuition and pre-vs-post analysis are valid for low-volume or obvious changes.

Tactical advice:

• If a sample size for an A/B test cannot be reached within one month, skip the test and use pre-vs-post analysis.
• Use 24-hour, 7-day, and 28-day readouts for non-experimental releases to monitor impact.
• Reserve formal A/B testing for high-traffic real estate or major strategic pivots.

Timestamp: 01:06:10

Eli Schwartz

Eli Schwartz

"When you do this top-down, it's a TAM forecast essentially. When you do a top-down, you're closer to the truth. Now, you probably aren't going to get to the truth... but it will help you make a better decision than if you just guess."

Insight: Top-down TAM (Total Addressable Market) forecasting is often more accurate for planning than bottoms-up keyword tool estimates, which are frequently unreliable.

Tactical advice:

• Calculate SEO upside by taking the total population and filtering by target demographic and internet purchase behavior
• Use keyword research tools for relative normalization rather than absolute traffic predictions

Timestamp: 01:41:20

Ethan Smith

Ethan Smith

"Take 100 different questions, half of them I will intervene, half of them I won't... you definitely want a control group, especially in Answer Engine Optimization."

Insight: In rapidly changing environments like AI search, rigorous A/B testing with control groups is essential to verify what actually works.

Tactical advice:

• Set up a test group of queries where you apply optimizations and a control group where you do nothing.
• Track 'Voice Share' over several weeks to account for the variance in LLM responses.

Timestamp: 00:44:19

Graham Weaver

Graham Weaver

"The first part of limiting beliefs, write them down, understand what they are, look at them in the cold day of light on paper, and then translate them into things that are just obstacles to be overcome. So, 'How would I fund this,' just becomes a plan, like, 'I need to design a plan where I'd get funding for this charity.' And then that just is a problem like any other problem. It's not this nebulous, scary fear. It's just literally a to-do item."

Insight: Externalizing fears by writing them down transforms paralyzing uncertainty into a manageable list of tactical problems.

Tactical advice:

• Write down every limiting belief or fear associated with a new project
• Convert each fear into a specific 'to-do' item or research task
• Treat nebulous fears as standard obstacles that require a plan to overcome

Timestamp: 00:19:15

Gustav Söderström

Gustav Söderström

"There are going to be two types of feedback. One is you did something and it was right, but people are upset because you changed stuff. The other is you did something and it wasn't right, and people are also upset but for good reasons. And so how do you separate these two?"

Insight: When launching big bets, leaders must differentiate between 'change aversion' and fundamental product-market fit issues.

Tactical advice:

• Use A/B testing to be scientific about big redesigns, but acknowledge that MVPs for new paradigms must be high-quality to avoid false negatives
• Be 'unemotional' and willing to change your mind 100% when data disproves a hypothesis

Timestamp: 00:54:45

Itamar Gilad

Itamar Gilad

"I created a tool called the confidence meter... It goes from very low confidence which is the blue area... all the way to high confidence which is the red area and you can see the numbers going from zero to 10. Where zero is very low confidence, we don't know basically anything we're just guessing in the dark and 10 is full confidence."

Insight: The Confidence Meter provides a standardized scale to grade the strength of evidence, preventing 'gut feelings' from being treated as facts.

Tactical advice:

• Assign low confidence (0.01 - 0.1) to opinions, pitch decks, and thematic alignment
• Assign medium confidence to user interviews and market data
• Require high-scale experiments or A/B tests to reach high confidence (above 5.0)
• Tie the level of resource investment to the current confidence score

Timestamp: 00:38:41

Janna Bastow

Janna Bastow

"You're saying that you want a quarter million dollars worth of investment, and you're going to spend it on your team who's going to run experiments... Some of these experiments are going to fail and some are going to succeed. You don't know which ones. But that's okay, you know that by the end of the quarter, enough are going to succeed that you're probably going to move the right numbers in the right direction."

Insight: Product development should be framed like a sales pipeline: an investment in a process of experiments where individual failures are expected but the aggregate moves the needle.

Tactical advice:

• Account for the number of experiments run and the resulting metric movements rather than promising specific feature delivery dates

Timestamp: 45:59

Jason Droege

Jason Droege

"Survival is just part of the game, and most people just give up before they get their timing right... Survival is a precursor to that. So let's not put ourselves in position that could potentially compromise the enterprise along the way. It doesn't mean don't take risks, but think about how you calculate it."

Insight: In uncertain markets, the primary goal is survival to ensure you are still present when the timing becomes right.

Tactical advice:

• Make asymmetrically positive decisions where the upside far outweighs the risk
• Avoid high-risk 'all-in' bets that could compromise the entire enterprise

Timestamp: 01:05:51

Jiaona Zhang

Jiaona Zhang

"how do to be clear about the phase that you're in? ... we are explicitly going to go learn these types of things... you just have to be very, very clear with your team on what phase you're in. 'Hey, we're in the learning phase and we explicitly are trying to learn these things' versus, 'Hey, we have this really big vision and we're just going to go at it.'"

Insight: Manage high-risk projects by explicitly defining 'learning phases' where unscalable prototyping is used to validate assumptions before full investment.

Tactical advice:

• Define explicit learning phases for new initiatives
• Use unscalable prototyping to gather data quickly
• Set go/no-go milestones for every quarter to avoid the sunk cost fallacy

Timestamp: 00:12:32

Lane Shackleton

Lane Shackleton

"He just stops and he's like, "You know what? Just test the extremes. Start the experiment tomorrow. We'll figure it out." Essentially. And I think his point was like, look, we can debate this forever. So I would rather us see the upper and lower bounds of how good this could be or how bad this is going to be immediately."

Insight: When faced with high ambiguity, test the 'extremes' immediately to find the upper and lower bounds of a product's potential.

Tactical advice:

• Launch experiments with polar opposite variables (e.g., a tiny button vs. a giant button) to gather directional data quickly.
• Prioritize making and testing over circular debates.

Timestamp: 00:59:49

Laura Schaffer

Laura Schaffer

"Roughly 80% of the times, ORs in the time are hypotheses and the things that we believe will be true... The closer you get to something that you go bear your head in the sand or go into an attic and build something for six months and ship it, the more likely it is that you are going to ship the 80% wrong stuff."

Insight: Because most product hypotheses fail, iterative experimentation is essential to avoid investing in the wrong solutions.

Tactical advice:

• Use 'painted doors' or mocks to validate concepts before committing to full builds
• Aim for 'embarrassing' first iterations to maximize learning speed
• Accept lower confidence intervals (e.g., 80% instead of 95%) to double or triple experiment velocity

Timestamp: 00:34:26

Lauryn Isford

Lauryn Isford

"Generally my advice is to experiment when you need to and to primarily see it as a risk mitigation tactic when you're making dramatic changes and to let the product development process do more work. So, spend more time with customers, be more rigorous in understanding precisely what problem you're solving, get mocks in front of people and see how they react, and hopefully have more conviction than you otherwise would when you ship something that it's okay if every customer sees it tomorrow and that the experiment doesn't actually matter as much."

Insight: Experimentation should be viewed primarily as a risk mitigation tool for major changes rather than a requirement for every product update.

Tactical advice:

• Use experiments to mitigate risk when making dramatic product changes
• Prioritize qualitative customer research and rigorous problem definition over A/B testing for every feature
• Avoid using experiments solely for the sake of metric precision if it slows down the shipping process

Timestamp: 00:06:09

Luc Levesque

Luc Levesque

"Experiments are great, but they can be slow... the subtlety is that experiments are great, but they can be slow... sometimes you just need to YOLO it because it's a better product experience or you just kind of know it's going to work. And if you're YOLO-ing 40 things and three of them work and you can look at pre-post... the speed can outweigh the cost and time it takes to do experiment."

Insight: Over-reliance on rigorous A/B testing can kill execution speed; sometimes 'YOLO-ing' obvious improvements is more efficient.

Tactical advice:

• Balance rigorous experimentation with 'YOLO' releases for obvious UX improvements
• Use pre-post analysis or holdout groups to monitor the impact of non-experimented changes

Timestamp: 01:16:56

Mayur Kamat

Mayur Kamat

"The moment you build experimentation, you've now made it scientific. Now, somebody comes up with an idea, say, that's a bad idea. Here, this is why it's a bad idea, because we have done this experiment six times and it has failed across this user groups at this exact level of impact created."

Insight: Experimentation democratizes performance and protects PMs from 'loudest voice' decision-making by turning ideas into testable science.

Tactical advice:

• Implement experimentation tools (like Statsig) to move from 'ideas' to 'data'.
• Use experiment results to provide objective 'no's' to stakeholders.

Timestamp: 00:48:43

Mike Maples Jr

Mike Maples Jr

"Any coin that says can't lose bad on one side of it might as well say, can't win big on the other side. It's your willingness to fail that lets you have breakthrough success."

Insight: Breakthrough success requires a willingness to be non-consensus and potentially wrong.

Tactical advice:

• Invest in projects with wildly asymmetric upside even if they have a high likelihood of failure
• Differentiate between 'forecasting' (extending the present) and 'backcasting' (betting on a different future)

Timestamp: 01:38:49

Nabeel S. Qureshi

Nabeel S. Qureshi

"One thing is probably just really fast iteration cycles. So placing a lot of bets and then being really rigorous about just going through that cycle very soon. I have this... principles, and one of the things on there is basically saying EOP successes goes up the more bets you make, and it's sort of a function of how many bets you make and the probability of success."

Insight: In uncertain environments, success is a function of the volume of bets placed and the speed of the iteration cycle.

Tactical advice:

• Maximize the number of small experiments to increase the probability of a 'hit'.
• Set tight deadlines for evaluating whether a bet is failing or needs more investment.

Timestamp: 01:16:27

Nicole Forsgren

Nicole Forsgren

"Draw four boxes on a piece of paper... the first two to the left of them write the word words. And below them, write the word data... always start with words. You do not start with data. You always start with words. And then you'll go around to a couple of people, stakeholders, managers, others, and you'll say, 'Do you agree with this? Is this actually what we're doing?'"

Insight: The 'Four-Box Framework' helps clarify hypotheses by forcing you to define conceptual relationships in words before selecting data proxies.

Tactical advice:

• Map out your hypothesis in words (e.g., 'Customer satisfaction leads to return customers') before looking at data.
• Identify data proxies for each 'word' box to ensure you are measuring the right things.
• Use the framework to identify 'spurious correlations' where the data might show a relationship that doesn't make sense conceptually.

Timestamp: 00:58:12

Nikita Bier

Nikita Bier

"Develop a reproducible testing process, and that will actually influence the probability of your success more than anything. It's so unpredictable whether a consumer product idea will work."

Insight: Success in consumer products is driven by the quality and speed of the testing process rather than the initial idea.

Tactical advice:

• Build a system for taking 'many shots at bat' to reduce the risk of unpredictability

Timestamp: 00:18:55

---

"If this is true, then what next needs to be true for this thing to work out? And these layers of conditional statements. And the more layers you have, the higher risk your product is, so you should try to condense it to about like four things that must be true for the thing to work."

Insight: De-risk products by identifying and validating the core assumptions in a logical sequence.

Tactical advice:

• List the 4 fundamental assumptions that must be true for the product to succeed
• Validate assumptions in sequence: core flow, peer spread, group hopping, monetization

Timestamp: 01:11:50

Ramesh Johari

Ramesh Johari

"Experimentation was never historically in science about winners and losers... Experimentation is always very hypothesis driven. It's about, what are you learning? And that's really an important distinction because what it means is if I go with something big, risky, and it, 'fails,' meaning that doesn't win. Nevertheless, if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot."

Insight: A healthy experimentation culture values learning from 'failed' risky bets more than safe, incremental 'wins.'

Tactical advice:

• Define clear hypotheses for every experiment so that a 'loss' still provides actionable business intelligence.
• Avoid judging data scientists solely on the number of 'wins' per quarter to prevent risk-aversion.

Timestamp: 00:43:53

---

"There's ways to take the past into account, to build what's called a prior belief before I run an experiment, and now take the data from the experiment, connect it with the prior, to come up with a conclusion... that falls broadly under the category of what's called Bayesian A/B testing."

Insight: Bayesian experimentation allows teams to incorporate historical knowledge into new tests, preventing the 'wasted' effort of ignoring past learnings.

Tactical advice:

• Use Bayesian A/B testing methods to reward experiments that move the 'prior belief' even if they don't reach traditional statistical significance.

Timestamp: 00:56:46

---

"What I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to all options... After the fact you're like, 'Treatment was better. What the heck were we thinking? Why'd we give all those samples to control?' ... you have to put yourself in the frame of reference of when you didn't have the answer. And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer."

Insight: Learning is not free; the cost of an experiment is the lost opportunity of not having the 'winning' treatment applied to the control group during the test.

Tactical advice:

• Acknowledge the 'cost of learning' when deciding which experiments are worth running.
• Use holdout groups to quantify the long-term value of a team's innovations, even if it has a short-term revenue cost.

Timestamp: 00:59:16

Ravi Mehta

Ravi Mehta

"At a startup, you can't do that. You just don't have those users to test with. And I think a lot of startups make the mistake of trying to use an experimental approach too early... I've had to shift my mindset from an experimental-oriented approach to making decisions to much more of a conviction-oriented approach."

Insight: Early-stage startups must rely on 'informed conviction' rather than statistical significance due to low user volume.

Tactical advice:

• Avoid paralysis by analysis; move forward once you have enough data for informed conviction.
• Stop digging for data when the sample size is too small to provide valid experimental results.

Timestamp: 00:11:33

Ronny Kohavi

Ronny Kohavi

"You have to allocate sometimes to these high risk, high reward ideas. We're going to try something that's most likely to fail. But if it does win, it's going to be a home run. And you have to be ready to understand and agree that most will fail. ... If you go for something big, try it out, but be ready to fail 80% of the time."

Insight: Treat high-risk product bets as a portfolio where an 80% failure rate is expected and acceptable.

Tactical advice:

• Allocate a specific percentage of resources to high-risk, high-reward ideas
• Prepare the organization for a high failure rate when attempting radical new designs

Timestamp: 00:00:22

Sahil Mansuri

Sahil Mansuri

"So the way I think about setting up a plan when you have limited visibility and some major headwinds is setting up a really conservative plan and then having milestones, short term milestones that unlock the ability to lean into growth and spend based on hitting those targets."

Insight: Use a conservative baseline plan with performance-based triggers to authorize additional spending or growth as visibility improves.

Tactical advice:

• Set a conservative baseline plan for the year
• Establish short-term milestones as 'unlocks' for growth spending
• Avoid 'floundering' between extreme optimism and extreme conservatism

Timestamp: 00:00:29

Shaun Clowes

Shaun Clowes

"Data is more like a compass than a GPS. If you look at data as a way of giving you the answer, you're always wrong. You're always wrong or you're slow. Wrong or slow or sometimes both, because mostly data doesn't give you the answer. It just tells you if what you just said is ridiculous or there's potentially something there."

Insight: Data should be used to validate or invalidate intuition rather than as a definitive roadmap for decision-making.

Tactical advice:

• Use data to disprove your assumptions rather than waiting for it to tell you exactly what to do.

Timestamp: 00:37:44

---

"If you're making a decision with less than 30% of the available data, you're making a big mistake. If you're making a decision only after you have 70%... you have waited far too long."

Insight: Effective decision-making requires finding the 'sweet spot' of data—enough to be informed but not so much that you lose momentum.

Tactical advice:

• Aim to make decisions when you have between 40% and 70% of the information you wish you had.

Timestamp: 01:13:32

Sri Batchu

Sri Batchu

"failure is not learning. So it's really important that you learn when you fail. And so we celebrate failure as long as you're learning and you can only learn if you've designed the right test and you failed conclusively"

Insight: In B2B, where sample sizes are small, you must maximize the 'treatment effect' to ensure an experiment fails conclusively if it doesn't work.

Tactical advice:

• Throw all possible tactics and resources at a hypothesis to maximize the chance of seeing a result
• If a 'maximized' version of a feature fails, abandon the hypothesis entirely rather than re-testing minor variations

Timestamp: 00:51:43

Teresa Torres

Teresa Torres

"The first thing is we have to learn how to take an idea and break it into its underlying assumptions. We have to learn how to prioritize those assumptions. Then we have to learn how to run tests that are small enough that they're just testing that assumption."

Insight: Sustainable discovery involves testing the underlying assumptions of an idea rather than the entire idea at once.

Tactical advice:

• Break solutions down into underlying assumptions
• Prioritize assumptions based on risk
• Run small, fast tests (half a dozen a week) to validate assumptions

Timestamp: 44:37

Tom Conrad

Tom Conrad

"One really, really big important lesson that I learned at Snap is about risk taking. And when you have the financial support and the foundational relationship with your investors that Evan has, it really allowed him to take these really big swings, acquire a technology that he thought was game changing, build features speculatively."

Insight: Strong investor relationships and capital allow for high-conviction, speculative bets that can lead to game-changing product wins.

Tactical advice:

• Leverage capital to make 'big swings' on speculative technology
• Accept that some speculative features will fail in pursuit of home runs

Timestamp: 01:00:14

Tim Holley

Tim Holley

"A/B testing... that's the highest bar. It proves with near absolute certainty that there's a causal relationship... But I think that it maybe misses the point in some changes or some areas where you are working towards a bigger net new thing or this specific change won't really be indicative of the greater whole you're building towards."

Insight: While A/B testing is the gold standard for causality, it should be balanced with other validation methods for long-term strategic changes.

Tactical advice:

• Use A/B testing for proving causal relationships in incremental changes.
• Look at cohorts over time or pre-post analysis for changes that don't fit standard A/B testing models.

Timestamp: 00:35:28

Tomer Cohen

Tomer Cohen

"I carved out two million members and I said, 'Those are my members. I'm going to focus on building that mountain peak. I'm going to build for them.' Full liberty and doing whatever, it doesn't hurt numbers, giving the scale and really focus on building a great experience for them."

Insight: When attempting a radical product shift in a large organization, isolate a specific cohort to experiment without the pressure of global metrics.

Tactical advice:

• Carve out a randomized cohort to act as a 'test country' for new product DNA.
• Run 'negative tests' to prove that legacy features or promotional content are actually hurting long-term engagement.

Timestamp: 00:22:45

Upasna Gautam

Upasna Gautam

"We always have to have the ability to A, pivot of course, but also have backup and buffers in those types of scenarios. So any time we're planning we build in buffers for all of that chaos that's happening on a daily basis."

Insight: In chaotic environments, planning must include explicit time buffers and contingency plans to remain effective.

Tactical advice:

• Build buffers into project timelines ranging from days to months depending on scope
• Assess situations objectively daily to decide whether to use buffers or move to the next phase

Timestamp: 06:33

---

"Stay firm on the goal, but flexible on the process."

Insight: Maintain rigid alignment on the ultimate objective while allowing the execution path to adapt to changing circumstances.

Tactical advice:

• Use OKRs as an anchor for the team while allowing squad-level autonomy on how to reach them

Timestamp: 21:12

Uri Levine

Uri Levine

"If you fail fast, you still have plenty of time to try another attempt and build another version of the product. Try another go-to-market approach. Try a different business model so you still have plenty of time to make more and more and more attempts, and the more attempts that you have, you simply increase the likelihood of being successful."

Insight: Success in startups is a function of the number of attempts you can make, which requires failing fast to preserve time and resources.

Tactical advice:

• Iterate quickly to increase the total number of attempts at scoring a 'hit.'
• Don't aim for perfection; aim for 'good enough' through rapid iteration.

Timestamp: 00:17:07

Yuriy Timen

Yuriy Timen

"The only thing that's worse than a channel or a tactic that you tried not working. The only thing that's worse now is when you didn't give it the appropriate shot, right? And you prematurely were erroneously concluded that it doesn't work and it's remarkable how often you find that to be the case when I talk to companies, "Oh, YouTube, we tried it. It doesn't work." I'm like, "Okay, can I see what you've tried?" And then you look at it and you're like, "Oh, this thing was not designed to even have a shot at working from the get go.""

Insight: Avoid prematurely abandoning growth channels by ensuring they were given a statistically significant and well-designed trial.

Tactical advice:

• Audit failed experiments to see if they were designed for success before writing off a channel.
• Ensure a channel is given an 'appropriate shot' before concluding it doesn't work.

Timestamp: 00:00:00

---

"I think with some tactics and some channels you can fairly objectively create some test guard rails where it's like, if it's YouTube, we know kind of minimum number of impressions that you got to get. Try two to three creative angles. Here's the click through rates range that you're looking for. If you get within these ranges on these KPIs, keep going. If you don't, abandon."

Insight: Use objective benchmarks and KPI ranges to decide when to persist with or kill an experimental channel.

Tactical advice:

• Set minimum impression thresholds for top-of-funnel tests.
• Test at least 2-3 creative angles before abandoning a channel.
• Define 'keep going' vs 'abandon' KPI ranges before starting the test.

Timestamp: 00:36:12

Naomi Gleit

Naomi Gleit

"Four life lessons... three, focus on what you can control. And four, for those things never give up."

Insight: In high-pressure or uncertain environments, maintain focus strictly on controllable variables and persist relentlessly on those.

Tactical advice:

• Identify which aspects of a project are within your control and ignore those that aren't

Timestamp: 00:18:43

Sanchan Saxena

Sanchan Saxena

"From an operating principle, we went into two week planning mode. Greg Greeley, who was the president of Airbnb used to say, 'Look, can't plan for a year, can't plan for a quarter. We're going to plan every two weeks. We're going to react to every two weeks.'"

Insight: In high-stakes crises, abandon long-term roadmaps in favor of ultra-short, reactive planning cycles.

Tactical advice:

• Shorten planning cycles to two-week increments during periods of extreme volatility
• Dissolve sub-teams to focus the entire organization on a single survival goal
• Be honest with the team about the lack of long-term clarity

Timestamp: 00:19:38

---

"How do you build conviction in a highly noisy world? ... The thing that I would take with me everywhere is, how do you build in that noise? How do you stay focused and still build what you believe is the right thing and still let the noise happen around you?"

Insight: Operating in emerging technologies like Web3 requires the ability to filter out market noise and execute on core convictions.

Tactical advice:

• Build a muscle for operating in ambiguity where data is scarce
• Focus on the 'Web 2.5' journey rather than jumping straight to idealized end-states

Timestamp: 00:47:40

Scott Belsky

Scott Belsky

"Resourcefulness brings you further than resources... If resources are carbs, resourcefulness is like muscle. It stays with you. It makes you stronger, and it helps you have a better intuition and better performance over time."

Insight: In uncertain or resource-constrained environments, focus on refactoring processes and leveraging technology rather than simply adding headcount.

Tactical advice:

• Refactor existing systems and processes to find efficiencies instead of requesting more budget
• View resource constraints as an opportunity to build the 'muscle' of resourcefulness

Timestamp: 00:50:35

Varun Parmar

Varun Parmar

"What you want to do is that you want to be the first one to hit the brick wall... speed is something that you should accelerate for the organization... can you be the first one to hit the brick wall where you have the learning faster than anyone else in the market so that you can decide, 'Oh my god, the path that I was going was not the right path.'"

Insight: In competitive markets, the goal of speed is to discover failure points before your competitors do.

Tactical advice:

• Optimize for 'time to insight' rather than just 'time to ship.'
• Use rapid prototyping (like Design Sprints) to hit 'brick walls' early and pivot 10-30 degrees based on findings.

Timestamp: 00:32:00