Evaluating New Technology

Evaluating New Technology

Help the user evaluate emerging technologies using frameworks from 22 product leaders who have made critical technology decisions at companies from Google to Shopify.

How to Help

When the user asks for help evaluating technology:

1. Start with the problem - Clarify what problem they're solving before discussing tools 2. Assess maturity - Determine if the technology is stable enough for their use case 3. Consider build and buy - Help them find the right mix rather than forcing a binary choice 4. Plan for change - Design for modularity since the landscape will shift

Core Principles

Tools solve problems, not the reverse

Austin Hay: "I have this adage I always say, which is tools are just meant to solve problems. And the problem set for marketing technologists and business technologists is you focus on the tools." Always define the problem and the people involved before selecting a system or tool.

Build AND buy, not build vs buy

Austin Hay: "Build and buy as opposed to build versus buy. Build and buy means that both of you can win." Buy tools to handle 90% of standard functionality and build the 'cool' 10% that is unique to your business.

Evaluate mental bandwidth, not just dollars

Dhanji R. Prasanna: "The savings and costs that there might be in replacing a vendor tool by something you build in-house is probably not worth it in the mental bandwidth that you've lost." Focus technical bandwidth on core competencies, not recreating vendor tools.

Update your priors constantly

Aparna Chennapragada: "The models couldn't do some things one year ago. My impression of it from trying it a few months ago - that prior needs to be updated. The baby just grew up to be a 15-year-old in a month." Re-test assumptions about what technology can do every few months.

Bet on abstraction layers

Asha Sharma: "You really need to bet on a platform or some app server type layer that allows you to swap things in and out and not really be beholden to any one technology." Invest in modularity as the AI stack evolves.

AI guardrails don't work

Sander Schulhoff: "AI guardrails do not work. If someone is determined enough to trick GPT-5, they're going to deal with that guardrail. When these guardrail providers say 'We catch everything,' that's a complete lie." Be skeptical of AI security vendor claims.

Use the tools yourself

Dhanji R. Prasanna: "I would say really try and use these tools yourself. We learn a lot about how our own workflow can change." Solve a specific, personal problem with new tools to understand their true strengths.

Context drives AI value

Jeanne Grosser: "Because this whole space is so nascent, often your own esoteric context, your content, your workflow is really key to unlocking the power of the agent." For AI agents, building internally often beats buying.

Questions to Help Users

• "What specific problem are you trying to solve with this technology?"
• "Is this technology stable enough for production, or still experimental?"
• "What's the mental bandwidth cost of building vs maintaining a vendor relationship?"
• "When did you last test your assumptions about what this technology can do?"
• "How will you swap this out if something better comes along?"
• "Have you actually used this tool to solve a real problem yourself?"

Common Mistakes to Flag

• Tool bias - Picking tools because you've used them before, not because they solve the problem
• Binary build vs buy thinking - Missing the opportunity to buy 90% and build the strategic 10%
• Outdated priors - Making decisions based on what technology couldn't do six months ago
• Vendor lock-in - Betting on specific tools without an abstraction layer for future flexibility
• Trusting security marketing - Believing AI guardrail vendors who claim to 'catch everything'

Deep Dive

For all 27 insights from 22 guests, see references/guest-insights.md

Related Skills

• AI Product Strategy
• Building with LLMs
• Platform Strategy
• Vibe Coding

Evaluating New Technology - All Guest Insights

22 guests, 27 mentions

---

Aishwarya Naresh Reganti + Kiriti Badam

Aishwarya Naresh Reganti + Kiriti Badam

"When someone comes up to me and says, 'We have this one click agent, it's going to be deployed in your system.' ... I would almost be skeptical because it's just not possible. And that's not because the models aren't there, but because enterprise data and infrastructure is very messy... I would rather go with a company that says, 'We're going to build this pipeline for you,' and that will learn over time."

Insight: Be skeptical of 'out-of-the-box' AI solutions for enterprises; real ROI requires a pipeline that accounts for messy data and infrastructure.

Tactical advice:

• Evaluate AI vendors based on their ability to build a learning pipeline rather than a static 'one-click' agent.
• Assess the readiness of your internal data layer before attempting to deploy autonomous agents.

Timestamp: 00:30:31

Aparna Chennapragada

Aparna Chennapragada

"The models couldn't do some things one year ago. I mean, image generation was full of spellings or reasoning. You just couldn't have deeper and smarter answers. You couldn't do data analysis. So my impression of it from change, trying it a few months ago, that prior needs to be updated. And it's hard to do that, right? You have to do something almost counterintuitive and against the grain to say, 'No, no, ignore what you learned about what this can or cannot do.' The baby just grew up to be a 15-year-old in a month."

Insight: To effectively build with AI, product leaders must constantly update their 'priors' because technology capabilities evolve faster than human habits or 'scar tissue' from past failures.

Tactical advice:

• Actively work to ignore 'scar tissue' from previous technical limitations
• Regularly re-test assumptions about what AI can and cannot do every few months
• Demand more from current technology rather than relying on old benchmarks

Timestamp: 00:31:17

Asha Sharma

Asha Sharma

"you really need to bet on a platform or some app server type layer that allows you to swap things in and out and not really be beholden to anything, any one technology or any one tool because the reality is the whole thing is going to change."

Insight: Avoid vendor or tool lock-in by investing in an abstraction layer that allows for modularity as the AI stack evolves.

Tactical advice:

• Bet on platforms that support model diversity and easy swapping of components
• Prioritize tools that offer high observability and evaluation capabilities

Timestamp: 11:24

Austin Hay

Austin Hay

"I have this adage I always say, which is tools are just meant to solve problems. And the problem set for marketing technologists and business technologists is you focus on the tools."

Insight: Technology should always be viewed as a means to solve a specific problem rather than an end in itself.

Tactical advice:

• Always define the problem and the people involved before selecting a system or tool.
• Avoid 'tool bias'—picking a tool just because you've used it before.

Timestamp: 00:10:11

---

"It's B and B as opposed to BVB. So, build and buy as opposed to build versus buy. People all the time just think the second that you're talking about implementing a tool or procuring a solution, it's, Hey, I want to build this thing or I want to buy this really expensive thing. Build versus buy is a very narrowly constricting decision tree. If it's only build versus buy, then you've already made the decision that you can only do one or the other... Build and buy means that both of you can win"

Insight: The most effective technology strategy often involves buying a core tool and building custom layers on top of it.

Tactical advice:

• Buy tools to handle 90% of the standard functionality and build the 'cool' 10% that is unique to your business.
• Use a financial model to show that building on top of a third-party tool often yields higher ROI than building from scratch.

Timestamp: 01:07:52

Brandon Chu

Brandon Chu

"if you're not technical, really lean into it and build something simple, learn how to build something simple for yourself, demystify the technology. That experience will take you far. I love telling people that literally don't even know what HTML is... you could build a clone of Twitter using a tutorial on Rails"

Insight: PMs should demystify technology by building simple applications themselves to break down the 'wall of obscurity' between product and engineering.

Tactical advice:

• Build a simple clone of a popular app using a tutorial to understand basic architecture
• Focus on demystifying how data flows rather than mastering the code

Timestamp: 48:03

Camille Fournier

Camille Fournier

"GraphQL... is one of the things that is both popular and thought relatively poorly of by most of the senior people that I know... if you're seriously thinking about it and you're not Facebook, you may really want to make sure you know what problem you're trying to solve because the impression that I have... is that GraphQL is kind of trying to promise front-end engineers that they don't really have to collaborate with backend engineers."

Insight: Popular technologies (like GraphQL) can be 'fads' that solve specific problems for giant companies but create unnecessary complexity for others.

Tactical advice:

• Identify the specific problem a new framework is intended to solve before adopting it
• Be wary of frameworks backed by VC-funded startups whose goal is adoption over utility

Timestamp: 00:28:52

Bret Taylor

Bret Taylor

"I do still think studying computer science is a different answer than learning to code... computer science is a wonderful major to learn systems thinking."

Insight: When evaluating emerging tech like AI, focus on systems thinking and foundational principles rather than just current tools.

Tactical advice:

• Prioritize understanding 'systems thinking' and complexity over learning specific syntax

Timestamp: 00:31:47

Christine Itwaru

Christine Itwaru

"We had someone at one point handle whatever tools connected to Pendo and make sure that those systems are set up for maximum outcomes for the product manager. So, Pendo's connected to Salesforce. We're connected to Looker. We're connected to all these different."

Insight: Managing the product team's internal tech stack ensures that data flows correctly between CRM, analytics, and product tools.

Tactical advice:

• Audit the product manager's tool stack to ensure maximum data integration
• Connect product analytics tools (like Pendo) to CRM systems (like Salesforce) to provide a complete customer picture

Timestamp: 00:13:45

Dan Shipper

Dan Shipper

"My first thing that I open is o3. I'm a ChatGPT boy... it has memory. And I just love that... I think Claude Opus is... Claude Code, everyone inside Every, that's basically what we use... Gemini... It's incredibly powerful and it's incredibly cheap, which is great."

Insight: A modern AI stack should be diversified based on specific model strengths like memory, coding autonomy, or cost-effectiveness.

Tactical advice:

• Use ChatGPT (o3) for tasks requiring long-term memory and personal style consistency
• Use Claude Code for autonomous engineering tasks and Gemini for high-volume, low-cost API applications

Timestamp: 00:35:51

Dhanji R. Prasanna

Dhanji R. Prasanna

"The savings and costs that there might be in replacing a vendor tool by something you build in-house is probably not worth it in the mental bandwidth that you've lost and the amount of the team's technical focus that's being taken away. ... I would always come back to what is the reason we're doing this? Why does it matter to us and to our customers?"

Insight: Avoid the trap of building internal tools just because AI makes it easy; focus technical bandwidth on core competencies.

Tactical advice:

• Evaluate build vs. buy decisions based on 'mental bandwidth' lost rather than just direct dollar costs
• Question if a process is even necessary before deciding to automate it or buy a tool for it

Timestamp: 00:40:31

---

"I would say really try and use these tools yourself. ... we learn a lot about how our own workflow can change, and that's going to tell you so much more about how are you going to change your organization's workflow than if you're reading a bunch of think pieces on LinkedIn."

Insight: Leaders must personally use and experiment with new technology to understand its ergonomics and potential organizational impact.

Tactical advice:

• Solve a specific, personal, real-world problem with a new tool to understand its true strengths and weaknesses
• Encourage the executive team to use the product daily to drive authentic adoption

Timestamp: 00:54:04

Howie Liu

Howie Liu

"I try to use as many different AI products, including not Airtable, as I can... I try to invent little, almost like side projects of my own, to have a real reason to use these products."

Insight: Deeply understand emerging tech by creating personal 'weekend projects' that force you to use the tools beyond a superficial level.

Tactical advice:

• Invent personal side projects to force deep usage of new AI tools
• Study the 'prior art' of AI products to understand emerging UX patterns and form factors

Timestamp: 00:40:38

Ivan Zhao

Ivan Zhao

"Before React wins, there's a competing technology called Web Component from Google... we're betting on that technology. And then we realize because it's so new, it's just so unstable. It don't know where the bug come from. It's from your source code or from the underlying libraries? Then we have to restart the company, rebuild the whole thing."

Insight: Betting on unproven, unstable technical foundations can force a total company reset if the technology fails to mature.

Tactical advice:

• Distinguish between 'orthodox' technology foundations and experimental ones.
• Be willing to 'throw away the code' and rebuild on a more stable foundation if the current stack is hindering progress.

Timestamp: 00:14:03

Manik Gupta

Manik Gupta

"One of the things that we used to talk about all the time at Google on Maps was how would we design a navigation product when people are in self-driving cars?... It's going to take years, but it is just such a different paradigm. It's like computers talking to computers, algorithms talking to other algorithms. Then there's a human in the mix in terms of serving the human at the end, but it's like the human is not initiating that much."

Insight: Evaluating emerging technologies like autonomous driving requires reimagining user experiences for a paradigm where the human is a passive recipient rather than an active initiator.

Tactical advice:

• Analyze how a new technology shifts the user from an 'initiator' to a 'passenger' or 'consumer'.
• Consider the 'computer-to-computer' interaction layer when designing for future tech stacks.

Timestamp: 01:01:35

Naomi Ionita

Naomi Ionita

"The modern growth stack... is the evolution of what you do with the data. So these are the workflows that the data enables to drive the business forward for product growth and revenue teams like I used to run. It's the modern replacement for infrastructure that teams like mine built or bought."

Insight: The 'Modern Growth Stack' consists of tools that automate growth workflows (experimentation, billing, reverse ETL) previously built in-house.

Tactical advice:

• Evaluate tools like Hightouch or Census (Reverse ETL) to break down data silos
• Look for tools that offer 'hard ROI' through either cost reduction or revenue generation

Timestamp: 35:37

---

"Eppo, which offers experimentation for the modern data stack. So unlike Optimizely, which focused on more kind of click through metrics, Eppo ties directly to the metrics in your data warehouse. So tying an experiment result to things like subscriptions or revenue or margins, really like board level metrics that you're trying to move."

Insight: Modern experimentation tools should tie directly to warehouse data and business-level KPIs rather than just surface-level clicks.

Tactical advice:

• Prioritize experimentation tools that integrate with the data warehouse (e.g., Snowflake, Redshift)

Timestamp: 43:29

Ronny Kohavi

Ronny Kohavi

"I think the key decision is whether you want to build or buy. ... It's usually not a zero one, it's usually both. You build some and you buy some, and it's a question of do you build 10% or do you build in 90%? I think for people starting, the third party products that are available today are pretty good."

Insight: For experimentation infrastructure, a hybrid 'build and buy' approach is often best, starting with third-party tools to establish trust.

Tactical advice:

• Start with third-party experimentation vendors to avoid the high initial cost of building a platform
• Evaluate the build vs. buy decision based on the maturity of the organization's experimentation culture

Timestamp: 01:07:17

Ryan J. Salva

Ryan J. Salva

"This product in particular, I probably spent more time with legal than any other products that I've ever kind of been responsible for. ... It is also privacy and security champions. It is, frankly, developers, like the people who are using it, listening to them. ... We're actually at a place now where we're able to partner with the Azure Department of a Responsible AI, and they've created some really extraordinary models that help detect I'll call it sentiment for lack of a better word, but basically when there is something that is patently offensive."

Insight: Evaluating high-stakes emerging tech like AI requires deep cross-functional collaboration with legal, privacy, and 'Responsible AI' teams to mitigate ethical and safety risks.

Tactical advice:

• Engage legal and privacy teams early when dealing with novel technology like LLMs
• Use secondary AI models to monitor and filter the outputs of primary generative models for safety

Timestamp: 00:41:09

Sander Schulhoff

Sander Schulhoff 2.0

"AI guardrails do not work. I'm going to say that one more time. Guardrails do not work. If someone is determined enough to trick GPT-5, they're going to deal with that guardrail. No problem. When these guardrail providers say, 'We catch everything,' that's a complete lie."

Insight: Current AI guardrail products are often ineffective against determined attackers and marketed with misleading security claims.

Tactical advice:

• Do not rely on third-party guardrails as a primary security layer
• Be skeptical of '100% catch rate' claims from AI security vendors

Timestamp: 00:00:00

---

"AI red teaming works too well. It's very easy to build these systems and they always work against all platforms... these automated red teaming systems are not showing anything novel. It's plainly obvious to anyone that knows what they're talking about that these models can be tricked into saying whatever very easily."

Insight: Automated red teaming tools often provide redundant information because all current transformer-based models are fundamentally vulnerable to trickery.

Tactical advice:

• Recognize that automated red teaming results against off-the-shelf models are expected and not necessarily a unique flaw in your implementation

Timestamp: 00:28:33

Yuriy Timen

Yuriy Timen

"Media Mix Modeling is now making a comeback... the company that's leading the charge of bringing the Media Mix Modeling methodology of the traditional advertising era and ushering it into the digital world is a company called the Recast."

Insight: Privacy changes (iOS 14) are reviving older attribution technologies like Media Mix Modeling (MMM) for digital spend.

Tactical advice:

• Consider MMM tools like Recast if spending over $100k/month across 3+ channels.
• Use incrementality testing (randomized control experiments) to determine true causality in ad spend.

Timestamp: 00:29:03

Andrew Wilkinson

Andrew Wilkinson

"I finally got one that actually works. It's called the Matic Vacuum... basically, I think it's like former Google Engineers basically built like a mini Waymo car. So it has machine vision and it will avoid absolutely everything."

Insight: When evaluating consumer robotics or hardware, look for products that utilize advanced machine vision and 'Waymo-style' engineering rather than basic sensors.

Tactical advice:

• Prioritize hardware built by engineers with backgrounds in autonomous vehicles or high-end computer vision.
• Look for 'machine vision' as a key differentiator in robotics to ensure reliability.

Timestamp: 01:23:16

Hila Qu

Hila Qu

"From infrastructure perspective, on data tool, my first tool usually, one is some sort of data hub segment, right? This next one is some sort of a product analytics tool. Think about Amplitude. I know PostHog is actually a pretty popular one... The third piece I think that's pretty essential, I counted in the infra, is some sort of a lifecycle marketing tool."

Insight: A robust PLG motion requires a specific data stack: a data hub, product analytics, and lifecycle marketing tools.

Tactical advice:

• Implement a data hub like Segment to allow for flexible tool integration
• Use lifecycle marketing tools that trigger based on in-product behavior rather than just email opens

Timestamp: 01:05:08

---

"The success of this is you identify the gaps and eventually you want to establish something called the data dictionary... The data dictionary will include, here are all the key actions, what's the event name for each of those, and what are the property and things like that."

Insight: Data integrity depends on a shared data dictionary that defines all key events and properties.

Tactical advice:

• Perform a data instrumentation audit to identify gaps
• Create a centralized data dictionary for the entire team to reference

Timestamp: 01:12:00

Jeanne Grosser

Jeanne Grosser

"I think the calculus on build versus buy is changing... because this whole space is so nascent, often your own esoteric context, your content, your workflow is really key to unlocking the power of the agent. And so I think there's value in experimenting with your own internal agent development."

Insight: For AI agents, building internally is often better than buying because internal context and specific workflows are the primary value drivers.

Tactical advice:

• Experiment with building internal agents for specific workflows before procuring fragmented AI tools.

Timestamp: 00:41:00

Tobi Lutke

Tobi Lutke

"Monorepo, now for companies, it's a very much one of those door A, door B kind of things. It's a very consequential choice that is incorrect to go say yes to at a certain size, and then it becomes very correct in my mind to say yes to, but at that point it's an enormous amount of effort. So, it's a kind of thing that actually is something I'm uniquely positioned to be involved with because it's actually a business strategy thing as well."

Insight: Major technical architecture shifts are strategic business decisions that impact how a company builds and scales.

Tactical advice:

• Evaluate technical infrastructure choices (like monorepos) as business strategy investments
• Recognize that technical choices that were 'incorrect' at one size may become 'correct' at another
• Use leadership influence to compress the time spent on contentious technical change management

Timestamp: 00:52:00