Episodes

  • Building AI Assistants, Not Autopilots: What Tony Zhang’s Research Shows About Automation Blindness

    Building AI Assistants, Not Autopilots: What Tony Zhang’s Research Shows About Automation Blindness
    Jun 24 2025

    Show notes will be available soon.

    Show More Show Less
    44 mins
  • Who Can Succeed in a Data or AI Product Management Role?

    Who Can Succeed in a Data or AI Product Management Role?
    Jun 10 2025
    Today, I’m responding to a listener's question about what it takes to succeed as a data or AI product manager, especially if you’re coming from roles like design/BI/data visualization, data science/engineering, or traditional software product management. This reader correctly observed that most of my content “seems more targeted at senior leadership” — and had asked if I could address this more IC-oriented topic on the show. I’ll break down why technical chops alone aren’t enough, and how user-centered thinking, business impact, and outcome-focused mindsets are key to real success — and where each of these prior roles brings strengths and/or weaknesses. I’ll also get into the evolving nature of PM roles in the age of AI, and what I think the super-powered AI product manager will look like. Highlights/ Skip to: Who can transition into an AI and data product management role? What does it take? (5:29)Software product managers moving into AI product management (10:05)Designers moving into data/AI product management (13:32)Moving into the AI PM role from the engineering side (21:47)Why the challenge of user adoption and trust is often the blocker to the business value (29:56)Designing change management into AI/data products as a skill (31:26)The challenge of value creation vs. delivery work — and how incentives are aligned for ICs (35:17)Quantifying the financial value of data and AI product work(40:23) Quotes from Today’s Episode “Who can transition into this type of role, and what is this role? I’m combining these two things. AI product management often seems closely tied to software companies that are primarily leveraging AI, or trying to, and therefore, they tend to utilize this AI product management role. I’m seeing less of that in internal data teams, where you tend to see data product management more, which, for me, feels like an umbrella term that may include traditional analytics work, data platforms, and often AI and machine learning. I’m going to frame this more in the AI space, primarily because I think AI tends to capture the end-to-end product than data product management does more frequently.” — Brian (2:55) “There are three disciplines I’m going to talk about moving into this role. Coming into AI and data PM from design and UX, coming into it from data engineering (or just broadly technical spaces), and then coming into it from software product management. I think software product management and moving into the AI product management - as long as you’re not someone that has two years of experience, and then 18 years of repeating the second year of experience over and over again - and you’ve had a robust product management background across some different types of products; you can show that the domain doesn’t necessarily stop you from producing value. I think you will have the easiest time moving into AI product management because you’ve shown that you can adapt across different industries.” - Brian (9:45) “Let’s talk about designers next. I’m going to include data visualization, user experience research, user experience design, product design, all those types of broad design, category roles. Moving into data and/or AI product management, first of all, you don’t see too many—I don’t hear about too many designers wanting to move into DPM roles, because oftentimes I don’t think there’s a lot of heavy UI and UX all the time in that space. Or at least the teams that are doing that work feel that’s somebody else’s job because they’re not doing end-to-end product thinking the way I talk about it, so therefore, a lot of times they don’t see the application, the user experience, the human adoption, the change management, they’re just not looking at the world that way, even though I think they should be.” - Brian (13:32) “Coming at this from the data and engineering side, this is the classic track for data product management. At least that is the way I tend to see it. I believe most companies prefer to develop this role in-house. My biggest concern is that you end up with job title changes, but not necessarily the benefits that are supposed to come with this. I do like learning by doing, but having a coach and someone senior who can coach your other PMs is important because there’s a lot of information that you won’t necessarily get in a class or a course. It’s going to come from experience doing the work.” - Brian (22:26) “This value piece is the most important thing, and I want to focus on that. This is something I frequently discuss in my training seminar: how do we attach financial value to the work we’re doing? This is both art and science, but it’s a language that anyone in a product management role needs to be comfortable with. If you’re finding it very hard to figure out how your data product contributes financial value because it’s based on this waterfalling of “We own the model, ...
    Show More Show Less
    50 mins
  • 170 - Turning Data into Impactful AI Products at Experian: Lessons from North American Chief AI Officer Shri Santhnam (Promoted Episode)

    170 - Turning Data into Impactful AI Products at Experian: Lessons from North American Chief AI Officer Shri Santhnam (Promoted Episode)
    May 27 2025

    Today, I'm chatting with Shri Santhanam, the  EVP of Software Platforms and Chief AI Officer of Experian North America. Over the course of this promoted episode, you’re going to hear us talk about what it takes to build useful consumer and B2B AI products. Shri explains how Experian structures their AI product teams, the company’s approach prioritizing its initiatives, and what it takes to get their AI solutions out the door. We also get into the nuances of building trust with probabilistic AI tools and the absolutely critical role of UX in end user adoption.

    Note: today’s episode is one of my rare Promoted Episodes. Please help support the show by visiting Experian’s links below:

    Links
    • Shri's LinkedIn
    • Experian Assistant | Experian
    • Experian Ascend Platform™ | Experian 
    Show More Show Less
    43 mins
  • 169 - AI Product Management and UX: What’s New (If Anything) About Making Valuable LLM-Powered Products with Stuart Winter-Tear

    169 - AI Product Management and UX: What’s New (If Anything) About Making Valuable LLM-Powered Products with Stuart Winter-Tear
    May 13 2025
    Today, I'm chatting with Stuart Winter-Tear about AI product management. We're getting into the nitty-gritty of what it takes to build and launch LLM-powered products for the commercial market that actually produce value. Among other things in this rich conversation, Stuart surprised me with the level of importance he believes UX has in making LLM-powered products successful, even for technical audiences. After spending significant time on the forefront of AI’s breakthroughs, Stuart believes many of the products we’re seeing today are the result of FOMO above all else. He shares a belief that I’ve emphasized time and time again on the podcast–product is about the problem, not the solution. This design philosophy has informed Staurt’s 20-plus year-long career, and it is pivotal to understanding how to best use AI to build products that meet users’ needs. Highlights/ Skip to Why Stuart was asked to speak to the House of Lords about AI (2:04)The LLM-powered products has Stuart been building recently (4:20)Finding product-market fit with AI products (7:44)Lessons Stuart has learned over the past two years working with LLM-power products (10:54) Figuring out how to build user trust in your AI products (14:40)The differences between being a digital product manager vs. AI product manager (18:13)Who is best suited for an AI product management role (25:42)Why Stuart thinks user experience matters greatly with AI products (32:18)The formula needed to create a business-viable AI product (38:22) Stuart describes the skills and roles he thinks are essential in an AI product team and who he brings on first (50:53)Conversations that need to be had with academics and data scientists when building AI-powered products (54:04)Final thoughts from Stuart and where you can find more from him (58:07) Quotes from Today’s Episode “I think that the core dream with GenAI is getting data out of IT hands and back to the business. Finding a way to overlay all this disparate, unstructured data and [translate it] to the human language is revolutionary. We’re finding industries that you would think were more conservative (i.e. medical, legal, etc.) are probably the most interested because of the large volumes of unstructured data they have to deal with. People wouldn’t expect large language models to be used for fact-checking… they’re actually very powerful, especially if you can have your own proprietary data or pipelines. Same with security–although large language models introduce a terrifying amount of security problems, they can also be used in reverse to augment security. There’s a lovely contradiction with this technology that I do enjoy.” - Stuart Winter-Tear (5:58)“[LLM-powered products] gave me the wow factor, and I think that’s part of what’s caused the problem. If we focus on technology, we build more technology, but if we focus on business and customers, we’re probably going to end up with more business and customers. This is why we end up with so many products that are effectively solutions in search of problems. We’re in this rush and [these products] are [based on] FOMO. We’re leaving behind what we understood about [building] products—as if [an LLM-powered product] is a special piece of technology. It’s not. It’s another piece of technology. [Designers] should look at this technology from the prism of the business and from the prism of the problem. We love to solutionize, but is the problem the problem? What’s the context of the problem? What’s the problem under the problem? Is this problem worth solving, and is GenAI a desirable way to solve it? We’re putting the cart before the horse.” - Stuart Winter-Tear (11:11)“[LLM-powered products] feel most amazing when you’re not a domain expert in whatever you’re using it for. I’ll give you an example: I’m terrible at coding. When I got my hands on Cursor, I felt like a superhero. It was unbelievable what I could build. Although [LLM products] look most amazing in the hands of non-experts, it’s actually most powerful in the hands of experts who do understand the domain they’re using this technology. Perhaps I want to do a product strategy, so I ask [the product] for some assistance, and it can get me 70% of the way there. [LLM products] are great as a jumping off point… but ultimately [they are] only powerful because I have certain domain expertise.” - Stuart Winter-Tear (13:01)“We’re so used to the digital paradigm. The deterministic nature of you put in X, you get out Y; it’s the same every time. Probabilistic changes every time. There is a huge difference between what results you might be getting in the lab compared to what happens in the real world. You effectively find yourself building [AI products] live, and in order to do that, you need good communities and good feedback available to you. You need these fast feedback loops. From a pure product management perspective, we ...
    Show More Show Less
    1 hr and 1 min
  • 168 - 10 Challenges Internal Data Teams May Face Building Their First Revenue-Generating Data Product

    168 - 10 Challenges Internal Data Teams May Face Building Their First Revenue-Generating Data Product
    Apr 29 2025

    Today, I am going to share some of the biggest challenges internal enterprise data leaders may face when creating their first revenue-generating data product. If your team is thinking about directly monetizing a data product and bringing a piece of software to life as something customers actually pay for, this episode is for you. As a companion to this episode, you can read my original article on this topic here once you finish listening!
    Show More Show Less
    38 mins
  • 167 - AI Product Management and Design: How Natalia Andreyeva and Team at Infor Nexus Create B2B Data Products that Customers Value

    167 - AI Product Management and Design: How Natalia Andreyeva and Team at Infor Nexus Create B2B Data Products that Customers Value
    Apr 16 2025
    Today, I’m talking with Natalia Andreyeva from Infor about AI / ML product management and its application to supply chain software. Natalia is a Senior Director of Product Management for the Nexus AI / ML Solution Portfolio, and she walks us through what is new, and what is not, about designing AI capabilities in B2B software. We also got into why user experience is so critical in data-driven products, and the role of design in ensuring AI produces value. During our chat, Natalia hit on the importance of really nailing down customer needs through solid discovery and the role of product leaders in this non-technical work. We also tackled some of the trickier aspects of designing for GenAI, digital assistants, the need to keep efforts strongly grounded in value creation for customers, and how even the best ML-based predictive analytics need to consider UX and the amount of evidence that customers need to believe the recommendations. During this episode, Natalia emphasizes a huge key to her work’s success: keeping customers and users in the loop throughout the product development lifecycle. Highlights/ Skip to What Natalia does as a Senior Director of Product Management for Infor Nexus (1:13)Who are the people using Infor Nexus Products and what do they accomplish when using them (2:51)Breaking down who makes up Natalia's team (4:05)What role does AI play in Natalia's work? (5:32)How do designers work with Natalia's team? (7:17)The problem that had Natalia rethink the discovery process when working with AI and machine learning applications (10:28)Why Natalia isn’t worried about competitors catching up to her team's design work (14:24)How Natalia works with Infor Nexus customers to help them understand the solutions her team is building (23:07)The biggest challenges Natalia faces with building GenAI and machine learning products (27:25)Natalia’s four steps to success in building AI products and capabilities (34:53)Where you can find more from Natalia (36:49) Quotes from Today’s Episode “I always launch discovery with customers, in the presence of the UX specialist [our designer]. We do the interviews together, and [regardless of who is facilitating] the goal is to understand the pain points of our customers by listening to how they do their jobs today. We do a series of these interviews and we distill them into the customer needs; the problems we need to really address for the customers. And then we start thinking about how to [address these needs]. Data products are a particular challenge because it’s not always that you can easily create a UX that would allow users to realize the value they’re searching for from the solution. And even if we can deliver it, consuming that is typically a challenge, too. So, this is where [design becomes really important]. [...] What I found through the years of experience is that it’s very difficult to explain to people around you what it is that you’re building when you’re dealing with a data-driven product. Is it a dashboard? Is it a workboard? They understand the word data, but that’s not what we are creating. We are creating the actual experience for the outcome that data will deliver to them indirectly, right? So, that’s typically how we work.” - Natalia Andreyeva (7:47)“[When doing discovery for products without AI], we already have ideas for what we want to get out. We know that there is a space in the market for those solutions to come to life. We just have to understand where. For AI-driven products, it’s not only about [the user’s] understanding of the problem or the design, it is also about understanding if the data exists and if it’s feasible to build the solution to address [the user’s] problem. [Data] feasibility is an extremely important piece because it will drive the UX as well.” - Natalia Andreyeva (10:50)“When [the team] discussed the problem, it sounded like a simple calculation that needed to be created [for users]. In reality, it was an entire process of thinking of multiple people in the chain [of command] to understand whether or not a medical product was safe to be consumed. That’s the outcome we needed to produce, and when we finally did, we actually celebrated with our customers and with our designers. It was one of the most difficult things that we had to design. So why did this problem actually get solved, and why we were the ones who solved it? It’s because we took the time to understand the current user experience through [our customer] interviews. We connected the dots and translated it all into a visual solution. We would never be able to do that without the proper UX and design in that place for the data.” - Natalia Andreyeva (13:16)“Everybody is pressured to come up with a strategy [for AI] or explain how AI is being incorporated into their solutions and platform, but it is still essential for all of my peers in product management to focus on the value [we’re] creating for ...
    Show More Show Less
    38 mins
  • 166 - How B2B AI and Analytics Product Leaders Should Think About Measuring UX Quality Along With Business Value

    166 - How B2B AI and Analytics Product Leaders Should Think About Measuring UX Quality Along With Business Value
    Apr 1 2025
    Today I am going to try to answer a fundamental question: how should you actually measure user experience, especially with data products—and tie this to business value? It's easy to get lost in analytics and think we're seeing the whole picture, but I argue that this is far from the truth. Product leaders need to understand the subjective experience of our users—and unfortunately, analytics does not tell us this. The map is not the territory. In this episode, I discuss why qualitative data and subjective experience is the data that will most help you make product decisions that will lead you to increased business value. If users aren't getting value from your product(s), and their lives aren’t improving, business value will be extremely difficult to create. So today, I share my thoughts on how to move beyond thinking that analytics is the only way to track UX, and how this helps product leaders uncover opportunities to produce better organizational value. Ultimately, it’s about creating indispensable solutions and building trust, which is key for any product team looking to make a real impact. Hat tip to UX guru Jared Spool who inspired several of the concepts I share with you today. Highlights/ Skip to Don't target adoption for adoption's sake, because product usage can be a tax or benefit (3:00)Why your analytical mind may bias you—and what changes you might have to do this type of product and user research work (7:31)How "making the user's life better" translates to organizational value (10:17)Using Jared Spool's roller coaster chart to measure your product’s user experience and find your opportunities and successes (13:05)How do you measure that you have done a good job with your UX? (17:28) Conclusions and final thoughts (21:06) Quotes from Today’s Episode Usage doesn't automatically equal value. Analytics on your analytics is not telling you useful things about user experience or satisfaction. Why? "The map is not the territory." Analytics measure computer metrics, not feelings, and let's face it, users aren't always rational. To truly gauge user value, we need qualitative research - to talk to users - and to hear what their subjective experience is. Want *meaningful* adoption? Talk to and observe your users. That's how you know you are actually making things better. When it’s better for them, the business value will follow. (3:12)Make better things—where better is a measurement based on the subjective experience of the user—not analytics. Usable doesn’t mean they will necessarily want it. Sessions and page views don’t tell you how people *feel* about it. (7:39)Think about the dreadful tools you and so many have been forced to use: the things that waste your time and don’t let you focus on what’s really important. Ever talked to a data scientist who is sick of doing data prep instead of building models, and wondering, “why am I here? This isn’t what I went to school for.” Ignoring these personal frustrations and feelings and focusing only on your customers’ feature requests, JIRA tickets, stakeholder orders, requirements docs, and backlog items is why many teams end up building technically right, effectively wrong solutions. These end user frustrations are where we find our opportunities to delight—and create products and UXs that matter. To improve their lives, we need to dig into their workflows, identify frustrations, and understand the context around our data product solutions. Product leaders need to fall in love with the problems and the frustrations—these are the magic keys to the value kingdom. However, to do this well, you probably need to be doing less delivery and more discovery. (10:27)Imagine a line chart with a Y-axis that is "frustration" at the bottom to "delight" at the top. The X-axis is their user experience, taking place over time. As somebody uses your data product to do their job/task, you can plot their emotional journey. “Get the data, format the data, include the data in a tool, derive some conclusion, challenge the data, share it, make a decision” etc. As a product manager, you probably know what a use-case looks like. Your first job is to plot their existing experience trying/doing that use case with your data product. Where are they frustrated? Where are they delighted? Celebrate your peaks/delighters, and fall in love with the valleys where satisfaction work needs to be done. Connect the dots between these valleys and business value. Address the valleys—especially the ones that impede business value—and you’ll be on your way to “showing the value of your data product.” Analytics on your data product won’t tell you this information; the map is not the territory. (13:22)Analytics about your data product are lying to you. They give you the facts about the product, but not about the user. An example? “Time spent” doing a task. How long is too long? 5 minutes? 50? Analytics will tell you precisely ...
    Show More Show Less
    26 mins
  • 165 - How to Accommodate Multiple User Types and Needs in B2B Analytics and AI Products When You Lack UX Resources

    165 - How to Accommodate Multiple User Types and Needs in B2B Analytics and AI Products When You Lack UX Resources
    Mar 18 2025
    A challenge I frequently hear about from subscribers to my insights mailing list is how to design B2B data products for multiple user types with differing needs. From dashboards to custom apps and commercial analytics / AI products, data product teams often struggle to create a single solution that meets the diverse needs of technical and business users in B2B settings. If you're encountering this issue, you're not alone! In this episode, I share my advice for tackling this challenge including the gift of saying "no.” What are the patterns you should be looking out for in your customer research? How can you choose what to focus on with limited resources? What are the design choices you should avoid when trying to build these products? I’m hoping by the end of this episode, you’ll have some strategies to help reduce the size of this challenge—particularly if you lack a dedicated UX team to help you sort through your various user/stakeholder demands. Highlights/ Skip to The importance of proper user research and clustering “jobs to be done” around business importance vs. task frequency—ignoring the rest until your solution can show measurable value (4:29)What “level” of skill to design for, and why “as simple as possible” isn’t what I generally recommend (13:44)When it may be advantageous to use role or feature-based permissions to hide/show/change certain aspects, UI elements, or features (19:50)Leveraging AI and LLMs in-product to allow learning about the user and progressive disclosure and customization of UIs (26:44)Leveraging the “old” solution of rapid prototyping—which is now faster than ever with AI, and can accelerate learning (capturing user feedback) (31:14)5 things I do not recommend doing when trying to satisfy multiple user types in your b2b AI or analytics product (34:14) Quotes from Today’s Episode If you're not talking to your users and stakeholders sufficiently, you're going to have a really tough time building a successful data product for one user – let alone for multiple personas. Listen for repeating patterns in what your users are trying to achieve (tasks they are doing). Focus on the jobs and tasks they do most frequently or the ones that bring the most value to their business. Forget about the rest until you've proven that your solution delivers real value for those core needs. It's more about understanding the problems and needs, not just the solutions. The solutions tend to be easier to design when the problem space is well understood. Users often suggest solutions, but it's our job to focus on the core problem we're trying to solve; simply entering in any inbound requests verbatim into JIRA and then “eating away” at the list is not usually a reliable strategy. (5:52)I generally recommend not going for “easy as possible” at the cost of shallow value. Instead, you’re going to want to design for some “mid-level” ability, understanding that this may make early user experiences with the product more difficult. Why? Oversimplification can mislead because data is complex, problems are multivariate, and data isn't always ideal. There are also “n” number of “not-first” impressions users will have with your product. This also means there is only one “first impression” they have. As such, the idea conceptually is to design an amazing experience for the “n” experiences, but not to the point that users never realize value and give up on the product. While I'd prefer no friction, technical products sometimes will have to have a little friction up front however, don't use this as an excuse for poor design. This is hard to get right, even when you have design resources, and it’s why UX design matters as thinking this through ends up determining, in part, whether users obtain the promise of value you made to them. (14:21)As an alternative to rigid role and feature-based permissions in B2B data products, you might consider leveraging AI and / or LLMs in your UI as a means of simplifying and customizing the UI to particular users. This approach allows users to potentially interrogate the product about the UI, customize the UI, and even learn over time about the user’s questions (jobs to be done) such that becomes organically customized over time to their needs. This is in contrast to the rigid buckets that role and permission-based customization present. However, as discussed in my previous episode (164 - “The Hidden UX Taxes that AI and LLM Features Impose on B2B Customers Without Your Knowledge”) designing effective AI features and capabilities can also make things worse due to the probabilistic nature of the responses GenAI produces. As such, this approach may benefit from a UX designer or researcher familiar with designing data products. Understanding what “quality” means to the user, and how to measure it, is especially critical if you’re going to leverage AI and LLMs to make the product...
    Show More Show Less
    49 mins