How Can GenAI Make Analytics More Accessible to Product Teams?

Read The Transcript:

We’re trying out transcribing our podcasts using a software program. Please forgive any typos as the bot isn’t correct 100% of the time.

Hannah Clark: Whether you prefer the term data-driven, or data-informed, or data-dazzled, it doesn’t matter—today’s tech cannot survive without high quality data sets AND the tools to use them effectively. But we also can’t afford to think about data as the responsibility of just one or two departments in the organization—instead, we need to be going into 2025 with a mindset of data democratization. In other words, our business is our customers, so what our customers are saying should be everyone’s business.

My guest today is Mario Ciabarra, Founder and CEO of Quantum Metric. Quantum is a Gen-AI powered analytics platform that allows companies to interpret data into useful insights for different stakeholders so that everyone—from the first day product designer to the company founder—can bake empathy for customers into every decision. We discussed how this works in practice, considerations around data hygiene, and some predictions on how Gen-AI will drive the evolution of analytics this year and beyond. Let’s jump in.

Welcome back to The CPO Club podcast. We are here today with Mario Ciabarra.

Mario, thank you for making time in your busy schedule to talk to us today.

Mario Ciabarra: Hannah, it’s an absolute pleasure. Thank you for having me.

Hannah Clark: Let’s start it off the way we always do. Can you tell us a little bit about your background and how you ended up where you are today at Quantum Metric?

Mario Ciabarra: Yeah, happy to. The background, as a young child, you could have seen the writings on the wall that I’d be in a future entrepreneur.

Maybe we can get into some of that in our discussion, but how did it ended up here today as part of that entrepreneurial journey? I was always passionate about software, computer engineering, software engineering, and really just uncovering how do we make experiences better just fell right into the center of that passion.

It could, it started off with how do I make things faster, how to make them more performant. And I started realizing, look, you can make them fast, but if they’re not great, it doesn’t matter. How do you make great experiences for audiences from my. 13 year old daughter to my 87 year old father, they have different needs.

How do we make sure our experiences work for everybody?

Hannah Clark: I do want to talk to you a little bit about your approach to leadership because you’ve had quite a journey in your career and you mentioned before that your passion has evolved from product and engineering to focusing on people and empathy. So how has that influenced your leadership approach?

Mario Ciabarra: If you ask me about, the company I was thinking about January is rolling around and it will mark my 10 year anniversary here at Quantum. And if you asked me nine, eight years ago, what I was passionate about at Quantum, it was absolutely creating product. It’s still my passion today. This weird thing happened and it’s my, this is my fourth company.

The other companies I had, they were small. They were one to three people. They were building product, their engineering product, iterating product to success. What snuck up on me is that we’re now 400 plus team members at Quantum and, it wasn’t, I had to get to hundreds. It was really in the dozens of people I realized successful companies are built on the back of the team.

And so you have to have an incredible team and then realizing that the CEO, the founder, the leader of the organization is responsible for leading the team. And that all doesn’t work well together. You can have all the great product that you want. It doesn’t work. And so really creating a great success is about people leadership, investing in people.

And it really comes down to why our first company objective is happy people, healthy and diverse culture. Our second one is customers as rating fans. Our third one is about achieving some economical success for the year. But those first two. They don’t change from year to year. Of course, our third one kind of, we move the goalposts each year about in terms of our success for the year.

I think some people, some organizations get that backwards. They focus on Hey, let’s go win a financial metric. That’s our purpose this year. I don’t think you can get there without getting happy people. I don’t think you can get there without having rating fans from your customers. It’s really about thinking about, yes, I want the economic success.

Of course I do. But let’s focus on our people and let’s focus on delivering for the customers that have invested their trust and financial wares with us as well.

Hannah Clark: And of course, empathy is a huge part of that. We’re going to be focusing today on Generative AI and how that can make analytics more accessible to product teams.

But before we get into the meat of that discussion, I want to talk about empowering empathy at scale, with so many users to take into account. How do you see analytics playing a role in fostering empathy across a large user base?

Mario Ciabarra: If you had people that you had to serve with whatever product or service that you had, and it was numbers in the single digits, I don’t think analytics is helpful.

You just go help the person, you go have a conversation with the two customers that you have. When that two turns into 2 million, like all of a sudden light bulb moment, I can’t talk to all 2 million. I need analytics to help me understand the customer. And so a thought process I’ve been having lately is less asking and more listening.

And that’s what analytics is. It’s really about listening with data. And I think about a graph or a chart on a screen, when I think about traditional analytics, yeah, it’s great. I saw the graph. Doesn’t it elicit empathy often for most people? I think about when I can connect that graph to real people.

That’s when empathy happens for me. And I’ve seen it in the eyes of our customers and the people I speak with, our team, it’s really about connecting the chart to the users that it represents. And that’s really the foundation of what Quantum has become. It’s like, how do I connect the experiences with the information and elicit empathies to enact change?

Hannah Clark: Yeah. And I love that quote too. That is all about the listening. I think that’s a really interesting approach there. And I think, we don’t often think about empathy when we think about analytics, those two almost seem like they’re counter to each other, but really, they do enable our teams to develop a sense of what are the users trying to tell us here.

Tell me a little bit about Generative AI and how it can make product analytics more accessible to teams to allow the teams working on the products to get a better sense of what users are trying to say.

Mario Ciabarra: There was a early part of this journey at Quantum and I sat down with a major retail product leader, awesome individual.

I’ve stayed friends with him over the last seven, eight years. He’s moved different organizations, but something that struck me when he took me into a room after we had our formal meeting with his team, and he said these words, Mario, you would not understand the guilt that product managers have for not using data enough.

And it’s stuck with me for the last seven years. Like, how do we make. The data accessible so people don’t feel guilty for not using it, that it’s easy to consume that essentially like eventually people are using that listening skill at scale for huge amounts of data to make the right decisions that both move their business.

But obviously that business is just serving the customer. Like, how do they serve their customers better with data? And it’s hard. That’s why he’s saying it’s that guilty feeling. It’s I’ve got my day job and yes, I want to use data, but it seems overwhelming to find the right data that speaks to the problem I’m trying to solve.

I don’t know how to interface with it. It’s in 16 different locations across all these different tools. I don’t know which tool is going to answer the right question. And frankly, I just give up. I’m going to go with intuition. I’m done. And so for us, like we’ve been collecting all this data, we’ve been trying to make it easier and easier to access.

In fact, like the premise of the whole company, in the beginning, we ended up making these t shirts of we make it simple. We make analytics simple and we’ve done that, like we’re always trying to figure out how to make it more simple, how to make it more accessible for a range of audiences from marketers, product owners, merchandisers, dev ops, executives, analytics teams.

All of these teams need to access that information about what’s happening from the customer’s lens and how can we make it better. And so for us, for Generative AI comes into play is what are the tasks that these teams are trying to do to understand that data. And for example, a lot of product leaders have spent time watching replays of a user and their journey.

Oh, wow. I didn’t know they’d click that. Oh, I didn’t know they’d go there first and then do this. There’s all these amazing insights. The problem with watching session replays is really slow. It’s really tedious. I’ve always laughed when people said, Oh yeah, we have movie night, we get popcorn and then we watch people and their journey for an hour in front of a conference room.

Like it’s intriguing because it’s valuable, but it’s not a good use of time. What if we could get generative AI to watch the replays for us? So that was this concept as January, because we know that Gen-AI is really good at summarizing things. What if we could point Gen-AI’s magic to summarize a user’s journey?

And it has, our businesses that has accelerated through this year of 2024, we released this product called Felix AI in the end of February, early March of 2024, and each quarter, our business is accelerated because it’s giving more teams access to understanding. Of what’s working and what’s not for the user and it’s doing it at scale because if it takes 20 minutes to understand an individual user and you have 2,000,000. You can do the math.

It just doesn’t work if we can take 2 seconds to understand a single user. And with that Generative AI also use it to help create a segment. Okay, this user was struggling. Does that struggle represent one user or 10 million? Help me do that automatically. And that’s giving that accessibility to listening to a larger group of people.

So we’re really excited about how Gen-AI can enable people to listen better.

Hannah Clark: To your point earlier, I think there’s a ton of pressure to be this data expert, really be able to parse and ask the right questions and go to the right sources. Break it down for me, like how easy exactly are we talking about when we say, I understand the idea of using the Gen-AI to parse the data when a, like a product manager, for example, is using the platform.

Can you walk me through maybe like a scenario, like a possible scenario, how they might actually tactically utilize that data and they use that platform?

Mario Ciabarra: One of my favorite stories that has just really made it clear the value of what we do is the ability for. Us to connect this information to different audiences.

And I’ll pick one that we all can relate to. So Hannah, when’s the last time that you called your bank, your telco, your healthcare provider, your airline. If you can think for me about when the last time you called one of those providers or service providers that you work with, you probably didn’t call them until you try to self service first, right?

You’re like you’re on their app, you’re on their website. You try to do it. It doesn’t work. And that’s when you call, you didn’t say, man, I woke up today. And the first thing I’m gonna do is I’m gonna call my healthcare provider. It’s like you want it to self service. But when you got to the end of trying to self service and it didn’t work and you called in, you probably ended up in an IVR pressing one through nine, a couple of times to get to the right audience.

They pick up, say, hi, Hannah. How are you? Let me tell you how I am, right? I’m trying to change my health care insurance for the year or look up, my coverage and it didn’t work. Oh, hold on, Hannah. Let me get to you with an expert. Hi, Hannah. How are you? You’re right. Wouldn’t it be amazing if we could change that conversation to, We know what Hannah was doing.

We know what she’s calling us about. We’re going to route her directly to the expert. Hi, Hannah. It looks like you’re looking up coverage in the app, but you got an error. Super sorry about the error. Can help you with that or something else. And really what I’m trying to convey is all of these companies have been collecting data about what Hannah has been doing on their website or app.

They just haven’t been able to activate that information into the call center. As just one example of an audience that could be valuable with that information. And Gen AI has allowed us to summarize Hannah’s journey in two seconds. In the past, we’ve always had it. It’s just was it available in real time?

Quantum has been, but even with the real time availability, hold on Hannah, let me watch what you’ve been doing for the last 30 minutes and then I’ll get back to you. No, it’s easier just to ask Hannah what she wants or tell her to press one through nine on the phone. The ability to summarize what Hannah was doing, and we know Gen-AI is good at that summary, that’s been transformational for our business.

Now, that use case of the call center, obviously not for our product audience as much, but think about the product manager. Think about, I’m trying to understand why this user didn’t convert, why they didn’t sign up, why they didn’t convert, why they didn’t book their flight or change their service correctly or change their address, whatever it was tasks that they’re trying to achieve.

How do I understand the why? Behind that journey, and it takes a lot of times, analytics session replay. There’s a lot of different tools that we have log files. How do we make that simpler? And so how do we make that understanding of the customer’s journey easy for the product owner? And that’s what Quantum is doing with Gen-AI.

Let’s just transform that understanding into seconds. And then, I think there’s that understanding of the individual is important, but then understand, okay, does this experience, this friction point or this optimization point represent a small use case or a big one? And I think. I was actually at a major bank here in the U.S. this week and I just, smiled from ear to ear when I heard a quote that one, one of the team members had said and it said, this is helping me be an operational product manager. I’ve never heard the term, but it really speaks to me a lot, which is, there’s product managers and I think we need to transform the product manager thinking to operational product management.

How do I use data to understand what’s working and what’s not for my customer in real time and pivot in real time on making sure that we’re making our customers happy? I think there’s a lot of product ownership that says, Oh, I’ve got my tasks. I’ve got my sprints. I’m going to go do that. I’m going to focus.

I’m going to be done. When I’m done with the release, I’m done and move on the next sprint. That’s crazy to me. I’ve got to use data. I got to know. Am I betting on the right change, right? Using data to make sure that the change or experiment or whatever I’m doing is the right thing for my users, in a prioritized way.

And then when I release it, whether it’s an experiment or a full release. Read the data correctly to understand, is it having the right impact? Don’t move on to the next sprint. Make sure I’m using data to like reinforce, I’m making good decisions or bad and have to change the way I’ve been thinking.

Because this isn’t moving the needle as I thought it would, or it is. And by the way, I want to elevate that to my leadership. I’m making good use of our resources. I’m impacting the customer experience. And hey, maybe I need a promotion or a raise or a bonus this year. But data can help confirm that I’m doing good decisions or bad.

And I think that’s what an operational product manager is.

Hannah Clark: There’s a bit of a, an issue right now in which a lot of pressures on product managers to make very sound decisions using data, even though they’re not really data scientists themselves. It’s very easy to make incorrect assumptions or kind of misinterpret data in a way that can mislead our understanding and interpretation and then thus lead to incorrect assumptions.

It seems to me that if you’re using Gen-AI as sort a middleman and you’re using it more conversationally, then you mitigate some of those assumptions. Is that correct? Or do we still need to be taking precautions to make sure that we’re still making sound decisions using this tool?

Mario Ciabarra: I don’t think it’s Gen-AI or not, but I will speak to the Gen-AI part of this. I think it’s what data source are we using. What I love about the data that we collect is undeniable truth. And these are words our customers say, when you see that a customer is trying to change their service or their telco or book their flight, and it’s not working, you can’t look away.

And there’s just so much data sources available to any team member, but it will pick on product managers. And if you say, Mario, I need you to go show me that my decision is right. I literally can find a data source in the organization that proves me right. That’s a terrible approach, by the way, don’t do it.

But that’s part of the problem that we have is like anyone can find data that, that confirms, confirmation bias Oh, I’m doing the right thing. And this is why I was speaking earlier about the operational product manager that’s using their business data to confirm that their decisions are accurate because when they do that release and it’s moved the needle for the business metrics.

Awesome. And so I think it’s about which data do I use for that confirmation? And I think about what we’re focused on is what does everything look like from the customer? I don’t care. I do care, by the way, about log files or backend server performance and stuff like that. I don’t want to dismiss it.

I don’t care. I only care about it when I need to care about it. I don’t think that we should start there, whether we’re product managers or executives and business owners. It’s not starting at the, Hey, look, the API is faster. That’s great. Everyone’s going to get a bonus this year. Who cares? And I’m not saying I don’t.

I’m saying, should I? And what I think about it is a customer doesn’t call or fill out a survey or tweet or email the CEO with a complaint saying, Hey, look, I’m in the B experiment. I’m in the C campaign and this API is broken. They just don’t talk that way. Hey, your website sucks, your app sucks, and I’m frustrated.

And so how do we get the product organization to think about it from the customer’s lens? I think that’s the key to, behind your question. And yes, absolutely. I want to add Gen AI to make it accessible, more easier to understand that data. Because I think. It’s not the struggle that organizations have access to the customer’s lens or not.

There’s lots of different ways to do it. It’s a question of like, how do they interpret that information to make the right sound decisions and not running off and Oh, but this log file says this. And this API says that I’ve seen that happen in organizations. And I’ve seen this toiling of hours of dozens of people trying to make the right decision just off the wrong data.

So I think it really is grounding in, are you looking at it from the customer’s lens? That’s really the only way a business should be run. It’s focusing on the customer first. And then, yes, you know what? The customer is frustrated here. Wow, I’ve tracked that down to an API or the log files tracking this line of code.

Let’s go focus our energy and efforts there. But sometimes it’s just, it wasn’t obvious that they needed to click this button or it was below the fold and they’re never going to see it. And there’s like this, all these parts from the user’s lens that get missed in log files and APIs and just. Really, the data source should be essentially listen to the customer, not necessarily ask the customer, which I would, my mind, I relate to, get a survey is a customer happy or not.

And I find value in surveys and I’m not dismissing any of these other techniques, but they should be used at the right moment. And I think it’s really about looking or listening from the customer’s lens first.

Hannah Clark: You’ve championed in the past democratizing data within organizations. I’d like to know what does that mean to you, first of all, and what challenges do companies tend to face when they try to make data more accessible across departments?

Mario Ciabarra: We’ve seen this evolution of consumption of data, I think to the year 2000, and I’m sure some listeners like, yeah, people use data before 2000, but I think like digital came to life for me using data and I’ll pick on things like for the the OGs listening like web trends, as an example, like I remember, 1995 or so using web trends to get analytical data from log files.

And then we progressed to companies like Omniture and Urchin was the predecessor name of Google Analytics. We started collecting like pixels and understanding journeys and experiences that way, so it started off in marketing analytics. If we put it somewhere, like the understanding of digital happened in marketing, we then saw the advent of the world of APM and we saw these kinds of DevOps analytics coming to life.

And then we’ve seen experience analytics with companies like tea leaf and click tail. Generally don’t exist in their former forms, today, but they were the initial take at how do we understand the user experience, from heat mapping to session replay. And then, in the last 5 to 10 years, we’ve seen product analytics come as a category.

I stand back and I listen to our customer and I really see the consolidation of all of these different kind of point solutions or siloed analytics to how do I take this data and democratize it across the organization to understand the why. And I think that understanding of like it’s been easy in any one of those systems to understand the, what Hey, look, there’s drop off at stage three of the funnel and we only have 2 percent conversion rate on our retail website.

Like that. I think every system has done pretty well. I think okay. Why are people dropping off at step three? I think that has been like the journey to understanding that why is different for a product owner, a DevOps owner, a merchandiser, a marketer, and because of that data is so similar across those use cases.

I don’t think it makes sense to have all these point solutions. So I think that the goal for us is how do we take that understanding of data and make it consumable by all of these different personas. And that journey is going to look different. The tools, like I’ll pick on a UX designer, for example, they typically don’t spend their time in analytics tools.

But when we overlay the analytical data in the format that they’re comfortable with, which is, just the screen, for example, so we take what are people clicking on and does that lead to success? And, I’ll pick on a retail or like a purchase, it starts to link the connection between the content, the layout, the merchandise to the financial goals that we’re trying to achieve.

And ultimately, that’s really what customers came to your website for. They didn’t come to. Browse and read content. Typically, they probably came to Hey, look, if you have the product I want and the right price point, I’m going to buy, how do we make that easier for that consumer? And so taking that data and making it consumable for the U S designer for us meant overlay the data onto the website that they’re on, or for an APM person, they get into the Y or, DevOps person.

It might be that the page loads fast enough. Were there any API errors and so on? And thinking about like how, for the product owner might be like, Hey, what’s the experiment where they in? And, what step did it break, at and just thinking about what are the different kinds of questions and the right formats of presenting that information for the different personas?

I think that’s what democratizing data means for me. It’s like making that same base data layer consumable by the different personas. So we don’t have 16 point solutions that are being maintained by our organization. And, I think going into these organizations, I see people fighting over like the most mundane, Hey, your data is different than mine.

I’ve got 2.3%. You’ve got 2.4%. Man, there’s something wrong here. What’s going on. And they just spend hours and days trying to narrow that down to man, if we just had one system and just agreed, because I don’t think the 2.3, 2.4 is the key point to like argue on it’s let’s just accept that our definition of a session or user could vary.

Where are they struggling? Where can we optimize the experience? How can we get them through checkout faster with less clicks, or so on, or less friction points? How do we make the experience better? And that’s an opportunity for Gen-AI to make this easier where, one of the things that we did with Felix AI once we realized we could summarize intent and a user is, wow, we can shape this for a different audience.

Sure, someone might want a quick summary. But I mentioned that call center persona. They don’t want to know all the detail. They just want to be able to, like, how do I interact with the customer in a very efficient manner? The product owner might want to know what experiment were they in and what steps did they go through and what features did they interact with?

A DevOps person might want to know which pages load slowly and fast, and we can shape the response of Gen AI for that audience. And that has been transformational for our customer base because It just makes it easier to access the information that they want for the right persona.

Hannah Clark: We’ve got a couple more topics to cover that are a little bit wildly different. So I’ll be a little bit rapid fire here.

Data hygiene. So let’s talk about that. A lot of companies struggle with data hygiene. How does Gen-AI help to mitigate some of the issues that we see commonly across organizations?

Mario Ciabarra: I’ll tell you, Hannah, I am like smiling from ear to ear because this is a project that’s been near and dear to my heart since 2015.

It was when we created our first explainer video. We said in that video, we’re going to go do this. I haven’t found a way to do it. Here comes Gen-AI. I am so excited that by the end of this quarter, so the next couple of weeks, we are releasing a product that does this, that actually finds the data that we need to make decisions for our business.

And I’ve been traveling the last couple of weeks and every customer and prospective customer I’ve been talking to, I asked a very simple question. How often is your data, right? And how often is it wrong? And you’d be shocked that all of them say on a weekly basis, they’re having to make decisions with.

Not having the full data. How difficult is that to make decisions and not having the right data? So what is the value of having the right data all the time? And there’s been a, some confusion in marketing and the marketplace here that the auto capture auto tag and all this stuff, we really haven’t hit the point where we can extract the right information, maybe going beyond tagging to collect the right information to run our business, because I keep seeing people not having the right information.

I think for product owners, what I’ve heard from them over and over is, we’re in a sprint. Analytics is in our timeline. We’ve got a bunch of bugs that creep up at the last part of the sprint. I can either tag the right data and get, my analytical information, or I can fix the bugs. And you know what they all choose.

And so they ended up going to production, not having the analytical data to understand, are they having success or not with the new capabilities they released? How crazy is that? What if we can automate that with Gen-AI? What if we can ask Gen-AI to say, Hey, I need these pieces of information, find it in experiment A, B, and C, without me having to say specifically where.

Don’t have to tag it with my engineers. Just extract the right information from our interactions from our website users and our app users. And the look on the face when we demo this product, I mean I was asked, I was talking to an industry analyst. I said, 1 to 10, I want you to tell me what you think about this.

And I showed, I demoed the product working. She said, 10 plus. This is amazing.

Hannah Clark: Yeah. It’s enough to make you a little misty in the eyes.

Mario Ciabarra: I’m excited about 2025 success, about how we empower organizations to have the right data to make decisions in real time.

Hannah Clark: That’s really exciting. And congratulations in advance.

Okay, so let’s talk about, since we’re talking about 2025, let’s talk about the future. So you’ve predicted in the next few years that product analytics will evolve into customer or experience analytics. What trends or innovations do you see driving this transformation?

Mario Ciabarra: Understanding the customer, listening to the customer is more than just digital analytics.

There’s a lot of information that we can get from that’s already been collected in organizations. It’s just not being pieced together. I’ll give you an example. In Quantum today, we pull survey data, like we, we integrate with the major survey providers. And sometimes we want we want to use data to listen without having to ask, where are their friction points.

Sometimes it’s just phenomenal. Just ask, Hannah, are you happy or not? Did we use data to understand Hannah’s experience successfully? And when that matches with the data we collect, awesome. When it doesn’t, okay, there’s something that we’re missing and we can go find part of the experience that we weren’t really understanding for Hannah.

But there’s more than just surveys. What about, when Hannah fills out a review? It’s not like we asked her. She just said, Hey, I love this product, or I’m really unhappy with this product. What about, I’ll pick on a telco as an example. What about when Hannah had three drops phone calls this month, do you think she’s going to turn or renew her subscription with a telco next month?

Maybe that’s not the only thing that has to happen, but maybe she ordered a new iPhone and it arrives, like literally one of my team members ordered a new iPhone with a telco this year, never got it, do you think that person wants to renew and stay with that telco still? So there’s all these pieces of data that we can understand at 360 degree view, both comprising our offline experiences, our online experiences.

And I think fundamentally what we’ve been trying to do the whole 10 years of Quantum is listen to customers. Are they happier today or less happy? Are they more loyal today or less loyal? It’s definitely more than surveys. It’s definitely more than digital experiences. How do we combine all that information to help executives, product owners, dev ops, all these different teams understand what are the points that matter to the customer, subscriber, whatever kind of audience that we have and how do we make it better for them?

How do we focus on the moments that matter?

Hannah Clark: Oh this has all been extremely interesting. It’s very exciting to hear how Generative AI is completely going to be changing the game for analytics and making it so much easier, more accessible for anyone, whether you’re a very data focused PM or whether you’re a lot more of a soft skills PM. So very exciting stuff to talk about.

Thank you for joining us, Mario. Where can our listeners follow your work online?

Mario Ciabarra: Feel free to follow me on LinkedIn, but for my work but the company quantummetric.com. We have so much content of sharing success stories from our customers. What’s up and coming and being released.

So join us on quantummetric.com, follow Quantum Metric on LinkedIn and listen to some future podcasts here with Hannah and I.

Hannah Clark: Awesome. Thank you so much for coming.

Mario Ciabarra: Thank you, Hannah.

Hannah Clark: Thanks for listening in. For more great insights, how-to guides, and tool reviews, subscribe to our newsletter at theproductmanager.com/subscribe. You can hear more conversations like this by subscribing to The CPO Club, wherever you get your podcasts.