How to Communicate the Right Product Analytics the Right Way

[00:00:00] Hannah Clark: On to the latest in our community events series. Good to see everybody here. Um, so, uh, we are currently we’re growing really quickly. We’re seeing really great success with these panel events. So I just want to throw a quick thank you to everybody who’s attended our events so far. Um, you guys are great and they make them really fun for us.

Um, and if you’re a new to the product manager and to our kind of content, my name is Hannah Clark, I’m the editor for the product manager. I’m happy to meet you. And today’s session is going to be focusing on how to communicate the right product analytics. We’ll be speaking with some great voices in the thought leaders in this space.

So I’m just going to take a second to introduce our amazing panelists. Um, so today we’ve got Moe Haliba. Moe is the CEO of DataWisp and that’s a natural language, uh, power, full featured data platform. So very relevant to this conversation. Uh, Moe is a leader in fast paced product [00:01:00] development, and he brings very deep expertise, uh, in Data driven decision making space and agile product management.

He’s also a big Arsenal fan, as you can see from his shirt. So a little known fact about Mo, is he actually caused the 2008 financial crisis. If you didn’t know, it’s all because of him. So we have him to thank. Or at least his friends tease him about it, obviously we’re joking. Mo, can you tell us what happened exactly in 2008?

[00:01:27] Mo Hallaba: Sure. I, I, so I graduated from college in 2007 and joined Merrill Lynch and literally within days, like the financial crisis began. And so everybody told me it was my fault. Like I started in equity research and on my first real day on the job, we downgraded every single stock that we covered. So fun times.

[00:01:48] Hannah Clark: A great start to your career. Uh, well, I’m glad that you’ve, uh, managed to recover from it and, uh, can’t say as much about the economy, uh, . And we also have Jeff Orange, uh, joining us today. So Jeff is a leader, uh, tiktoks product team. [00:02:00] Uh, he boasts over 14 years of experience as a product manager, and he’s a master, uh, disambiguate.

He constantly pushing towards clarity to deliver products that meet customer needs and grow business strategically. So, Jeff. Awesome. Back to you. Um, you have been to 37 concerts this year, which to me sounds nauseating, but that’s amazing. Is there anyone you haven’t seen yet that you’re hoping to get a chance to hear in person now?

[00:02:23] Jeff Orange: Uh, yeah. So, uh, 37 in the last calendar year. So it’s, we’re mid May or mid, what, uh, September now. Um, so, you know, um, not quite 37 this year, but in the past calendar year. The one I’m most excited about is I am getting ready to go see Weezer again, I think for the third time, but being the old guy here, um, they are actually playing their entire first album, which is the blue album.

It’s what I, uh, you know, listen to all throughout my high school years. So I’m very excited about that.

[00:02:54] Hannah Clark: That’s awesome. Okay, I, I would love to see Weezer. Um, and Victoria Koo is joining us as well. [00:03:00] Uh, so she was most recently the head of product and design at Hynote. And prior to that, she spent eight, uh, as she describes, eight long years at Airbnb during its hyper growth phase, uh, building, uh, various products, uh, in business lines, including Airbnb business travel, co hosting, pro hosting, magical trips, and she also scaled, uh, Or was, uh, highly responsible for scaling global payments.

Um, so Victoria, in your spare time, I understand that you are a sculptor, amateur sculptor. Have you been sculpting anything lately?

[00:03:28] Victoria Ku: And I’ve been sculpting, um, a newborn. So, I have shipped and delivered that product, and looking forward to going back to plaster level sculpting now.

[00:03:39] Hannah Clark: Wonderful. Do not blame you.

So. We’re going to get started in just a moment. Um, I’d love to hear, uh, for folks who are tuning in and in the chat, uh, please throw in where you’re tuning in from. We’d love to see kind of where everybody is, uh, joining us from. And I’ll just go through a couple of just, uh, need to knows. Um, so first of all, the session is being recorded and will be made available shortly [00:04:00] after the call.

Uh, we may use clips from it on our website and our social channels, uh, or your cameras and microphones currently are turned off by default, and you will not appear on the recording. So don’t worry about that. Uh, we’ll also be running a Q& A session towards the end of the session, so please, uh, you’re encouraged to post your questions in the chat.

Uh, nothing’s off the table. Uh, if it is off the table, we just won’t answer it. Uh, but we’ll get to as many as we can in the last 10 to 15 minutes. Okay, so we’re going to get started with the discussion, um, which, and we, our discussion today is going to be in three parts, so, uh, our, our sections will be collecting and interpreting data, second section will be effective strategies for making data visible and digestible for all stakeholders, and then we’re going to finish off with a section on storytelling using data, um, so we’ll kick it off with collecting and interpreting data, uh, and this question goes to Jeff.

Um, so we can get analytics from almost anything related to our product. How do we decide what to track and maybe more importantly, what not to track?

[00:04:54] Jeff Orange: Yeah. Um, so, I mean, the biggest part of this is, and I mean, this is, you know, uh, product [00:05:00] management one on one is how do you measure and, uh, show that you have achieved what you said you were going to put forth in the product management development process.

It has a start at the beginning. Um, and it has to be part of your requirements as you write the features, how it’s supposed to look, how it’s supposed to interact with your back end, so forth and so on, how your opera, how you’re going to operationalize. That decision making has to be done at that level of how you’re going to track.

Um, there are so many times I’ve seen in so many organizations that the, uh, analytics team and the analytics knowledge actually lies outside of the product team and it’s kind of like the product team does their thing. Then they go, Hey. Let’s check on how we would measure this and that always causes a huge disruption and it’s never a really good output in my opinion.

So like making that part of the process of as you’re writing the requirements, you’re like, okay, how am I going to measure this? And if you’re not the right person to have that knowledge, Making sure that you have your development team also part of that. Uh, I was most [00:06:00] proud of an organization I was part of to where like, I actually got to a point of where I had engineers going, how are we going to track that?

And that’s like the dream kind of state, just because that means that your data and analytics thought process kind of becomes your culture.

[00:06:20] Hannah Clark: Uh, does anyone want to add on to that or build on to that?

[00:06:26] Mo Hallaba: I think tracking as much as possible all the time is, is never bad. Uh, obviously, you know, if you’re a smaller company and you have, um, resource, uh, bandwidth issues, then yeah, maybe not everything, but at least, you know, when you’re launching a new feature, you want to be able to tell how people are using it.

And if it matches your expectation of how you designed it. And so if you don’t think ahead of like, Hey, what things can we track to, you know, to just point, um, you’re kind of flying blind.

[00:06:58] Hannah Clark: All right, this next question goes out to [00:07:00] Victoria. So what are some challenges or errors that your teams have made in the past when it comes to collecting data, and what do you do differently now to avoid those mistakes?

[00:07:08] Victoria Ku: Yeah, at Airbnb, there was a great example of this where we have a very extensive data platform.

We track, as Jeff said, we track like nearly next to everything, right? Every little piece of detail. And during the pandemic, you know, we decided to double down in this area and see if we could optimize for transactions in the payment system. And we, we did the whole shebang, right? We, we made sure that we asked like the experts that like, if we moved this lever, would this actually net us the results that we thought we got?

Like the industry data so that we could compare, we, you know, we did the double check. Um, and so we did all the experiments, we like moved all the little levers and all the little like metadata. And like, really, it didn’t do much. So that was the funny thing is that like, we did all the things that were right.

And so the, the key here was that, um, just because you [00:08:00] actually collect the data, does it always mean that you’re going to net out the solution that you think you will, and that’s the really exciting part is theory versus execution. You need both, right? You need to prep, you need to also try, and then. What makes wisdom and experience so valuable is that you’ve actually done it in real life and you can say, Hey, given this circumstance, we saw a different result.

Or you could say, this is exactly what we thought at Airbnb. We saw multiple situations, um, and both, both of which in which we thought we would move hugely. Huge amounts of nights with a small amount of work and, um, and didn’t, or when we just like added a single word, um, just out of curiosity and we moved like 0.

21 percent of total nights books.

[00:08:43] Hannah Clark: That’s, um, okay. So, uh, let’s, let’s assume now that we have a reasonable amount of high quality data on a product usage. Uh, and this, this goes out to Mo by the way. So what do you start looking for and how do you interpret raw inputs?

[00:08:56] Mo Hallaba: Um, I think people are always looking for some [00:09:00] kind of like, You know, direct correlation between the data that they’re looking at and like exactly what their users are going to do.

And then, you know, Oh, if we move this button here, it will do this or, uh, and that’s just not really how it works. Um, the way we like to think about it is just. informing more decisions with data. And so rather than saying these people did this thing, and therefore X or Y or whatever, um, we like to just dig into the data a bit and see how people Uh, stayed on our website in general for 60 seconds or people left after 30 seconds.

And so if you’re trying to create a, um, a feature and that feature can be faster or it can be slower and it could be, you know, more detailed or less detailed. If you have an idea of how long people tend to hang out on your website before they bounce, you could use that as a, as an estimation, you can use that to kind of drive, um, [00:10:00] You know how long you anticipate people to stay there, but that’s not to say that people are going to stay exactly that amount of time, right?

And you do that over and over and over. And and if every decision that you make is now informed slightly more by some bit of data, then you can start to piece together a story. Um, so I would just shy away from from, you know, correlating like one data point with with something and just making a decision based off that.

It’s not as simple as that. So if that was long winded,

[00:10:29] Jeff Orange: no, no, I think it was good. I’m oh, I was just going to kind of carry on top of that. Another thing that we used a lot in other organizations is kind of like a peer review from a, uh, uh, analytics perspective. Like this is how I’m thinking about measuring this.

And that’s something that I’ve always found to be an effective strategy to kind of. Make sure that we’re all on the same page, and it’s not just one person’s view of that. You know what I mean?

[00:10:52] Mo Hallaba: There can easily be other things that can influence that, that you might not be aware of, or like some detail about how the data is tracked, or how [00:11:00] it’s measured even.

Um, hey, actually, you know, these things reset at midnight, or this is eastern time, and that user It’s all these little details, right? And so, you know Yeah, double checking with the other people that you work with, not just flying, you know, based off of that one data point.

[00:11:18] Hannah Clark: Yeah, I’m sure that that’s really important for avoiding confirmation bias and trying to see the narrative that you want to see in the data that you’re collecting.

A lot of

[00:11:26] Mo Hallaba: people use data to justify the things they already want to do. And that’s like, that’s not really how you’re supposed to do it.

[00:11:34] Hannah Clark: I think the opposite.

[00:11:36] Mo Hallaba: Yeah.

[00:11:38] Hannah Clark: Uh, does anyone have any other, uh, thoughts kind of on that topic before we move on to section two?

All right, so we’ll move into effective strategies for making data visible and digestible for all stakeholders. Uh, so this one goes out to Victoria. So how do you walk the line between making your point clear without, uh, causing data overwhelm, especially when speaking to, to stakeholders who are [00:12:00] very far from, uh, data science and, and much closer to, uh, the soft skills?

[00:12:05] Victoria Ku: Yeah, definitely. Definitely. When, when it comes to, like, the strategy of, um, communicating, like, obviously, I think things are pretty complex. You always want to start where the stakeholders coming from. But if I were to generalize it a little bit and say that this is the, this is an effective general strategy that I saw just happen time and time again at Airbnb, um, The key here is just building a great story and narrative before you even start going into the data.

So at Airbnb, we always started from the user, and to Jeff’s point, culturally this does need to be supported for you to be like fantastically successful, right? You’d be surprised how not narrative crafting or storytelling some executives and some companies are. But I’ve always seen that if you are able to craft a narrative and start from the user, um, you can’t, you can’t not be successful with building empathy for that user, which is ultimately what your company is there to do.

And so when, oftentimes we would [00:13:00] start with like, here’s the scenario for the user, right? They had this amazing goal in mind. They want to travel. They’ve taken, you know, all this PTO, they do this one side of the year and it’s for four days, right? And you kind of highlight how hard it is for the user to just like get by and like now imagine this amazing product that you’re building, right?

And like, we’ve seen that, like, um, with this product, you can reduce X amount of friction, right? And this is where you start weaving the narrative and creating the narrative by using the numbers to kind of prove that this narrative is going to do. In fact, like the most amazing user experience the product Going to, to build on what the company is set out to do for the world.

And that’s where I find that you don’t have as much, um, data well, um, uh, data overwhelm, like versus if you just came in and said X amount of users experience X, Y, Z, and then bam, this is what it’s like. And, um, people start having questions and then you start going deeper into the complexity and then, and that’s an opportunity sometimes to lose people when they’re coming from [00:14:00] a different area of the business.

[00:14:02] Hannah Clark: I really, I like that approach. I kind of like having the data clear in order to support your points without having to just kind of data dump like a bunch of information before people have got that natural curiosity. Um, does anybody have anything that they wanted to add to that?

[00:14:21] Jeff Orange: I kind of do, but it scales into my next, uh, into the next question you’re getting ready to ask. So it’s like, it’s kind of actually a perfect segue.

[00:14:28] Hannah Clark: Oh, perfect. Okay. Well, the next question was not all teams will benefit from seeing the same dataset. Um, so how do you, you, uh, Uh, tailor that information depending on the audience that you’re presenting to scalably.

[00:14:40] Jeff Orange: Yeah, exactly. And I mean, I think this is kind of what Victoria was saying is, you know, product management is 90 percent know your audience. And you need to be able to have flexibility in your BI layer to where you can meet. Um, the V. P. Um, that maybe doesn’t have deep analytics experience and understand the difference between the [00:15:00] server side and client side analytics event or something like that.

And you need to just be able to tell the story to them. You need to understand what their needs are and then provide a tool and platform to where your resources or that resource themselves. Can kind of pull that information and have what they need, but it needs to be flexible to the point of where it’s executive level kind of copy and paste my charts.

Uh, down to the level of, I need to actually see those analytic data points and do validation. Make sure that they’re all the way through, um, our platform and exactly how we want to, uh, see them. And also have regression testing for every time that you launch. Um, that’s really important as well, because I’ve seen that many times before a new launch comes and we’re like, Hey, where’d that tracking go?

It all of a sudden disappeared. Um, but specifically around that, you know, flexibility in the front is it’s a hundred percent about knowing your audience, but it’s also Speaking to your audience and understanding what their needs are.

[00:15:58] Victoria Ku: I would love to add one more thing, [00:16:00] um, which is just on top of what Jeff said.

Um, I think it’s really different between like big tech and startups, right? Like, unfortunately you’re just not going to have that much data when you’re starting out. And so that narrative becomes even more important versus if you work for like the Amazons. Microsoft’s like the air B and B’s, you’re gonna have already infrastructure built for this area that you want to experiment on.

And so you can, you can exercise some creativity because you have these resources and decide what is the optimal path, depending on who you’re talking to. With startups, that’s not the case, right? You might have to really double down on that narrative, or you’re just gonna have to do a ton of research and maybe just research line, right.

And just like double down on that one research line and say, Hey, the industry says that if we do this right, we’re going to net out positively by X percentage, and I believe it and just kind of have to double down on that. But, um, it goes back to what Jeff was saying, right? Like it’s 90 percent knowing who your user is and crafting that story.

So that, that’s, that’s great. that [00:17:00] the user can be supported.

[00:17:04] Hannah Clark: Mo, did you have anything to add before I pass the question your way?

[00:17:07] Mo Hallaba: Everyone’s kind of said what was on my mind already.

[00:17:11] Hannah Clark: Awesome. Okay. Well, I’ll toss something different to you. Uh, so what are some strategies for data visualization that can be helpful for keeping stakeholders informed?

I feel like you have a very good answer for this one.

[00:17:20] Mo Hallaba: Yeah. So, and in general, I would say, keep it simple. Um, a lot of folks when working with data have a tendency to go into like scientist mode, which is what, which is like, I conducted this experiment and I measured these 800 data points and this is my methodology and this is my, and like people don’t care about that.

Like we have a data scientist and also my CTO does this whenever we work on stuff internally, they, they always follow this. Kind of pattern. And I always tell them, just do it backwards. So show me the result first, show me what you tested and then, and then [00:18:00] tell me about this stuff. Because for, for product managers or business people, or, you know, people managers or whatever, um, they, they’re more like results oriented or more outcome.

So, okay, what decision are we making? What change to the product are we making? What strategy shift are we, and, And you kind of have to start with like, what is, what is the result or what, what decision do we want to make? And then how did we come to this conclusion? Okay. So we looked at these data points and here’s some charts that back up this idea that we had, and it just kind of helps them like digest things.

If you just put way too much data or like a giant dashboard with a hundred charts on it, Nobody’s going to retain all the necessary, um, stuff. There’s also in terms of just visualization, some chart types are just easier for people to understand than others. Uh, our data scientist cam hates pie charts with a burning passion.

Um, simply because if you have like five [00:19:00] slices of a pie, it’s just never as easy to visualize, like how much bigger one section is than the other, like versus a column chart. It’s like little things like that. Sometimes, you know, people are really good at Tableau and they make these like super, super fancy, you know, area charts and things like that.

And they’re, and just for regular business people, they’re incredibly hard to understand. Or, you know, the worst is like your data scientist always wants to show you like a bubble plot with like, you know, the area corresponds to this metric and the color corresponds to this other, just make column charts and line charts and keep it simple until people understand what you’re Um, explain.

[00:19:41] Jeff Orange: Oh, I, I,

[00:19:43] Mo Hallaba: Victoria,

[00:19:44] Jeff Orange: we both want to jump on that. We’re like immediately the product managers. I know, right?

[00:19:49] Victoria Ku: Um, I would, I would say that like, in a similar sense, right? You just can’t go wrong with a basic column chart. That goes from top to down, like greatest to least, like [00:20:00] the human mind looks that way, looks at the real estate on the page and just thinks that way from top down, left to right.

And so like, if you’re on the fence about how to visualize data, you can’t go wrong with a very simple column charts. That’s top to down organized. Um, that’s something that I learned the hard way and being the type that love to do beautiful visualizations. Um, the second thing that I’ll say before I hand it over to Jeff is, um, Again, as someone who loved to give more information, a former mentor used to say, like, don’t volunteer information just yet, right?

Craft the narrative. All else goes to appendix, right? You can have the, like, gorgeous amount of scientific information. on the RET presentation, no problem. Put it in the appendix. That way when someone asks a follow up problem or follow up question, you look uber prepared. You scroll down to your appendix, you say, great question.

I’m prepared, right? Here’s, here’s the appendix to showcase like what I mean. Um, and it’s just a win win. So handing over to you, Jeff.

[00:20:59] Jeff Orange: Yeah, [00:21:00] no, that’s perfect. It’s like, and I’m kind of just echoing the same thing in the perspective of like, you know, um, just making sure that, uh, you know, as far as far as I go is like from a visualization standpoint that you’re, um, understanding that audience, uh, and knowing what exactly their needs are.

But also like, you know, I and I apologize. I kind of lost track of something. Somebody stopped by and I’ve lost my train of thought. Go ahead and skip to the next one. I apologize.

[00:21:35] Hannah Clark: Oh, no worries. Uh, we can move into a story, their storytelling section now. Um, but yeah, Jeff, if you, if you remember what you, you were, uh, going to say, we can come back to that last one as well.

Uh, and just a reminder for folks, uh, if you have any questions that you wanted to ask our panelists. Um, we would love to answer the questions towards the end of the event. So, you know, stream of consciousness, whatever comes to mind, please feel free to pop them in the chat. Um, so we’ll move on to [00:22:00] section three, storytelling and deriving action items from analytics insights.

Um, so we could run an entire workshop on storytelling and, uh, probably will. Um, but, uh, what are the most valuable lessons, uh, in your experience and strategies that you’ve learned for piecing together stories from data? And this is from Oh,

[00:22:19] Mo Hallaba: sure. So I find that in general, like when we talk to our clients that data was the first thing that we do is ask them, you know, what are your goals, right?

Like what are you trying to achieve? And so, um, you really need to understand, um, your objective and you need to understand the data that you have at your disposal and kind of what your tools are for, for digging into it. Um, it is really important. And I think if you, a lot of people fall into the trap of.

You know, we have clients that come to us and they go, we want to get insights from data. And we go, great. About what, you know, what are you trying to accomplish? Right. And they go, well, here’s all our data, you know, [00:23:00] and then it’s just giant database with tons of tables and stuff. And we go, cool. Like, what are you trying to do?

And until they answer that question, it’s really impossible for us to help them in any meaningful way. Right. So, so generally it needs to be, well, we’re having an issue with retention. We go, okay. Um, why are you having an issue with retention? While people are leaving, uh, within the first X amount of time of using the app and we’re trying to figure out why, okay.

What do you think it could be? Well, we think it could be these five things. Okay, cool. What data do you have to support that? And then we can dig into that data and actually investigate. And, and, and, and that way you can, you know, you have an idea, you, you investigate, you can create a story around what you’re trying to accomplish and what data you’ve looked at and what data supports the, the changes that you want to make, right?

Otherwise it’s just chaos.

[00:23:56] Hannah Clark: Does anyone have any, uh, thing to add before we move on to our next question? [00:24:00]

[00:24:00] Jeff Orange: Yeah, I, I kind of, I, I remembered what I said was going to say, and essentially, uh, it’s kind of, we’ll build on that a little bit. It’s a matter of like, that’s kind of, you know, what product management’s job is, is kind of being the middleman.

Um, you know, like as Bee was talking about and your talking to a data scientist, I’ve worked at some. Companies that had some very, very smart data scientists. And I was owning a project. We were building our own in house analytics tool. I didn’t understand half of what they were saying. And, you know, like being part of that, like you need to know, like, uh, from a product perspective, you don’t need to know everything about analytics.

You need to know and understand the basics and the frameworks. The same thing you do for, you know, software development. You need to understand basic software development frameworks and different types of code that people use and so forth and so on. But you don’t have to code, you know. It’s the same thing with data, except for you don’t need to know how it’s actually implemented.

You need to know how to visualize it and how to prepare [00:25:00] a presentation to meet your client’s needs. So that’s kind of like my main point that I was trying to get to there.

[00:25:05] Mo Hallaba: Different, uh, different companies have different models when it comes to data analytics and data science. And on, in some companies there’s like a central like data practice, um, and they kind of like help everyone with their stuff.

And then on some companies, every kind of team has their own like internal data team that helps them. And it’s really important, you know, because based on how your company functions, um, it might be more important to. Help your external data team understand what it is that you’re trying to accomplish because they’re not part of your team.

They don’t work with you every day. They don’t understand what your objectives are. They don’t understand what you’ve tried. Uh, they don’t understand what your, what your customers like. Points of friction are on often, as is the case with a lot of companies, because there’s so few data scientists and there’s so many people that are asking them, [00:26:00] um, stuff they’re just like going through a queue of like tickets, but essentially, you know, people have submitted these questions and we’re just cranking out whatever, and sometimes it’s easy for them to lose the context of like why this question is being asked or who it’s relevant to and stuff.

So it’s really important. Then when you’re talking to your data, people that you communicate all this context and why things we’ve tried. This is what we’re trying to achieve. Um, and then. Use that to, to help them better serve you.

[00:26:29] Victoria Ku: I would love to answer

[00:26:29] Hannah Clark: that

[00:26:31] Victoria Ku: really quickly. Um, agreed on all the points.

And I would say that if you have extra cycles, like, like everyone has extra cycles in the world right now, um, Understanding that business intelligence is always just gonna net you like additional an additional skill set. Um, these data scientists are thinking about their models. They’re thinking about the pipelines.

They’re kind of thinking about data in aggregate. But if you can come in and truly understand like what each Data element [00:27:00] actually means and like how does it actually net value for your user and your product? You’ve already done probably 70 of the leg lifting for them So just like just being able to understand this part of the product scope Is going I would say it’s it’s it’s what differentiates a junior level with a mid level Product manager.

[00:27:21] Hannah Clark: I’m seeing a common thing. I was just going to say, and this also supports my point that it seems like a common theme here is like really understanding, like putting yourself in the perspective of like, what are we trying to achieve? Uh, where does the user fit into this and kind of like using, you know, some emotional intelligence and psychology to understand on like how to communicate with stakeholders.

Very interesting how, uh, all this kind of fits together. So

[00:27:44] Mo Hallaba: Victoria, to Victoria’s point, um, we, you know, we build a product. There’s other companies that have similar products that allow, you know, business people to just ask questions in plain language and like get the answers that they’re looking for.

Uh, and there’s increasing, you know, [00:28:00] amounts of, of tools that allow you to do that stuff. And so to her point, if you, if you know a little bit. If you understand what data is being collected and you have just even a basic understanding, you know, you can leverage tools like that to ask some of your own questions, be curious, whatever, and really speed up the cycle.

Because right now, if you have to ask someone else a question, wait two weeks for them to get back to you. There’s only so much that you can do, right? But if you’re more informed and you can ask them better questions, you can get, you know, better answers faster and you can iterate more efficiently. Um, so definitely, you know, get involved a little bit, understand what data you track and how it’s tracked.

And that knowledge can only help you as a, as a product manager.

[00:28:41] Victoria Ku: You earn the respect of your engineers, which is only a plus.

[00:28:44] Jeff Orange: A hundred percent. Yeah. They’re more willing to work

for you.

[00:28:48] Jeff Orange: Yeah, I know one way that I always found successful to do that though is like whenever you’re at that situation, like writing the release notes as the product manager, you know, like, Hey, here’s the new tracking that we’ve had take that as [00:29:00] ownership and just write all that down because there’s no way you won’t know everything as long as you’re that person that’s doing the documentation.

[00:29:10] Hannah Clark: Are we ready to move on to the next part of the next question? All right. Uh, so the next one is for Victoria. Um, so what are some of the pitfalls to watch for when trying to use storytelling to communicate user insights? I guess we kind of a little bit touched on this, but yeah, pitfalls. Yeah,

[00:29:24] Victoria Ku: yeah, definitely.

So we talked about crafting the narrative and crafting the like, um, the story, um, so that people can create a mental model before you give them all the data. You know, the main pitfall that I see is that People sanitize the data too much in in response to crafting an understanding or a narrative. And that part is really detrimental because, um, our leaders rely on us to give them the necessary intelligence for them to make the necessary decisions at their level, right?

So if we if we overly sanitize it, we’re essentially washing away the root [00:30:00] problem, right? And you’re gonna hear me say this all the time. Like, Correlation versus causation. Most things are actually correlation. Um, so you can’t actually say that because we made this one change, this other change occurred, and we know a hundred percent that this is why, right?

Most things at complex scales are correlation, not necessarily causation. And so, um, that’s kind of like the main issue that I see, especially as you ascend the ranks, um, is that I want to put a call out to our industry is that like, you don’t want to sanitize. the data too much, right? Insist on giving the necessary data.

However complex it is, either in an appendix or either through crafting the perfect narrative, but insist that this data does actually get through to the person that is consuming that. Um, I would say the second thing that I see as a pitfall is just, okay, you’ve created the narrative. Um, the follow up is what about this data at scale?

Does it scale? Right? That’s the second question that I hear almost 100 percent of the time, [00:31:00] and it’s not always going to be the case, right? In smaller companies, you’re gonna, you’re gonna go out on a limb, you’re gonna have a gut feeling, right? And that’s your, that’s your experience talking as your job as a product manager in this area.

And that’s okay, right? You can boost your narrative with as much data as you can, but ultimately, it’s okay to say, You know what? This is a gut instinct. The industry heads in this way. I think that, like, you know, we have this level of tolerance for risk, right? I’m willing to put resources against it. And here’s the, here’s the bottom line.

If we don’t reach this number, then, like, let’s, let’s end the pilot, right? And so that’s okay. But these two areas are overly sanitizing and in pursuit of the narrative, um, and also, like, forgetting about what happens at scale and what to do otherwise. Like, those are usually the two pitfalls that I see in this area.

[00:31:49] Jeff Orange: Yeah, and just to carry on top of that, Victoria, I think like, you know, um, as much as you possibly can do an AB test, that’s how you get that early validation about your assumptions. Um, [00:32:00] and like most organizations I’ve been part of, whenever we have a new feature that rolls out, we’ll start out with the percent of traffic and have a cutoff point at each one of those and iteration as part of that.

And that only works if you have the proper tracking and the product proper data to lead it in the right direction. So, um, yeah, great point.

[00:32:19] Hannah Clark: Um, I, I’m curious if Mo has anything to add, but before we get into that, we actually did have a question popped up that I think is relevant to answer in the moment. And someone asked if Victoria can expand a little bit on what she means by overly sanitized data.

[00:32:33] Victoria Ku: Um, so I’m going to be a little bit of a contrarian.

I apologize if I insult anyone. Um, the mental model that I want you to imagine is Are what I call Davos types, right? Davos is this beautiful industry in Davos, Switzerland. Um, I mean, it’s this beautiful conference and all the world leaders go to Davos and then they discuss about how to solve the world’s problems, right?

Do we create infrastructure? We create like, you know, um, uh, additional nonprofits. [00:33:00] In my honest opinion, a lot of times, the root cause is just rich people need to pay their taxes, right? That’s a good example when you overly sanitize, um, a, a root cause, a root cause, a narrative, right? You, you give this beautiful demonstration and story about what the world could be like if we had these nonprofits, if we had these, like, um, beautifully set up infrastructure.

But in reality, the, um, the root cause is much more simpler than that. It’s just been kind of like washed away. And so you kind of focused on creating this mental model. And now the person you’re talking to understands that, but they don’t quite understand what it takes to solve the problem and truly what it takes to solve that problem for several reasons.

One is you might forget, right? The narrative is, is beautiful, right? You’re having a lot of fun with it. Um, and, and like, number two is like maybe leadership has really enjoyed going down a certain, um, Solution path, right? They really want to solve with this solution and that’s kind of where our jobs as [00:34:00] product managers, we kind of have to keep it on the problem because there could be many solutions for Properly articulated problem, right?

Um, and the third one is just like, sometimes we don’t know. Um, and so rather than to show that we don’t know, we would prefer from a career perspective, um, to focus on what we do know. And so that sanitization can also occur again, a little bit cynical, but in my, in my honest opinion, it is a realistic situation.

It’s not meant to involve anyone, but that is what I mean specifically.

[00:34:33] Hannah Clark: Hey, I like that. I wonder also, there’s a factor of sometimes the recommendation is not what the people in the room want to hear. When I think of Davos, I kind of think of, I don’t know if you saw that clip of, uh, uh, someone’s kind of calling out the billionaires in the room.

That was very. Yeah, yeah.

[00:34:51] Victoria Ku: Yes, I was like, bravo, right, bravo, because that’s exactly, we just have to pay our taxes sometimes. But, um, but that, that’s kind of the mental model that I want people to [00:35:00] think about, right? The narrative is fun, it is a great skill set, just don’t forget your user, right? Ultimately, who you’re building for is your user and the problem that they are experiencing.

[00:35:11] Hannah Clark: Hey, does anyone want to be a contrarian to the contrarian or go back to?

[00:35:16] Mo Hallaba: No, I think, um, in line with what Victoria said, you know, causation and correlation are not necessarily the same thing. And You know, it’s, it’s more about making more in general data driven, uh, or not, not data driven. I kind of hate this word.

I’m just kind of conditioned to saying it because everyone else says it. We like to say data informed, data informed decisions. So you’re aware of what something is and then you make your decision yourself. Um, there can be, there, there’s, you know, if you’re just looking at one or two data points, like there, there’s so many things that could be misleading.

For example, you introduce a bug that causes your, your product to take longer to spit out the thing the user is looking for. And then you go, Hey, our users are [00:36:00] spending more time in product. It’s like, yeah, in this session they are, but they’re not going to come back for another session because they had a miserable experience.

And so you’re looking, if you’re, if you’re only tracking, you know, Time and product on first session, right? And you go, Hey, time and product on first session is skyrocketing. And it’s like, but we’re not getting more paying customers. Why could that be? Um, it could be because you’re, you know, simplifying things too much and you’re only looking at one data point and using it to, to mean something that it’s not necessarily what it means.

[00:36:34] Hannah Clark: I

[00:36:35] Jeff Orange: think another thing that kind of pops up to me is like, you know, especially like, you know, in e commerce, a lot of teams are broken down like in different parts of, you know, the purchase path. So like you’ll have a team that owns checkout. And you have a team that owns like, you know, the product page and maybe search and then someone else owns home and content pages, you know, just like the main components and each of these teams operate usually pretty [00:37:00] independently, but they’re so reliant on each other.

So like, even just understanding your, you know, roadmap, Your roadmap also needs to include the data points that are going to be part of that roadmap and make sure that you’re all aware and aligned on how you’re going to track and surface this data. Um, and it has to be a team thing. It can’t be a, an organization thing.

It can’t be a team thing.

[00:37:21] Mo Hallaba: I have an amazing example, and it’s of us, so we won’t embarrass anyone else. Um, for a while, we were running these ads, and we had outrageously high conversion rate, like on the ads. So, so basically, uh, we were just looking at parts of our funnel, and we go, oh, 50 percent of people that click an ad for DataWisp, like they were click ads.

End up creating an account like our landing page conversion was insanely high, right? And we were so happy about this. And then we realized that these are all students who are trying to do their homework and data wisp, and they would show up and they would just literally copy paste their homework. And then it wouldn’t work [00:38:00] because that’s not how data was works.

And then they would leave and never come back. Right. And, but if you’re grading, just the, you know, the marketing side of this, we were doing phenomenal. Right. And so you have to understand like, who are these people? Are they the right kind of people? Like when these people make it into your product, what are they doing in your product?

It’s not just that you had an amazing conversion rate, right? You have to track them kind of all the way through. Um, so that’s from, from personal experience, an easy way to make that mistake. Yeah.

[00:38:29] Hannah Clark: Wow. And it’s coming from a marketing background that would be, uh, setting up for very, very unrealistic expectations for future campaigns.

I have to say, sorry, Jeff, were you going to jump in?

[00:38:39] Jeff Orange: No, it’s just, yeah, same thing. I mean, we all probably have some sort of horror story about this. Um, you know, it’s just like, I mean, if I think about that same example, if I’m the product manager, why don’t I just auto add to cart? And then I have like a 90 percent add to cart rate, but well, that’s going to kill the next person that’s like, you know, well, I can’t get these people to check out now.

Just [00:39:00] auto checkout. It’s on your credit card. Well, I think we just discovered a business. We’re going to be okay. But yeah, no, I mean, it’s, it’s just a matter of, um, having conversations and having alignment, uh, not only with the people that are on the output, but the people that are in the same teams as you, or, you know, uh, teams that are siloed similar to you, and being able to make sure that you guys are all on the same page is really crucial, and it’s a great learning opportunity as well.

Oh,

[00:39:31] Hannah Clark: yeah. Everything should be a learning opportunity, good or bad, I think. Um, yeah. That’s kind of like a non thing to say, but, uh, all right, so we’ll go to our last question before we head into Q and a so just the last call for anyone who has got some questions brewing from what they’ve heard today.

Anything at all that you’re curious about does not have to be related to these three sections. Also, you know, if you’ve got a specific situation that you’ve been dealing with, we can always toss it to our panelists. Um, so. Last question is for Jeff. How [00:40:00] do you ensure that the insights derived from your story lead to clear, actionable steps, and how do you track the execution of those actions?

[00:40:07] Jeff Orange: Uh, yeah. So, you’re asking about project management, and I hate project management. Uh, the way that I found best to deal with project management is just from the perspective of, like I said from the very beginning, it has to be part of your project plan. Like that has to be like, we’re developing something and as we develop it, this is how we’re going to do the testing and make sure that we’re doing the tagging correctly.

This, this point is when we’re going to build the dashboard that ultimately will serve this new feature. Um, and so forth and so on and have that part is the complete plan from end to end. I’m thinking from the obviously perspective of a new product development and how you make sure that you keep that there.

Um, but I think the other thing to call out is, you know, that final step of presenting to your audience can sometimes be a wild card. You know, um, you’re sometimes [00:41:00] going to have challenges and like, you know, being able to actually. Get people to believe the data and, you know, like that actually happens too.

So, you know, just keep all those things in mind and make sure that you understand what the complications could be, make sure that you have a supportive team that you work with and the right tracking, you should be set up for success in a, from an analytics standpoint.

[00:41:23] Victoria Ku: I would add to that, um, bringing it back to culture, um, This is why culture is so important.

Um, and, and why I guess, like at Airbnb, like culture was such a famous part of the recruiting, um, in good environments in which, uh, like operators are incentivized to take data and, like, create solutions and, and kind of like there is no bureaucracy in which you have to, um, cut through, like, a ton of process in order to just, like, get one little experiment tested.

You are incentivized and almost encouraged to go through and experiment and [00:42:00] glean insights and learn and create like wisdom and experience for your team and yourself, um, in bad culture mindsets, right? Um, it’s that all of that is not existing. Right? You have to. It’s almost like to even make a little change.

You have to go through like three different bureau committees, and this is just for a tiny little word change. Right? So that kind of gives you an understanding of like why we we talk about culture and why the environment does need to be set up, setting you up for success because sometimes you cannot push against the environment, right?

And it’s not any problem of yours or your product. This is an area that is very hard to change. So know where to put your energy.

Very true.

[00:42:43] Hannah Clark: Any final remarks before we move on to Q& A?

Okay, I think we can, so we’ve got some questions here going, um, and folks if you want to keep on asking, and there’s still time, we’ve still got time for more questions. Um, so I’ll start with a question [00:43:00] from Michael Murphy. Uh, I’m trying to transition from project management to product management. I have some strategies on how to prepare for this, but would like to know what the single best thing I should do is to make this transition.

So I guess not so much analytics related, but more just general career advice. Um, anyone want to speak from their own experience?

[00:43:18] Jeff Orange: That’s actually the path that I took. Uh, and I think, you know, like anytime that you can get in as part of a project team, uh, and be part of that product team, um, that ultimately is supporting that you’re going to get a lot of exposure and understand, like what product management is and find if it’s the right thing for you.

My assumption would be that you’ve already went through that experience because you’re a program manager, but I think Victoria actually said it perfectly. And, you know, Hey, focus on like. Learning data, how to analyze data, how to present data in an effective way. If you’re able to do that, that takes you from that mid to that senior level, uh, from [00:44:00] a product perspective, that’s one of the key things that I think you can glean as part of this is just, you know, making sure that, you know, you’re In that process, make sure that you understand that product management is something that you want to get into, because I think a lot of people think they want to get into it until they’re actually in it.

Uh, but then also, you know, just making sure about those other aspects as well.

[00:44:22] Victoria Ku: I’ll add to that. Um, if again, there’s a lot of complexity that we could answer with, but I think if I were to just answer one general statement, I would say lead with an expertise. At the end of the day, Lead with an area that you know better than anyone else.

If it’s data, right? Lead with understanding exactly end to end how this all works that you could build a better system, right? If it’s a certain user, right? Let’s just say, you know, hypothetically it’s like tax, right? If you understand taxes and you understand the legal structure, right? That’s your leverage, right?

That’s when the engineers, the data scientists, the designers come to you because you understand what needs to be built. Um, so, [00:45:00] Summarizing that like lead with an expertise and then find like a really great mentor or sponsor to kind of transition you into that initial, that initial product area.

[00:45:16] Hannah Clark: Before we move on to the next question, while we still have everybody in the room, I just want to give a little call out to our next event. So if you enjoyed this event so far, we’re going to get back to Q& A in just a moment. But I would love to see you guys at our next event, which is very exciting.

It’ll be an online workshop. So the workshop is called How to Use Improv Practices to Build Team Dynamics. If you have experience in improv, you’ll find it really exciting to put some of those skills to good use. If you’ve never done improv before, don’t worry, you’re not going to be on stage. It’s going to be a lot of fun, very dynamic, uh, uh, Session with Samantha Gonzalez, who is a wonderful, uh, friend of ours from Dockyard.

And she’s also an improviser and a leader. Um, it’s going to be a really fantastic and fun session, so we’d love to see you there. Registration for that event will be opening later this week. [00:46:00] So, uh, you’ll want to bookmark the link that Michael is just about to put in the chat. Uh, so when that goes live, uh, we’d love to have you on there to pre register.

Uh, and you can also check back on our website later this week if you’d like to RSVP that way. Okay, we’ll get back to the questions now. Um, let’s see, where are we? Uh, so this one is from Diane. Can you speak about product roadmap as a communication tool? How far out are your roadmaps? Do you have a different roadmap you share with customer partners?

And how often is the roadmap experience, um, items moving from one quarter or half a year to another? Or do you find that they typically are stable and don’t change? It’s kind of a few questions. Maybe we can do that, uh, bit by bit. Uh, let’s start with, can you speak about Product Roadmap as a communication tool and kind of, um, how far out they tend to be planned?

I think that it probably will be a little bit different for each of you, since you’re kind of coming from different size companies.

[00:46:51] Mo Hallaba: We had a product roadmap actually, um, that we just got rid of. Um, ultimately, you know, as a small startup, it [00:47:00] wasn’t kind of outward facing product roadmap. Um, and we found that typically customers want stuff now, they don’t want stuff in six months. So we found that it wasn’t super helpful. Um, internally we have a product roadmap, uh, only as far as like being able to, you know, budget development resources and strategize around.

Uh, how things are going to get executed. Um, In a startup, these things kind of get moved a lot, especially when you’re trying to, uh, onboard your first few clients. And then you realize they need things that aren’t on your roadmap and you, and you have to shelf your roadmap kind of in favor of some more urgent stuff.

So in general, roadmaps have not been particularly useful for us. They’re great to put in your investor deck. If you’re a smart, you know, a startup, because it shows vision of what you want to build, but. Not much use besides that we have a, um, what we do now is we just have a kind of product backlog where things that we identify as useful.

We kind of just put in there [00:48:00] and then every now and then we’ll just kind of re re, uh, shuffle the priorities on our backlog and, and that’s, we’ll just toss a bunch of stuff into our next sprint and, and take care of it then. But we’re also a very small startup, so it’s, you know, that’s not necessarily advice for everyone else out there.

[00:48:19] Hannah Clark: Yeah, I was just thinking, like, Jeff is coming from a much, much different size team. What does roadmapping look like for you guys?

[00:48:27] Jeff Orange: So I, it’s all over the place and it’s completely different. And Mo, I’ve been in that environment to where it’s like, uh, roadmap is just like an Excel sheet, you know what I mean?

Um, then I’ve been in other organizations where we build, um, you know, pretty robust, uh, product roadmapping tools. Uh, and we talk about what happens 18 year or 18 months out. Um, but. Honestly, I, I kind of agree with you a lot of those things because whatever you put on your roadmap, even three, six months down the road, like think back to 2019, maybe October, [00:49:00] you had a clear roadmap for 2020 that completely changed, and there’s nothing you can do.

What you need to understand is what the business needs are. And make sure that you’re have those as goals for each quarter, and then you write the requirements and the features and the operation changes to meet those goals. So setting more like goals and like objectives for each quarter versus like, Hey, here’s a hardcore roadmap and we’re going to do this, this, and this, which seems less flexible.

Um, that is the ultimate goal, but very large enterprise organizations. Or have no tolerance for that, uh, just because they need predictability. They can’t, they don’t want to take on that much risk and so forth and so on. So it’s kind of like, you have to find that happy medium to where you don’t overcommit, uh, but you are committing to a point to where it’s showing value within the organization.

It’s the magic and, uh, uh, uh, [00:50:00] uh, art, I guess there is a product management right there.

[00:50:03] Hannah Clark: Yeah. Yeah, I tend to agree. I tend to be a big fan of benchmarking of like, oh, this is kind of, you know, these are the goals. These are kind of where we should be tracking, but not, uh, I just kind of think like having a very specific deadline for when everything has to be done and just as a recipe for things just running over and causing a train wreck.

Um, did, uh, Victoria, did you wanna, uh, add anything before we move on to the next question?

[00:50:26] Victoria Ku: Yeah, I guess from a like tactical perspective, I think the smaller the company or like it also depends on what type of company you are. So B to C versus B to B, right? Um, it really depends on the environment that you’re dealing with.

That is where the little like the levers will change. So that’s where like frequency of when you have to create the road map will change. Um, that’s also like the stability of like ranking and priorities will also change. I mean, in general, like the bigger the company, the more the resources, um, The more like stable it could be just because you already have a ton of infrastructure built.

And that’s when you’re looking at like half a year cycles [00:51:00] and with the end of the year doing a full year, um, you know, road map like draft, right? And then the smaller companies or B to B companies that also deal with smaller companies, you’ll probably see a lot of shift, um, or especially if in our compliance area.

Right. You, you might even deal with monthly at first, which is exhausting, right? And which is you want to like get to the point where you’re doing quarterly because four times a year versus 12 times a year is just a lot of time saved. Um, so that’s kind of like just how I think about it. You’ve gotten a lot of good information from both Mo and Jeffrey, like on both sides of the spectrum.

So this is, this is how I would in general think about the environment and how it dictates like how you should act.

[00:51:41] Hannah Clark: We’ll move on to the next one from Allison. Um, do you have any tips on getting data from on, uh, on prem enterprise software when you have limited visibility into how your customers use it, uh, and they’re very reluctant to give you any data? That’s, that’s a very interesting question. Has anyone dealt with that situation before?

[00:51:57] Victoria Ku: I, I have, um, most recently at high note. [00:52:00] Um, so this is kind of where you get into B to B territory, right? You’re dealing with the businesses and kind of a layer of abstraction. Maybe you’re dealing with the finance types or like the CEO, um, instead of the actual users that they are, um, solving for, um, I think it’s super helpful to as a product manager to dive deeply into that customer and like actually look at their app, actually look at their application or product and like use it like you are a user so that you understand like the business that they are in and also how to potentially solve for some of the problems.

If you were a product manager on that in that business, um, the software isn’t always great. Everyone, Everyone always says that they can give you something, but the reality is that the nuance is just, is the nuance is more than that. Um, so doing your own research, um, while, while it can be tedious is very much a part of product management and we’ll net you pretty significant gains.

[00:52:58] Mo Hallaba: If you’re, if you’re talking about [00:53:00] extracting data from so, so let’s say we have data waste that runs on data on our website, right? Like, and we track stuff that our customers do if, if the question is referring to, like, if we had an on prem deploy, you know, for, for a custom for a B2B customer, and how do you get the data from that when they’re reluctant to give it to you?

My answer would be give them a reason to give it to you because ultimately you need to convince them that you collecting this data is going to make the product better for them. And really, this is kind of how it works across the board. Nobody wants you to collect their data for anything these days, especially because it’s so abused everywhere.

And so, you know, with GDPR, like the, The stipulation is that you are using it to make the product better, right? That’s the only data you’re allowed to capture. So just communicate that, um, make them understand that they’re going to get better features. They’re going to get better service if they let you collect this data.

That’s, that’s the only advice I can give.[00:54:00]

[00:54:02] Hannah Clark: Um, just for the sake of time, we’ll move on quickly to Jason’s question. Um, this one, I think it’s more of a stakeholders relations, uh, question. So what’s the fastest way outside of using data to solve stakeholder conflict on what to be built first? I

think fastest way is maybe a trickier question. Maybe more what’s the most effective way is maybe the how I would interpret that question.

[00:54:28] Jeff Orange: I would say the fastest and most effective way is finding the right person in the organization that makes the decision. Narrow in on that person and make sure that they fully understand, become their best friend.

That’s the best way to do it.

Yeah.

[00:54:44] Jeff Orange: Oh, there’s so much, so much of product management is politics and building relationships. And most importantly, it’s not just building relationships. It’s building trusting relationships. And if you have the right data sets. And you’re able to provide the answers to those [00:55:00] people in a quick and easy way, which is pretty much everything that we’ve talked about here today.

You’re going to be very successful within your organization. I

[00:55:12] Hannah Clark: really like that advice. Um, does anyone, uh, have any other, uh, stakeholder management tips they wanna add before we to our last question?

[00:55:20] Victoria Ku: I think that is in fact the fastest way. Leverage your relationships.

[00:55:26] Hannah Clark: Get in, get in early, get get in before, uh, , before you have to ask them for favors. Uh, all right. How do you navigate challenging or messy datasets?

Let’s say you have data coming in from multiple sources, or maybe there isn’t a dedicated owner, so data is stored in multiple locations. Do you have any tactics or go to tools that help you analyze, process, and share data with your wider team? This one’s from Brianna.

[00:55:48] Mo Hallaba: I, uh, bite the bullet and clean up your data.

It’s worth it. Just do it. Just do it. Invest the time in it. You net no one ever wants to do it. Just do it. Um, [00:56:00] the data analytics tool that you use, whatever it is, is going to give you awful results. If you put bad data into it, um, there’s just no, way around it. The, the, the hardest part for most companies is just getting the clean data.

Once you have clean data, there’s so many things that you can use to extract insights from it efficiently and quickly. We build one, there’s a bunch of other people building them. Uh, but the most important thing is clean data going in and there’s no shortcut to that. You just gotta invest the resources.

[00:56:33] Hannah Clark: Okay. That’s, that’s a mic drop moment. And I think, yeah, I think that ultimately It sounds like that is worth the investment in all cases. You know, we, I think we’ve kind of heard sort of in different ways. Um, you can’t make a good decision if your data isn’t good. So that, uh, makes a lot of sense to me.

[00:56:49] Mo Hallaba: We’ve had, we’ve had people show up with like missing rows and stuff that’s labeled incorrectly and stuff. And so even if you do all the right charts and graphs and stuff, they just don’t tell you the right thing because your [00:57:00] data’s wrong. So,

[00:57:02] Hannah Clark: um, and, uh, with that, we actually just had somebody ask, uh, if it’ll be possible to get a recording for this webinar.

Uh, the answer to that is yes, this webinar has been recorded. We are going to be, uh, putting it out publicly for everybody on a landing page, uh, which we’ll, you know, publish, uh, very shortly. I, we might even have, uh, I’ll have a link for that already. I’m not sure. Michael will have to take care of that in the chat.

Uh, but I just want to take a moment to thank everybody. Definitely want to thank our panelists, uh, Mo, Jeff, Victoria. It’s been really great chatting with you guys. Uh, such great energy in this conversation. Lots of really, uh, useful and very, uh, practical points, uh, discussed today. Uh, and thank you so much to our, uh, everyone who’s attended the webinar for making time in your day to be here.

We really appreciate, uh, Uh, your, uh, your attention and your questions, um, and, uh, just a reminder also that, um, our improv workshop that I mentioned earlier in the session, um, will be happening October 24th, uh, you can RSVP at the link that Michael will be adding to the chat, um, so we would love to have you there, uh, it’ll be tons of fun, you will [00:58:00] definitely laugh, if that’s not a reason to attend, I don’t know what is, uh, so thank you guys, uh, for making time and enjoy the