Product Testing (with Erin Hess from Crema)

Michael Luchen

There’s so much time spent talking about building product, but what about testing product? What are the approaches, postures, and disciplines that go into ensuring a product is thoroughly tested so that it supports the users that it was built for? Joining us on the show today, we have a field expert in test engineering to dive deep with us into what defines this critical role on a product team, how they partner with product managers, and why they just might be a key empathy champion on product team. Stay tuned.

This is The CPO Club podcast — the voices of community that’s writing the playbook for product management, development, and strategy. We’re sponsored by Crema, a digital product agency that helps individuals and companies thrive through creativity, technology, and culture. Learn more at crema.us. Keep listening for practical, authentic insights to help you succeed in the world of product management.

Hey, everyone. I am so excited today to welcome Erin Hess to the show. I’ve had the personal pleasure of getting to know Erin when she was recently brought her talents to our team at Crema as a Senior Test Engineer. Erin has a unique set of creative and technical talents gained over 12 years of experience across multiple roles within software development and engineering.

She started her career as a customer support representative before becoming a self-taught coder who develop a love for software testing. A user advocate at heart, some of Erin’s key strengths include empathy, visualization, and the anticipation of behavior and communication. Erin enjoys to create maps of processes and imagines where pain points might be. Through walking in other shoes, looking at problems from multiple angles, and seeing the long-term consequences and benefits of today’s actions, Erin has been able to make a difference for any team and product that she joins.

Erin also helps organizations adopt and implement inclusive design practices by creating and spreading awareness to universal usability barriers. She champions accessibility and continuously contributes resource materials to ever growing online testing communities.

Hey, Erin. Welcome to the show.

Erin Hess

Thank you so much for that great intro, Michael. I’m happy to be here.

Michael Luchen

Awesome. I have to say, I am personally really excited about this discussion. Um, I think one of the things about the test engineer and product manager period is it can create some of the best results, uh, yet it isn’t really talked about that much.

You ready to jump in?

Erin Hess

Oh, absolutely.

Michael Luchen

All right.

So I just want to start high level for maybe some of our listeners who might be coming in to understanding maybe they don’t have a test engineer on their team and they really wanting to understand the role. So maybe you could kick us off by, uh, letting us know what are some of your favorite activities to do as a test engineer?

Erin Hess

Oh, totally. Um, there’s a bunch and this is really a topic I could go on and on about. Um, there’s so many activities that you get to do as a tester. Um, one of my absolute favorite things in general is building relationships and getting to know those teams that I’m working with. So it’s the developer, the designer, the product managers, um, other engineers, client support, uh, even marketing sometimes.

And, um, on unique projects, working on like a collaboration project, uh, getting to know the client side resources as well. Um, I’ve found a niche and writing technical documentation. So I really appreciate like having the time and a space to give the right amount of detail and description, um, with useful visual aids.

So if I can get creative and add a picture of a cat sitting in front of a computer, then I’ll do that. Um, and, um, let’s see. I like learning new tools and pairing with other engineers. It’s something I really look forward to.

Michael Luchen

Awesome. How do you approach the relationship building, um, with your teams, particularly coming from, uh, the angle that a test engineer needs to come from?

Erin Hess

I’m super open-minded and I know that, uh, everybody’s a little bit different that I’m not going to have the same kind of conversation with the designer as I might the developer and you know, that same thing in mind with the product owner. Um, so I, I’m very open-minded and I try to have as much room and empathy for the team as I can.

Michael Luchen

How do you find that balance between like creating that, that room and empathy, especially when you know, everybody is different. Everybody has their own, uh, skills and disciplines that they’re bringing to a product team.

Erin Hess:

I will say for me, that, that discipline has been just patience and practicing, uh, learning, giving myself and my team retros of like, how did we do for, you know, caring about each other and, you know, going on through this sprint, you know, what things might’ve been difficult? Um, not just the work itself, but like as human beings and giving them that space to talk to me and have those kind of conversations.

Michael Luchen

That’s awesome. You know, kind of stepping out a little high-level for a second.

What are some of the, uh, typical activities that a test engineer does on a product team? Um, it’s certainly not and you know this is a joke, but it’s certainly not just saying it can release, right?

Erin Hess

Oh, no. Yeah. It’s not, I don’t just sit here with like a stamper with the word pass or fail on it. Um, it’s a lot of intricacies, a lot of relationship-building more than you would think, um, for being a technical role. A lot of working with and collaborating with different parts of the team and that’s going to be seen as like pairing up with the developer to, you know, maybe get better at doing pull requests and reviewing pull requests.

Pairing up with a product manager to, um, get the right level of detail in your acceptance criteria, um, to, you know, set the standard and the expectation of what’s going to be in the acceptance criteria.

All the different user stories and features, um, need to be looked at, you know, not just by the designers and the dev and everyone, but stakeholders and having all these different moving parts and pieces. As the test engineer, you’re part of all of that. Like you have to, to keep up and communicate with all of these people, because if you don’t, you’re going to miss something and you’re going to lose something, and the risk of the product, not meeting requirements increases.

Michael Luchen

Interesting. I mean, and then it kinda connects to, I think what you’re talking about in terms of really finding that niche for yourself in technical documentation because it helps pull all that context together, right?

Erin Hess

I also use it as my second brain. So it’s like, I don’t expect myself to remember everything, but I know that my documentation is pretty excellent, so I can reference it when I need to on a project or anyone else to go use it.

Michael Luchen

By that reference to the second brain I think, at least something I’ve seen firsthand is you keep all this documentation publicly accessible from the product team, right?

Erin Hess

Yes.

Michael Luchen

That is.. sorry, go ahead.

Erin Hess

I was going to say it is just out there.

It’s not hidden or locked away or behind any weird paywall. It’s, it’s right out there for the team to consume. And when I do create stuff, I definitely share.

Michael Luchen

That is really cool. Um, you know, and it just transparent documentation is something that I think where product teams if they’re not already doing, should really push themselves to do. Not, not only from test engineering but from all crafts that make up a product team, because with that comes to context, which the context is why you have communicated is what helps your role thrive.

You know, for others who might be listening, who, um, or again, new to the test engineering role, maybe they haven’t had the privilege of collaborating with one on their team. Where is the best point, uh, on a product team for the test engineer to start collaborating with that team? Is it sometime after that product work kicks off? Is it towards the end? Is it sometime else..

Erin Hess

I, I will say I’ve learned over time that the sooner you can bring the test engineer in, the better. Um, some companies have this stance of they, they’re just kind of there to fill it, fill it task and write a test and pass or fail it. And there’s not a whole lot of involvement, but when you want those good test engineers and the ones that are going to bring more than just the testing abilities to the table, bringing them in on that pre-kickoff, bring them in on that context week and let them kind of see what they’re getting into and they can start imagining a lot.

From the very beginning and even help contribute to, you know, influencing, um, designs for testability, influencing development, um, bringing up accessibility [00:09:00] testing, uh, instead of it being an afterthought.

Michael Luchen

That’s awesome. I completely agree. Um, it is always become, it is evolved to become a huge risk for me.

If I see that there’s a team that’s kicking off without a test engineer at the beginning, because that, um, context that you’re gathering from all those angles is so key too in helping us ensure that we have a quality tested product that meets the user’s needs. Um, it’s much more than, as you said, kind of that rubber stamp that maybe, um, sometimes gets passed around.

So as a test engineer, what would you say is the things to keep in mind, um, during a sprint?

Erin Hess

Oh, good question. Um, so I’ve got a couple of things for this. I would say, uh, first off, like you don’t know what you don’t know, um, so don’t be afraid to reach out to your team, uh, when you got a doubt or concern, regardless of how severe you feel it is. Like sometimes a small concern can develop into a significant improvement. Um, another reason to speak up would be to that, uh, it would reduce the risk and cost to fix an issue, if, if whatever it was, you didn’t speak up and it got released.

And then there’s no “I” in product development, uh, like there’s no “I” in team. Um, you’re not doing this to yourself. You’re on the team, um, and you’re not going to do the team any good when you don’t tell them you’re struggling. So I would say, you know, one of the worst things you could do is wait till the end of a sprint and notify your team that you’re unable to do something. So, speak up.

Michael Luchen

Yeah. Yeah, I think it’s, it’s, it’s so crucial because especially coming from kind of the product management side of things, um, that helps identify, is there any major risks in this sprint?

Whenever we start a sprint as a full product team, we run into a position where we have a limited assumption of what the risks are going into. Um, but I think as you’re pointing out from a test engineer perspective, there could be something that’s flagged early in the sprint that could change the trajectory of that sprint for very good intentions.

Erin Hess

Yes.

Michael Luchen

So getting into maybe some, uh, inner craft collaboration, um, what are some of your tips for how tests engineers, um, and designers and developers should be set up for success to collaborate?

Erin Hess

Ah, yes. So for this, I would say, um, it’s definitely beneficial that design and devs, um, pair up with the test engineers during the design and development of the user stories, uh, whenever possible.

And I can say from experience, like we’ve learned, um, that there’s opportunities to pair with design and development during the early stages of a project and it gives me more information to prepare test scenarios, test files, write up automation scripts. Like right now I’m, I’m enhancing our acceptance criteria and requirements with including Gherkin so that when we’re ready for automation, the big chunk of the work’s already out of the way.

Michael Luchen

That’s cool. And you know, I’m kind of curious, you just mentioned several concepts there, like test scenarios, for example. Um, you know, what, what is that? Uh, is it, is it the, is it going through the acceptance criteria that, like as a product manager, I write or is it something more intricate?

Erin Hess

It’s a little bit of, it’s both. Yes. So it’s taking the acceptance criteria that the team and the PDM have gone together and improved, and then with those test scenarios, it’s going to be as this type of user on this type of device. You know, what am I doing? What’s the expected outcome? Um, what are some risks? What are some things that I might want to try to, uh, mistreat, you know? Um, it’s coming up with a different end to ends flows of like what a user could potentially do in that test.

Michael Luchen

So when we deploy a product, I mean, there’s just, there’s so many in places where that product might be used. Um, multiple web browsers, different screen resolutions, um, different devices and more, and through like those test scenarios where you’re kind of saying, like, you want to, you want to give the, put the product in kind of a tough situation.

Um, how do you determine in this like massive scope of all the things you could to, um, where to focus those efforts at?

Erin Hess

I’ve learned overkill is a thing with testing. You can definitely overkill it with accessibility testing, doing too much too often on pages that aren’t necessarily needing to be tested that way.

You can just, just, go nuts with testing, um, and to reign it in, it takes time and practice and like some forethought into, Okay, you know, this is the type of project that you’ve got and, you know, it’s going to be like a web app, um, versus like, you know, a native app or, you know, it’s only going to run on iPhones.

So for, you know, different things that it could work for and you think about all the ways that you’re going to be testing this and you just start to weed out the things that you know you’re not going to need a test. And you start building out their test major C’s of, all right I need this device with this browser and this type of thing for this type of user.

Michael Luchen

So as you’re looking through figuring out your test scenarios and where you want to focus, um, you, are you regularly looking at like browser usage statistics, device usage statistics, things like that. And then how old are you looking at what’s relevant to the product that you’re working on?

Erin Hess

There are a lot of resources that I use, a lot of test engineers use that kind of inform us on what’s the most practical matrices of things to test on, you know. What’s the browser usage right now for the any given device or any given browser version?

And you know, how many people are actually using these things with these types of, you know, applications, or products and services? And then we kind of used that to inform, all right, you know, we’re definitely wanting to check Safari and Chrome, you know, iOS and Android. Um, and then you kind of list out maybe like lower risk things that you would like to test if you have time, like, you know, iOS or like Edge or, you know, Netscape, some other random browser, you know, that’s not super important, but it would be nice to know if the app works on it.

Michael Luchen

Yeah. Yeah. That’s really cool. Um, you know, I’m also curious, are you, is there like any user testing or surveys that can inform, uh, where to be testing as well?

Erin Hess

When you say like forms, like, uh, is that like from a service, from a website you’re using..

Michael Luchen

Like surveys, like a, yeah.

Erin Hess

I personally haven’t done a ton with like user surveys, but I, I like filling them out. I like giving that feedback. Um, that would be something really unique to get into that end of feedback from that deep, the real users. Um, I definitely take into account, like what’s, like the average, what’s the average user experience expected for this type of application, and then I kind of go from there.

Michael Luchen

Awesome. Awesome. Um, before we move on, I also got to ask about something else you recently talked about, which is you write automation scripts. Um, what is the value of automation on a product and why and when should it be introduced?

Erin Hess

Oh, cool. Very good question, by the way. Um, this is also one of those ‘it depends’ scenarios. If the project is long-term and ongoing and it’s got a robust team and it’s like a whole, like, there is no end in sight kind of thing, which can be good for automation. You can actually start, you know, with your manual tests and rank them based on risk and feature.

And then, you know, do some thoughts and thinking about what kind of automation framework you want to use, or do you want to go with a low code web-based solution? Lots of options with automation and how to do it. And from there, you’re just thinking, okay, well, what’s, what’s the most, what makes the most sense to do first and that can be on a project if it’s a brand new project that’s starting from the ground up.

Thinking about automation doesn’t hurt, because there’s some parts with the, you know, pre-development and design where you can have opportunities to work with those teams. Like for example, uh, creating accessibility tags on components.

That’s something we’re actually doing right now in Gherkin to, uh, basically have some code and some scripts already written before there’s even a product yet. So it can save time, um, it helps you kind of plan a little bit better if you’re doing it kind of in parallel with, uh, design and development and testing.

Um, it adds another layer of complexity to the test engineers, um, what they get to do, uh, each sprint because on top of the, um, acceptance criteria and, you know, writing and reviewing their test cases you’re also almost a little developer by, you know, writing and reviewing your, um, acceptance criteria written out for your automation scripts.

So, early is great, but it’s kind of like, whenever you really feel the need is there and it’s usually going to be because you want to speed up these and the process overall for testing and also, um, stabilize the testing.

Michael Luchen

Interesting. Yeah, that’s really fascinating. Um, so I’m curious, shifting gears into another role on the team, selfishly, how do you approach collaborating with product managers?

Erin Hess

Oh, this is fun. Um, Like I had mentioned before, like I, I like building relationships with my team. So it isn’t going to be based on hierarchy, uh it’s, it’s because I want everyone in the team to, to be, um, collaborating and working with each other and I like kind of helping that process along. So I like working with the PMs just as much as I do with the devs and the designers.

Some of that comes out of just collaborating on and helping with, um, acceptance criteria or doing research, um, gathering information about [00:20:00] something that, you know, they’re more interested in about. Maybe we can do this approach for this type of test or, you know, any, any task, really, if there’s something the PM needs, uh, and I can assist, then I step in. Yeah.

Michael Luchen

That’s cool. I have to say, as, as I alluded to earlier, um, I think the test engineer and a product manager partnership is one of the most impactful on a product team, um, because of that, particularly that focus around the acceptance criteria and how things are going to be validated before they’re released out to users.

Personally for me, I find there to be a really nice balance when product managers and test engineers collaborate early on that acceptance criteria. Um, what I have found is that, um, the product managers might be bringing a sense of, but not limited to, you know, a sense of focusing on what are the outcomes for the user we’re trying to get to, but the test engineer is bringing a sense of, okay, like what are the details?

How do we make sure, like all the gaps are covered and inappropriate level that still allows creativity from the rest of the product team to complete this, um, so that we can make sure something quality is released and tested that supports the user needs. Um, would you say that balance? Okay, cool. Cool. I was like, I’m assuming something here.

But yeah, I think there can be, um, so much when, uh, when that, that pairing starts early. And that’s one of the reasons why I always advocate for, um, test engineers need to start at the beginning of the, uh, product, um, just like all the other roles are starting, um, for that to happen.

Erin Hess

I totally agree. And I’ve been on the ends of where, where you’re not, and now I have like really good insight and I really appreciate being able to get brought in early at the very beginning.

Michael Luchen

Yeah. It’s very clear like once you experience a product team without a test engineer, it’s such a night and day like it’s, it’s very clear [00:22:00] how terrible things can be, um, I think.

Alright, so, um, something that you’ve shared about before in the past is you’re a huge accessibility advocate. Can you talk a little bit about what that is?

Erin Hess

Totally. Yup. Um, so the word accessibility gets thrown out a lot right now. It’s kind of a hot topic. Um, it’s also known as ‘A11Y’. Um, and that is because there’s 11 letters between the A inaccessibility and the Y. Um, and what that, what, uh, I would say the advocate means is that you’re a, you’re a voice for all the different end-users. And you’re going to champion for like features and development of products and services to be accessible by everyone. So lots of inclusion, lots of talking and, and finding out like where can I use my voice in a way to help someone else?

Michael Luchen

That’s awesome. You know, I’m, I’m curious, like, um, the, something that, you know, maybe unfortunate I’ve seen is, um, sometimes stakeholders might be like, uh, we don’t have the ability to prioritize, um, these accessibility things.

Um, how do you, kind of challenged that, or, or kind of ride that wave?

Erin Hess

I’ve done this in a few different methods and I found that the most successful one for some stakeholders is going to be showing them the numbers of how expensive it could be for the business if they do not have a compliant website. Um, the risk that is involved in getting a lawsuit, uh, it, it’s, it’s messy.

It’s really messy. It’s hard to recover and it does leave a bad taste in people’s mouth if they start to learn that, you know, you didn’t care about this whole other group of end-users. And that might not have actually been the case, but that’s what [00:24:00] they’re going to think.

Michael Luchen

Interesting. Can you talk a little bit more about empathy as the foundation for testing?

Erin Hess

Yeah. So you’ve likely, or most people have heard like the term, “walking in someone else’s shoes”, uh, which is also another phrase to say empathy. Um, and it’s the capacity to understand or feel what another person is experiencing from within their frame of reference. So like the definition of empathy could really encompass like a broad range of emotions.

But to further press the topic, um, I have actually done a presentation on empathy specifically, uh, and testing. And it was at the Ministry of Testing, um, TestBash home event that was actually earlier this year, um, where I presented with my friend, Jenny Bramble. And we talked about our feelings and we [00:25:00] talked about how, uh, “Empathy is the Foundation of Testing”, um, and what that looks like and how our histories put us in a unique place to talk about empathy and customer needs.

Michael Luchen

Pretty cool. So kind of like in a nutshell, like what are, what, what makes it the foundation of testing? Like what are the tactics there?

Erin Hess

Oh boy, uh, like ultimately, um, there’s an intersection of testing and empathy, and that looks like, um, like you can’t know someone’s expectations if you don’t know their experience.

So stepping into someone else’s experience is like an important part of testing and we do it all the time and we may not even think about it. Like testing a high contrast theme, or a dyslexie font, like it’s, it’s there for a reason. Um, so ultimately like that intersection of testing and empathy falls squarely into the realm of accessibility testing.

Michael Luchen

Interesting. That’s really interesting. It’s just, just chatting with you over the last 25 minutes it’s, it is interesting to see a few themes emerge, um, such as like the intersection of, um, being that accessibility advocate, um, how empathy goes through everything you do as a test engineer. Um, and just how important collaboration is for the test engineer’s role.

Not only with the product manager but with the product team as a whole.

So before we wrap up, I’d love to ask some personal lightning round questions, if that’s okay?

Erin Hess

Shoot it.

Michael Luchen

Alright. Um, so the first one, uh, which of your personal habits has contributed most to your success?

Erin Hess

Oh, I will say this absolutely is like my ability to create and structure and organize vast amounts of information, developed over time.

It’s like one of my superpowers. Um, and I think that [00:27:00] this came from my obsession with organizing an organization in general and, um, passion for writing good, useful, meaningful documentation.

Michael Luchen

Awesome. Um, and what is your favorite tool that you use regularly?

Erin Hess

I will say that’s definitely going to be a Chrome DevTools, um, for so many reasons, but I’ll shout out the Device Mode, which you can use to simulate mobile devices, test responsiveness, and different specific viewports. Uh, the Network Panels, my best friend. I definitely use that when I’m going into dev tools to report like anything I can if I find a defect in something, I, I’d start digging in there first. Um, and then Lighthouse and Axe, like for accessibility testing.

Michael Luchen

Awesome. So last question, uh, for someone at the start of their journey in the product, um, what is one piece of advice that you would give them?

Erin Hess

One piece of advice I would give for someone like at the start of this would be, um, be open to learning anything, um, and no matter who it’s from. So if you’ve got a developer or a designer or the product manager wanting your ear or your time, to do something, take it. Use it as a learning experience and gain from it.

Michael Luchen

Very well said. I have certainly learned a lot today. Thank you so much, Erin, for coming on the show.

Erin Hess

You are so welcome. Thank you for having me and I look forward to talking to even more about accessibility testing and test engineering in general.

Michael Luchen

I am so excited to get into it. And for those of you listening who want to learn more from Erin, you can find out more about Erin’s work at crema.us. Find her on LinkedIn. Erin is also active on Twitter and you can find her at her handle @QueenTester.

If you enjoyed today’s show, check out our related publication, the QA lead, where we explore the latest and best of automation and testing. You can check it out at theqalead.com.

Again, thank you so much, Erin, for joining the show today. This has been a fantastic episode. Thank you everyone for listening. Please be sure to leave a review for this podcast. I would love to have your feedback and then also be sure to follow and join our community at theproductmanager.com. Thank you.