Product Talks with Greg - ProductFTW #35

For our fourth post in the Product Talks series, we chatted with Greg Isaacs. I'm sure you'll be surprised to read that Greg is another former AT&T Interactive colleague of mine. He's been CPO at multiple companies and brings a wealth of knowledge, especially on the topic of hiring product roles through many of his roles but particularly through the work he's currently doing with Criteria Corp.

Greg:

All right, let's roll! Thank you for the opportunity, by the way.

Matthew:

Yeah, of course! So, let's start with what your job is today. Where do you work, and what do you do?

Greg:

I'm the Chief Product Officer at a company called Criteria Corp, which has been in existence for 16 years. The company's mission is to help companies of all sizes find and develop better talent. I think we’re unique because there's this industry that I didn't know much about around psychometrics. These are commonly known as assessments that will measure someone's cognitive ability, personality, emotional intelligence, and soft and hard skills. But really what we do is we give clients, who range from small businesses to large enterprises, confidence that they are making the right hire and then ultimately helping those people be successful. 

Prior to working at Criteria and being the company’s first Chief Product Officer, I worked at ZipRecruiter for a number of years running International Operations. I learned a lot about talent acquisition while I was there, so that’s kind of what introduced me to this field. The thing that always struck me about talent acquisition is that—we've been in a lot of organizations where technology has had a fundamental impact. So think about Salesforce and what it did and the idea of a CRM, marketing automation, financial systems, et cetera. But HR, and specifically talent acquisition, didn't really leverage data as much as it could have. And when I say data, I mean things that help with understanding what makes a person successful: their cognitive ability, their personality, and how those things align with the role that they're looking to be hired for.

In a nutshell, we give a data-driven approach that gives clients confidence they're hiring the right person but also makes sure that it doesn't create unintended bias. It's a really fascinating field, I've learned so much. I get to work with a lot of brilliant PhDs, and so, for me, it really melds the art and the science of what I think makes Product Management great. So I’m really excited about that. I've hit year four, which is exciting. We've executed our product vision pretty consistently, and I would have imagined at year four, I would think that it was time to move on to the next opportunity because we'd done everything we wanted to do. But I have to say, I'm even more excited now than I was when I started because what AI has done to our business and to the product that we can build has been pretty game-changing.

Matthew:

I bet. That's awesome. So you mentioned ZipRecruiter, but what's the longer arc? How did you get here? This is not your first Chief Product Officer role, so what’s the path to getting up to this point that you took?

Greg:

I had kind of an interesting path. I've always just generally tried to follow two rules when looking for job opportunities. I didn't have a great vision of becoming a Chief Product Officer or even getting into Product—I actually didn't know what it was. But in general, I tend to follow or try to work with really smart people; people I admire and know are different in a really positive way. 

When I look back to earlier in my career, when I started working at eBay, I just knew someone there, and he hired me to work for him. Working there opened my eyes to so many different things. For example, the idea of a marketplace and what marketplaces are, how they tick, and how they work and the dynamics of them, or the idea of a platform and what can be built on top of it. So again, the first thing for me is I generally try to work with really smart people. And the second rule is, is it something that I can understand that I feel like, on day one, I could add value? Does it make sense to me? Can I explain it to my friends, my wife or family, or even my kids? I found opportunities where I had a tough time explaining it to them, or they didn't understand it, and I didn't get excited about it. But when they understood it, and they're like, “Oh, that's cool.” I got excited by that. 

With Product, as I found out later, it’s this really unique mix of left brain/right brain and also, strategic and tactical. For me, the context-switching in any given day is really exhilarating. I could talk to a Fortune 100 Company about some of their challenges. And then the next meeting could be about making small changes to our design. Like, “Have we thought about modifying the UI/UX a little bit?” It just really gets me pumped. But I would say that the arc was not, to be honest, one where I knew I wanted to be a Chief Product Officer. I knew what I liked, and I just sort of circuitously found it. And now that I've done a couple of different Chief Product Officer roles, and I've also done Chief Marketing Officer roles, Product for me is the 1a; it's the thing I really love, and hopefully, I get to continue to do it.

Matthew:

Yeah, you're only the third person I've interviewed, but no one I’ve talked to knew that this was what they wanted to do. I'm not even sure when the title of Chief Product Officer first appeared. The first time I heard it was probably six or seven years ago.

Greg:

The first time I'd ever heard about the role was when I was at eBay. There were a few people who worked at world-class tech organizations like Microsoft, and so they were Product Managers. I feel like at eBay we did have a Chief Product Officer, so that was the first time I heard it. But yeah, unless you're in tech, nobody really knows what it is. Even people ask me what I do, and I tell them I’m a Chief Product Officer, they’re like, “So what does that mean exactly?”

The way I explain it is I get to work with business people who want things. They could be salespeople, for example, could be marketing people. And I also get to work with technology people who can build that something, and I try and find the commonality and get those things done. Obviously, I don't want to diminish what we do by saying we're not business people because I think we are. And in some cases Product people are quite technical. But that's the way I describe it to my kids or to people who aren’t really familiar with tech. And generally, I just show them something. I’m like, “You see this thing on my phone or this thing on my computer? My job is to figure out if we should build it and then, candidly, make money off of it.”

Matthew:  

Well, that kind of leads to one of my questions, which is, what does Product Management mean to you? So that was kind of how you explain the role to someone, but maybe take it a step further. What do you think it really means, or what's its role in the business?

Greg:

I've actually put together a presentation about what a product manager does at the last few companies I’ve been at because, at least at the organizations that I get attracted to, they haven’t had a product function. And what I like to describe it as is being the closest thing to a CEO at a company without actually being the CEO. Because, you need to be multidisciplined in your capabilities as a PM. So you need to understand the business dynamics, the competitive landscape, monetization, user behaviors, and you need to understand your clients intimately -  there are a lot of facets you need to understand. Which doesn't mean you need to do them all perfectly. But for me, at least, the way I like to run product teams is that all product people should have a fundamental understanding of many of those things that I described. 

So, when I look to hire people, and I interview them, I tell them what the product role in an organization is. I say, “You are the GM or the CEO of your feature or your product line.” 

The reason I convey it like that is because sometimes product is about shipping the widgets, and then your job is done. To me, that's only the beginning. It’s figuring out what you’re building, shipping it, and then figuring out how to make it successful. 

The other thing about product is that it’s not just about the capabilities of a product manager; it is also the mindset. The deck I put together, which I alluded to earlier, was really about, what does that mean? What is a good product manager? Well, they're intellectually curious, in my view. They are really data-driven, but they also can understand that they need to make decisions with imperfect information. They generally try to move quickly. They're decisive.

I like to do this during interviews because I want the people that I'm interviewing to hear my vision for the role, at least at the organizations that I'm running, and see how they react. Does that excite them? More often than not, if they're not excited by it, it means they may not be the right individual for the product organization that I'm trying to run. I've been in a few different jobs, and so I think I've seen what works well and what doesn't. From there you just try and take the things that work well and do more of them, and you take the things that don’t work well and try to eliminate them or at least put them in a box, so to speak.

Matthew:

Yeah. I was thinking about the days when we worked together at AT&T Interactive, we had this concept of these different types of product managers. I was a business manager, which meant I had a P&L, and we also had PMs, and there was a lot of frustrating back and forth about who was technical or not technical, which used to drive me nuts. So to your point about the product  manager being like a GM and having some ownership of a product, not just over delivery but over its lifecycle - can you expand on that thought? And also a bit about how you think about technical backgrounds versus non-technical backgrounds, and what maybe some of the distinctions in your mind, that you make about ultra technical PMs? 

Greg:

At the end of the day, I think having a technical background is incredibly important for a product manager role. Maybe I'm biased because I don't have a true technical background; I don't have a computer science degree. But I don’t think it’s a requirement to be a successful product manager. I think the other things I mentioned earlier, like having the right personality for the role, and a few things I didn’t mention, like being patient and collaborative while still being able to reach a consensus and willing to take calculated risks, are important. I think being technical is helpful because you can always think about if there’s a better way to do things and speak with credibility on that option. 

But candidly, I've been in this world for 20+ years. I ran a developer program at eBay and worked with technical people all day long. I think if you're honest, you don't know what you don't know. You learn to seek counsel and advice, and you develop a pretty good BS meter when you know someone's not being completely intellectually honest with you, but these skills come with time. 

As much as I'd love to be technical in terms of being able to code, I think if you've been around enough organizations, you understand how technology works. You may not know down to the bit level, but you have a really good understanding of what solid architecture is, how we build things, and how you structure teams. Ultimately, I think it's important because I've run technology organizations, and I personally prefer to have a really great partner in a CTO. I have been in organizations where I've seen technology teams work really well, whether I was running them or someone else was. I think I know what to look for in a great partner. So that’s a long way of saying; for me, I don't feel like it's a requirement to be technical. It certainly helps, but I wouldn't sacrifice being technical for not being good at those other things I mentioned earlier.

Matthew: 

That makes sense—and I agree. It's funny because I have a technical background; I used to write code and stuff, even though I don’t have a CS degree. So, when people used to ask me this, I would say that, yeah, you should be technical. But then I looked back at all my favorite PMs who have ever worked for me, and none of them are technical. So I was like, okay, well, forget that rule.

Greg:

I tried learning Ruby a bit when I was at AT&T, and I thought it was cool. But, at the end of the day, it wasn't going to help. The amount of time I would have had to spend to become really technically proficient and to become a real coder just wasn't worth it because, at that point, I'd be sacrificing my time on other things.

Since we’ve been mentioning AT&T, I think it would be worth discussing something that I learned and keep going back to. One of the most valuable lessons I learned at AT&T was not to assume and to go and actually talk to customers. 

So, if you remember back when we started at AT&T, or I guess yellowpages.com at the time, I went on a few sales calls with one of the local salespeople, and I started to sit in meetings with their clients or on their sales calls, and one was with an industrial scale company, which I didn't even know what that was. We talked about algorithms, cost per click, being performance-based, and all these different metrics, and I was watching how the rep sold and how the clients responded. Like, “ If you spend this, you get your ad here.” It was just so eye-opening to see, like, we're tech guys, and we're thinking about the world this way. But the client base we're selling to, or the go-to-market team and the way they sell, it was so different. It was just so eye-opening to me. And so the lesson is, I don't assume anything anymore; I will talk to clients. When I first joined my current company, I spoke to 50 clients just to hear about their pain. And that, for me, was a really valuable lesson. The other lesson was just about how you talk to go-to-market teams; in this case, I’m referring to revenue and understanding them and their pain points. Their pain points are so fundamental, you really need to understand how they sell, the friction they run into, the challenges, and what they're selling. In my opinion, from a product perspective, you can build the most wonderful product you can, but if they can't sell it right or they're not convinced they can sell it, then it often doesn't result in success. And so those were two painful lessons to learn, but ones that I've taken with me in pretty much every other role I've had.

Matthew:

Yeah, I think that's extremely well put. And it goes also to your point about PMs and their responsibilities. PMs are responsible for the delivery of software, but that's not all they're responsible for. If there isn't a channel strategy or clear reasons to buy, it's not going to work. And I agree, I thought that was a really interesting part of AT&T. I did a ton of sales calls as well and flew out to different regions and went to the call centers, and I spent a ton of time with channel marketing on how we could make the sales super obvious and easy. If you do that, you can move a huge amount of product, but if you don't think about the buyer properly and what they really care about, it doesn't matter how cool your product is. This is a theme that comes up a lot when I talk to other product leaders, which is, “I was really good at building widgets, but I didn't know how to sell them.” I would always tell people that sales is not a dirty word and every product manager should be figuring out how to go sell stuff.

Greg:

Yeah, absolutely true.

Matthew:  

Maybe along those lines, if someone who is just starting their career comes to you and is like, “Hey, I want to get into product.” What kind of advice do you give them about how to get started on the path?

Greg:

Normally, I first like to ask them about what it is that they're trying to achieve and what excites them about it. Because sometimes they'll say, it just seems like a cool role, and it's an exploratory question. And they'll want to ask me about what it's like to be a product manager or running a product team. I normally can tell what kind of advice they are looking for based on their why; no matter what, I'll hopefully give some helpful advice. 

If they are looking to break into product and let's say, for example, they want support from me. Where can I help? Well, I can help in my network. I can tell them, go talk to this person, and maybe they will have a role for you.

But then I also tell them that the people they’ll be speaking to, their time is really valuable, right? In general, they're going to be nice because I wouldn't send someone their way if they weren't. But they're also going to be looking at how you can help them at the end of the day. They may not say it, but they're going to think of it like, I'm taking the time to speak with you; what's in it for me? 

Sometimes, people just want to pay it forward and help you out because they were once in your shoes. But other times, they are hoping to get something out of talking to you, and your job is to sell yourself or show them why they should be speaking to you. And ideally, you turn that into some type of engagement, like an interview or a job offer. 

So what I say is, go into those calls with ideas or suggestions; own the meeting. For example, if you want to go have an intro conversation with a Chief Product Officer at a specific company, don't just go in with basics like, “Tell me about your job, or tell me about the company, what are your pain points?” Those are helpful for context, but come in with a position that’s your own. You could say something like, “I was on your website; I signed up for your product as a consumer or as a business, and I just wanted to share my observations with you. I don't know a lot about your business, only what's publicly available, but this is what I think.” And give them your feedback. You're putting yourself out there, and doing this is exactly what product is; you don't have all the answers. A lot of it's opinion-based. Hopefully, you've got some data, but in the beginning, you don't. 

There are a lot of hypotheses, but part of product's job is to evangelize, convince, and influence. So, this is a great way of doing that at the most senior levels. When I used to interview more regularly and I would have a conversation with a recruiter, or even a CEO, I would have a deck. That deck would usually be a presentation of what I thought worked and didn’t work and observations that I came to based on my own research I did. That shows them how I think and how I influence my approach.

So that’s what I tend to tell people if they are starting out their careers. And you’re not going to know everything. We’ve been around for a little bit, and we don’t know everything - we are still learning every day. But you're going to show you're earnest, you're curious, you're motivated. Some of your other traits might pop out. And if your goal is to get hired, and they don't hire you, stay in touch with them anyway. You can send them a deck later saying, “I was back on your website, and I thought this…” Use this as an opportunity to cultivate a relationship. 

Just remember to try and own a conversation because, at the end of the day, especially when we're talking over Zoom or Teams, people will get distracted really easily, so you need to keep them engaged.

If you can build something, I went to PowerPoint immediately because that's sort of my go-to, but if you're technical and you can build something, even simple, you can show them and say, “I just built something because I had a couple of hours or a couple of days, and I just wanted to share with you” so they can see how you think and work.

Matthew:  

Since you're talking about hiring and you work at a hiring company, what is your favorite question (or set of questions) if any, that you like to use in interviews with PM's?

Greg:

Let me answer this in two ways. The first way is regarding our business; we not only do assessments, but we also enable what we call structured interviews, which basically means that every candidate gets the same question. Ideally, those questions are aligned with the competencies for the role, and the people who are evaluating the candidates are all using the same rubric. 

The reason for doing this is that data has shown, over decades, that structured interviews are twice as predictive as unstructured interviews. An unstructured interview is usually the “Tell me about your experience.” and then they ask three random questions. 

In general, I like to ask specific questions, but it's not as important to me as everyone looking at the same rubric and all the candidates getting the same questions. So, for me, the questions I typically go to are: 

• Give me an example where you had to make a decision with imperfect information. What was that decision? What information did you have, and what was the outcome? 
• Give an example of a time when you didn't have enough resources to do something, and how did you accomplish it? Because you hear people saying that they don’t have the resources every day. 
• Give me an example where you had to influence someone who wasn't bought in initially on what you wanted to do. How did you influence them, and what was the outcome? And give me specifics. 

I'm really focused on specifics because I like to hear if they have numbers. Like, did they take this KPI from X to Y? Do they even have a KPI? How did they have that conversation? So those are the three or four of my questions. 

But what I really like them to do is I'll give them a homework assignment. And sometimes, it'll be very specific. Like, “Go through our signup flow and tell us how you would optimize it, or go to our website, just do your thing and come back.” And I'd like to hear how you would improve whatever you think needs improving. So part of the design of that is you give them a task that's very specific and you see how they do it. Sometimes it's vague because you want to see how they’ll work with a lot of imperfect information or no information and no guidance, which is sometimes, candidly, part of the job. You want to see how they think, how they get excited. Typically, I'll give them a week, or maybe less, to see how they manage under a little bit of time pressure. 

So think of it as a situational job test, and at a minimum, you get something out of it because, hopefully, they've got some good nuggets of ideas. Hopefully, at the maximum, you find someone who's just a great fit for your organization.

Matthew:  

It's super interesting, and we could probably spend much more time talking about hiring than we certainly can now, and obviously, it’s your business now. But, I've come away from all the hiring I'm doing, and, well, A: it's a crapshoot, and B: almost all of my instincts are incorrect, so they're totally unreliable. 

Like, when I go back, and I score myself on who I thought was going to be great or who I hired even though I was skeptical about them, I’ve almost always had it backward. Then I’ve thought about this idea of assessments a lot; how do you test? Especially because I think one of the challenges with product is it means a lot of different things in a lot of different organizations. And I've had experiences where I've hired someone who's had a senior PM title at a company, and they show up, and I'm like, you don't know how to do this job at all, what were you doing over there? You can’t just go on titles and experience, necessarily. 

So, how do you address some of those things, or what are some of your brief thoughts on it?

Greg:

Not to get super deep into our product, but we can measure two things when candidates go through a hiring process. 

First, their cognitive ability; we can look at things like attention to detail and numerical reasoning, for example. I believe that, for certain kinds of roles in product, candidates should be pretty strong numerically, and you can test that really rigorously. 

Secondly, we can test from a personality perspective, and how suited a candidate is for a specific role like product management. We give guidance on a product manager role, for example, and their various traits, like, assertiveness, openness, competitiveness, or whatever it happens to be. You can decide in your organization what's important for you. You get a data-driven output, and you’ll see if the candidate has the cognitive ability and the right personality for the role. 

It gives you a numerical comfort, if you will. It's by no means foolproof; people can have the raw ability and the desire to do the role, but if they don't have the relevant experience, in some cases, it’s a disadvantage because you want them to have the relevant domain expertise. It's less important earlier in their career, though, when you want people who could just have the ability to do things. 

The way I test someone’s expertise is through that homework assignment and very specific questions. I seldom look at a resume; honestly, it's not a very good predictor. And there's going to be unconscious bias. I mean, there's data that shows over and over, if you see Google on a resume, you're going to automatically think a candidate is going to be great. But the reality is, who knows? 

I get very specific with my questions. So for example, I’ll ask, “Give me an example of a time when you had to create an initiative that didn't have any buy-in. What were the KPIs that you had? How did you move them?” If they aren't very crisp, or if they don't have the data or KPI on the tip of their tongue, they don't know about the metrics that they moved; you can tell pretty quickly if they may not be a great fit for the role. 

But again, hiring is hard; there are lots of different variables there. Humans are interesting people, and so, if I can do a plug for what we do, having a little bit of a more data driven approach to hiring can give you that edge. If I could say to someone, you can get a 25% edge in whatever you do, you would do it, right? And that's why I think I'm a big believer of assessments and structured interviewing.

Matthew:  

I should have hired you at my last company. Geez, I missed out. So far, I've exclusively hired people I already know, and that's been my trick at this company.

Greg:

I love doing that because you know what you're getting. But generally, at some point, we have to hire junior people. And it's hard because I just don't have as much interaction with junior people outside of my organization or people who are starting their careers. And so that comes a lot harder.

Matthew:

I agree.

Greg:

This is not to answer any specific question, but the reason I love hiring junior people early in their career is that, in general, if you hire the right person who is open-minded and has a different way of thinking about things, I feel like I learn as much from them as, hopefully, they learn from me. 

One of my favorite meetings we have at the company is a flex topic meeting we have once a week where there is no agenda. We just come in and talk about things you've seen that you really like, products, technologies, whatever it happens to be. And it could be related to our business, but more often than not, it’s not, and it's just kind of general. 

The two things I love about this meeting are that, firstly, I just love listening and watching. I actually no longer attend the meetings, I just watch the recordings because I noticed that when I was there, the tone shifted a little bit, maybe because I was so engaged and started talking too much. 

But secondly, we use Figma to vote on topics and so it's like the wisdom of the crowds and people get to vote on what they want to talk about. I listen to the recordings, and I just love it because I learn so much about so many different disciplines that I wouldn't typically learn about. 

Even in my one-on-ones, like, for example, with our Head of UI/UX. She’ll show me all these different UI/UX patterns and how some work and some don't. She'll show me websites or I'll watch her go through websites and best practices, or how someone else has been using AI or a bunch of different large language models to do really amazing things. And I get excited about these meetings because I learn more in that 45 minutes watching than I probably would reading TechCrunch or the Wall Street Journal.

Matthew:

That's awesome. Yeah, I completely agree with that point about large organizations, where just being in the room as an executive can really change things. And sometimes, you have to avoid meetings very purposefully so that they can be more productive.

Greg:

Yeah. The downside of watching the recording is that I don't want to feel like Big Brother.

Matthew: 

Totally. I mean, it's really interesting with recording, I always joke that if you make it an expectation and it becomes background noise, then usually it's fine because they only get watched with good purpose, but it does make some people very unhappy, for sure, to have everything recorded.

Greg:

I will say, going back to the AT&T experience of client calls, I’ve discovered Gong while at this job. Gong is really focused on go-to-market teams to help build and support better. I started playing around with it, and I realized that it’s gold for product discovery. So, fast forward four years, I get alerts every single day about keywords like, roadmap, feedback, friction. The AI is also good at summarizing and giving me nuggets, and then I also get to listen to the client. Sales and CS are hugely important for giving you feedback, but they can also have recency bias, meaning the last thing they hear can be the most important. Or, some will be very focused on (as they should be) figuring out how to best help a specific client because I can make money off of that client. 

Product people need to step back and be a little bit more realistic. Gong, for me, isn’t perfect because you are only getting your feedback in one way. But it helps with a better understanding of what the client is asking for. So, for example, a rep may tell me that a feature is super important to a client, but when I listen to the call, I can tell that that’s not what the client is saying at all. So when you talk to the client, subsequently, you have a really valuable conversation. I thought I would just mention that because I found it is a gold mine.

Matthew:

I agree. We used a similar tool at my last company, Apto, called Wingman, which had the same basic premise. I owned both product and sales, which was an interesting position to be in. But it was interesting because we hired a lot of new people in product as we were building out the organization, and I needed them to learn a lot about our customers and our interactions. But it kind of got to the point where it became a drag on sales to have to invite all these product people to all of the calls. We had to explain to our customers why they were there, and they’d ask all of these questions, so it actually became amazing to just have the calls recorded. And we were all remote, so I could just give them a set of calls I wanted them to listen to. The instinct was great, to want to jump on calls and talk to customers, but there was actually an expense to the salesperson at a certain point. 

Greg: 

That's right.

Matthew:

To your point about AT&T, I remember I flew to Portland one time for two days and just drove around with the sales guys to get an understanding. And, I mean, that's a whole trip, you couldn't just do that every month, right? So it was very hard to keep up with what was going on in the market.

Greg:

Yeah, great point.

Matthew:

All right, well, I know we've gone for a while, so I just have two more questions for you. The first one I’ve been asking everyone is, how do you really build stuff day-to-day? What methodologies or tools do you use? Are you guys pure Agile, doing your own thing, or using Jira or something new and cool? I just think it’s interesting because I’m a tools nerd, so that’s why I ask.

Greg:

It's a “depends” answer. Let me start from the beginning. We're a very transparent company in terms of ideas that come in and where those ideas stand, meaning will they become a feature, and what is the chance of them becoming a feature? We use Aha! to gather feedback, mainly from the go-to-market team, who will give us the Gong link, the revenue of the client, and the feature.

So every day, I get an alert when the requests come in, and I’ll assign the request to a product manager. The goal for the product manager is to make a very quick decision about whether to consider building the feature or not. They can’t let it hang in purgatory; that's the worst. Our sales leader says it's like a maybe. And “maybe” is the worst in sales; you want to have a yes or a no. It’s very important for the go-to-market team to give the PM all of the background information because if that doesn’t happen, the product person isn’t going to listen to it. So we’ve been trying to train our go-to-market team to give us enough information we can act on. It’s only a one-minute process which helps us a ton. 

Once we have that, then the product person has to do the work to see if the request is something we’re going to focus on. Some of that is going to be their opinion, like, “I don’t think this is going to add a lot of value,” or maybe there won’t be a lot of revenue to it. This is where the go-to-market team has to make their case initially; why they think we should focus on it. But the product person needs to make some quick decisions on if it’s something they want to pursue. If they do, then they’ll probably do some discovery, meaning they’ll talk to some clients. 

My view is—and sometimes I think there’s some disagreement on this—is that I think you can talk to clients, and you need to ask the question, “Will you pay for this?” I don’t know if it’s a really great question, because sometimes clients will say yes, only because they don’t want to say no. But when the time comes for them to pay, they actually don’t. But what the product person is trying to figure out is if they believe that the feature would actually be something unique that we can sell. 

So, everything is in Aha!. Once they have conviction that it’s something we should build, we put a product value score on it. This is based on the revenue opportunity and the cost it would take us to build it. Then, every two weeks, we look at our roadmap and try to reprioritize things so we don’t do things that don’t meet a certain threshold, and the things that do meet a certain threshold, we do. And again, everything is in Aha! So it’s meant to be lightweight, but it gives us some idea of where we should be investing our energy. 

Now, I’d say for most of the product team, that’s the way it works. For newer initiatives, we’ve just gotten into a new business for us, which required a startup within a startup. So, we constructed a team of one product manager, one designer, and two engineers. The way they build is completely different. Basically, they’re just talking to customers, talking internally, then deciding if they want to build something. There are requirements, and there’s design, but there’s no prioritization process because the team is talking all the time and they’ve decided to make it truer to Agile. 

So, to answer your question a little more directly, we’re not quite waterfall at all. We release pretty regularly, and we try to make it as nimble as possible, but we’re not quite Agile if that makes sense. So we try and find a happy medium for most of our business. But, for the other part of our business, we’ve been quite agile and I’ve been ecstatic by the results of that team in terms of them delivering product.

And then from a tooling perspective, it's really Aha! And Jira. Our design team uses Figma quite a bit. There are some other edgy tools that we use, but there's nothing, there's no tool that I think you haven't heard of before.

Matthew:

Yeah, well, it is interesting to hear what people are using because it does vary. There are so many out there. I've heard some interesting ones so far.

Greg:

I know our design team loves Figma, and it has great plugins for developers too, so they're using it as well.

Matthew:

Figma is everywhere, they totally took over. It's really, really amazing. 

So my last question may be the AI question that we talked about earlier, but how has product changed over your career, and where do you think it's going?

Greg:  

I can talk about how we use AI. Personally, I love using it, and there are two main ways that we’ve used it; one internally, and one in our product suite. 

Internally, I would say about 20% of my team’s time is focused on inbound questions from the go-to-market team, which are usually from them getting inbounds from clients or prospects about feature functionality and capabilities like, “Can you do this?” I would say that many of them are repetitive, meaning that we’ve had those questions before, whether that be six months ago or a year ago. What happens is, that a go-to-market team member will post it and tag the PM who is in charge of that request. What we’ve done, is we’ve leveraged one of the large language models to build our own instance to support the go-to-market team; we call it Criteria Chat. 

We publish our roadmap internally to the go-to-market team on a quarterly basis about a year out so they know what's going to be coming, they can speak to it, and they're knowledgeable on our roadmap. You can ask Criteria chat questions like, “When is this feature going to be available”, but it can actually help you. You can prompt it to say “I have a customer or prospect who objected to this. How would you answer this, or how would you respond?” And now you start getting really good responses, either on the fly or in email, that are tailored to our company’s delivery and what we do. And it’s specific because we’ve trained, or fed, our large language model with a lot of our internal data. So that’s one way we’ve been using it that has been really helpful for us, and it’s just going to get better.  

The other way we use it is from a product perspective. We’re in a fairly regulated industry with hiring rules and regulations, and with AI in particular, and there are a lot of laws in existence, especially coming up in New York City in particular. So the way we’ve used AI is, it has to actually solve a business problem; we’re not just going to use AI just to say we do AI, it has to allow us to do something better. With that mindset, we are able to build products that we simply couldn’t have done without using AI. 

For example, we recently launched an English Language proficiency assessment, which looks at how well someone can speak, write, read, and understand English. Some of those things, like reading, are easy to do; you can do multiple-choice, for example. For listening, you can have them listen to an audio and then give them multiple choices. But writing has been really hard because historically, you would have to outsource those writing components to a third party that would look at the writing samples. It would take 24 to 48 hours if you were lucky, and it was really expensive.

We were able to build something using our organizational psychology expertise, and it took a long time. But what we can do with our model, is that it not only looks at what an individual wrote grammatically, sentence structure, use of words, et cetera. But it actually has the context and is context-aware. So they see a picture, and they have to write about it. So, if I wrote a perfect paragraph or a couple paragraphs, but had nothing to do with the image, the system would score it poorly. And so we've had to train this model, and the clients get an instantaneous result. So we've been able to do something that is much faster, and much cheaper that we wouldn't have been able to do without leveraging AI.

Matthew:

Does that picture, in part, help you avoid the cheating problem? This was just in the journal the other day, about AI buffed-up resumes, right?

Greg:

I read that article as well, and it was interesting because I think that a resume has some value, but it’s not great for hiring. The data has shown, I mean it was bad before, but now it’s even worse because you can use ChatGPT or whatever model to say, “I’m applying for this job, give me the perfect resume.” But the reality is, is that the resume is probably inaccurate or at least embellished. It doesn’t tell you, “Matthew is a great fit personality-wise and cognitively.” So the reason we show the image is for exactly that reason. We don’t want someone to just copy and paste. We have measures that don’t allow them to do that anyway, but we don’t want someone to just write some random stuff. So the idea of the picture is to make sure they understand what they’re seeing and can write about it in a way that is contextually relevant. We can talk about AI and cheating another time, though. It’s a bit of a hot topic at the moment. 

Matthew:

Yeah, one of my old interns from way back when I did Wallaby started an AI cheating detection company actually. But it is funny, we use a bunch of it internally. We use ChatGPT to help with stuff, I think we use it very in the assistive model. We have this whole funny thing where we're very honest with each other, because it's not that anyone's cheating by using it. But we’ll be like, was this ChatGPT? And sometimes you're like, yeah, absolutely. And it loves certain words, so sometimes you can kind of tell. The other day someone said to me, “This sounds like ChatGPT” I was like, “Hey that’s offensive.” And they said, “Well, you used the word delve.” And I said, “You're right, it does love that word. But so do I!”

Greg: 

Was it ChatGPT?

Matthew:

No, it wasn't. I had actually written it. That's why I was offended because I would totally own up to it. But I did actually write this. It was very funny. But all those tools like Grammarly, which I’ve been a user for years, but I’ll use it to do my own first pass for editing because it will see stuff you missed or you can apply style guide rules, and I think all of that stuff is super helpful. 

Greg:

Yeah, I find if you can find a good tool, continue to use it. I’m in this networking group, and we go over tools we use all the time. There’s this one woman who spends like $600 a year on all these random tools, and I just think, I would rather just wait for Microsoft to come out with AI, because those are three tools I know they should be doing, and I would, candidly, rather just wait a year for it to come out rather than switching tools all the time. Which I know is odd for me to say, because I figure that as a product person, we should be an early adopter of those things. But if you go back to Jeffrey Moore crossing the chasm, most people are not early adopters, they’re kind of in the late adopting stage.

Matthew:

Yeah, and our industry is so tool-driven. It’s interesting because, I still do a little bit of writing random codes, like little scripts to do things, and I’m a terrible engineer which is why I’m not an engineer. But it’s super great for debugging. I can give it something and say “Hey, this doesn’t work” and it will tell me that I was missing a semicolon. Or, rather than spending an hour searching Stack Overflow, it can do that for me. 

It’s funny because there’s really a trade-off, right? Especially in our industry, where we’re relatively highly paid people. If your average salary is $180,000 a year as a product manager, and you have the option to spend $200 on a tool that will make you way more productive, it’s easy to think, “Why the hell not?” But next thing you know, your tools budget is enormous. It’s a challenge, but individually, it feels like a really smart move. Like, if I can save them 2 hours a week for $200, what a deal. 

Thank you so much for your time today, Greg, it’s been great chatting with you.

Greg:

Thank you for the opportunity! Hopefully, it wasn't too meandering.

Matthew:

No, I loved it. It was super interesting. And it's fun to talk to people and ask them questions that you don't normally ask people because you already know each other. I’m having a blast doing this, so thank you!

About ProductFTW

ProductFTW is a biweekly newsletter about product management, with a focus on real-life experiences in startups. We want to help product leaders be successful by giving realistic approaches that aren’t for giant tech companies. We know you don’t have a full-time product designer on each team. We know your software probably hasn’t been used by millions of people worldwide–yet. We’re here to bridge the content gap from building your product and team to scaling it.

Part of the Product Talks series — interviews with experienced product managers across HopSkipDrive, Smartsheet, The Zebra, perigon°, ClosedLoop AI, and Totavi.