Cassie Kozyrkov on How AI Can Be a Leadership Partner

MOLLY WOOD: Today I’m talking with statistician and decision-making expert Cassie Kozyrkov. She advises companies on how to approach decision making and AI strategy. She is also the founder of a discipline called decision intelligence, which is the name of her popular newsletter. Cassie joins us to share her insights on decision making, how people often get it wrong, and how to understand the value AI can bring to an organization. And now my conversation with Cassie. Thanks so much for being here.  

CASSIE KOZYRKOV: I’m so excited to be here, Molly.  

MOLLY WOOD: Cassie, you’re credited with founding a field, which all by itself is amazing, and that field is called decision intelligence. Could you give us, broadly, a definition of what that means?   

CASSIE KOZYRKOV: Decision intelligence is the discipline of turning information to better action—any scale, any setting. So what it does is it annihilates the silos between the decision disciplines, and perspectives on decision making come from the classic ones like psychology and other social sciences, managerial sciences, and, of course, the data and mathematical sciences. Decision intelligence is a kind of end-to-end approach, and if we think about why we might need it—if you have technology that makes the actual execution of something relatively effortless, you might say, hey, machine, do this thing for me, and you get an answer like that. Two questions for you. Did you ask the right thing? And do you know what you’re looking at when you get an answer? We are beginning to speak more and more powerful words to machines. Are we aware of the consequences of what we’re saying, and are we aware of what we’re actually saying? That’s the decision intelligence approach.  

MOLLY WOOD: So, one of the things that you’ve written that I found completely compelling is this Harvard Business Review article saying many decision makers think they’re being data-driven. You brought up this idea of the gap sometimes between data and intelligence: they think they’re being data-driven when they look at a number, when they form an opinion and execute their decision. Unfortunately, such a decision will be data-inspired, at best. What do you mean by that?   

CASSIE KOZYRKOV: You can look at that as data-decorated—data as a decoration or as something that makes us feel better about what we’re going to do anyway. We don’t always realize when we’re doing this. We can be completely convinced that we’re integrating information from the real world, but all we’re doing is using it so much more like a mood board and less as a recipe plan or blueprint for decision making. And this really jumps into the concept of confirmation bias. The way that we see information changes based on what we would like to be true or what we believe already. If you have already made a decision, the way that you’re going to look at a number, a fact, is going to be very, very different from, if you haven’t made the decision yet and you intend to use that number to actually drive your decision. When it comes to confirmation bias, there’s a very simple antidote to it—the discipline of pre-committing to how you’re going to use information to drive your decision. In other words, the structure for your decision has to be there before the data. That’s kind of like saying, I’m going to set my goalposts before I actually kick the ball and see where it lands. Not afterwards, where I could just put the goalposts around the ball and say, yay, I scored. And that pre-commitment process that happens way before the data, that is something that leaders, decision makers have to be responsible for. So it’s really about that gap bridging and fluently speaking both languages: the language of engineering and data, and the language of leadership and decision making.  

MOLLY WOOD: So then we introduce this big, endless opportunity for data, and I believe you have referred to it as endless right answers. How do we think about decision intelligence in the age of generative AI?   

CASSIE KOZYRKOV: Right. So—   

MOLLY WOOD: Now things get really messy... [laughter]  

CASSIE KOZYRKOV: Yeah. Now things do get very messy. So there’s a lot of work done by psychologists where they would show things like, people find it a lot easier to choose between two options. Would you like to have this flavor of gem or that flavor of gem than, say, 16 options? Having more choice doesn’t necessarily make things better and easier. Sometimes having a structure that limits your options can be healthier because we just can’t deal with optimization as humans on that scale. And if 16 is too many for us, what do we do when it’s 16,000 or 16 million? So the thing about generative AI is that it will generate as many as you like, as many as you can afford, compute-wise. What does it mean to have a good customer service interaction where a chatbot is interacting with your customer? What does it mean to draft a good email? What is good in this situation? If you haven’t really thought about that and you start, maybe, going down a rabbit hole, you have to learn how to cut it off and limit your own options and get to where you’re trying to go faster, because if you don’t, here you are looking at potentially infinite good-ish possibilities. How do you choose in those situations? One of the hardest types of choices that you can face is the—good problem to have—situations where the distance between two options is actually quite small. So, a classic example here is going on vacation. And if I asked you whether you would prefer to go to vacation in, let’s say, your local landfill or Paris, right. [laughter] I mean, that’s a fairly easy vacation choice. But let’s say it’s Paris versus another place that you feel quite similarly about—let’s say Paris and Madrid. They’re both great. So how do I then choose between these two if they are so similar, and how do I find what would break that tie? I may find myself overspending effort on that minuscule distance between these two pretty good options. With generative AI as well, you now start to get this proliferation of fairly good answers, and the distance between them might be really small. And then how do you figure out how to inform a choice between all those options? How you would do that would be similar to how you would break a tie between Madrid and Paris. There’s not one right answer.   

MOLLY WOOD: But it is interesting because it points to what you were saying, which is that you sort of have to go to the end. You have to go to, even if it’s individual, what you value the most. So for example, I might prefer croissants to tapas, and therefore I can optimize backwards. But, and what I like about what you’re saying is that, there’s really still a human, there’s really still goal setting. There’s not this sort of blind following of whatever generative AI is telling you.  

CASSIE KOZYRKOV: Hundred percent. Hundred percent. Connecting with your personal reason, your why, is how you break these ties. What AI can help you with is generating a bunch of options for you. There’s this tendency to maybe skip a step when we see something that calls itself AI, or it’s computer-y, there’s math somewhere around it, there’s data somewhere around it, that people think that now what we get is access to objectivity, access to the only possible answer. It still comes back down to who is driving, what is important to them, and how they create the criteria for what happens next. So how do we set everything up so that at the end, the technology, the tools, the outputs really do serve the people who are behind all of it? A piece of advice that I have for absolutely everybody is, find the practice, the discipline, of seeing the humans in any technological system. There are so many trade-offs and choices that happen before we get to the mathematical stuff, and understanding that there are people making those trade-offs—we hope that they’re doing it wisely—is maybe the best skill that we can have as decision makers in an increasingly complex and technological world.  

MOLLY WOOD: This sort of leads us naturally, then, into what you have called the generative AI value gap—the difference between individuals finding enormous value in generative AI and organizations struggling to measure that value. How do they get across that gap?   

CASSIE KOZYRKOV: When you get a new tool or a new toy, it is enough that it feels useful to you, quite often. I feel like I go faster at writing email if I have generative AI do some pre-drafts for me. That feels good. But if we were actually to dig in and say, well, how do you measure, do you actually know how much of a speed-up you’re getting? And now you want to implement this tool at some kind of scale in an organization. Scale demands to be measured. The first question is, okay, what’s the ROI here? And it’s going to be fairly straightforward to figure out what it costs to put it in, this much headcount, this much processing power, this much technical debt. Then what do we get out of it? These technologies don’t come with that concept built into them. The leader has to take responsibilities down and say, This is why we’re doing it. This is what it means for the system as a whole to succeed. This is the cutoff where the answers are good or better. I want to create a system that generates social media copy automatically, let’s say. Well, then, how do I determine whether one piece of copy is better than another piece? How much better? And that articulation is something that a lot of people find very difficult.   

MOLLY WOOD: What this is raising for me is the other kind of interesting question about making decisions with AI, being able to use this potential thought partner to break out of some of those patterns, to say, I know that I could be working toward a better outcome that I have not yet determined, because I’m still only human—even if I’m a really good leader and I know I need one. Imagine, then, how could we engage with AI thought partners to help us think differently, get to a different end goal before we start putting in all the data?   

CASSIE KOZYRKOV: One of your procedures that you would want to do as you’re structuring a decision is to think about what you haven’t thought of. One approach to doing that is analytics. You can also go to an AI brainstorming partner and say, What haven’t I thought of? This is how I’m structuring my decision. What am I ignoring? What hasn’t occurred to me? What assumptions might I be making that I don’t even realize I’m making? AI will keep pushing you. You say, give me 50 more. It will try. A lot of them will be garbage, but you might go, huh, that 47th one, I really didn’t think of that. Maybe that’s much more important than what I’m focusing on.   

MOLLY WOOD: And that feels magical because it takes a little bit of that pressure off. Like, yes, you still have to lead, but you maybe have a partner in getting you to the leadership part.  

CASSIE KOZYRKOV: Right, right.   

MOLLY WOOD: With that in mind, are there problems that come to mind for you that we might be able to solve that we would’ve had a hard time solving before?   

CASSIE KOZYRKOV: Drug discovery is a great one, right? That timeline is shortening because you now have this ability of a machine that really supplements two things that we used to think of as very uniquely human. One of these was memory, and the kind of memory that can hold abstract concepts and layer Lego blocks of abstractions in a way that we haven’t really found evidence of animals doing. So, what a fantastic property. Data is really good for memory, data is really good for attention. Machines, they’re pretty cool memory prosthesis. The other thing that’s quite special about us is language, and that we are able to transmit information with language. AI is really participating in both of these topics, suddenly giving us access to vast amounts of shared memories. And then with language, the reason that generative AI, I think, is really wowing people is that, before, if you wanted to talk to a computer, you would have to learn a language—your C++, your Python, whatever it is. Whereas now, you have this democratization where you can speak your own language and have a shot at the machine being able to do things for you. The trouble with our own language, though, is that it is not precise. Mathematics is a great way to say very little, very precisely. So that gives you a lot of control. Poetry is a great way to say a lot, very imprecisely. Now we can express ourselves poetically and be a little bit surprised by what we get. Now think about that element of being surprised by what we get, and put it in the context of generating ideas, of brainstorming. How wonderful. And then put it in the context of something mission-critical, where the system has to work. How do you put guardrails and safety nets on what is essentially a kind of proto-genie, and the prompt is a kind of proto-wish. And are we sure that we are able to express ourselves properly, particularly when we’re going to scale that wish up? How do we think about what we actually mean? How do we do it well? That prompt is more like a wish. You might make a terrible prompt and get something that, you know, you definitely don’t deserve, based on the effort you put in—  

MOLLY WOOD: Your poor construction. [laughter] 

CASSIE KOZYRKOV: Yeah, right, exactly. Your poor construction. You didn’t know what you were asking for, and somehow you got something good back. That’s possible. You might also have done a really great job of asking, gotten something garbage back, this surprise factor. As we put in this surprise factor and we start to scale it up beyond the individual user, we start to take it into the organization. What does it mean to have a system that has this greater propensity for surprise, uncertainty, for complexity, for chaos?   

MOLLY WOOD: If you wouldn’t mind sharing, how are you using AI in your work and, ideally, your personal life?   

CASSIE KOZYRKOV: What AI needs to do for me is make me more effective. Does it make me better? Does it augment me, does it help me do something faster, smarter, or in a more inspired way than before? So of course, I look at things I work on and find all the drudgery—a lot of it is translation. So language translation is what we’re actually talking about quite often when we’re thinking about these generative AI systems. Language is the interface to human collaboration. Naturally attempting to express myself is more convenient, and so I can get my wishes translated. I can get them translated into code, I can get them translated into action. So if I am really dreading writing a particular kind of email, I might ask for a draft—edit the email a bit and put it into my voice. If you think translation is like English to Spanish and back, that’s too narrow. Translation also includes taking bullet points and translating them into a fleshed-out email, and taking a fleshed-out email and translating it back into bullet points, which was a use case that I found that a lot of people were doing with generative AI, which tells you a lot about the human condition. What I don’t use AI for is thinking on my behalf. A classic thing—so my dad absolutely does this, or at least pretends to do this. So he will be looking at a menu, he will be stuck between two fairly good options. Maybe it’s the Caesar salad that he likes. Maybe it’s the steak. Those to him are quite similar, as it turns out. And then he will take out a coin. He will flip that coin and that will tie-break for him. He knows there that he’s fairly indifferent. He’s thought about it, and that’s why I say it’s like something he pretends to do, to say the coin makes the decision. It is very possible to use AI in this way. And in the same way that I don’t recommend letting a coin run your life, I also don’t recommend having what is also a very similar process. An AI system is composed of much smaller Lego blocks, which if you take them down to their atoms will look a lot like coin tosses. It’s a probability engine. You don’t want that running your life either. So you, the human, you have to stay in control. You have to say, this is how I’m setting things up. This is what’s important to me. This is what I’m choosing not to pay attention to. This is what I’m choosing to pay attention to. You are the author of meaning as a human. You choose what’s important, and then you use AI afterwards.   

MOLLY WOOD: Knowing that this is how you’re going to approach this question, I feel like it’s going to be an extra interesting answer to the question we always ask, which is, if you’re fast-forwarding three to five years, what do you—not necessarily think—what do you predict may be some of the most important changes in the way we work, or the biggest changes.  

CASSIE KOZYRKOV: So I have this concept of thunking versus thinking. Thinking is exactly what it sounds like. Thunking is where, it’s like the sound of a dull brick—thunk, thunk, thunk. It’s where, if we’re honest with ourselves, we are executing on something that we’ve already decided how we’re going to approach, and now we’re a little bit checked out. It’s the difference between a conversation where you are engaged and a conversation where you’ve already pre-planned what you’re going to say, you’re not listening anymore. The thing with AI and other kinds of machine automations is that they will automate more and more and more of the thunking, and every job has a thunking component, where you’ve figured out what to do. What’s going to be interesting, challenging, is how we approach managing thinking as we take out a lot of the thunking, because I’ll tell you what not to do. What not to do is to say, great, I had this employee and I have automated out so much thunking that I’ve taken out seven hours out of a nine-hour workday. Great. Let me compress that and make them do thinking for two hours. And now they come to work at 9 a.m., they leave at 11 a.m., and they’re just going to do pure thinking. If anything’s wishful thinking it’s that. That is not how we optimize for the creative and engaged moments. I’m not sure that we know how to optimize for them. What we’ve been measuring this entire time is the most repetitive, the most digitized, and the least creative aspects of work. That’s what we know how to measure, because they’re easy to measure. How do you define creativity so you can measure it? That’s hard. But you can measure the amount of time someone sat in their chair, words per minute that they typed, the number of customers that they served. These are all the things that AI sees. What AI doesn’t see is the creative bit. So then if you’re going to take away the thunking, what are you going to do to make sure that the thinking still happens well? Okay, I am in charge of myself as my own boss and CEO of my company. So no one tells me how to spend my thinking, thunking, creative, not creative time. Every time that I find a way to automate some of the thunking, which I do quite aggressively, I try to remove as much of it as possible, I find that I still need to put something like that back into my schedule so that I have the creative thoughts. Now, it’s nice that I can choose between, you know, I find data entry quite soothing, so sometimes I’ll enter data into a spreadsheet that I don’t even need to enter, just for the soothing relaxation that I think a lot of people seek when they play games on their phone. That when you distract yourself from pure thinking, you may be more likely to be creative, you may find that you actually need those things. What we’ll see is that work is trying to push those things out, because that’s what we used to optimize for. We used to optimize for those things. Now we will find how to really optimize for those things, and then we’ll have empty space. Workers will have empty space. How will leadership deal with that empty space, and will they deal with it in a way that really optimizes for creative ideas, healthy cultures, and productive work environments? That’s going to be a massive challenge, and in three to five years we will have to solve this challenge. And so that’s something we’d better start with today. 

MOLLY WOOD: Thank you again to Cassie Kozyrkov, AI and decision intelligence expert. You can find her Substack at decision.substack.com. Thank you so much for the time.  

CASSIE KOZYRKOV: Thank you so much for having me.   

MOLLY WOOD: Thank you all for joining us, and keep checking your feeds. We have more fascinating guests on the way with actionable insights that can help leaders develop an AI-first mindset, reimagine their business for a new era of work, and maximize the ROI of AI. If you’ve got a question or a comment, please drop us an email at worklab@microsoft.com, and check out Microsoft’s Work Trend Indexes and the WorkLab digital publication, where you’ll find all our episodes, along with thoughtful stories that explore how business leaders are thriving in today’s new world of work. You can find all of that at microsoft.com/worklab. As for this podcast, please, if you don’t mind, rate us, review us, and follow us wherever you listen. It helps us out a ton. The WorkLab podcast is a place for experts to share their insights and opinions. As students of the future of work, Microsoft values inputs from a diverse set of voices. That said, the opinions and findings of our guests are their own, and they may not necessarily reflect Microsoft’s own research or positions. WorkLab is produced by Microsoft with Godfrey Dadich Partners and Reasonable Volume. I’m your host, Molly Wood. Sharon Kallander and Matthew Duncan produced this podcast. Jessica Voelker is the WorkLab editor.