AI Pilots: 95% Flop. 5% Don’t. Glass Half Empty or Half Full?

Welcome to the prompt. There's been a lot of numbers flying around this week, and I'm glad those data people were finally using our data. MIT says 95% of AI pilots are flopping. Gartner's latest hype cycle has gen AI in the trough of disillusionment. I wasn't surprised. The Harvard Business Review is saying maybe it's time to slow down on AI investments and get your foundation right. Not a shocker. What a concept, right? But so now here's the kicker. I read the 95% and my immediate reaction was like, aha! That's not exactly news from my experience. Like having lived through the.com boom and bust and seeing the explosion of excitement, and then how that shrank down to a couple key players. I always had it in my mind that this AI boom was in that same cycle. But my question was, all right with that 95%. Great. What's up with the 5%? Because there's obviously some value in wins there. What are your thoughts? It's not. Yes it is news. So I don't think it's fair to say it's it's not news. More specifically it's not surprising. I think this is more commentary of rather consistently the nature of how we'll say proof of concepts pox and the nature of how they're facilitated and also how you're qualifying Rois. All of this is broken. To me, this is more what's being highlighted rather than AI being a feasible option in terms of missed expectations of what it's supposed to do for you. That's not a surprise. I would rather question that. And to your point, the 5% someone clearly is doing it. I think the bigger picture is that AI is actively being used, and I think in at least one of those articles, there's a call out of shadow AI, which is that someone is using AI. Personally, you don't quite know what they're using it for, but they are using some variation of ChatGPT, Gemini or a helper in their day to day. There's no question of that. I think that's what's lacking, uh, in terms of metrics to match up of attempted corporate adoption through the proof of concept model as compared to the rampant shadow AI adoption that's happening everywhere. My immediate reaction was AI is the scapegoat for our broken processes and our laziness to evaluate whether or not activities that our teams are doing actually tie back to a strategic value driver. Absolutely. And this has been the challenge that's been going on well before AI, right? I mean, it's just the most current, uh, goal topic excitement in. Whatever metaphor you want to use. AI is not magic. By virtue of just using it. Your PNL isn't going to sprinkle the AI pixie dust and it just magically improves. I mean, there are specific use cases where it has a material change in how you do business. However, I would take a step back to understand probably what are those 5% doing and how can we explain or extrapolate that to identify what's going to work best and really go from there? Well, I started to dig into the 5% to say, like, what was different for those 5% versus the big headline grabbing 95%. And one of the things that struck me was two thirds of those initiatives were driven by vendor partner relationships. Well, and that's pretty consistent. I don't know if you've read Soul Rashid's book on AI and her experiences. She's had success implementing AI and speaks very candidly about her success and and failure along the way, and that the metrics basically match up to what she's memorializes in her book. But that is that if you're starting off, you should use something off the shelf first and start there. It is baked. When using a partner that already has a finished product, you're not asking for a lot of configuration or modifications just so you can use it, which is usually part of a POC of saying, cool, uh, we have a concept, but how does that apply to us? And there's nothing to feel ashamed about by getting something off the shelf. That too is also using AI. I think there's a lot of purists who say, you know, well, if I buy a product and use it, well, that's not really using AI. Yeah, and that's interesting. I keep going back to the dotcom days. Right. We had to learn how to create a website. We had to learn HTML, JavaScript, and there weren't a whole lot of low code. No code tools out there at the time. That wasn't a vernacular at the time at all. Exactly. Haha, exactly. Especially coming out with a degree that was not a thing that we talked about or really imagined would be available to us. We had to learn and actually build it from scratch. And now you go to GoDaddy or Wix. Com and they've got all sorts of templates and starting points available for you. So if I use one of those existing templates or website starter kits, am I not building a website? You totally are right. I mean, and sure, there are limitations of what you can or can't do. However, I think you get to the 80 over 20 concept of Pareto where you know if you get that 80% done, you're doing the job and achieving the goal. You don't have to hit 100% of all the nuance and detail to deploy a website that perhaps you could do without using those web parts, and that's okay. Because isn't the goal of, you know, is this necessary and can it be simplified? And if both answers are yes, then you're getting the job done. And don't haggle over that 20%, right? Because there's a lot of other things to do that are a lot more high value. Isn't that the point? Exactly. I'm trying to think through any tech implementation that I've been through that was 100% perfect because the business value or business needs change, right? Well, that's usually for any of these POCs. Also is like the business moves on and you're not necessarily competing with, uh, one option versus another. It's more, uh, you're competing with the groups, figuring it out for themselves and creating their own solution while you're going through the POC cycle. Right. Which is probably some version of personal AI that they can just do themselves. And I think that there's a meta's attitude of, you know, move fast. Or you could say fail fast, break things. I think that's a whole other subject altogether of the politicization of POCs, and that it's very high stakes of if you are putting yourself forward to advocate for whatever solution is in the POC, there's a lot of incentive to be successful. But if you don't have a quick win, then you're lost and the whole thing dies and then everybody claims, well, gen AI is a complete failure and we can't do it. I think the whole point is you learn that in the POC, it was not a good fit, and maybe take what you learned and roll that into a take too. But I think often the will or the interest and time you get from everyone, the runway that you might have had is just really kind of gone. And that's what's unfortunate, but common. Yeah, I think that's one side of it. I think the other side is have we over indexed on what the success of a POC truly means? I've experienced that organizations believe the POC must 100% align with the business requirements versus the 8020 rule of. It mostly gets us there. I can see the vision and let's learn from it and enhance in the next incremental build. And I'm going to play my governance. And I'll use a Stephen Colbert terminology as well. How many POCs actually focus on the truthiness of the data? Oh good God yes of course. Actually, let's just get some data in there so I can get my POC up and running. How often does that happen? Or that's used as a double edged sword in the sense of like it's used as a reason to call out failure if you don't achieve certain lofty goals for data that hasn't been touched before. Often it's like, well, if we cannot get this to 80% after working on it for 90 days, like that's, I think setting expectations so high to where there's no other way for you to go other than to fail rather than acknowledge, like we learned a lot. About our data and probably how bad it is. And then as a result of that, we can do a lot of things with our discovery. I feel like we've played a fun little blame game so far in terms of the problem. Is not AI, right? AI is the scapegoat. I think the challenge in truly recognizing ROI is one this desire, I'll call it, to build versus buy. Instead of seeing what's out in the market and partnering with a vendor that's been there, done that, and has actually proven value to having unrealistic expectations of what a POC will deliver versus looking at it in terms of is this directionally correct? And three, not investing enough upfront in actually the truthiness of the data versus just using what's there, because it's the first time teams have started to look at data. And then I think the follow up question is what is it that you can control? What is it that is completely out of your control? And how do you influence what is in your sphere of influence to better. And I feel like that's what the bigger question is for everyone. When I have these conversations with organizations, my first question is, what exactly are you trying to do? What exactly do you think AI is going to bring you? Because we haven't really articulated the problem we're trying to solve and what good looks like. How do you know when you've actually solved the problem? How will you measure success against that? The technology doesn't even come into the conversation for me when we're talking about value. AI is not a strategy. Just like moving to the cloud is not a strategy. That's just a means to an end. That's a loaded statement, lady. You're not wrong. Oh, thank you, thank you. I mean right. Strategy is more. Ah. How are we going to increase revenue or reduce costs, you know, improve our operational efficiency to do the thing faster and how the data helps you get there. Perhaps it's unspoken and that's maybe something we can do better in the data community. But we do need to say that out loud. And for example, the you know, our our strategy is moving our data to the cloud. Why? Because it's going to save us costs so that we're spending less and focusing our attention on helping you with your PNL, for example, like that. I think that's perhaps the bigger conversation that we need to call out is that that's not immediately obvious to anyone outside of the data team. And while it may directly improve your day to day, why does your CFO care? If you're spending a bunch of budget and it takes you six months to get one dashboard, that has to be made really updated? Does it matter given how many times I read in a day the dashboard is dead? No, I don't think it matters. I mean, it's. I say that. And I think you know this too, because it's way too common, right? I think asking why often is a very powerful activity that really helps you prioritize. If you get day to day requests of modifying your report. You know why? Why do I need to modify the report? And is this a good use of my time as opposed to saying, is this report even something that we should be maintaining. But it went through probably the the helpdesk and got prioritized there doing what they were asked to do and closing tickets. And that's how it happens. I think that's the greater problem when you are talking about going back to the original, you know, failure of the AI POCs, the bigger challenge, which is change. You have to think about the incentives of the folks in the organization. Why would they take the time to speak up or change their processes, when really they're just looking to get their jobs done, hit their goals? Like I've heard that story and seen in organizations so many times that clearly it's just a common human behavior that's repeated over and over again. We see a lot of good intentions to do better, but perhaps the data team's interpretation of what better looks like, which does help their ecosystem, may not really make sense or make a big dent in someone who in the finance team, who is submitting the requests, why should they change their process? That to me, how different POCs end up dying on the vine. There are so many folks that have input, and rightly so, but they end up basically declaring said POC a failure because it's not meeting these this pile of needs or success metrics that have been created when you were just getting started. I mean, it's like death by a thousand POCs, right? Girl, you should see on that. I will, I will. At what point do we say enough's enough? Like, enough with the POCs. We've done enough to know what's going to work, what needs a little bit more thought, and what just isn't going to match, what drives value for us? At what point do we say that? That's a really great question. Maybe that's something that we should throw out to everyone to see what they think. I think that's an ongoing question of what will be the tipping point for us to move beyond those 5% outliers being successful, and expand that to being the norm. Does the ROI of those 5% success stories outweigh and overcome the sunk cost of the 95% failure rate? Is that 5% enough to say, like, actually, we see enough benefit that we're able to absorb the 95% that didn't get to fruition because we're actually in a better place than we were before we started. I feel like you could start asking questions of, you know, are those 5% really success stories? If you started asking more questions? Or how did. They get to those numbers. And further, of the 95% that were declared failures. Were they really failures or how did you get to that in the first place? Um, it's really moving target, I think, and that's more where the market will suss out those that are just a lot of great marketing and those that really have something tangible that the market will support and be profitable. And then you do have to have tangibles and strong financials to stay viable. Exactly. You know, we have the dot coms. Then I think about the app economy and how for the longest time it was always there's an app for that. There's an app for that. But there were some really dumb apps. Like do you remember the app where it looked like you were drinking a beer? And as you tilted your phone, the beer level went down? I can't say that I do, but that doesn't mean it didn't happen, right? Right. And then there's another one that I found highly problematic, especially in a post Sandy Hook Columbine world where the app was a shotgun, and you could point it at people that look like it was shooting them. Oh, God. Yeah, that was not great. But hey, there was an app for that. Well, right. And I guess on one hand you can say most successful entrepreneurs have failed a lot. And I think just zooming out more, there's a lot of failures coming from all the apps SaaS, micro app, APIs, whatever way you want to qualify it that I think are mostly wrappers around Llms. However, I think that's a part of the experiment to try and learn if we're in the trough of disillusionment. I also wonder, while there are good success stories, when the expectations for what AI is supposed to do for you and I think that's fair to say, are pretty high, and the reality of it doesn't materialize. You're disappointed, possibly disillusioned. And so I think you might question, well, Maybe we started with expectations way too high, but I do think it's very realistic to say that there are very real, tangible use cases in enterprises where it's making a difference in brick and mortars, not just cloud forward companies like AWS or Microsoft. As much as we're questioning how the success metrics for POCs are set up, I would also question just more the lofty goals of what it's supposed to do for us, perhaps what the discussion of things like singularity and dystopian concepts about how much it's supposed to change our lives. I think it's more reality that we're saying like, well, yeah, that's not happening exactly as it was described. That's not to say it's not happening. And I think that it's happening in a way that is not smacking you in the face, where you can put your finger on being like, that is a material change where my life is not the same because of it, but I think our expectations for the rate. Of change were probably unrealistic to some degree. There's been so much excitement around the potential that I think we expected to see a net benefit overnight, and change takes time. Things are happening very fast, but maybe we were just expecting them to happen faster than was humanly possible, because humans are still part of the equation. Well, right. And going back to the original question, it's not a question of what AI can or can't do. It's a question of what the humans can do to exploit the benefits of it. And that's really what's what's the what's the slowdown. I think that's the greater call out. Right. So then what do we do? I mean, HBR, there's articles saying maybe we need to just take a pause and breathe and think about what to do. I don't think that organizations are prepared to completely stop, because there's always the FOMO effect. The I can't slow down because my competitor is speeding up. What I do think is realistic is to maybe pump the brakes a bit and put some processes, some governance, dare I say, in place to actually determine or understand how AI efforts align back to organizational value drivers and shut down efforts where you cannot clearly tie back to a strategic objective of the organization. That's really what had me buzzing this week. I was just like, vanity metrics. Here we go. Hey! Well, that's very clickbait too. I guess that's the call out that everyone is really caught up in that, considering how much penetration that's gotten, where that's all anyone's talking about. Yeah, exactly. Including us. That was the conversation he did have this week to just get it out on the table. That yeah, I see the 95%. But what's really underneath that you know let's talk about that first. All right Karen appreciate the conversation as always. And we'll do this again real soon. Until next time.