Dave Heywood, MD & Growth Partner at OX7 Partners (00:00)
It’s a pretty well worn idea that AI makes us more productive. But what if we just think it is?

There a fascinating study recently involving some very experienced software developers using some AI tools to code. Before they started, they thought AI would make them around 24% faster. Afterwards, they still thought it made them about 20% faster. The reality? they were 19% slower.

perhaps it’s time we started asking.

Whether AI is actually earning its keep. This is Scale, an OX7 Partners podcast and I’m Dave Heywood

Today I want to get into what happens when we stop being impressed by AI parlor tricks and start actually holding it to account. Because I can’t help feeling we’ve learned remarkably little from previous tech hype cycles.

And in the world of AI, every time a major new model is launched, the whole hype cycle seems to start again. Some new benchmarks, new capabilities, really incredible slick demos, and people trying to squeeze every last token out of their subscriptions. businesses start looking for more things for AI to do. And then eventually the reality catches up.

we come across a whole host of limitations, we find stuff that needs checking and rechecking. The amount of rework the costs getting bigger and bigger,

And things that perhaps shouldn’t have been handed over to AI in the first place. And just as we start getting into the slightly less exciting business of working out whether any of this has actually been worthwhile, a new model comes out, and off we go again.

and as AI starts to become just part of the normal toolkit for a business, I think we’re gonna have to start getting much, much better at separating that capability from value.

Look, there’s no question that AI can make us faster. there’s studies left, right and centre showing significant productivity improvements in customer service, writing, software development and consulting. And I’ve benefited from this myself. I use AI a lot for market research competitor to monitoring.

Data analysis, getting a first pass on something and perhaps repurposing some original content I’ve already created. Now these are things I can do in minutes that would probably have taken me hours, if not days. and that for me is AI really earning its place. but I’ve also come across

Plenty of areas where the opposite is true. Content origination is one. I find deck production a little bit lackluster. I’ve experimented with synthetic personas and found that particularly disappointing. I ask AI to represent a particular type of customer, and it has a tendency to reach towards that average sum of the experience.

The obvious objections, the typical tropes,

But we know real customers aren’t averages. They’re messy and they’re full of contradictions. They make decisions for reasons that don’t always make sense. Two people who look almost identical on paper

can be worlds apart when it comes to behaviour.

so yes, AI can give me something incredibly quickly.

But if I then spent ages rewriting it, correcting it, challenging it, removing the formulaic bits and putting back in all the nuance that it stripped out, I have to ask myself how much time did I actually save?

we’re still in this place where we’re very impressed by how quickly AI can produce something.

But that’s quite different from understanding how quickly we produced the outcome that we wanted.

So AI might save me an hour, but then create forty-five minutes of work for someone else who has to deslopify whatever I’ve created. Well, we haven’t saved the business an hour anywhere. We’ve just shifted some of the work around.

So I think the question we really should be asking is quite a simple one.

Is the final outcome better than it would have been without AI? And what was the true cost of us getting there?

So this is moving beyond just the licence cost or any additional tokens, but this is about checking, correcting the mistakes, the management that goes around that work. The stuff that typically ends up getting pushed somewhere else.

Because if this was a junior colleague, brand new in your business, you would not employ them day one and let them loose across your entire business unchecked just because they can work incredibly quickly. you’d work out what they were good at, you’d understand where they needed a little bit of extra supervision or tuition. You’d check their work, you’d feed back on their work, you’d look at the total cost of employment.

And eventually you’d look at are they actually making a useful contribution? and AI should be held to the exact same standard.

I do think that points to a slightly bigger problem with the way some firms are approaching this. The starting point has very quickly now become what’s our AI strategy?

Whereas in fact it should always be what’s your business strategy? What are you actually trying to achieve?

What’s stopping you? What are the opportunities? What are the constraints? and then we can ask whether AI could help us do something about that. And sometimes that answer will be absolutely one hundred percent yes.

Sometimes another technology will be better. Sometimes putting a human being on it will get us the outcome that we want. And sometimes doing absolutely nothing Might be the sensible answer. Now AI is an incredibly powerful tool, right? But finding jobs for it to do simply because we’ve got it and paid for it isn’t a strategy.

But then this leads me on to another question that I find perhaps even more interesting.

Let’s say deploying AI genuinely makes your team twenty percent more efficient. Brilliant. What are we gonna do with that 20%?

Maybe you need fewer people, and we have to be realistic about that. There are areas where AI almost certainly means we’ll need less headcount, but reducing headcount isn’t the only option available to us. We could produce more, we could improve the quality of what we produce, we could serve customers better, we could open ourselves up to experiment more, we could give people more capacity to do the things very actually.

really, really good at and that clients value,

they’re all valid business decisions. But then we have to ask, what happens when your competition suddenly becomes 20% more efficient too?

We’ve all got access to broadly the same technologies and capabilities. and if we’re both using it in similar ways to make broadly the same things, faster or more efficient.

We come back to that central question of why should somebody choose you over them?

there’s already a lot of research out there suggesting that AI assisted work is becoming more and more homogeneous.

Now this isn’t a viewpoint of humans will always win over technology. I’m not quite sure that’s true all of the time here. but if AI raises the floor and makes that baseline easier for everyone to reach, well what becomes more valuable?

Is it professional judgment? Some original thinking? Any proprietary knowledge or insight that you hold, your brand, the strength of the relationships you’ve built, human connection?

Probably all of the above to a degree, right?

but I’m a hundred percent convinced that just becoming hideously efficient at producing the same stuff as everyone else is not the pathway to a competitive advantage.

So here’s something I think you can do right now.

Pick one workflow in your business where AI is supposedly saving you time. It only has to be one. And actually follow that all the way through.

Talk to the person using the AI who may have developed and deployed it, obviously. But then also go and talk to the person receiving the output. Ask them what’s actually improved, what do they have to check, what gets rewritten, what mistakes are appearing

and get to the heart of has any of that work actually disappeared or have we s simply had a big game of musical chairs and moved it around the business?

And ultimately, is that final outcome actually any better or more effective?

That then helps you answer the question of if we took AI away tomorrow, what would you genuinely miss?

And I reckon that answer will tell you far, far more than how many licenses you’ve deployed, how many prompts people have written or how many tokens you’ve used in a given period.

So of course, the next round of models of AI will be better. And the models after that will be better again, and so on and so on. But better AI doesn’t automatically guarantee that we’ll have a better business.

At some point we’re just gonna have to stop being impressed by how much this technology can produce and become a little more demanding about what it’s actually doing for us.

Like anyone and anything else, AI has to earn its keep. I hope you found that useful. This is Scale, an OX7 Partners podcast. I’m Dave Heywood and I’ll see you next time.