Avoiding the AI Experimentation Trap

The MIT Media Lab/Project NANDA recently released findings showing that 95% of investments in generative AI have produced zero returns [1]. This revelation comes at a time when the underwhelming launch of OpenAI’s GPT-5 has provided ammunition for sceptics questioning whether AI’s progress is slowing, with The Economist suggesting that generative AI is entering its “trough of disillusionment” era [2].

Yet these headlines tell only part of the story. The MIT report itself is more nuanced, acknowledging that whilst individuals are successfully adopting generative AI tools that increase their productivity, such results aren’t measurable at a profit and loss level, and companies are struggling with enterprise-wide deployments [3]. More tellingly, most spending on AI experiments goes to sales and marketing initiatives, despite the fact that back-end transformations tend to produce the biggest return on investment [4].

For business leaders, this presents a familiar dilemma. How can companies learn to harness these transformative tools without falling into the same traps that ensnared so many during the digital transformation era? The answer lies in understanding that experimentation, whilst essential, must be purposeful rather than scattershot.

Déjà vu

The current AI experimentation frenzy bears uncomfortable similarities to the digital transformation mistakes of the previous decade. When many leaders felt confused about digital transformation and the path forward, they embraced innovation and experimentation with a “let 10,000 flowers bloom” approach, hoping that a few experiments would produce unicorn-level returns [5].

This lack of focus proved to be a blunder. Without a clear connection to real business opportunity, the result, as described by Professor of Strategy at INSEAD Nathan Furr and John H. Loudon Chaired Professor of International Management at INSEAD Andrew Shipilov, was “a morass of unfocused, under-resourced teams that produced few scalable results” [6]. Facing disappointing returns, many leaders naturally concluded that experimentation with digital was broken and shut down the experiments, either returning to business as usual or refocusing on safer bets like replacing ageing IT systems [7].

The fundamental error was losing sight of business’s most basic objective: solving problems for customers. By framing AI as radical and disruptive, organisations often disconnect from this core purpose [8]. As one CEO admitted to Forbes’ Andrew Binns, “we are going to spend $2 billion on AI, I don’t know what on, but we are going to invest” [9]. This statement reveals the danger of investment without strategic direction.

The Broader Context

To avoid repeating past mistakes, leaders must first understand AI within the larger arc of transformation. The true change organisations are wrestling with isn’t simply about AI. Rather, it’s about a fundamental shift from digital technology operating at the periphery of organisations to digital technology operating at the very core [10].

Previously, IT was about laptops, Wi-Fi, printing, and databases for registry of core activities. Now, organisations are built around digital workflows and customer journeys rather than their own production activities [11]. In essence, every company is becoming a technology company, moving from people performing tasks based on human judgement and intuition to a world of data- and AI-driven decisions, overseen by humans but not necessarily with people as the core engine of activity [12].

Consider how Ant Financial makes lending decisions or Amazon makes pricing decisions, with humans only overseeing, not doing, the activity [13]. This represents a profound shift that will take many years to complete but will ultimately lead to a fundamentally different kind of organisation.

Understanding this bigger picture helps leaders remember that the point is to transform the business to use technology to serve customers better, faster, easier, and cheaper. All forms of AI, including generative AI, are simply tools — one of many — that can help accomplish this objective [14].

The Four Traps

Whilst strategic misalignment represents the overarching challenge, companies face several specific traps that can derail their AI initiatives. The first is resisting adoption altogether due to discomfort with the unknown [15]. Yet as Holly Shipley, a Google alumna and strategic leader spearheading generative AI efforts at a Fortune 200 tech company, notes: “Employees should be focused on the basic functionality of genAI tools like ChatGPT. This will reduce the learning curve when genAI is integrated into tools employees already use” [16].

The second trap involves fearing AI will replace human workers entirely. However, research reveals that more than 70% of workers would delegate tasks to AI to lighten their load, with organisational psychology professor Adam Grant observing, “It’s fascinating that people are more excited about AI rescuing them from burnout than they are worried about it eliminating their jobs” [17]. The reality is that individuals won’t be replaced by AI, but they will be replaced by someone who knows how to use AI [18].

The third trap represents a fundamental misunderstanding of technology’s role in solving problems. Companies often expect AI to fix cultural problems that require human intervention. For instance, AI can speed up email responses, but if companies judge employee performance by response time, it may actually increase email volume rather than reducing it [19]. Similarly, AI might help prioritise meetings, but it cannot address the underlying cultural issue of using meeting invitations as affirmations of influence and worth [20].

The fourth trap involves immediately backfilling any time AI saves with additional tasks [21]. The margins that AI could create would be beneficially used for rest, white space, relationships, and creative brainstorming, but organisational obsession with busyness often leads to filling any created space with more work [22].

The Perils of Overinvestment

The current AI investment frenzy recalls other innovation trends, particularly the “big data” revolution of the early 2000s. General Electric’s ambitious bid to become a “top ten software company” serves as a cautionary tale [23]. Seeing the possibility for an Industrial Internet of Things, GE forecast a market worth $500 billion by 2020 and committed itself to achieving first-mover advantage. It tripled its R&D budget, built a 1,000-person software division, and launched its own big data platform, Predix [24].

Five years later, the strategy had failed spectacularly. The CEO was fired, the company dropped out of the Dow Jones 30 for the first time, and GE’s software ambitions folded [25]. The problem wasn’t technical. It was that GE built a big data platform that was a mismatch with the diversity of the manufacturing sector, treating an emerging, uncertain market the same way they handled mature ones [26].

Most of GE’s toxic assumptions were about non-technical topics like customer priorities, similarities between manufacturers, ease of capturing data, and IT organisations’ readiness for new roles [27]. These critical assumptions were knowable in advance, but GE lacked the patience and humility to discover what potential customers actually wanted before launching.

Focused Experimentation

Furr and Shipilov argue that successful AI experimentation requires balancing three elements: connection to true value creation, low costs that allow multiple learning cycles, and design with an eye toward eventual scaling [28]. This sounds simple but proves difficult in practice, with leaders either charging ahead without considering scalability or becoming bogged down obsessing about enterprise readiness from day one.

The solution lies in striking a balance through what Furr and Shipilov call the IFD framework, standing for intensity, frequency, and density [29]. When evaluating potential AI applications, leaders should assess how intense the problem is, how frequently it occurs, and how many users or instances of the problem exist. For example, developing digital tools to help apartment managers order repair services might seem valuable, but parents wanting to ensure their child’s safety every night represents a more intense, frequent, and dense problem worth solving [30].

Once experiments prove their value, scaling requires careful attention and dedicated resources. Someone with power to create change must own the initiative, leading a “ninja” team with air cover from senior leadership, company-wide connections to secure resources, and focus to scale effectively [31]. These teams, observed at companies like Amazon, Qualtrics, and 7-Eleven, possess the organisational clout necessary to transform experiments into enterprise-wide solutions [32].

Going Forward

As organisations enter what some call the post-enthusiasm wave of AI, many leaders risk misinterpreting implementation challenges as signals that AI cannot create value [33]. They face the same danger that plagued digital transformation. That is, falling behind whilst competitors advance.

The truth is that AI can create significant value. But creating value always returns to the initial moment of experiment design, when teams can see how new tools create value for customers.

Success requires de-risking business models through small, relatively cheap experiments that test hypotheses before committing substantial resources. As Henley Business School’s Narendra Laljani argues, every business has a “mental model” that explains the world through unconscious, unarticulated assumptions about success [34]. When pressure mounts, organisations default to these models, which for corporates often means “go big or go home” [35].

Just because generative AI could represent innovation on the scale of the printing press doesn’t justify indiscriminate spending. Instead, leaders must de-risk their innovations with rapid experiments testing critical assumptions underlying their investments. Fortune favours the learner, not merely the brave.

The fundamental lesson remains unchanged. Regardless of what new tools emerge, business’s purpose will always be solving important problems for customers. Companies that remember this whilst thoughtfully experimenting with AI will avoid the experimentation trap and unlock genuine transformation. Those that don’t risk joining the 95% whose investments produce zero returns, a fate as avoidable as it is expensive.