Let’s start with an example. A company is deploying an AI assistant to handle email inquiries in the B2B department. The goal is sensible: to lighten the team’s load, speed up responses and reduce routine work. One day, one of its strategic clients submits a request to return a batch of goods. The AI analyzes the inquiry, checks it against company policy, and sends a polite refusal based on that policy.<\/p>
However, the algorithm doesn’t know what every salesperson in the company knows: key partners have flexible, individually negotiated terms. So the decision was in line with company policy and at the same time completely wrong. The sales director only learned about the problem when the client threatened to terminate the contract.<\/p>
No one noticed the malfunction. Because there was no malfunction.<\/p>
This is not an exceptional story. The results of the report “Connext Global AI Oversight Survey”, <\/mark><\/a>published in mid-February 2026, show that there are many more such quiet failures. As many as 42% of respondents openly admit that the models routinely overlook business-critical details. The same survey shows that only 17% of employees believe that AI can operate fully autonomously in their workplace. And in 1 out of 5 cases, the new technology made the customer’s situation worse instead of better.<\/p> So it comes as no surprise that, according to Gartner’s analyses, at least 30% of corporate projects based on generative AI are abandoned right after the testing phase. The well-known and widely cited MIT research (“The GenAI Divide. State of AI in Business”<\/a><\/mark>) goes even further, claiming that as many as 95% of pilots don’t deliver a fully measurable return.<\/p> What’s the reason for that?<\/p> Over the past few years, the market has focused on buying access to all sorts of platforms. However, the fundamentals have been overlooked, without which even the most advanced system behaves like a highly educated person on the first day of a new job\u2014eloquent, full of energy, but operationally useless.<\/p> Most AI failures in business are due to tools simply generating answers that sound convincing but are factually incorrect. Such hallucinations are far more dangerous than an obvious error because they\u2019re harder to detect. 31% of surveyed employees report that AI sounds very confident when giving wrong answers. Is it the technology\u2019s fault? Not quite…<\/p> Professor Ethan Mollick of the Wharton School, one of the leading analysts of AI’s impact on business, describes modern language models as an (“alien Intelligence”<\/a><\/mark>). His concept of the jagged frontier aptly captures the nature of this issue: algorithms can brilliantly draft a complex strategy, only to completely fail moments later at a trivial, company-specific task.<\/p> Why is this happening?<\/p> Public models have been trained on billions of publicly available documents. They don’t know the history of the relationship with a key client. They don’t know that certain procedures have unwritten exceptions. They don’t understand the internal jargon or why this particular partner gets different terms than everyone else. As a result, employees waste time on constant rework, management doesn’t see the promised return on investment, and no one fully understands why\u2026<\/p> A natural response to such situations is to turn to the RAG (Retrieval-Augmented Generation) architecture. The idea is simple: instead of relying solely on the knowledge learned by the model, the AI should use the company’s internal databases: regulations, policies, customer histories, and so on.<\/p> But there’s another pitfall here. Not all RAGs are created equal.<\/p> Throwing all corporate documents into a one repository and expecting the system to draw the right conclusions on its own is a dead end. What\u2019s needed is a well-thought-out data architecture\u2014an organized environment that includes not only plain records but also the full history of interactions from CRM systems, negotiation context, exceptions and their causes. McKinsey & Company data <\/a><\/mark>makes it painfully clear that information architecture is the Achilles\u2019 heel of companies implementing AI, as 70% of enterprises cite problems with data integration and a lack of appropriate data governance as the main barrier.<\/p> But let’s assume that the system of the company mentioned at the beginning had access to a well-organized database that contained this client’s relationship history. Would it make a mistake? Probably not…<\/p> It\u2019s just that “probably” makes a significant difference.<\/p> Even the best-designed system won’t catch every nuance. Like how a particular salesperson has been talking to a particular client for the past three years. Or that there’s an unwritten rule that this client is sometimes worth “pampering”, and the definition of that “sometimes” is based purely on the salesperson’s intuition and the context of the conversation.<\/p> And that’s precisely why mature AI implementations don’t eliminate humans, but position them differently in the process.<\/p> Mollick lists “human presence in the decision-making loop” as one of four ironclad rules for effective collaboration with AI. However, many automation purists regard the “Human-in-the-Loop” approach as an admission of automation’s failure. That\u2019s a misinterpretation.<\/p> An employee who reviews and, if necessary, edits a system-generated draft response needs a fraction of the time it would take to write it from scratch. The company gains speed without losing control over relationships that have real business value.<\/p>AI doesn’t know what it doesn’t know<\/h4>
A knowledge base is not a folder on a drive<\/h4>
Human + AI<\/h4>
Check before you deploy<\/h4>