According to Infosecurity Magazine, the UK’s National Cyber Security Centre (NCSC) has raised a major alarm, stating that prompt injection vulnerabilities in AI may never be fully solved. NCSC technical director for platforms research, David C, warned security teams not to treat the threat like SQL injection, as large language models (LLMs) don’t distinguish between data and instructions. Because of this inherent confusion, he argues that common mitigation attempts, like training models to prioritize instructions, are likely doomed to fail. Instead, the agency advises focusing on reducing the risk and potential impact of attacks, suggesting that if a system’s security can’t tolerate the remaining risk, it might not be a good fit for LLMs. The NCSC aligns its guidance with the ETSI TS 104 223 standard for AI cybersecurity. Exabeam’s chief AI officer, Steve Wilson, agreed, stating that defending AI agents is more like defending humans against insider threats than securing traditional software.
The Unfixable Problem
Here’s the core of the issue, and it’s a bit mind-bending. With something like SQL injection, you can theoretically fix the root cause: you separate the user’s data from the system’s instructions. You sanitize inputs. You use parameterized queries. The problem is solvable. But with an LLM? The entire mechanism is different. It’s just predicting the next token. It doesn’t “understand” a command versus the data it’s supposed to process. It’s all just tokens in, tokens out. So when the NCSC says this might be a category of vulnerability that’s never fully mitigated, they’re not being lazy. They’re pointing out a fundamental, architectural reality. The model itself is, as they put it, an “inherently confusable deputy.” You can’t teach it to not be confusable at its core.
Shifting The Security Mindset
So what do we do? We stop trying to build a perfect wall. The advice from both the NCSC and industry experts like Steve Wilson is a massive shift in perspective. It’s about operational discipline, not magic bullets. Think containment. Think monitoring. Assume your AI agent will get tricked or confused, and design systems that limit the blast radius when it happens. Does it really need access to that critical database? Can its actions be put through a human or automated review loop? This is a far cry from traditional appsec, and it’s going to be a tough pill for many CISOs to swallow. It means accepting a level of inherent unpredictability. Wilson’s analogy to defending against insider threats is spot-on. You monitor behavior, you limit privileges, you look for anomalies. You don’t expect to eliminate human error or malice; you manage the risk.
The Coming Wave Of Breaches
The most sobering part of the warning is the historical parallel. David C from the NCSC points out that we spent years dealing with rampant SQL injection flaws because apps weren’t designed with that threat in mind from the start. He fears we’re on the exact same path with generative AI. We’re rushing to embed LLM-powered agents into everything—customer service, data analysis, workflow automation. But if we’re not baking in this new containment-and-monitoring mindset from day one, the consequences are predictable. We’ll see a similar wave of breaches, data leaks, and manipulated outcomes. The urgency isn’t about finding a cure; it’s about installing seatbelts and airbags before we all start driving at high speed.
What It Means For Builders
Basically, this changes the calculus for using LLMs in any serious application. The NCSC’s line is crucial: if the system’s security cannot tolerate the remaining risk, it may not be a good use case for LLMs. That’s huge. It means we need to be brutally honest during design phases. Is this a low-stakes creative brainstorming tool? Fine. Is it an agent that can execute financial transactions, alter sensitive records, or control physical systems? Now you have a massive risk management problem that can’t be coded away. The focus moves from the model itself to the architecture around it—the gates, the logs, the failsafes. It’s a less sexy, more engineering-heavy approach to AI security. But according to the experts, it’s the only one that has a chance of working.
