A Python package called huggingface-cli once pulled in more than 30,000 downloads. Nobody set out to build it. It existed because AI coding assistants kept telling developers to install it, so a researcher at Lasso Security registered the empty name to see what would happen. Thousands of people ran the install command without a second thought, because their AI told them to.
That is a security hallucination in one sentence: code that looks right, runs clean, and quietly opens a door. The syntax is perfect. The logic is wrong. And the tools most teams lean on never flag it, because they were built to catch typos and known bad patterns, not confident mistakes.
If you build with Cursor or Bolt.new, you have almost certainly shipped a few of these. Here are the five we run into most, and how to catch them before they reach production.
What are AI security hallucinations?
A security hallucination is when an AI agent writes code that compiles and passes a quick review but carries a flaw it either invented or skipped. It happens because the model optimizes for finishing the task, not for defending it. Ask for a working feature and you get a working feature. Whether that feature checks permissions, validates input, or imports a package that actually exists is a separate question, and the model rarely asks it for you.
In practice the damage shows up in three places: packages that do not exist, authorization checks that were never written, and inputs the model trusted when it should not have.
The data nobody wants to hear
The awkward part is that AI does not just miss security. It makes you more confident while it does. A Stanford study (Perry et al., 2022) found that developers working with an AI assistant wrote less secure code than those without one, and were more likely to believe their code was safe. Models have come a long way since then, but that confidence gap has not closed. You feel fast, so you check less.
The gap has a price tag. IBM put the average cost of a data breach at 4.88 million dollars in 2024, up from 4.45 million the year before. When AI lets a two-person team ship a new integration before lunch, the flaws ship just as fast, and they tend to sit in production a long time before anyone notices.
EchoLeak: when the input is the attack
The clearest recent example of how this goes sideways is EchoLeak (CVE-2025-32711), found by Aim Security in 2025. It targeted Microsoft 365 Copilot rather than a coding agent, but the mechanism is exactly what should worry anyone wiring an AI into their product.
An attacker hid instructions inside an ordinary-looking business document. When Copilot read the document, it followed those instructions and leaked confidential data. No link to click, no file to run, no action from the victim at all. Researchers called it a zero-click attack, and it works because the AI cannot reliably tell the difference between content it is meant to summarize and commands it is meant to obey.
Your Cursor or Bolt.new app inherits that same blind spot the moment it feeds untrusted input into a model or an agent, whether that input is a support ticket, a webhook payload, or a file a user uploaded. If you never draw a hard line between data and instructions, the input becomes the attack.
The five hallucinations we see most
1. Phantom packages (slopsquatting)
This is the supply chain attack that AI handed to attackers for free. Models routinely recommend libraries that do not exist. A USENIX Security 2025 study ran 576,000 code samples through 16 models and found that roughly one in five suggested packages were hallucinated. Of those, 51 percent were pure inventions, names with no real counterpart anywhere.
Attackers read the same output you do. They harvest the names the models keep inventing, register them on npm or PyPI, and wait. The huggingface-cli story from the top of this post is the proof: a fake name, an empty package, 30,000 downloads, zero marketing. The tactic now has a name, slopsquatting, and it is one of the cleaner ways to slip malware into a codebase. Before you run an install command an AI handed you, confirm the package is real and actually maintained.
2. The missing permission check (BOLA)
AI is great at fetching data and careless about who is allowed to see it. Ask for an endpoint that returns a user's orders and you usually get one that returns a user's orders, with no check that the person asking is that user. Change the ID in the URL and you are reading someone else's records. This is Broken Object Level Authorization, and it is the most common serious flaw we find in AI-built apps.
Here is the shape of it:
// Hallucinated: fetches the record, never checks who owns it
app.get('/orders/:id', async (req, res) => {
const order = await db.orders.findById(req.params.id);
res.json(order);
});
The fix is the one line the model skipped. Confirm the record belongs to the authenticated user before you hand it back:
app.get('/orders/:id', async (req, res) => {
const order = await db.orders.findById(req.params.id);
if (order.userId !== req.user.id) return res.status(403).send('Forbidden');
res.json(order);
});
3. Secrets left out in the open
Credentials are what attackers want most, and AI exposes them in two ways. First, agents love to hardcode a key to get a feature working, then leave the mock token or temporary credential sitting in the source. Someone forgets to move it into an environment variable, and it ships. Second, the credentials leak after the fact. IBM's 2026 X-Force Threat Index reported that infostealer malware exposed over 300,000 ChatGPT credentials in 2025. Once someone is inside your AI account, they can read every prompt you ever pasted in, which for most developers means code, configs, and the occasional secret.
Scan your repo for anything shaped like a key before every deploy. If a secret ever landed in a commit, rotate it. Deleting it from the latest version does not help, because it is still sitting in the history.
4. Routes that quietly went public
To make a feature work on the first try, agents skip the boring parts. In Next.js or Express, that often means adding a new API route and leaving off the authentication middleware. The endpoint works in testing, so it looks finished. It is also open to anyone on the internet who finds the URL.
// New route, no auth. Works perfectly, for everyone.
app.post('/admin/refund', async (req, res) => {
await issueRefund(req.body.orderId);
res.send('done');
});
Every new route is a door. Make sure each one has a lock before it ships, and review the full route list, not just the endpoints you remember writing.
5. Poisoned workspace rules
This is the newest one and the easiest to miss. Files like .cursorrules give your AI agent standing instructions for a project. Attackers have started hiding malicious instructions inside these files in public repositories. Pillar Security documented the pattern and named it the Rules File Backdoor. Clone a compromised repo, open it in Cursor, and the hidden prompt comes along for the ride. From then on the agent quietly writes weaker code or leaks environment variables as it works, and nothing in the diff looks out of place.
Treat config files from outside sources the way you treat any other untrusted code. Read the .cursorrules before you trust a repo, especially one you forked from a stranger.
Why your scanner misses all of this
Look at what these five have in common. Not one of them is a syntax error. The install command is valid. The endpoint runs. The route responds. A traditional scanner reads code as text and hunts for known bad strings, so it finds nothing, because at the level of text there is nothing to find. The flaw lives in the relationships between files: a route that trusts an ID, a fetch that skips a check, a config that came from somewhere else.
Catching that means reading the codebase the way an attacker would, following how data moves from one file to the next. That is what CodeHalo's deep repo audit is built for. Take the wide-open setting an agent reaches for when it just wants something to work:
// Hallucination: open to every origin so the feature works right now
app.use(cors({
origin: '*',
methods: ['GET', 'POST']
}));
A text scanner is perfectly happy with that. CodeHalo flags it and hands back the restricted version:
// Fix: only allow origins you actually trust
const allowedOrigins = [process.env.FRONTEND_URL];
app.use(cors({
origin: function (origin, callback) {
if (!origin || allowedOrigins.includes(origin)) {
callback(null, true);
} else {
callback(new Error('Not allowed by CORS'));
}
}
}));
Ship fast, just check the logic
AI is the best thing to happen to small teams in years. The catch is that it writes code that looks done before it is safe, and the usual tools cannot tell the two apart. You do not have to slow down to fix that. You have to add one step that reads your project the way an attacker would, before your users get the chance to.
If you want to see where your own code stands, run a free lite scan on any GitHub repo, or read more on why vibe-coding security is its own discipline.
FAQ
What is an AI security hallucination?
It is code an AI agent writes that looks correct and runs without errors but contains a flaw it invented or skipped, like a call to a package that does not exist or an endpoint with no permission check. The syntax passes. The logic does not.
What is slopsquatting?
Slopsquatting is a supply chain attack that preys on AI package hallucinations. Attackers find the fake library names models keep recommending, register those names on npm or PyPI, and fill them with malware. Developers who trust the AI's install command pull the malicious package straight in.
Is it safe to use Cursor or Bolt.new?
Yes, as long as you check the output before you ship it. The tools are not the problem. Shipping their code unread is. They are fast and confident, which is exactly why the security gaps slip through. Audit the logic before you deploy and you keep the speed without the exposure.
What is BOLA?
Broken Object Level Authorization. It happens when an endpoint returns a record without checking that the requester is allowed to see it. Change the ID in the request and you get someone else's data. It is the most common serious flaw in AI-generated apps.
How do prompt injection attacks actually work?
The attacker hides instructions inside content the AI is going to read, such as a document, an email, or a config file. The model cannot reliably separate data it should process from commands it should follow, so it follows the hidden instructions. EchoLeak (CVE-2025-32711) used this to leak Microsoft 365 Copilot data with no user action at all.
Can traditional security tools catch AI hallucinations?
No. Static scanners look for known bad patterns in text, and these flaws are valid, error-free code at the text level. The danger is in how the pieces connect across the codebase, which only a tool that traces data flow can see.