aisecurityrisks.com

# R-07·OWASP LLM07:2026·cases 1·checked Nov. 15, 2026

Misinformation and overreliance

A model produces something wrong that looks credible, and a person or system acts on it.

What it is

OWASP frames the security problem as trust: incorrect output becomes a risk when someone or something acts on it. In agentic systems that might be a wrong assumption about system state that leads straight to an action.

NIST describes the human side as automation bias, people deferring too much to automated output. A fabricated package name, a wrong configuration step or an invented policy citation can all turn into an incident once acted on.

Documented cases

Incidents happened in the wild. Research demos were shown by researchers, often disclosed to the vendor first.

  • Research demodisclosed March 28, 2024·AML.CS0022

    Hallucinated package name downloaded more than 30,000 times

    A Lasso Security researcher asked four models thousands of coding questions across five languages and found they often recommended packages that did not exist, sometimes the same invented name again and again. He uploaded an empty package under one of those names, huggingface-cli.

    The empty package received more than 30,000 downloads in three months, and an Alibaba research repository told readers to install it. OWASP calls registering such names in advance slopsquatting.

    Sources: Lasso Security; OWASP GenAI Security Project; MITRE ATLAS AML.CS0022

Controls to start with

Questions to ask vendors

Our suggestions, not taken from a framework.

  • Where does the assistant get its answers, and can it show the source for each one?
  • Which decisions does the system make without a person checking the output?

Sources

  1. OWASP Top 10 for LLM Applications 2026. OWASP GenAI Security Project, Aug. 3, 2026.Standard or framework · read: full text (122-page PDF) · retrieved Nov. 15, 2026 · license CC BY-SA 4.0note: Community list. OWASP says the 2026 ranking combines a practitioner vote with incident evidence. Appendix A maps each entry to other frameworks.
  2. OWASP Top 10 for Agentic Applications 2026. OWASP GenAI Security Project, Dec. 9, 2025.Standard or framework · read: full text (PDF) · retrieved Nov. 15, 2026 · license CC BY-SA 4.0note: Covers risks once a model acts as an agent with tools, memory and delegated authority.
  3. MITRE ATLAS data, release 2026.10. MITRE, Oct. 28, 2026.Standard or framework · read: full data file (tactics, techniques, case studies) · retrieved Nov. 15, 2026 · license Apache-2.0note: ATLAS labels each case study as an Incident (real-world) or an Exercise (research or red-team demonstration).
  4. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. National Institute of Standards and Technology, July 2024.Government guidance · read: sections 2.4 (Data Privacy), 2.7 (Human-AI Configuration) and 2.9 (Information Security) read in full · retrieved Nov. 15, 2026 · license US government worknote: Voluntary profile. Names 12 risks that generative AI creates or makes worse.
  5. Diving Deeper into AI Package Hallucinations. Lasso Security, March 28, 2024.Security vendor research · read: full page · retrieved Nov. 15, 2026note: Lasso Security sells AI security products. Rates come from the researcher's own test set and models available at the time.