OOmnistu

Blog / AI Development

AI Development

Treat AI model output as untrusted input: improper output handling explained

Treat AI output as untrusted data before it reaches web pages, databases, commands or other systems that interpret content.

OOmnistu
Reviewed 2026-11-16 · 3 min read

A common shortcut in AI features: take what the model returns and pass it straight to the next part of the system, such as a web page, a database or a command line. OWASP's Top 10 for LLM Applications (2025) calls the failure to check that output LLM05, Improper Output Handling. We read the project's page in full on 15 November 2026. We did not read its references. This article summarises that one source.

The idea

OWASP defines the risk as insufficient validation, sanitisation and handling of model output before it goes downstream. Because model output can be steered by the prompt, passing it on unchecked is "similar to providing users indirect access to additional functionality." Our article on prompt injection covers how text steers a model; this one covers what happens after the model answers. OWASP separates it from overreliance, which is about trusting the accuracy of answers, not about where the output goes.

Per the page, successful exploitation can mean cross-site scripting (XSS) and cross-site request forgery (CSRF) in browsers, and server-side request forgery, privilege escalation or remote code execution on backends.

Examples the page lists

  1. Model output goes into a shell or a function like exec or eval.
  2. The model returns JavaScript or Markdown that the browser interprets.
  3. Model-written SQL runs without parameterisation.
  4. Model output builds file paths without sanitising them (path traversal).
  5. Model output goes into email templates without escaping.

What OWASP recommends

  1. Treat the model like any other user (zero trust) and validate what it sends to backend functions.
  2. Follow OWASP ASVS guidance on validation and sanitisation.
  3. Encode model output before showing it to users.
  4. Use context-aware encoding: HTML encoding for HTML text. For SQL values, use parameterised queries or prepared statements rather than relying on manual escaping. Parameterization does not make an arbitrary model-written SQL command authorized: restrict allowed operations and database permissions as well.
  5. Use strict Content Security Policies.
  6. Log and monitor model outputs for unusual patterns while applying redaction, access and retention controls. Do not create a second leak by retaining private prompts or credentials in raw logs.

The page also says impact rises when the app gives the model more privileges than a normal user, when third-party extensions do not validate input, and when there is no rate limiting or anomaly detection on model use.

THREE DESTINATIONS, THREE HANDLING BOUNDARIESModel outputTreat it as untrusted,like user input.Web pageEncode for the output context;use a strict Content Security Policy.Database valuesUse parameterised queries;restrict operations and permissions.Shell / exec / evalDirect unchecked use is a routeto remote code execution.ALTERNATIVE DESTINATIONSModel outputTreat it as untrusted, like user input.Web pageEncode for the output context;use a strict CSP.Database valuesUse parameterised queries;restrict operations / permissions.Shell / exec / evalDirect unchecked use can leadto remote code execution.
Three of the OWASP LLM05 page's examples, shown as alternative destinations, not consecutive steps. Its recommendations include validation at backend functions, context-aware encoding, parameterised queries and strict CSP. Restricting database operations and permissions is an additional safety check in this article; parameterisation alone does not authorise an arbitrary model-written command. The redirection risk is downstream handling, not whether the answer sounds correct.

Scenarios from the page

A short checklist

Related reading on this site: SQL injection prevention, input validation basics, Content Security Policy basics, CSRF prevention basics.

Limits

One OWASP page, read in full; references and ASVS were not read. No claims about specific models or products, and no statistics.

By Omnistu.

Sources

Previous
When an AI feature gets it wrong: misinformation and overreliance for product teams