Tigera Logo

Guides: OWASP Top 10 LLM

Quick Guide to OWASP Top 10 LLM: Threats, Examples & Prevention

What Is OWASP Top 10 for LLM?

The OWASP Top 10 for Large Language Model Applications is a framework that identifies and ranks the top ten security vulnerabilities commonly found in applications involving large language models (LLMs).

This initiative is designed to educate developers, designers, architects, and organizations about the security risks associated with deploying and managing LLMs. It addresses vulnerabilities that are prevalent, easily exploitable, and have significant potential impact on the systems using these models. Learn more in our detailed guide to generative ai security risks. It provides practical security guidance for designing, building, and deploying applications that leverage LLM technologies.

You can access the official OWASP Top 10 LLM resource center here.

This is part of a series of articles about LLM Security.

In this article:

Why Is the OWASP Top 10 LLM Important?

As organizations rapidly integrate LLMs and Generative AI into their operations and client-facing offerings, the need for adequate security protocols becomes critical.  However, the speed of AI adoption has often outpaced the development of matching security measures, resulting in major security gaps.

The OWASP Top 10 LLM list is a collaborative effort involving over 500 experts and 150 active contributors from various sectors. These contributors include individuals from AI and security companies, independent software vendors (ISVs), cloud service providers, hardware manufacturers, and academia.

Learn more in our detailed guide to generative ai security

OWASP Top 10 Threats for LLM and How to Prevent Them

LLM01: Prompt Injections

Threat
Prompt injection is when an attacker manipulates a large language model (LLM) through crafted inputs, causing the LLM to unknowingly follow the attacker’s intentions. This can lead to issues like data exfiltration, social engineering, and unauthorized actions. For example, a user may trick the LLM into ignoring system prompts and revealing sensitive information or performing unintended actions.

Examples
A notable example is when an attacker uses an indirect prompt injection in a webpage that an LLM summarizes. The LLM may then solicit sensitive information from the user and exfiltrate it via JavaScript. Another case is when a malicious resume with hidden prompt injections is uploaded, causing the LLM to falsely evaluate the candidate favorably.

Prevention
To prevent prompt injections, enforce privilege control on LLM access to backend systems, separating external content from user prompts. Adding a human-in-the-loop for critical operations can also reduce the risk. Establish trust boundaries between the LLM, external sources, and extended functionality to mitigate risks.

LLM02: Insecure Output Handling

Threat
Insecure output handling refers to the inadequate validation, sanitization, and handling of outputs generated by LLMs before they are passed downstream to other systems. This vulnerability can lead to issues such as cross-site scripting (XSS), cross-site request forgery (CSRF), or remote code execution (RCE) in backend systems.

Examples
An example is when an LLM generates JavaScript or Markdown code that gets interpreted by the browser, leading to XSS. Another scenario involves a user utilizing an LLM-based summarizer that inadvertently includes sensitive data in its output, which gets transmitted to an attacker’s server without proper validation.

Prevention
To mitigate insecure output handling, treat the LLM as any other user by adopting a zero trust approach and applying input validation to responses coming from the model to backend functions. Follow the OWASP Application Security Verification Standard (ASVS) guidelines to ensure effective input validation and sanitization.

Additionally, encode the model output before sending it back to users to mitigate the risk of undesired code execution by JavaScript or Markdown. The OWASP ASVS provides detailed guidance on output encoding, which can help prevent vulnerabilities and ensure the security of the application.

LLM03: Training Data Poisoning

Threat
Training data poisoning occurs when attackers manipulate the training data of an LLM to introduce vulnerabilities, backdoors, or biases that compromise the model’s security and effectiveness. This can result in the model producing incorrect or harmful outputs, degrading its performance and damaging the reputation of its operators.

Examples
For example, a malicious actor might inject biased or false data into the training set, causing the model to generate misleading information. Another example is when an LLM is trained with data from unverified sources, which can lead to the inclusion of harmful or inaccurate information in the model’s responses.

Prevention
To mitigate training data poisoning, verify the supply chain of training data and maintain attestations via the ML-BOM (Machine Learning Bill of Materials) methodology. Implement strict vetting or input filters for specific training data or categories of data sources to control the volume of falsified data.

LLM04: Denial of Service

Threat
Denial of Service (DoS) attacks on LLMs involve consuming an excessive amount of resources, resulting in degraded service quality or complete unresponsiveness. Attackers may exploit the model’s context window by sending a continuous stream of inputs that exceed its capacity, causing it to use excessive computational resources.

Examples
A common example is an attacker flooding an LLM with variable-length inputs crafted to reach the context window limit, exploiting inefficiencies in processing such inputs. Another scenario involves recursive context expansion, where the attacker forces the model to repeatedly expand and process its context window, leading to resource exhaustion.

Prevention
To prevent DoS attacks, implement input validation and sanitization to ensure user inputs adhere to defined limits and filter out malicious content. Cap resource use per request and enforce API rate limits to restrict the number of requests an individual user can make within a given timeframe.

LLM05: Supply Chain

Threat
Supply chain vulnerabilities refer to the risks associated with third-party components, models, or data used in LLM applications. These vulnerabilities can lead to security breaches, biased outcomes, or system failures. Traditional software components and pre-trained models supplied by third parties can be tampered with, leading to significant risks.

Examples
For example, using a vulnerable pre-trained model for fine-tuning can introduce security issues. Another scenario is when poisoned crowd-sourced data is used for training, resulting in biased or inaccurate model outputs. Outdated or deprecated components can also be exploited by attackers to gain unauthorized access or perform malicious actions.

Prevention
To mitigate supply chain vulnerabilities, carefully vet data sources and suppliers, including their terms and conditions and privacy policies. Use reputable plugins and ensure they are thoroughly tested for your application requirements.

Maintain an up-to-date inventory of components using a Software Bill of Materials (SBOM) to prevent tampering. Implement strict access controls and authentication mechanisms to limit unauthorized access to LLM model repositories and training environments.

LLM06: Permission Issues

Threat
Permission issues arise when LLM plugins or tools are granted excessive permissions or autonomy, leading to potential security breaches. This can occur when an LLM agent has access to functions or systems that are not necessary for its intended operation, allowing it to perform harmful actions.

Examples
For example, a plugin designed to read documents might also have the ability to modify or delete them, posing a significant risk if the LLM is compromised. Another example is when an LLM plugin accesses a database with excessive permissions, such as the ability to insert, update, or delete records, rather than just read access.

Prevention
To prevent permission issues, limit the plugins and tools that LLM agents are allowed to call to only the minimum functions necessary. Implement strict access controls, granting only the permissions required for the plugin’s operation. Avoid open-ended functions where possible and use more granular functionality.

LLM07: Data Leakage

Threat
Data leakage occurs when an LLM inadvertently reveals sensitive information through its outputs. This can happen if the LLM memorizes and reproduces data from its training set or if it mishandles input and output data, leading to unauthorized access to sensitive information.

Examples
An example of data leakage is when an LLM trained on proprietary datasets unintentionally includes proprietary or confidential information in its responses. Another scenario is when sensitive data from one user session is inadvertently exposed to another user due to inadequate isolation mechanisms.

Prevention
To mitigate data leakage, integrate data sanitization techniques to prevent user data from entering the training model data. Implement strict input validation and sanitization methods to filter out potentially harmful inputs. Apply access control methods to external data sources and maintain a secure supply chain.

LLM08: Excessive Agency

Threat
Excessive agency occurs when an LLM-based system is granted too much autonomy, allowing it to perform damaging actions in response to unexpected or malicious inputs. This can lead to a range of impacts across confidentiality, integrity, and availability, depending on which systems the LLM-based app can interact with.

Examples
For example, an LLM granted access to a user’s email account might send emails without user approval if manipulated by a malicious input. Another scenario is when a plugin with excessive functionality allows an LLM to execute shell commands, potentially leading to remote code execution.

Prevention
To prevent excessive agency, limit the plugins and tools that LLM agents are allowed to call to only the necessary functions. Ensure that plugins implement strict parameterized input and enforce least-privilege access control. Avoid open-ended functions and require user authorization for high-impact actions. Implement logging and monitoring to detect and respond to undesired actions.

LLM09: Overreliance

Threat
Overreliance on LLMs can occur when users or systems trust LLM-generated outputs without proper oversight or validation, leading to security breaches, misinformation, or other issues. LLMs can produce inaccurate or unsafe content, known as hallucinations or confabulation, which can mislead users.

Learn more in our detailed guide to ai safety.

Examples
For example, an LLM might generate incorrect code suggestions that introduce vulnerabilities into a software system if not properly reviewed. Another example is when an LLM provides authoritative-sounding but inaccurate information, leading to poor decision-making.

Prevention
To mitigate overreliance, regularly monitor and review LLM outputs using self-consistency or voting techniques to filter out inconsistent text. Cross-check LLM outputs with trusted external sources and implement automatic validation mechanisms to verify generated content. Communicate the risks and limitations associated with using LLMs and establish secure coding practices to prevent the integration of potential vulnerabilities.

LLM10: Insecure Plugins

Threat
Insecure plugins refer to the risks associated with using third-party plugins in LLM applications. They can introduce vulnerabilities if they do not properly validate input, implement access controls, or handle data securely. Malicious inputs can exploit these vulnerabilities, leading to data exfiltration, remote code execution, or privilege escalation.

Examples
For example, a plugin that accepts raw SQL statements without validation can be exploited by an attacker to execute arbitrary queries. Another scenario is when a plugin accepts all parameters in a single text field, allowing an attacker to inject malicious payloads.

Prevention
To prevent insecure plugins, enforce strict parameterized input and implement type and range checks. Apply OWASP’s recommendations in the Application Security Verification Standard (ASVS) to ensure adequate input validation and sanitization. Design plugins to minimize the impact of insecure input parameter exploitation and use authentication and access control mechanisms.

Generative AI Security with Calico

Calico offers numerous features to address the many network and security challenges faced by cloud platform architects and engineers when deploying genAI workloads in Kubernetes.

Learn more in our detailed guide to container security.

Here are five Calico features for container networking and security for GenAI workloads:

  1. Egress access controls – Calico provides granular, zero-trust workload access controls between individual pods in Kubernetes clusters to external resources such as LLMs. It provides fine-grained workload access controls using DNS egress policies and NetworkSets (using IPs/CIDRs in network policy).
  2. Egress gateway – The Calico Egress Access Gateway assigns a fixed, routable IP to a Kubernetes namespace. All egress pod traffic from that namespace with an assigned routable IP address identifies the workload running within that namespace. This enables the cluster to securely scale while preserving the limited number of routable IPs and leveraging non-routable IPs for all other pod traffic within the cluster. The Calico Egress Gateway works with any firewall, enabling Kubernetes resources to access endpoints behind a firewall securely.
  3. Identity-aware microsegmentation – Calico enforces microsegmentation to achieve workload isolation and secure lateral communication between pods, namespaces, and services. It enables teams to logically divide workloads into distinct security segments and then define granular security controls for each unique segment. Teams can isolate workloads based on environments, application tiers, compliance needs, user access, and individual workload requirements.
  4. Observability and troubleshooting – Calico’s Dynamic Service and Threat Graph provides a graph-based visualization of your Kubernetes deployments, including images, pods, namespaces, and services. It has built-in troubleshooting capabilities to identify and resolve security and compliance gaps, performance issues, connectivity breakdowns, anomalous behavior, and security policy violations.
  5. Cluster mesh – Calico provides a centralized, multi-cluster management plane to enable security, observability, and advanced networking for workloads and services across multiple clusters in hybrid and multi-cloud environments. Calico provides unified security policy controls and federated endpoints and services.

Next steps:

X