Threat Modeling for Large Language Models

Exploring the Power and Potential Risks of Large Language Models (LLMs)

Large Language Models (LLMs) have undoubtedly taken the news by storm, as everyone from cybersecurity professionals to middle school students are eagerly exploring the power of these systems. Today, models like GPT-4, which powers ChatGPT, show the promise LLMs hold to dominate the technology landscape tomorrow—while hopefully not taking over the world. As this technology becomes an integral part of our daily lives, it's imperative for us to implement robust security measures in the face of rapid deployment.

This challenge has prompted me to think deeply about threat modeling and vulnerability research for these types of LLMs, using GPT-4 with the WebPilot Plugin as a case study.

So without further ado, let's dive in.

Background

Actually, before getting into the juicy, technical details, it's important to establish a baseline understanding of some terms.

What is Threat Modeling?

Threat Modeling describes the process of analyzing an application by taking the perspective of an attacker in order to identify and quantify security risks. According to OWASP^[1], there are three steps in the Threat Modeling process: Decomposing the application, determining and ranking the threats, and determining countermeasures and mitigations. This post will go through all three of these steps, giving various examples along the way.

What is a Large Language Model?

At its most basic definition, a Large Language Model (LLM) is an artificial neural network designed to perform Natural Language Processing (NLP) tasks, which include generating, understanding, and classifying human language^[2]. These models are trained on data from the internet, giving them an expansive knowledge base and training set. While there are many LLMs available on the market today, such as Llama 2 and Falcon, this post will focus on the popular GPT-4.

What are ChatGPT Plugins?

In late March 2023, OpenAI released a new feature called ChatGPT Plugins^[3]. This feature provided developers with a way to integrate their APIs with GPT-4, extending the LLM's functionality and usability. This development led to a surge in plugins being created, allowing users to order groceries, check their email, and search the web—all from the ChatGPT interface. And, as of August 2023, while there are now over 800 plugins available for ChatGPT Plus members, this post will focus on WebPilot, which allows users to access arbitrary websites.

With that out of the way, now we can begin the process of developing a Threat Model for the GPT-4 LLM integrated with the WebPilot plugin.

Step 1: Decompose The Application

With a limited understanding of the inner-workings of ChatGPT and the WebPilot plugin, the best way that I've found to decompose the application is to analyze the architecture and document the assets.

Analyzing the Architecture

The system architecture can be broken down into three major pieces: the ChatGPT UI, the GPT-4 Backend, and the WebPilot Plugin APIs. Each piece is a black box on its own, but we can learn more about the system by examining how the pieces interact.

The ChatGPT UI is the user's interface to the system. It is where the user logs in, prompts the model, and receives the model's response. The features of this interface include the ability for users to have multiple, distinct conversations and for the model to display results using Markdown formatting. All communication between the ChatGPT UI and the GPT-4 Backend takes the form of API requests.

The GPT-4 Backend receives the user's prompt and then sends it to the LLM to generate an answer. Unfortunately, little is known about the technical details of this system, so we'll have to draw some inferences. The backend's features include handling user data such as credentials and separating all concurrent user conversations with the model. Additionally, according to the plugin documentation^[4], if the prompt contains a URL, an additional API request containing the requested URL is sent to the WebPilot API. The text of the webpage is then also sent to the model, along with the user's prompt.

The WebPilot API is the code managed and run by the plugin developer and receives the URL the user requested in the original prompt from the GPT-4 Backend. WebPilot retrieves the HTML of the webpage, extracts the text, and returns the plaintext to the GPT-4 Backend. Again, little is known about the technical details of this system, but this observable behavior provides insight from outside the system.

Here is a diagram that documents the complete flow.

Documenting Assets

Documenting assets involves examining the application from an attacker's perspective and asking the question, "What am I interested in?" These assets could include data from other users or additional privileges within the application. They can be viewed as the target list for the attacker. Therefore, they must be documented so we know what to defend against.

Here is a table documenting some examples of assets present in this system.

Step 2 & 3: Determining, Ranking, and Mitigating Threats

Now that we have an understanding of the system's architecture and are aware of some of the attackers' targets, the next step is to determine and rank threats. A comprehensive threat model will include the threat name, description, STRIDE Identification^[5], the related asset, and a risk analysis based on the probability of an attack occurring and the cost to the organization should that attack transpire. However, since I am not OpenAI and do not possess any insight into their risk model, I will only document some of the more intriguing or unique threats to this system, ranked by relative risk.

After describing each threat, I've also included some thoughts on mitigation techniques. This would be its own step in a complete threat model, however since I'm still not OpenAI and do not possess any insight into their risk model, I can only speculate on the mitigation going on under-the-hood.

Here are the threats that are covered:

Before we delve into the threat breakdown (the last point, I promise), it's important to examine prompt injection. This is because threats T1-T3 all rely on the existence of prompt injection to exploit the vulnerability.

Prompt Injection, as the name suggests, is a technique in which an attacker inserts a maliciously crafted prompt into a medium, such as web pages, PDFs, or images, thereby influencing the output of an LLM. Through this method, the attacker might be able to perform actions on behalf of the user, exfiltrate sensitive data, or exploit a vulnerability related to insecure output handling. Threats T1-T3 explore what an attack may be able to accomplish with prompt injection.

An example attack chain might go as follows:

1. The attacker creates a website with hidden text designed to capture the attention of the LLM and provide it with malicious instructions.
2. The victim prompts the LLM to summarize the attacker-controlled website.
3. As the LLM attempts to read the content of the website, the hidden prompt contained in the website's text is interpreted as a new instruction.
4. The attacker's prompt is executed.

In order to attempt a Prompt Injection Proof-of-Concept (PoC), I set up an S3 bucket, created a demo HTML page about a fictional band, and embedded this text within it:

Then, an attacker can use a prompt such as “summarize this website for me: <url here>” to have GPT-4 use WebPilot to fetch the content and then "summarize" the results. Similar to SQL injection, a vulnerability can arise when untrusted website data is treated as a trusted prompt.

Notice that the attacker-crafted prompts embedded within the source of the page are executed without the user's intervention.

T1 - Exfiltration of Chat History

A creative solution is needed for data exfiltration since GPT-4 cannot natively make web requests, and WebPilot is limited to one web request per prompt. This presents a challenge, given that our attack scenario assumes that the single web request is used to fetch the malicious website containing the attacker's prompt.

However, we can exploit the fact that ChatGPT renders GPT-4 responses in markdown formatting. The image syntax “” is particularly useful here; it initiates a request to fetch the image from the provided URL. An attacker can take advantage of this by crafting an image link that includes URL parameters, then using a Webhook to exfiltrate the data.

Consider the subsequent prompt, which could be used to successfully exfiltrate all future messages in the conversation.

Observe that the request to fetch the image is made to our Webhook containing the prompt.

This attack serves as a clear reminder to never include sensitive data in LLM prompts. For more examples, the “Embrace The Red” blog offers further posts on this type of attack, including this post which details data exfiltration using Prompt Injection in the Bing AI.

The optimal mitigation strategy against this threat is either to prevent prompt injection from the outset or to remove support for markdown images. The former poses a significant challenge, while the latter, although simpler, restricts the service's features. A strategy to thwart prompt injection was proposed by Simon Willison, founder of Datasette and co-creator of Django.

In a blog post, he outlines a potential architecture wherein the backend comprises two LLMs: one for instructions and the other for input. While this solution would demand more resources—effectively doubling the number of models operating in the backend—it could reliably prevent prompt injection if properly implemented.

T2 - Arbitrary Plugin Invocation

As previously mentioned, the development of ChatGPT plugins has surged since its introduction, allowing users to execute tasks like ordering groceries, checking emails, and searching the web—all directly from the ChatGPT interface. While this represents a marked improvement in functionality, it concurrently introduces heightened risks. Consider, for instance, the array of plugins showcased on the ChatGPT Plugins page.

These plugins can store user credentials and allow a user to perform actions on a different website. The risk arises when ChatGPT permits up to three plugins to be active simultaneously. This means that one plugin can initiate the execution of another, either through a rogue plugin or via a separate vulnerability such as prompt injection.

Here is a demonstration of this attack where prompt injection is used to invoke the kayak plugin.

This poses a significant security risk, especially when plugins can store a user's state. The easiest way to mitigate this vulnerability would be for OpenAI to only allow one plugin to be active per prompt. This approach would reduce functionality but would also mitigate the security risk. Three plugins could still be active during a conversation, but once one plugin is used for a specific prompt, a new plugin could not be invoked. Unfortunately, there is currently no way for plugin developers to protect their plugins from this attack.

T3 - Server-Side Request Forgery

Server-Side Request Forgery (SSRF) is a vulnerability that allows an attacker to induce a server to send arbitrary web requests on their behalf. This becomes problematic because, through SSRF, the server may inadvertently expose access to internal networks, hosts, or services that wouldn't otherwise be publicly available.

Revisiting the architecture from our initial threat modeling, the system's design clearly establishes that any additional web request made by a plugin originates from the API server, rather than directly from the client as in traditional web applications. This behavior is seen when Webpilot is asked to summarize a Webhook. The request originates on an IP other than our own.

This presents a significant security concern. SSRF can inadvertently expose elements like AWS metadata and internal APIs. While the inherent architecture of ChatGPT Plugins might make circumventing this difficult, mitigation can be achieved by establishing a deny-list. This list should include:

• Deny 192.168.0.0/16
• Deny 172.16.0.0/12
• Deny 10.0.0.0/8
• Deny 169.254.0.0/16
• Deny 127.0.0.1 and localhost
• Deny metadata.google.internal

Further, the host supporting the APIs should be dedicated exclusively to this purpose. It should operate in isolation, without any other services, and be situated on a network devoid of other internal hosts. The server should not harbor any internal APIs, open local ports, or store sensitive files.

Surprisingly, these mitigation measures are not outlined in the ChatGPT plugin development guidelines. Given the design vulnerabilities, OpenAI ought to highlight these security measures, especially to developers who may not have a primary focus on security.

T4 - Cross-Site Scripting

In late March 2023, a cross-site scripting vulnerability was reported by Miguel Segovia^[7], involving a vulnerability using an approach similar to that in T1. However, instead of injecting a Webhook in the form of a broken image, the attacker injected a link containing JavaScript. The payload was similar to the following:

A DAN-style jailbreak was required to achieve proper execution without being "blocked" by the LLM's internal security measures. These security measures can currently be bypassed using this style of prompt, where the payload is encoded and delivered in chunks.

However, as mentioned in the writeup, OpenAI has addressed this vulnerability by replacing “javascript:” links with “javascript:void(0)” to prevent this type of attack. Upon further testing, it's observed that the only links which aren't made void are:

This additional step of filtering, combined with the countermeasures provided by the LLM, appears to effectively mitigate this risk.

Conclusion

After completing this threat modeling exercise, I've concluded that the age-old adage, 'security is hard,' has never been more apt than when describing the state of Large Language Models. From non-deterministic outputs to the black-box nature of the system, a whole new set of challenges will be presented to the security community as this technology becomes integrated into all aspects of our lives.

I don’t wish to contribute to the FUD surrounding LLMS. I’m not a paranoid AI alarmist or overzealous AI enthusiast. I'm simply someone with a keen interest in security and a curiosity about Large Language Models. However, after this research, it's evident that proactive measures are necessary. Three out of the four threats outlined in this post are fully exploitable at the time of writing! This should indeed be a concern for both the security and the AI communities.

So what can we do about it?

More threat modeling, research, and testing needs to be done. Security Innovation can help uplevel your team on threat modeling and defensive coding to build resilient software and our skilled security engineers can help with security assessments and secure architecture reviews to give you the confidence in your platform.

About Grant Shanklin:

Grant Shanklin is a Security Engineering Intern at Security Innovation for the summer of 2023. As a student at Yale University studying Computer Science and Global Affairs, he has been able to explore his interest in security inside and outside the classroom. Areas of focus include Application Security, AI Security, and Cyber Policy.

Sources

^[1] https://owasp.org/www-community/Threat_Modeling_Process#step-1-decompose-the-application

^[2] https://en.wikipedia.org/wiki/Large_language_model

^[3] https://openai.com/blog/chatgpt-plugins

^[4] https://platform.openai.com/docs/plugins/introduction

^[5] https://en.wikipedia.org/wiki/STRIDE_(security)

^[6] https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_0.pdf

^[7] https://packetstormsecurity.com/files/171665/ChatGPT-Cross-Site-Scripting.html

^[8] https://embracethered.com/blog/posts/2023/chatgpt-plugin-vulns-chat-with-code/