A couple of months ago, I sat down with Steve Wilson, Chief Product Officer of Exabeam and a security expert with decades of infosec experience, to have a conversation. We delved into one of the most pressing issues in today’s enterprise landscape – data privacy and security in Large Language Models (LLMs). As more organizations integrate AI technologies into their operations, LLMs, like ChatGPT or Claude, have become indispensable tools. However, with this adoption comes a critical challenge — ensuring that sensitive data doesn’t get leaked into/by the LLMs. I had written about this hot topic earlier in a previous post of mine as well.

The Untapped Risk of LLMs

Steve’s extensive work in AI-driven cybersecurity highlights the growing threat landscape around LLMs. While LLMs are powerful tools for productivity and automation, they also come with significant risks, particularly around data privacy. Steve’s initiative, the OWASP Top 10 LLM Security Risks, aims to address these concerns. Initially launched as a small project, it rapidly gained traction, drawing hundreds of industry professionals who recognized the importance of understanding these risks.

However, despite the growing interest in LLM security, Steve pointed out that most enterprises, especially their CISOs, still struggle to grasp the full implications. While many organizations are concerned about the potential for data breaches, they often focus on theoretical risks rather than the concrete challenges posed by integrating LLMs into their systems. For example, there is a widespread fear that employees might unintentionally expose sensitive data through LLMs, but the deeper, systemic risks — such as how LLMs access, store, and retrieve data — remain underappreciated.

The Data Privacy Dilemma

A significant part of the conversation centered around the data privacy risks associated with LLMs. Steve emphasized that while enterprises often draft policies to control the use of AI tools, many fail to address the underlying privacy concerns. One of the most common examples is the use of public LLMs in enterprise settings, where sensitive data can be exposed without proper safeguards.

Steve mentioned a common fear among CISOs: the accidental leakage of confidential data through AI tools. This fear was amplified by a headline-making incident involving Samsung in 2023, where employees were believed to have exposed proprietary information through ChatGPT. While the specifics of the Samsung case remain unclear, it has become a cautionary tale in the corporate world. However, as Steve noted, this type of incident represents only a fraction of the real risks. The more pressing concern is that LLMs, when used in enterprise environments, have access to vast amounts of internal data, which can be inadvertently exposed through improper configurations or insufficient data governance policies. A more recent example in this regard is the case of Microsoft Copilot sharing sensitive information such as CEO emails and HR documents with customers “accidentally”.

Who Owns the Data Privacy Problem?

A recurring theme in the conversation was the question of responsibility. In many enterprises, there is confusion about who should be in charge of AI data privacy. Is it the responsibility of the CISO, the CIO, or the Chief Data Officer (CDO)? Steve observed that while CISOs are typically the ones tasked with drafting AI policies, they often lack the authority to enforce these policies, especially when it comes to data governance.

Steve explained that many CISOs focus on creating restrictive policies to prevent employees from using tools like ChatGPT (as in the Samsung case), but this approach is shortsighted. Instead, he advocates for organizations to embrace AI tools while implementing strong privacy safeguards. By purchasing enterprise versions of LLMs with built-in data privacy features and training employees on responsible usage, companies can reap the benefits of AI without exposing themselves to unnecessary risks.

This fragmented responsibility creates a gray area in which crucial decisions about data privacy are often delayed or ignored altogether. In Steve’s experience, many organizations treat data privacy in LLMs as someone else’s problem — until it’s too late. The result is that AI deployments in many companies are either overly cautious or recklessly permissive, with little middle ground.

The Case for Stronger Data Governance

One of the standout examples that I raised was about Snowflake, a leading cloud data warehouse company. Snowflake provides tools for data masking and column-level access controls, ensuring that structured sensitive data is protected from unauthorized access. However, these protections only apply at the query level. The data itself remains in plain text within the system, leaving it vulnerable if someone gains access to the backend. Moreover, this type of columnar masking is easier when the column is already identified as a specific type of PII, such as SSN, Credit Card number, etc. But, if it is a column that contains unstructured text, there is no easy way to detect PII within that text and no easy way to apply a mask on that. And this problem is not unique just to Snowflake. There are several databases, data lakes, and data warehouses that have the exact same problem today.

This raises an important question. In scenarios like this, who is responsible for ensuring that sensitive data is properly protected — the CISO, the CDO, or the CIO? Steve noted that in many cases, no one takes full ownership of the problem. This lack of accountability is one of the main reasons why data privacy in LLMs is such a complex and unresolved issue.

Moreover, the problem is exacerbated by the rapid adoption of AI technologies across various industries, including healthcare, finance, and legal. In highly regulated industries, like healthcare, where HIPAA compliance is critical, the risks of improper data handling are even more pronounced. Yet, as Steve noted, many organizations still proceed with AI deployments without fully understanding the privacy risks involved.

Shadow IT and the Explosion of GenAI Tools

Another significant risk factor is the rise of shadow IT — the use of unauthorized tools and software by employees. Steve explained that with the advent of generative AI (GenAI), every employee has become a potential risk factor. Tools like ChatGPT are used widely by knowledge workers across industries, often without the oversight of IT or security teams.

Rather than banning these tools, Steve advocated for a more balanced approach. He believes organizations should acknowledge that employees need access to AI tools to do their jobs efficiently. Instead of restricting their use, companies should invest in enterprise-grade versions of these tools that offer enhanced security features, like data masking and compliance with privacy regulations.

Steve pointed out that while many CISOs are aware of the risks, their response is often reactive. They create policies that prohibit the use of AI tools, but they fail to implement the necessary frameworks to enable their secure use. This reactive approach leaves companies vulnerable to data breaches and other privacy risks.

Building a Safer Future for LLMs

Steve’s insights made it clear that the industry is still in the early stages of addressing the privacy and security challenges posed by LLMs. While organizations like OWASP are working to develop standards and best practices, there is still a long way to go. In fact, Steve mentioned that the OWASP Top 10 LLM Security Risks, which was originally designed for software developers, has found a larger audience among CISOs and security professionals. This shift in audience underscores the need for cross-functional collaboration in addressing these risks.

Looking ahead, Steve emphasized the importance of building stronger governance frameworks for AI technologies. He believes that data privacy and security must become integral parts of AI adoption strategies, with clear roles and responsibilities assigned to CISOs, CDOs, and CIOs. Without this clarity, companies will continue to face the risk of data breaches and other privacy violations as they integrate AI tools into their workflows.

Call to Action

As LLMs become more entrenched in enterprise environments, the need for robust data privacy measures becomes increasingly urgent. Steve’s insights offer a sobering reminder that while AI has the potential to revolutionize industries, it also introduces new risks that many organizations are unprepared to handle. The key to mitigating these risks lies in proactive governance, clear accountability, and a commitment to educating employees and security teams about the unique challenges posed by LLMs.

In a world where AI adoption is accelerating, data privacy in LLMs is not just a technical problem. Rather, it’s a business-critical issue that demands attention at the highest levels of the organization. As Steve wisely noted, companies that fail to address these challenges today are likely to find themselves in the headlines tomorrow.

If you are interested in learning more about Steve’s viewpoints, I highly recommend that you read his book – The Developer’s Playbook for Large Language Model Security. It is pretty much on the same topic that we have been discussing in this post but goes into a lot more detail. The book delves into –

Understanding LLM Security Challenges: Explains why LLMs present distinct security issues compared to other AI technologies.
Identifying Risks and Vulnerabilities: Outlines common threats such as prompt injection, data poisoning, and AI hallucinations, while offering methods to detect and mitigate these risks.
Establishing Trust Boundaries: Emphasizes the importance of defining and maintaining critical trust boundaries within LLM applications to ensure secure operations.
Implementing Defensive Measures: Provides actionable steps for deploying defenses against top vulnerabilities, including input filtering, adversarial training, and rate limiting.
Integrating Security into Development: Advocates for embedding security considerations throughout the LLM development lifecycle, from initial design to continuous monitoring.

In the next post, I will write about pragmatic solutions for this problem space. In the meanwhile, please do share your thoughts on this topic.