There's a lot of foundational work that goes into building a HIPAA compliant application, both regulatory and technical. 

As always, it's about 80% paperwork and 20% engineering. And while working with Large Language Models (LLMs) isn't vastly different from traditional machine learning, a few risk areas stand out:

Processing and storing Personally Identifiable Information (PII) in unstructured formats.

Here are some architectural patterns we've adopted to address high-risk the more LLM specific vulnerabilities. The main considerations are:

PII Substitution at LLM Request Time: Substitute PII before sending requests to the LLM to minimize risk.

Understanding Contracts with LLM Providers: Ensure that contracts with third-party providers clearly define how they handle and protect PII.

A lot of our specific architecture is related to AWS, but the principles and tools are similar across other cloud providers.

Chat, and increasingly voice, applications are making it very easy to take in and store protected information, potentially without realizing it. An obvious example is taking in data via structured input vs chat messages.

In the first scenario above, it's pretty easy to know that the SSN should be treated as PII, and subsequently stored in an encrypted manner.

The chat scenario however becomes much more difficult. You need to parse each message to identify any PII that is disclosed by the user. 

Whether you use a third-party or self-hosted LLM, implementing PII redaction in messages enhances security. However, simple redaction isn't enough for LLM applications since the models need context to function correctly. Replacing a user's name with "**** *****" can lead to unpredictable results.

Instead we create a lookup table that substitutes the PII for a semantically similar value. That means `Dan Smith` is substituted for a randomly generated `John Baker` at query time. And the real user information is stored in a secure fashion. 

Deploying a LLM is hard. They’re huge (Llama3 70B is a 

) and require 1000x the compute that a traditional web server might need. Even if models get to half the size, it’s likely most companies stick to dedicated model providers.

There is increasing demand for clearly defined terms governing model use and the ownership of prompts and inputs. Especially with stories of OpenAI’s questionable 

. To address these concerns, many model providers now offer Business Associate Agreements (BAAs) or Data Processing Agreements.

Despite these offerings, many health tech organizations still prefer to use self-hosted models. This trend is driven by the desire for greater control over data security and compliance with regulatory requirements, ensuring that sensitive health information is handled appropriately and securely within their own infrastructure.

 that likely pass muster under HIPAA requirements. The process is a little opaque (just email baa@openai.com) and the standard copy is not public, so it’s difficult to fully ascertain.

 is to go through the Microsoft Azure managed services which has specific coverage for 

AWS Bedrock is a service that AWS has published the EULA for each model provided on the Bedrock platform. Note the terms in each of these differ from the official terms of underlying model providers. In the AWS variants, there’s very clear protections around user data.

Here are the legal terms of use for each model class available on AWS Bedrock:

The OSS Models (Llama / Mistral) are covered under the standard AWS Customer agreement. While Anthropic, Cohere, and Stability have distinct agreements covering use. Each agreement should be validated independently, but on initial review they each contain language that’s in keeping with most HIPAA compliant agreements.

 The technology provided by Anthropic to AWS to enable Customer’s access to the Services 

does not give Anthropic access to Customer’s AWS instance

, including Prompts or Outputs it contains (“Customer Content”). Anthropic does not anticipate obtaining any rights in or access to Customer Content under these Terms. As between the Parties and to the extent permitted by applicable law, Anthropic agrees that Customer owns all Outputs, and disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs. 

Anthropic may not train models on Customer Content from Services

. Anthropic receives usage data from AWS to monitor compliance with the AUP (as defined below) and understand Service performance (“Usage Data”).


. For example, Anthropic receives notifications about the volume of Customer’s usage of the Services.

The explicit exclusion of Customer Content within training data,

Clarification that “Usage Data” does not include any Customer Content.

. Recipient will destroy Discloser’s Confidential Information promptly upon request, except copies in Recipient’s automated back-up systems, which will remain subject to these obligations of confidentiality while maintained.

Unlike GDPR, HIPAA does not explicitly provide a right for individuals to request data deletion. The primary data destruction requirements within HIPAA are focused on proper disposal mechanisms of (

How do you run a HIPAA compliant LLM

I want to do a quick dive into a new Omni feature that saves an enormous amount of headache for data engineers. This is our 

At the click of a button, Omni manages the full lifecycle of:

Reading realtime data from your warehouse

Hosting and running an LLM to transform data

Writing the enriched data back to the orginal warehouse

If you've been in the trenches of data engineering and pipeline building, you've probably built this exact flow a dozen times by now. And it only gets worse as more and more tools are added to the "Modern Data Stack". Eventually everyone ends up with a rube goldberg of Airflow, dbt, Fivetran and a handful of Lambda's to do the actual transformation work.

On top of the orchestration headache, each new platform you add to the flow increases infosec and data governance concerns.


Even simple enrichment tasks can become a data orchestration nightmare

If you've ever been tasked with doing some form of data enrichment or cleanup, it usually involves quite a few steps. Lets go through a basic example. Imagine you're using a tool like Fivetran to pipe data from Zendesk over to Snowflake, and you want to run sentiment classification on each issue. The steps would probably look like:

Set up and deploy a sentiment model. You might pick a model from Huggingface, or use one of the AWS/Azure text analytics services.

Create a Lambda to call the sentiment model with your data.

Create a new column or table for your output data.

Set up a listener on your warehouse for any new data. You would likely setup Snowflake Streams, or SnowPipe, to continuously pick up on new rows.

Run a batch job to backfill of all your existing data.

This setup isn't terrible if you're doing a single data transformation, but it quickly balloons if your requirements become more complex.

If you're working with an unstructured data source like support tickets you'll need to do more than a sentiment model. It's likely you'd want to run categorization, topic identification, find common complaints, etc. And if you need to test out 2-3 different variants of each model to evaluate results, it quickly becomes a huge engineering burden.

Unstructured data requires a lot of experimentation

LLMs have opened up a ton of new capabilities for data teams. A lot of previously inaccessible data is now much easier to access. But working with brand new data sets will always requires iteration and experimentation. Things like:

Which models give the best performance / cost ratio on your specific data

What set of categories are most appropriate for the dataset.

How do we fine tune prompts to improve accuracy

Today each iteration can take days of engineering time. It's not easy to spin up a dozen LLMs to do these kind of test runs!

Focus on transformation, not the orchestration

Our focus is on empowering you to focus on the transformative aspects of data engineering, rather than getting bogged down by the orchestration. With Omni’s In Warehouse transformations, you're equipped to implement powerful LLM capabilities directly within your existing data environment.

This means you can run complex data transformations like sentiment analysis, categorization, and topic identification without the hassle of managing multiple integrations or worrying about data governance issues. And instead of being stuck in the setup phase, your team can move off orchestration, and onto building something useful with that data!

Enriching data, without moving it

> "GPT would be great if it didn't hallucinate all the time" 

> "How do we make sure GPT doesn't hallucinate the wrong answer" 

> "OpenAI need to fix the hallucination problem"

Hallucinating, the popular term for GPT confidently spewing incorrect data. It's often cited as the main reason not to use an LLM in a production environment: "we don't know if it will hallucinate wrong information". The prevailing sentiment seems to be that the "hallucination problem" will simply get patched out in an new version.

While I sympathize with the desire, this represents a failure of understanding of how LLMs operate. The word "hallucination" is likely the culprit. It's easily framed as an 'error in thinking' that needs to be resolved. But in GPT's case, "hallucination" is part of thought. 

"Explain the iPhone like Abraham Lincoln"

"Write me some code to generate a 3d model of an Owl" (pictured above)

When GPT answers these questions, it's not looking up an actual Lincoln quote about iPhones. Solving a new problem requires stringing together data in a novel fashion. Eliminating "hallucination" eliminates the magic that separates GPT from a Google search. 

More data helps, but can only get you so far

The easist option would be to keep feeding GPT more data. Over time it should fix the problem itself right? Yes and no. While GPT 3.5 to GPT4 has some pretty obvious improvements, it's unlikely we'll see another huge jump from training data alone.

This is part of the reason OpenAi isn't full steam ahead on 

. Additional training data has diminishing marginal returns. Simply piling more data on top of the same model will less and less noticable effects. 

Hallucination is easy to solve if you *know* the right answer. Just take the GPT response and douple check it against the proper answer. But then, if you know the right answer to the question, why not pass it back directly rather than asking GPT.

But there are some cases where this works. LLMs fail pretty regularly when presented with null data (

Here's we have a user requesting a search and the api failing. 

In this scenario, we know the data returned by the API (null) and we know the LLM response. So we can run another request to the LLM with a fact-check query:

If the answer is 'YES', pass along the first result. Otherwise retry. 

This method is relatively effective for fact checking single responses. A frequent source of hallucination is overfitting to conversation history. By passing a context free request a lot of that surperflorus information is removed. 

However it still relies on the underlying LLM to answer correctly. A "who will guard the guards" scenario.  

The technology is new and I expect that LLM use cases will change over time. GPT is not the sanitized customer chat bot that many companies desire. No amount of prompt engineering will prevent GPT from making up a fact, or operating outside it's boundries. 

It is still highly useful for these cases, but as a classifying tool rather than a drirect response. (i.e. 'classify the follwing question into one of [x,y,z] categories`). 

But when some amount of creativity and problem solving are necessary , GPT's ability to imagine and construct unique responses can be invaluable. It can generate unexpected solutions and innovative ideas, leading to novel perspectives and insights that wouldn't be achieved with mere factual regurgitation.

The key to leveraging this technology is to acknowledge its strengths and weaknesses, to understand when to rely on its 'creativity' and when to validate its 'facts.' 

Do you need a database or a mathematical equation solved accurately every single time? Then, it might be best to resort to a more deterministic system. But when faced with abstract queries or problems that require an inventive approach, GPT truly shines.

"Hallucination" is part of thought

Lately there have been a lot of companies building AI 

 on top of LLM platforms. Essentially, connecting an AI model to a series of tools and allowing the LLM to interpret and coordinate a series of actions at the user's direction. This is generally referred to as the 

A couple popular libraries implementing this are 

. I've been building with both a fair bit lately, and noticed a really interesting trend. The best way to reliably interact with LLM Agents is to 

Give them simple, single purpose, functions

When you're building tools for an end user, you're typically not throwing them raw JSON and asking them to interpret it. Instead, you're giving them paragraphs, buttons, bulleted lists, etc. All the nice GUI elements we are used to.

Now while LLMs seem pretty proficient at parsing JSON, in practice they are much less reliable than if you pre-parse that JSON into a more clear value.

The techniques I highlight below have helped me reduce a lot of the variability in LLM Agent performance.

For the purpose of these examples, lets pretend we're dealing with a Realtor AI Agent. It'll help users shop for, compare, and purchase a home.

Now if I were typically building an API, I would think a generalized 

 endpoint would be valuable here. It would take in zip code, price range, bedrooms, bathrooms, acreage, etc. and return an array of houses that meet this criteria.

Unfortunately, in practice this performs quite poorly. The more arguments a function takes, the higher risk of the LLM "forgetting" what arguments go where.

 function is working fine for the first couple searches, but then through some miswording, or simply a LLM hiccup, the earlier search parameters are ignored, and the LLM decides to just search any house under $500k.

This is an issue that is going to come up time and time again if you have complex functions, and are relying on the LLM to use its conversation memory to handle requests.

So lets think, how would a user facing application typically go about handling this sort of functionality?

Rather than relying on the chat memory to correctly parse the search arguments out for each additional user request, we could emulate a 

. In a front-end application, you can imagine storing the current state of a filter component in some useState or redux store. Likewise here we can create a search_filters object as a store of value, and have the Agent tools update this state, rather than execute the search directly.

An updated set of functions might look like:

, and returning the result to the LLM to format and display. This way you've removed ambiguity, and you've created a more reliable store of information than the chat history.

This one is pretty straight forward. If you have an input that the LLM can get wrong, make sure you give it verbose error responses.

, failed requests can be retried. So your error responses can be used by the LLM to correct the input and resubmit.

The above example is pretty common. The LLM treating a zip-code starting with `0` as a number and just passing the last 4 digits. With it gets an opaque 500, the LLM is forced to come up with a (often wrong) justification. But here with a good error handling the LLM can auto retry the request.

Imagine dealing with a user that can only enter strings, and has zero front-end validation. The end tools need to have a combination of flexibility and graceful error handling.

LLMs read string. LLMs write strings. Nothing else.

Just as you need to give back well formatted error responses, you also need to format successful responses in a way that is easily parsed by the LLM.

In the above JSON response, there are a couple items that might be confusing to the LLM.

It might get this, it might not. It might round 1.5 to 2. It might assume that all bathrooms have a decimal value and start displaying 2.0, 3.0 for other bathrooms.

One again, seems clear to us, but definitely another opportunity for a mistranslation. Better to be specific.

Here we've made the response very explicit, and taken away some risk of the LLM mis-interpreting results. But we can improve it further.

A devloper is testing a bar. He orders 1 beer, 0 beers, 999 beers, -1 beers. Everything works fine. A user walks into a bar, asks where the bathroom is, the bar bursts into flames.

Some of the biggest issues I've had revolve around the LLM lacking data and hallucinating a response. If data is unknown to you, it's best to be explicit. This is especially true when you have inconsistant data between responses (i.e. some houses have acerage info, some do not).

It might seem overly verbose, but passing the LLM an explicit "unknown" or "no data on XYZ" will significantly reduce hallucinations.

If I'm designing an API endpoint to search for homes, I expect it should return an Array of JSON. And if no homes are found, that array will be empty. Return type should always be an array.

Unfortunately with an LLM this tends to create a lot of uncertainty. Often they seem afraid of not returning anything, and will take the `[]` as an opportunity to hallucinate a few homes for us.

Here the AI has hallucinated a few homes for us because it didn't have any information to go on. What might seem obvious to us (empty array === no houses found) will not always translate to the LLM.

Or an even better response would include which filter is the limiting factor in this search. Since right now the LLM doesn't know whether there are no houses at all in 94114, or if there are none under $100,000.

Building tools with LLMs is a constant rollercoaster ride. Some days you're amazed at all the insights an LLM can pull from data, or its ability to chain together tools to accomplish a task.

Other days you're yelling at the screen while GPT tells the user a house has 

 bathrooms. The frustration is exactly the same as watching a user navigate your UI in a totally unpredicted way.

 mental model has helped me quite a bit in reducing the variability of responses. The superpowers of AI models comes at a cost. Your tools will be much more flexibile, but lose the predictability that we have come to love.

Blog

How do you run a HIPAA compliant LLM

Enriching data, without moving it

"Hallucination" is part of thought

Treat the AI Like a User

Ready to try OmniAI?