(4/6/2025 Update: Llama 4 is out and wins by a large margin!)

For several months, we’ve been evaluating how well vision models handle OCR. Our 

 focused on the closed-source models (GPT, Gemini, and Claude) and their comparisons to traditional OCR providers (AWS, Azure, GCP, etc.). 

However this week has been particularly exciting for open source LLMs. In the last few days we got:

 model. It seemed like a good time to put together a separate OCR benchmark post focusing specifically on the open source vision models.

 do not yet have native vision support, so they could not be included. 

Additionally, while there are tons of other open source OCR providers (Paddle OCR, Docling, Unstructured, etc.), this particular evaluation focuses on open source VLMs only. A follow-up benchmark will revisit traditional OCR models.

As with our main OCR Benchmark, the dataset and methodologies here are entirely open-source. You can review (and reproduce) the evaluation yourself via our 

After evaluating 1,000 documents for JSON extraction accuracy, cost, and latency, here are the major takeaways among the open-source models:

 are by far the most impressive. Both landed right around 

 accuracy (equivalent to GPT-4o’s performance).

 (72.2%), which is specifically trained for OCR.

, which is particularly surprising given that it's architecture is largely similar to Gemini 2.0 which topped the accuracy chart.

We spent some extra time verifying Gemma-3’s performance because it came in significantly below our expectations. We didn’t observe any single, standout failure mode. 

Instead, the errors reflected typical VLM challenges, including hallucinations, omitted values, and swapped words. Below is an example output that appears correctly formatted at first glance, but closer inspection reveals numerous inaccuracies and missing data.

For a more details on methodology and scoring check out the original benchmark report: 

The primary goal is to evaluate JSON extraction from documents. To evaluate this, the Omni benchmark runs Document ⇒ OCR ⇒ Extraction. Measuring how well a model can OCR a page, and return that content in a format that an LLM can parse.

We are using GPT-4o as a judge. Passing in the OCR result + JSON schema using the structured extraction mode. We compare that to the ground truth JSON values to determine accuracy. Several LLMs with structured output were evaluated as judge (including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash). GPT 4o most consistently conformed to the JSON schema.

A ground truth accuracy is calculated by passing in the 100% correct text to GPT 4o along with the corresponding JSON Schema. There's always some variability with GPT even given the perfect text input, and we typically see 99% (+/-1%) accuracy on the ground truth.

Getting up and running with the OS models was a bit more challenging than the hosted APIs from traditional providers. Here’s a quick overview of the infrastructure challenges and solutions we encountered:

 by far provides the best early access to models like Gemma-3 and Qwen-2.5-VL-32B but has stricter rate limits on some of the newer models (~200 requests/day).

, while having Gemma-3 support, does not recognize Gemma as a vision model. Compared to Openrouter which had the model and correctly configured image support.

 has really scalable inference but typically lags on supporting the newest models. But for models like Llama 3.2 this was easy to connect.

. However it was a bit of a pain to set up for each provider, so we opted for the hosted versions.

. Initial testing was done on Openrouter, but we needed a bigger rate limit to run the full benchmark. 

These hosting setups may change as new providers add vision model support or increase rate limits. But these are the endpoints live in the benchmark runner today.

Questions or suggestions? Feel free to open an issue on GitHub or reach out on our community channels.

The best open source OCR models

Vision models (specifically VLMs) are the next and possibly final leap forward in OCR. Right out of the box models like 4o and Llama 3.3 do an incredible job on the documents where traditional OCR models fail. They do particularly well with tables, color coded charts, infographics, and handwriting.

They’re getting cheaper too. On average we see ~$4/1000 pages for GPT 4o vision. Which is getting pretty close to Azure OCR at $1/1000 pages.

So how to do we make VLM perform more reliably? We’ve learned a lot in the last several months deploying these models. Our findings boil down to:

Small volumes of fine tuning data makes a huge impact on accuracy. Especially on specific document formats.

High quality labeled data is particularly expensive to produce for this application. And we need lots of training data.

We should build an infinite PDF generator

For our purposes, training data means a straightforward input-output pair: image in, Markdown out. Now proper synthetic data is hard. But documents have enough built-in structure to set some natural limits on complexity. We’ve seen some terrible documents come through, but there is a logical upper bound to how much content can be crammed into a single page.

, our open source document ⇒ markdown library. This takes any document format, turns it into a series of images, then uses a VLM to transcribe each image into Markdown. 

You can test out the demo instance here: 

JSON ⇒ Markdown ⇒ Image ⇒ LLM ⇒ Markdown ⇒ JSON

Convert that JSON into Markdown, following the same document schema. Supplement with some generated text.

Convert that JSON into HTML, again using the same schema.

Apply post-processing to clean up or degrade the image.

Which gives us three sets of training data:

: Train the vision model to parse image data and produce clean Markdown. This is where 80% of extraction inaccuracies come from.

: Improve extraction from clean markdown.

: If we’re feeling ambitious, we can train the model to jump directly from a series of images to JSON, no Markdown required. Although it’s likely this approach will only work on smaller page documents.

Our PDF generation starts by stacking page elements in a logical manner. Headings before text blocks, tables in appropriate sections, and so on. We then fill these elements with generated values that reflect the variety you’d find in real documents.

We break down pages into three high level concepts:

: These are the building blocks of the page. Each element might be a heading, a paragraph, a list, or a table. We introduce variability here; lists can be ordered or unordered, arranged horizontally or with wrapping text, and styled differently.

: This refers to the overall “look and feel” of the page. Maybe the PDF is crisp and digital, or maybe it’s been printed, marked up, scanned, and then recompressed. The image degradation/noise/blur logic can be applied retroactively to any document.

: Once we have a basic layout, we apply further tweaks. We might change the font family, alter the language, experiment with different page sizes, or even tilt and rotate the layout for added realism.

These are the variations we currently apply:

In our workflow, we generally work with two types of synthesized documents:

: Starting with a general schema, we randomly produce entire documents. This includes content, layout, and various page conditions.

: If we have a known document structure (like a standard tax form) we automate the form filling with representative data. We then apply the same page conditions as with fully generated documents.

The world exchanges billions of PDFs every day, but luckily a lot of them are the same templates. Synthetic generation for templated forms is a pretty well established practice, and how a lot of traditional single document OCR models are trained.

For templated documents, we have two main approaches:

: Rebuilding the document layout in HTML. This is a huge pain, and we haven’t really found a way around it. But it’s the most reliable way to get a “fillable” variant of any form. Fingers crossed someone like Figma creates a really good 

: If the document has fillable form fields, we can simply map our JSON schema to these fields. This is much faster and requires less overhead.

The example below uses the fillable IRS Form. We start with a clean form, populate it with random data, and then pass it through the document degradation flow.

At the end of the day we want synthetic docs to be indistinguishable from real docs. For maximum realism, each document needs a coherent theme. An income statement shouldn’t suddenly morph into a refrigerator manual halfway through.

To avoid this, we generate content generation hierarchically, with that each section logically dependent on the one before it. This is a lot slower, since we’re processing page elements synchronously. but keeps the narrative consistent throughout the page.

This is where LLMs are a huge unlock for synthetic data. By itself, model outputs become pretty repetitive. For example, an LLM would typically do a terrible job at randomly generating an income statement (numbers and line items would be overly simplistic).

. And as long as we can feed in enough external randomization, the LLM can create plausible text. In this example, we’re giving it a company, the type of document it’s working on, the text element we want it to generate (i.e. footnote, definitions, etc.). Then we pass in the HTML table from the previous step as context.

Example below of the hierarchical content generation. The table element will be generated first, and footnotes, justifications, definitions, etc. will reference the parent table element.

Lots of documents have embedded images. This is one of the harder elements to incorporate. So far we’ve kept to logos / branding, which is pretty easy to conditionally render on a page.

We start with a data set of a few thousand companies (pretty much the Russell 2000).

Assign a probability of a Header / Footer containing a logo, and pass the company name reference downstream so that any references in the text align with the imagery. That way, the entire page feels more cohesive, rather than just randomly stamped with a logo.

We use a mix of labeled and synthetic data for training. And it’s unlikely the human labeled data element will go away any time soon. We’re always finding uniquely bad documents that we need to account for.

However labeled data for our use case is particularly time consuming to produce. Compared to something like image classification, document ⇒ Markdown takes way more time and nuance.

Redundant verification. Manual labeling is very error prone. People just aren’t particularly well suited for retyping entire documents by hand.

Data ownership. We’re limited to representative documents that are public domain. And public facing documents often differ significantly in layout from internal documents.

(manually annotating docs to pass time on a flight)

Until we hit 100% on OCR + extraction accuracy, this is going to be a continuous work in progress. And on that note, we are actively hiring for engineers who want to really dig in on document extraction. If you’re interested, check out our 

! As part of this, we’ve published an open-source validation dataset consisting of:

As mentioned earlier, our main goal was LLM-based document extraction. To benchmark this, we validated the following workflows:

In this flow, we ran OCR on a document and passed the resulting text to an LLM along with the JSON schema for extraction. We only evaluated providers on their OCR accuracy, using GPT-4o structured output for the extraction.

For multimodal LLM providers, we will run a separate test of direct extraction without the OCR step (GPT 4o & Anthropic PDF).

We launched the benchmark with the following providers, and now that the test runner is live, it’s easy to add more:

Infinite PDF Generator

There's a lot of foundational work that goes into building a HIPAA compliant application, both regulatory and technical. 

As always, it's about 80% paperwork and 20% engineering. And while working with Large Language Models (LLMs) isn't vastly different from traditional machine learning, a few risk areas stand out:

Processing and storing Personally Identifiable Information (PII) in unstructured formats.

Here are some architectural patterns we've adopted to address high-risk the more LLM specific vulnerabilities. The main considerations are:

PII Substitution at LLM Request Time: Substitute PII before sending requests to the LLM to minimize risk.

Understanding Contracts with LLM Providers: Ensure that contracts with third-party providers clearly define how they handle and protect PII.

A lot of our specific architecture is related to AWS, but the principles and tools are similar across other cloud providers.

Chat, and increasingly voice, applications are making it very easy to take in and store protected information, potentially without realizing it. An obvious example is taking in data via structured input vs chat messages.

In the first scenario above, it's pretty easy to know that the SSN should be treated as PII, and subsequently stored in an encrypted manner.

The chat scenario however becomes much more difficult. You need to parse each message to identify any PII that is disclosed by the user. 

Whether you use a third-party or self-hosted LLM, implementing PII redaction in messages enhances security. However, simple redaction isn't enough for LLM applications since the models need context to function correctly. Replacing a user's name with "**** *****" can lead to unpredictable results.

Instead we create a lookup table that substitutes the PII for a semantically similar value. That means `Dan Smith` is substituted for a randomly generated `John Baker` at query time. And the real user information is stored in a secure fashion. 

Deploying a LLM is hard. They’re huge (Llama3 70B is a 

) and require 1000x the compute that a traditional web server might need. Even if models get to half the size, it’s likely most companies stick to dedicated model providers.

There is increasing demand for clearly defined terms governing model use and the ownership of prompts and inputs. Especially with stories of OpenAI’s questionable 

. To address these concerns, many model providers now offer Business Associate Agreements (BAAs) or Data Processing Agreements.

Despite these offerings, many health tech organizations still prefer to use self-hosted models. This trend is driven by the desire for greater control over data security and compliance with regulatory requirements, ensuring that sensitive health information is handled appropriately and securely within their own infrastructure.

 that likely pass muster under HIPAA requirements. The process is a little opaque (just email baa@openai.com) and the standard copy is not public, so it’s difficult to fully ascertain.

 is to go through the Microsoft Azure managed services which has specific coverage for 

AWS Bedrock is a service that AWS has published the EULA for each model provided on the Bedrock platform. Note the terms in each of these differ from the official terms of underlying model providers. In the AWS variants, there’s very clear protections around user data.

Here are the legal terms of use for each model class available on AWS Bedrock:

The OSS Models (Llama / Mistral) are covered under the standard AWS Customer agreement. While Anthropic, Cohere, and Stability have distinct agreements covering use. Each agreement should be validated independently, but on initial review they each contain language that’s in keeping with most HIPAA compliant agreements.

 The technology provided by Anthropic to AWS to enable Customer’s access to the Services 

does not give Anthropic access to Customer’s AWS instance

, including Prompts or Outputs it contains (“Customer Content”). Anthropic does not anticipate obtaining any rights in or access to Customer Content under these Terms. As between the Parties and to the extent permitted by applicable law, Anthropic agrees that Customer owns all Outputs, and disclaims any rights it receives to the Customer Content under these Terms. Subject to Customer’s compliance with these Terms, Anthropic hereby assigns to Customer its right, title and interest (if any) in and to Outputs. 

Anthropic may not train models on Customer Content from Services

. Anthropic receives usage data from AWS to monitor compliance with the AUP (as defined below) and understand Service performance (“Usage Data”).


. For example, Anthropic receives notifications about the volume of Customer’s usage of the Services.

The explicit exclusion of Customer Content within training data,

Clarification that “Usage Data” does not include any Customer Content.

. Recipient will destroy Discloser’s Confidential Information promptly upon request, except copies in Recipient’s automated back-up systems, which will remain subject to these obligations of confidentiality while maintained.

Unlike GDPR, HIPAA does not explicitly provide a right for individuals to request data deletion. The primary data destruction requirements within HIPAA are focused on proper disposal mechanisms of (

How do you run a HIPAA compliant LLM

I want to do a quick dive into a new Omni feature that saves an enormous amount of headache for data engineers. This is our 

At the click of a button, Omni manages the full lifecycle of:

Reading realtime data from your warehouse

Hosting and running an LLM to transform data

Writing the enriched data back to the orginal warehouse

If you've been in the trenches of data engineering and pipeline building, you've probably built this exact flow a dozen times by now. And it only gets worse as more and more tools are added to the "Modern Data Stack". Eventually everyone ends up with a rube goldberg of Airflow, dbt, Fivetran and a handful of Lambda's to do the actual transformation work.

On top of the orchestration headache, each new platform you add to the flow increases infosec and data governance concerns.


Even simple enrichment tasks can become a data orchestration nightmare

If you've ever been tasked with doing some form of data enrichment or cleanup, it usually involves quite a few steps. Lets go through a basic example. Imagine you're using a tool like Fivetran to pipe data from Zendesk over to Snowflake, and you want to run sentiment classification on each issue. The steps would probably look like:

Set up and deploy a sentiment model. You might pick a model from Huggingface, or use one of the AWS/Azure text analytics services.

Create a Lambda to call the sentiment model with your data.

Create a new column or table for your output data.

Set up a listener on your warehouse for any new data. You would likely setup Snowflake Streams, or SnowPipe, to continuously pick up on new rows.

Run a batch job to backfill of all your existing data.

This setup isn't terrible if you're doing a single data transformation, but it quickly balloons if your requirements become more complex.

If you're working with an unstructured data source like support tickets you'll need to do more than a sentiment model. It's likely you'd want to run categorization, topic identification, find common complaints, etc. And if you need to test out 2-3 different variants of each model to evaluate results, it quickly becomes a huge engineering burden.

Unstructured data requires a lot of experimentation

LLMs have opened up a ton of new capabilities for data teams. A lot of previously inaccessible data is now much easier to access. But working with brand new data sets will always requires iteration and experimentation. Things like:

Which models give the best performance / cost ratio on your specific data

What set of categories are most appropriate for the dataset.

How do we fine tune prompts to improve accuracy

Today each iteration can take days of engineering time. It's not easy to spin up a dozen LLMs to do these kind of test runs!

Focus on transformation, not the orchestration

Our focus is on empowering you to focus on the transformative aspects of data engineering, rather than getting bogged down by the orchestration. With Omni’s In Warehouse transformations, you're equipped to implement powerful LLM capabilities directly within your existing data environment.

This means you can run complex data transformations like sentiment analysis, categorization, and topic identification without the hassle of managing multiple integrations or worrying about data governance issues. And instead of being stuck in the setup phase, your team can move off orchestration, and onto building something useful with that data!

Enriching data, without moving it

> "GPT would be great if it didn't hallucinate all the time" 

> "How do we make sure GPT doesn't hallucinate the wrong answer" 

> "OpenAI need to fix the hallucination problem"

Hallucinating, the popular term for GPT confidently spewing incorrect data. It's often cited as the main reason not to use an LLM in a production environment: "we don't know if it will hallucinate wrong information". The prevailing sentiment seems to be that the "hallucination problem" will simply get patched out in an new version.

While I sympathize with the desire, this represents a failure of understanding of how LLMs operate. The word "hallucination" is likely the culprit. It's easily framed as an 'error in thinking' that needs to be resolved. But in GPT's case, "hallucination" is part of thought. 

"Explain the iPhone like Abraham Lincoln"

"Write me some code to generate a 3d model of an Owl" (pictured above)

When GPT answers these questions, it's not looking up an actual Lincoln quote about iPhones. Solving a new problem requires stringing together data in a novel fashion. Eliminating "hallucination" eliminates the magic that separates GPT from a Google search. 

More data helps, but can only get you so far

The easist option would be to keep feeding GPT more data. Over time it should fix the problem itself right? Yes and no. While GPT 3.5 to GPT4 has some pretty obvious improvements, it's unlikely we'll see another huge jump from training data alone.

This is part of the reason OpenAi isn't full steam ahead on 

. Additional training data has diminishing marginal returns. Simply piling more data on top of the same model will less and less noticable effects. 

Hallucination is easy to solve if you *know* the right answer. Just take the GPT response and douple check it against the proper answer. But then, if you know the right answer to the question, why not pass it back directly rather than asking GPT.

But there are some cases where this works. LLMs fail pretty regularly when presented with null data (

Here's we have a user requesting a search and the api failing. 

In this scenario, we know the data returned by the API (null) and we know the LLM response. So we can run another request to the LLM with a fact-check query:

If the answer is 'YES', pass along the first result. Otherwise retry. 

This method is relatively effective for fact checking single responses. A frequent source of hallucination is overfitting to conversation history. By passing a context free request a lot of that surperflorus information is removed. 

However it still relies on the underlying LLM to answer correctly. A "who will guard the guards" scenario.  

The technology is new and I expect that LLM use cases will change over time. GPT is not the sanitized customer chat bot that many companies desire. No amount of prompt engineering will prevent GPT from making up a fact, or operating outside it's boundries. 

It is still highly useful for these cases, but as a classifying tool rather than a drirect response. (i.e. 'classify the follwing question into one of [x,y,z] categories`). 

But when some amount of creativity and problem solving are necessary , GPT's ability to imagine and construct unique responses can be invaluable. It can generate unexpected solutions and innovative ideas, leading to novel perspectives and insights that wouldn't be achieved with mere factual regurgitation.

The key to leveraging this technology is to acknowledge its strengths and weaknesses, to understand when to rely on its 'creativity' and when to validate its 'facts.' 

Do you need a database or a mathematical equation solved accurately every single time? Then, it might be best to resort to a more deterministic system. But when faced with abstract queries or problems that require an inventive approach, GPT truly shines.

"Hallucination" is part of thought

Lately there have been a lot of companies building AI 

 on top of LLM platforms. Essentially, connecting an AI model to a series of tools and allowing the LLM to interpret and coordinate a series of actions at the user's direction. This is generally referred to as the 

A couple popular libraries implementing this are 

. I've been building with both a fair bit lately, and noticed a really interesting trend. The best way to reliably interact with LLM Agents is to 

Give them simple, single purpose, functions

When you're building tools for an end user, you're typically not throwing them raw JSON and asking them to interpret it. Instead, you're giving them paragraphs, buttons, bulleted lists, etc. All the nice GUI elements we are used to.

Now while LLMs seem pretty proficient at parsing JSON, in practice they are much less reliable than if you pre-parse that JSON into a more clear value.

The techniques I highlight below have helped me reduce a lot of the variability in LLM Agent performance.

For the purpose of these examples, lets pretend we're dealing with a Realtor AI Agent. It'll help users shop for, compare, and purchase a home.

Now if I were typically building an API, I would think a generalized 

 endpoint would be valuable here. It would take in zip code, price range, bedrooms, bathrooms, acreage, etc. and return an array of houses that meet this criteria.

Unfortunately, in practice this performs quite poorly. The more arguments a function takes, the higher risk of the LLM "forgetting" what arguments go where.

 function is working fine for the first couple searches, but then through some miswording, or simply a LLM hiccup, the earlier search parameters are ignored, and the LLM decides to just search any house under $500k.

This is an issue that is going to come up time and time again if you have complex functions, and are relying on the LLM to use its conversation memory to handle requests.

So lets think, how would a user facing application typically go about handling this sort of functionality?

Rather than relying on the chat memory to correctly parse the search arguments out for each additional user request, we could emulate a 

. In a front-end application, you can imagine storing the current state of a filter component in some useState or redux store. Likewise here we can create a search_filters object as a store of value, and have the Agent tools update this state, rather than execute the search directly.

An updated set of functions might look like:

, and returning the result to the LLM to format and display. This way you've removed ambiguity, and you've created a more reliable store of information than the chat history.

This one is pretty straight forward. If you have an input that the LLM can get wrong, make sure you give it verbose error responses.

, failed requests can be retried. So your error responses can be used by the LLM to correct the input and resubmit.

The above example is pretty common. The LLM treating a zip-code starting with `0` as a number and just passing the last 4 digits. With it gets an opaque 500, the LLM is forced to come up with a (often wrong) justification. But here with a good error handling the LLM can auto retry the request.

Imagine dealing with a user that can only enter strings, and has zero front-end validation. The end tools need to have a combination of flexibility and graceful error handling.

LLMs read string. LLMs write strings. Nothing else.

Just as you need to give back well formatted error responses, you also need to format successful responses in a way that is easily parsed by the LLM.

In the above JSON response, there are a couple items that might be confusing to the LLM.

It might get this, it might not. It might round 1.5 to 2. It might assume that all bathrooms have a decimal value and start displaying 2.0, 3.0 for other bathrooms.

One again, seems clear to us, but definitely another opportunity for a mistranslation. Better to be specific.

Here we've made the response very explicit, and taken away some risk of the LLM mis-interpreting results. But we can improve it further.

A devloper is testing a bar. He orders 1 beer, 0 beers, 999 beers, -1 beers. Everything works fine. A user walks into a bar, asks where the bathroom is, the bar bursts into flames.

Some of the biggest issues I've had revolve around the LLM lacking data and hallucinating a response. If data is unknown to you, it's best to be explicit. This is especially true when you have inconsistant data between responses (i.e. some houses have acerage info, some do not).

It might seem overly verbose, but passing the LLM an explicit "unknown" or "no data on XYZ" will significantly reduce hallucinations.

If I'm designing an API endpoint to search for homes, I expect it should return an Array of JSON. And if no homes are found, that array will be empty. Return type should always be an array.

Unfortunately with an LLM this tends to create a lot of uncertainty. Often they seem afraid of not returning anything, and will take the `[]` as an opportunity to hallucinate a few homes for us.

Here the AI has hallucinated a few homes for us because it didn't have any information to go on. What might seem obvious to us (empty array === no houses found) will not always translate to the LLM.

Or an even better response would include which filter is the limiting factor in this search. Since right now the LLM doesn't know whether there are no houses at all in 94114, or if there are none under $100,000.

Building tools with LLMs is a constant rollercoaster ride. Some days you're amazed at all the insights an LLM can pull from data, or its ability to chain together tools to accomplish a task.

Other days you're yelling at the screen while GPT tells the user a house has 

Blog

The best open source OCR models

Infinite PDF Generator

How do you run a HIPAA compliant LLM

Enriching data, without moving it

"Hallucination" is part of thought

Treat the AI Like a User

Turn Documents into Data