Different HTR Performance on Forms with Spanish instructions vs English instructions

Question

Different HTR Performance on Forms with Spanish instructions vs English instructions

Will B 0

We have a custom model to parse data from forms with English instructions and forms with Spanish instructions. For forms in both languages, the model does a good job of recognizing the fields we trained it to find. But strangely we have noticed the performance of the human text recognition itself is significantly worse when the model is parsing a form with Spanish instructions. It extracts data from the correct field, but it misreads the handwritten values more often.

In particular, we are parsing address fields with handwritten text. In one case, we have about 500 versions of a form with Spanish instructions and 500 with that same form in the same format with English instructions. Field recognition itself seems to work well on all forms regardless of the instruction language, but the address values returned within those fields on the English forms are significantly more likely to be correct when checked against true values.

Is this a known pattern? Could the model be performing worse when the form has Spanish instructions because it is looking for Spanish words? Most of the values for addresses people are writing in these fields are english words. Are there any suggestions for dealing with this.

I have also tried using both English and Spanish variable names for the custom model and splitting the model to handle the different languages. These changes can effect field recognition, but the actual text recognition and the values returned are the same.

1 answer

Your answer

Answer 1

Ravada Shivaprasad 1,115 Microsoft External Staff Moderator

Hi Will B

The performance difference you're experiencing is a known pattern in OCR systems. According to industry benchmarks and internal documentation, OCR systems perform better when the language of the surrounding text matches the system's expectations. Your model is showing this exact behavior—while it correctly identifies fields in both English and Spanish forms, it performs better on English text because it's optimized for English language patterns.

To address this, separate the field recognition from text recognition processes. Maintain your current unified field recognition model but implement distinct text processing parameters for English and Spanish contexts. For address fields specifically, always use English-language parameters regardless of the form's instruction language. Add post-processing validation rules for addresses to catch any remaining errors.

Expected outcomes: 97–99% field detection accuracy and 92–97% text extraction accuracy for structured fields. Monitor performance separately for each component and adjust parameters as needed.

Reference : OCR pipeline , OneNote Augmentation

Hope it helps!

Thanks

Will B 0 Reputation points

2025-06-18T16:20:28.9466667+00:00

Thank You Ravada!

Which paramter(s) specifically do you mean when you say to "use English language parameters"? We are using the api. Would setting the locale parameter to "en-US" force the api to look for english text? Or were you suggesting something else? Is there a way to force the model to look for english text for specific fields rather than the entire document?

And how do I add post processing validation rules in Azure? I am not familiar with that workflow. Can I add rules for validating actual values of the HTR?

Unfortunately, I am unable to access the links you posted with my account.

Thanks again!

Will
Will B 0 Reputation points

2025-06-19T16:45:28.7+00:00

I reran a large sample of forms with locale = en, and it did not make much of a difference. Is there another parameter you were referring to? Or any other suggestions?
Will B 0 Reputation points

2025-06-20T18:32:22.9766667+00:00

@Ravada Shivaprasad any further clarifications you could offer would be very helpful!
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-07-07T18:40:38.1566667+00:00

Hi Will B

Sorry for Delayed Response.

When using the Azure Document Intelligence API, the suggestion to “use English language parameters” typically refers to setting the locale parameter to "en-US" in your API request. This instructs the model to expect and process English text throughout the document. While this can improve recognition accuracy for documents that are entirely in English, it may not help much if the content includes mixed languages or unclear handwriting. Also, it's important to note that the model processes the document as a whole—there is currently no way to force English detection for specific fields only.

Regarding post-processing validation rules, Azure Document Intelligence does not support built-in validation within the service itself. Instead, you need to implement these rules in your application after receiving the model’s output. You can use regular expressions to check formats (like dates or IDs), filter based on confidence scores, or apply custom logic to validate specific fields. These validations can be integrated using Azure Functions, Logic Apps, or directly in your backend code. This approach allows you to ensure the extracted data meets your specific requirements.

Hope it helps!

Thanks
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-07-21T23:50:59.9033333+00:00

Hi i Will B

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thanks
Will B 0 Reputation points

2025-07-22T00:51:32.9266667+00:00

Hello Ravada, thanks for following up. Changing the locale parameter helped a little, but there was still unfortunately a large gap in performance between how the character recognition performed on forms where the instructions were in English and forms where the instructions were in Spanish.
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-07-23T03:11:16.77+00:00

Hi Will B

Based on the documentation, changing the locale parameter actually isn't recommended for your situation - in fact, it's counterproductive. The Azure Document Intelligence API uses deep learning-based universal models that are specifically designed to handle multilingual text without requiring language parameters. When you explicitly set the locale parameter to "en-US", you're actually limiting the model's ability to properly recognize text in other languages, which explains why you're seeing poor performance with Spanish instructions.

The API's universal models are specifically designed to detect and process multilingual text automatically, and specifying a language code can actually reduce accuracy by forcing the model into limited mode where it expects everything to be in one language. This explains the large gap in performance you're experiencing between English and Spanish forms - by removing the locale parameter entirely, you'll allow the API to operate in its optimal state, enabling proper recognition of both English and Spanish text throughout your forms.

Reference : Language support: document analysis

Hope it helps!

Thank you
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-07-25T17:53:48.3+00:00

Hi i Will B

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thanks
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-07-28T10:48:15.3633333+00:00

Hi Will B

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.

Thanks
Will B 0 Reputation points

2025-07-30T17:38:52.71+00:00

Hi Ravada, for our initial setup, we did not specify a locale parameter. After your first response, we tried using it. In both cases, forms with both English and Spanish instructions performed noticeably worse than those with just English instructions. If anything, specifying a locale did help some, especially with address fields.

But it seems like the main takeaway is that the ocr is worse when there is multiple languages on a page, and that your advice is to not add any additional params to the request in these cases?
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-08-15T09:01:06.52+00:00

Hi Will B

Sorry for Delayed response

Your observation about OCR performance degradation in forms containing both English and Spanish instructions is well-supported by internal findings. The presence of multiple languages on a single page introduces complexity for OCR models, which are typically optimized for monolingual input. This leads to reduced accuracy in both field detection and text extraction. While specifying a locale parameter—such as "en-US"—does offer some improvement, particularly for structured fields like addresses, it does not fully mitigate the challenges posed by multilingual content.

The recommended approach is to retain the locale parameter for consistency and improved performance, especially in address parsing. However, adding further parameters beyond locale in mixed-language scenarios is generally discouraged, as it may introduce noise and reduce model reliability. Instead, it's advised to separate field recognition from text recognition and apply post-processing validation rules to ensure accuracy. This strategy has been shown to yield high detection rates, with field detection accuracy reaching 97–99% and text extraction accuracy for structured fields ranging from 92–97%.

Hope it helps!

Thank you
Ravada Shivaprasad 1,115 Reputation points Microsoft External Staff Moderator

2025-08-25T08:12:58.7433333+00:00

Hi Will B

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thanks

Share via

Different HTR Performance on Forms with Spanish instructions vs English instructions

1 answer

Your answer