How to distinguish handwritting vs printed text, and be able to understand where the handwritting is placed?

Hi Stikker Uipko,

Yes, this is a common real-world scenario when working with scanned delivery notes or invoices that include both printed and handwritten information.

You’re successfully extracting printed content using Azure Document Intelligence. However, you're encountering difficulties with:

Identifying and isolating handwritten notes such as “Not received”.

Mapping these handwritten annotations to the corresponding table rows or items.

Interpreting handwritten overwrites or strike-throughs in context.

To address this scenario effectively, you can consider the following approach:

1. Enable Handwriting Extraction

Ensure you're using the latest Azure Document Intelligence API version (e.g., 2023-10-31-preview) and enable the handwriting feature in your request:

"features": ["handwriting"]

This allows the model to extract handwritten content in addition to printed text.

2. Use Bounding Box Geometry for Association

Both table elements and handwritten annotations come with bounding box coordinates. You can use spatial analysis to match each handwritten note to the nearest table row or cell:

Compare the handwritten text’s position with the table row’s bounding box.

If it overlaps or is near a specific row, assign the note to that row programmatically.

This can be achieved by writing post-processing logic that evaluates the geometric overlap.

3. Add Custom Post-Processing Logic

Since the model does not natively associate handwriting with specific rows:

Build a custom script or logic layer that appends a "handwritten_notes" field to each row.

Use the bounding box comparison results to determine the correct mapping.

Example Output:

"tableRows": [
  {
    "rowId": 5,
    "fields": {
      "ItemNo": "330703",
      "Qty": "2"
    },
    "handwritten_notes": ["Not received"]
  }
]

4. Optional Enhancements

Use Azure Form Recognizer Studio to visualize bounding boxes and fine-tune your logic.

If working with large volumes, consider training a custom layout model to recognize handwritten overrides directly.

For advanced use cases, integrate with other services like Ink Analysis API (if you’re capturing digital pen strokes).

Apart from the above methods please ensure that you have

• Make sure handwriting is legible and has good contrast in the scan.
• Strike-throughs or overwritten digits may be inconsistently picked up, depending on clarity.

Let us know if you need further assistance—we’d be happy to help!

Best regards,

Chakravarthi Rangarajan Bhargavi

If this answer was helpful, please click "Accept Answer" and upvote it so others in the community can find it more easily.

Hello Stikker Uipko,

Thank you for your question regarding distinguishing handwriting vs. printed text and associating it with specific rows/columns in your scanned invoices/delivery notes.

Current Behavior

• Azure Document Intelligence is indeed strong at extracting printed/structured content (tables, fields).
• Handwritten text is detected, but it is not automatically anchored to table rows/columns when it’s overlaid or added after scanning. Instead, it is returned as free-floating text spans with their bounding box coordinates.

Thanking community member Chakaravarthi Rangarajan Bhargavi for the detailed explanation – your guidance around handwriting extraction, bounding box association, and custom post-processing is very valuable and addresses most of the customer’s scenario well.

To further complement the response and add clarity for the community, here are some additional points:

1. Layout API for structural context
   • In addition to enabling handwriting recognition, you may also consider running the Layout API in Azure Document Intelligence.
   • This provides the structural context (tables, bounding boxes, reading order), which when combined with handwriting extraction makes spatial matching of annotations to rows/columns more reliable.
2. Schema design clarification
   • Document Intelligence itself does not natively support extending a schema to dynamically attach handwritten notes to table rows.
   • Instead, you need to enrich the extracted schema downstream in their pipeline.
3. Alternative approach with custom classifiers
   • For common recurring notes such as “Not received” or “Qty changed”, training a custom classification or entity model can simplify the workflow.
   • This way, instead of handling raw text, customers can output structured flags such as received = false.
4. Strike-throughs and overwrites
   • While handwritten text is extracted, strike-through detection is not a native feature. Overwritten or crossed-out values often appear as degraded OCR results.
   • Additional business rules or heuristics are recommended to interpret such cases reliably.
5. Integration with Azure AI Search (optional)
   • If organizations want to make handwritten annotations searchable across scanned documents, the extracted notes can be indexed into Azure AI Search, enabling queries such as “find all invoices marked Not received”.

Please feel free to accept this as an answer.
Thank you for reaching out to the Microsoft QNA Portal.