Auto loader and schema changes

Hi @azure_learner ,

Welcome to the Microsoft Q&A Platform.

you’re on the right track.

Schema drift detection and evolution should happen only once, at the ingestion edge (Landing → Bronze). This is where raw feeds can introduce new or unexpected columns, and Auto Loader is purpose-built to handle them incrementally (including capturing anomalies in _rescued_data). Running schema evolution again from Bronze → Silver or Silver → Gold doesn’t add value — it just reprocesses the same anomalies at higher cost.

Layered Approach

• Bronze: accept everything, lose nothing.
  Allow Auto Loader to evolve schemas, capture rescued data, and surface drift for observability.
• Silver/Gold: serve exactly what we promised.
  Schemas here should be explicit, stable contracts for downstream BI, ML, and apps.
  No auto-evolution — instead, enforce constraints and data quality checks.

Practical Benefits

1. Lower compute & complexity — schema inference isn’t free; don’t repeat it unnecessarily.
2. Stable contracts — Silver/Gold tables remain predictable for consumers, avoiding unexpected breaks.
3. Better governance — Unity Catalog works best with well-defined schemas for lineage, permissions, and sharing.
4. Controlled change management — additive changes can flow intentionally from Bronze to Silver/Gold (with defaults, backfills, and communication); breaking changes are handled via versioning (e.g., _v2 tables or compatibility views).

Unity Catalog Best Practices

• Default to managed tables unless you have a specific need to control raw storage externally.
• You’ll still get governance, lineage, and access management with managed tables.

Special Case

The only time you might consider schema evolution beyond Bronze is if you have a “staging Silver” that’s effectively acting as another raw landing zone (e.g., direct partner feeds).
Even then, it’s usually clearer to model that as a second Bronze.

Medallion Architecture (Bronze → Silver → Gold)

What is the medallion lakehouse architecture?
This article outlines the layered approach of Bronze, Silver, and Gold tables, emphasizing schema stability and governance in Silver/Gold layers.
🔗 learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion

If you have any further questions, feel free to respond in this same thread.