Azure Data factory V2 for each loop run twice in pipeline with couple of seconds delay

Question

Azure Data factory V2 for each loop run twice in pipeline with couple of seconds delay

Kartheek Vasantha 0

Hi,We had one incident in our production environment of Azure data factory v2 instance, where for-each iterator loop run twice in one single run of pipeline with some couple of seconds delay.

In foreach loop iterator, we are doing some data load steps based on some criteria. Due to loop running twice, Data copy activities has duplicated data into target tables.

Please let me know, any details you need to investigate further.

1 answer

Your answer

Answer 1

Hello Kartheek Vasantha,

The unexpected double execution of a ForEach loop in Azure Data Factory, resulting in data duplication, typically stems from an automatic platform-level retry following a transient internal error.

This occurs when the initial pipeline run succeeds but a failure acknowledgment triggers the service to reprocess the activity. Investigation requires analyzing pipeline run logs in the monitor section and querying diagnostic data in Log Analytics to identify retry events and duplicate iteration IDs.

To definitively prevent duplication, the solution is to design idempotent workflows by replacing simple insert operations with upsert logic or merge statements. Configuring copy activities to use built-in upsert capabilities ensures that repeated processing of the same data does not create duplicate records, rendering the pipeline resilient to such operational anomalies.

Please, let me know the response helps answer your question? If the above answer helped, please do not forget to "Accept Answer" as this may help other community members to refer the info if facing a similar issue. 🙂

Share via

Azure Data factory V2 for each loop run twice in pipeline with couple of seconds delay

1 answer

Your answer