Azure Function cost and performance question

Hello,

Welcome to Microsoft Q&A,

• Azure Function is suitable for incremental loads (200-500 records) if you use a Premium Plan or App Service Plan (for longer execution times), but not ideal for full loads (2-3 hours) due to timeout limits.
• For full loads, consider Azure Automation, Azure Data Factory, or Azure Batch as they handle long-running tasks and large datasets better.
• Bandwidth Costs: Minimal if both services are in the same Azure region. Outbound transfer costs depend on regions and data volume.
• Performance: Ensure SQL Server is optimized for bulk inserts and high-volume data. Use batching or compression if necessary to manage network and database load.

Incremental Loads: Azure Functions (Premium or App Service Plan)

• Full Load: Azure Automation, Data Factory, or Batch.

Please Upvote and accept the answer if it helps!!

Hi Ananay Ojha, Thanks for Reaching out Microsoft Q&A

I support Divyesh Govaerdhanan, yes you can run all 19 scripts in a single Function App using Timer Triggers to schedule them daily at 11 PM, with automatic scaling and a pay-per-runtime cost model.

Additionally, I want to share few points and documents

1.Is Azure Function a good service for it (or we can consider any other) like azure automation Account?

• Azure Functions → Best fit for your daily incremental loads. Supports scheduling (Timer Trigger), scales automatically, and has strong integrations with SQL/Key Vault.
• Azure Automation → Better suited for infra/maintenance tasks (patching, VM jobs). It can run Python, but not ideal for data-heavy ETL-like workloads.

To get clear idea on azure automation refer: https://learn.microsoft.com/en-us/azure/automation/overview

• If full loads are rare, you might run them in Azure Batch or Azure Container Instances (ACI) instead of Function

2.What will be bandwidth cost (provided SQL server is hosted on Azure VM) any performance bottlenecks expected?

• If your SQL Server VM and Functions/Batch/ACI are in the same Azure region, there’s no extra data transfer cost; if they’re in different regions, inter-region bandwidth charges apply.

Refer document for Bandwidth Costs: https://azure.microsoft.com/en-us/pricing/details/bandwidth/

The main performance bottlenecks will come from Jira API response speed, SQL bulk insert efficiency, and network latency if services run in different regions.

3.We are focused to deliver first requirement ASAP, but what if someday we need to fulfill 2nd requirement?

• Daily incremental loads (200–500 records, 20–25 min) → run smoothly on Functions Consumption/Flex plan.
• Full loads (17–20M records, 2–3 hrs) → exceed the 60-min limit, so use Functions Premium/Dedicated (unlimited runtime, higher cost) or handle them with Azure Batch (parallelized) / Azure Container Instances (on-demand, no timeout).

Refer document for function app hosting options: https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale

4.But can we use Azure Pipelines with cron job trigger for this?

Yes,

• You can run existing Python scripts without refactoring, schedule them with cron-like YAML triggers, and define both incremental and full loads as separate pipeline stages.
• Pipelines integrate seamlessly with repos, Jira, and SQL, while automatically storing logs in DevOps for easy auditing.

Refer document: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml

Following are the few Caveats:

Azure Pipelines are designed for CI/CD (build and deploy) rather than long-running ETL/data jobs, so while they work, Microsoft doesn’t recommend them for production ETL scenarios.

Runtime Limits: Microsoft-hosted agents allow up to 6 hours per job (enough for your 2–3 hr full load, but not longer), while incremental runs (~25 min) are safe.

Refer: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#timeouts

Scalability: Pipelines run sequentially on an agent and don’t auto-scale, so parallelizing 19 scripts requires splitting into multiple jobs/stages.

Cost: Microsoft-hosted agents charge per pipeline minute after the free tier, while self-hosted agents avoid this but essentially put you back to managing a VM.

Operational Fit: Using Pipelines for ETL may raise governance concerns since they’re not the recommended service for production data workloads.

I would suggest if the priority is to deliver ASAP with minimal changes, yes, you can use Azure Pipelines with cron triggers. It will run your Python scripts almost the same way as your current scheduler.

For the long term, Functions + Batch/ACI is the more cloud-native and scalable design. You might want to start with Pipelines (quick win) and later migrate to Functions/Batch.

I hope this helps you clarify your doubts. If you have any further queries, I am happy to assist