Optimizing Bulk Merge into Azure SQL Hyperscale Without Staging Tables
Janice Chi
620
Reputation points
JDBC Bulk Update Support
Can the Microsoft JDBC driver handle true bulk updates (not just inserts), or is the recommendation always to use a staging table and in-database MERGE?
Are there driver settings (e.g., `sendStringParametersAsUnicode`, `packetSize`, `batchSize`) that improve throughput for batched updates/inserts at our scale (3k–70k events/sec)?
**TVP (Table-Valued Parameters) from JDBC**
Is TVP fully supported in the Microsoft JDBC driver for Azure SQL Hyperscale in production workloads?
What are the practical limitations of TVPs (row size, number of rows per TVP, transaction log impact) when we push 5k–20k rows per micro-batch?
Is there any benchmark data comparing TVP throughput vs. staging table MERGE for high-volume streaming?
**Latency & Concurrency**
At high concurrency (multiple Databricks streams writing into Hyperscale), what are the recommended limits for batch size and transaction duration to avoid log contention or throttling?
How does Hyperscale handle concurrent MERGE operations against the same table if we choose TVP or direct JDBC batch instead of staging?
**Deletes & CDC Handling**
For deletes (marked in CDC payloads), is it more efficient to batch DELETE statements, or to rely on a TVP MERGE that includes delete flags?
Any gotchas with delete-heavy workloads when avoiding staging?
**Temporary Tables**
If we create a **#temp table inside a stored procedure** to load rows and then MERGE, does Hyperscale optimize this as efficiently as a permanent staging table?
Is there any guidance on temp table usage in Hyperscale for high-frequency micro-batch merges?
**Bulk Copy API vs. TVP**
Bulk Copy (`SQLServerBulkCopy`) is very fast for inserts. Can it be combined with an update/delete pattern in Hyperscale, or is it strictly insert-only?
For insert-only bursts, is Bulk Copy recommended over TVP, and what are the trade-offs?
**Resiliency & Retries**
If a TVP or JDBC batch transaction fails mid-micro-batch, what is the best retry pattern: re-send the entire micro-batch, or break into smaller chunks?
Any Hyperscale-specific features (retry logic, partitioning hints) that can help avoid deadlocks or long rollbacks?
**Future Roadmap**
Is Microsoft planning any enhancements to allow direct MERGE INTO from external sources (e.g., Delta/Parquet in ADLS) to Hyperscale, avoiding the need for staging or TVP workarounds?
Any upcoming features in the Microsoft JDBC driver that improve bulk update/merge performance?
Azure SQL Database
Azure SQL Database
An Azure relational database service.
Sign in to answer