ADF Copy Activity Queue Stuck 7 Root Causes and Enterprise Fixes
A Copy activity that sits in queued state is rarely a random Azure Data Factory issue. It is usually a capacity, connectivity, or downstream throttling problem showing up at the orchestration layer.
The hard part is that the delay often appears in one place, while the cause sits somewhere else. A pipeline run may look healthy, the source may respond, and the sink may still reject throughput. Or a Self-Hosted Integration Runtime may be online but too CPU-starved to pick up new work.
This guide breaks down seven common root causes of ADF Copy activity queue delays, how to diagnose each one in an enterprise architecture, and what to change so high-throughput batch pipelines run predictably.
Start by separating queue time from transfer time
Before changing settings, confirm what kind of delay you are seeing.
A Copy activity has several phases. It must be scheduled, assigned to an Integration Runtime, connect to source and sink, read data, transform formats if needed, and write to the destination. Queue-related problems happen before or around resource allocation. Throughput problems happen after the copy starts moving data.
Check these places first:
Signal | What it tells you | Where to check |
Activity run status stays as `Queued` | ADF has not assigned or started execution | ADF Monitor activity runs |
Long duration before first bytes are copied | Resource allocation or connection setup is slow | Copy activity output and logs |
Copy starts, then throughput falls sharply | Source or sink is throttling | Copy activity output, source logs, sink metrics |
Many pipelines queue at the same time | Shared Integration Runtime or downstream bottleneck | ADF Monitor, Azure Monitor, IR metrics |
Only private network pipelines queue | DNS, firewall, private endpoint, or routing issue | Network logs, NSG flow logs, firewall logs |
The most useful troubleshooting pattern is simple:
Check whether the issue affects all Copy activities or only a subset.
Group failed or delayed activities by Integration Runtime.
Compare queue duration with CPU, memory, connection count, sink throughput, and firewall events.
Look for a single shared dependency.
The phrase ADF Copy Activity Stuck in Queue can describe many symptoms, but the fix depends on which shared dependency is saturated or blocked.
Seven root causes and how to fix them
1. Azure Integration Runtime concurrency limits are exhausted
Azure Integration Runtime is the default compute layer for cloud-to-cloud movement. It manages execution capacity for Copy activity, but it still has concurrency limits and regional service boundaries. When too many activities compete for the same runtime, new activities can wait in queue.
This often happens in large nightly loads where dozens or hundreds of tables start at the same minute.
Architecture diagnostics
Look for these signs:
Many Copy activities queue at the same trigger time.
The same Azure Integration Runtime is used by several pipelines.
Queue duration rises during batch windows and drops outside them.
Retrying the same activity later works without any change.
Source and sink metrics look normal while ADF shows delayed starts.
Check whether the pipeline uses high fan-out patterns, such as `ForEach` with high concurrency, many tumbling window triggers, or multiple upstream systems landing files at the same time.
Enterprise fixes
Spread demand instead of launching everything together.
Reduce `ForEach` batch concurrency where the downstream system cannot handle parallel writes.
Split workloads across multiple Integration Runtimes where architecture permits.
Stagger triggers by workload class, such as critical finance loads first, then reporting refreshes.
Use pipeline parameters to process table groups in controlled waves.
Review Azure subscription and ADF limits, then request quota increases where supported.
For large estates, treat Integration Runtime capacity as a shared platform resource, not a per-pipeline detail.
2. Activity fan-out creates more parallel work than the architecture can absorb
A common design pattern is metadata-driven ingestion. One pipeline reads a control table, loops through source objects, and runs Copy activity for each item. This works well until the control table grows from 50 objects to 2,000.
The Copy activity may queue because ADF is asked to schedule more work than connected systems can serve.
Architecture diagnostics
Check the pipeline shape:
Does one trigger launch many child pipelines?
Does each child pipeline run several Copy activities?
Are multiple schedule triggers overlapping?
Is `retry` creating duplicate pressure after a slow sink response?
Are table groups mixed without priority?
The issue is not just ADF. The source, sink, Integration Runtime, network path, and metadata store all receive a burst at once.
Enterprise fixes
Build controlled fan-out.
Use smaller batches from the metadata table.
Add workload classes such as small dimension tables, large fact tables, and archive loads.
Set separate concurrency levels for each class.
Prevent overlapping pipeline runs unless the design needs them.
Add dependency logic so heavy tables do not start together.
A good batch architecture has a queue by design. It is better to control that queue in metadata than to let the runtime queue randomly under pressure.
3. Self-Hosted Integration Runtime CPU or memory is exhausted
Self-Hosted Integration Runtime is often used when ADF connects to on-premises databases, file shares, SAP systems, private networks, or sources behind enterprise controls. If the machine running SHIR is overloaded, Copy activity can queue while waiting for an available worker.
The runtime may still appear online. That can mislead teams because “online” does not mean “healthy under load”.
Architecture diagnostics
Check host-level and runtime-level signals:
CPU remains high during the batch window.
Available memory drops and paging increases.
Disk queue length rises, especially when staging or compression is used.
Network throughput is close to the VM or host limit.
SHIR logs show worker allocation delay or connectivity retries.
One node handles most of the work while others sit lightly used.
Also review antivirus scans, backup jobs, file indexing, and patching windows. These can consume CPU and disk during the same window as ingestion.
Enterprise fixes
Right-size and scale SHIR like any other production compute tier.
Move SHIR to dedicated VMs instead of shared utility servers.
Add nodes to the SHIR cluster for scale-out and resilience.
Increase CPU and memory for sustained high-volume loads.
Place SHIR close to data sources to reduce network latency.
Exclude SHIR working directories from heavy security scans where policy allows.
Keep SHIR updated to avoid known runtime issues.
For critical pipelines, run at least two SHIR nodes. This reduces single-node pressure and supports maintenance without stopping ingestion.
4. Self-Hosted Integration Runtime nodes are uneven or poorly placed
Adding SHIR nodes does not automatically fix queueing. If nodes have different CPU sizes, memory, network paths, or access rules, the cluster can behave unevenly. Some nodes may pick up work and fail slowly, while others remain underused.
This often appears after rapid expansion, where one team adds a new node in another subnet or region without matching the original configuration.
Architecture diagnostics
Compare every node in the SHIR cluster:
VM size and CPU generation
Available memory
Network bandwidth
Latency to source and sink
Installed drivers
ODBC or JDBC versions
Firewall and proxy settings
Access to staging storage
SHIR version
Look for pipelines that queue or fail only when a specific node handles the job. SHIR logs and Windows event logs can help correlate activity timing with node health.
Enterprise fixes
Standardise the cluster.
Use the same VM size and configuration for all SHIR nodes.
Keep drivers and runtime versions consistent.
Place nodes in the same network zone unless there is a clear reason not to.
Validate connectivity from each node, not only from one working node.
Remove weak nodes from the cluster during batch peaks.
Use infrastructure as code to rebuild nodes consistently.
A SHIR cluster should be boring. If every node behaves the same way, troubleshooting becomes much faster.
Downstream systems can make ADF look stuck
5. Sink DTU or throughput throttling slows allocation and writes
Azure SQL Database, Azure SQL Managed Instance, Synapse dedicated SQL pools, storage accounts, and other sinks all have throughput limits. If the sink is saturated, Copy activity can appear stuck, slow, or repeatedly delayed.
For Azure SQL Database, DTU or vCore pressure is a frequent cause. When CPU, data IO, log IO, locks, or tempdb pressure is high, writes slow down. ADF may wait longer, retries may increase, and the next batch may queue behind unfinished activity.
Architecture diagnostics
Check sink metrics during the exact activity window:
DTU percentage or CPU percentage
Data IO and log IO
Deadlocks and blocking sessions
Tempdb pressure
Storage account throttling
Synapse workload queueing
Request rate and server-side latency
In ADF, compare rows copied per second and write duration. If read performance is strong but write performance is poor, the sink is the likely bottleneck.
Enterprise fixes
Tune the sink path, not only ADF.
Increase sink capacity during batch windows where cost policy allows.
Use staged loading into files, then bulk load into the database.
Partition large writes by date, key range, or source object.
Disable or defer non-critical indexes during large loads where safe.
Avoid many parallel writers into the same hot table or partition.
Use batch size and degree of copy parallelism carefully.
Separate ingestion tables from serving tables.
For SQL sinks, watch transaction log pressure. A load that looks small in row count can still be heavy if rows are wide, indexed, or updated rather than inserted.
6. Source throttling and file enumeration delays block the batch
The source can be the hidden limiter. REST APIs may enforce rate limits. SaaS connectors may restrict concurrent sessions. File stores with millions of small files may take a long time to enumerate before copying starts.
ADF then looks queued or idle because the activity has not reached the high-throughput transfer phase.
Architecture diagnostics
Look at source behaviour:
API responses include throttling or retry-after messages.
Source database waits increase during extraction.
File listing takes longer than the data transfer.
Many small files are processed one at a time.
Source owners report connection pool exhaustion.
Extract queries scan large tables without partition filters.
For file-based loads, separate listing time from copy time. A folder with lakhs of small files can perform worse than a smaller number of large files with the same total data size.
Enterprise fixes
Reduce source pressure.
Use incremental extraction based on watermark columns.
Partition reads by date, ID range, or source-native partition.
Avoid full scans for every batch.
Combine small files upstream where possible.
Use source-specific limits for parallel sessions.
Cache metadata instead of listing huge directories repeatedly.
Schedule heavy extracts outside source maintenance windows.
The goal is not maximum parallelism everywhere. The goal is steady throughput that the source can support without pushing back.
7. Firewall, private endpoint, or DNS misconfiguration blocks runtime access
Network misconfiguration is a classic reason for Copy activity queueing or long startup time. This is common in locked-down enterprise environments with private endpoints, firewalls, user-defined routes, proxies, and hybrid connectivity.
The tricky part is that a basic connection test may pass from one place while the actual runtime path fails.
Architecture diagnostics
Trace the path from the runtime to both source and sink.
For Azure Integration Runtime, check:
Managed private endpoint approval status
Private DNS zone records
Storage account firewall rules
SQL firewall rules
Trusted Azure services settings, where used
Region and network integration design
For SHIR, check:
Outbound access to Azure Data Factory service endpoints
Proxy authentication
DNS resolution to private endpoints
Firewall rules from SHIR subnet to source and sink
Port rules for database and storage protocols
TLS inspection or SSL interception behaviour
Common symptoms include activities that queue for a long time, then fail with timeout errors, or pipelines that work from public endpoints but fail after private networking changes.
Enterprise fixes
Validate the real execution path.
Test connectivity from each SHIR node.
Confirm DNS resolves to the expected private IP.
Review firewall denies during the activity time window.
Approve managed private endpoints and check their state.
Keep separate firewall rules for dev, test, and production runtimes.
Avoid mixing public and private endpoint assumptions in the same linked service.
Document the required outbound endpoints for SHIR and review them after network policy changes.
Network teams need timestamps, source IPs, destination FQDNs, and ports. Provide those details rather than a general “ADF is slow” ticket.
Set queue duration alerts in Azure Monitor
Queueing becomes expensive when teams discover it after the SLA is already missed. Azure Monitor can help detect the pattern early if ADF diagnostic data is sent to a Log Analytics workspace.
Enable diagnostic settings for Azure Data Factory and send pipeline runs, activity runs, and trigger runs to Log Analytics. Then build alerts around activity duration patterns, queued states, and repeated long waits.
A practical alerting model has three layers.
Alert layer | What to detect | Why it helps |
Early warning | Copy activities queued longer than the normal baseline | Gives support teams time before SLA breach |
Batch health | More than a set percentage of activities delayed in one window | Detects shared IR, sink, or network pressure |
Critical workload | Any priority pipeline queued beyond tolerance | Protects finance, compliance, and operational loads |
A query can filter activity runs by pipeline name, activity type, status, runtime, and time window. The exact schema can vary by diagnostic configuration, so validate field names in the workspace before building production alerts.
Good alert design avoids noise. Do not alert every time one low-priority table waits briefly. Alert when queue duration breaks the normal pattern for a workload class.
Best practices for high-throughput batch pipelines
High-throughput ADF pipelines succeed when orchestration, compute, network, source, and sink capacity are designed together.
Use these practices as a baseline.
Design workload waves
Do not start every table at the same time. Group workloads by size, priority, source system, and sink target. Run heavy tables in planned waves.
Keep metadata-driven control
Use a control table to store concurrency class, priority, partition strategy, expected duration, and retry rules. This gives operations teams a safe way to tune load behaviour without editing every pipeline.
Right-size Integration Runtime capacity
Review Azure IR and SHIR usage before each major onboarding. New sources and tables add shared pressure. Capacity planning should be part of release review.
Use partitioning with care
Partitioning can improve throughput, but too many partitions create connection storms. Match partition count to source and sink limits.
Separate landing from serving
For large loads, land data into storage first, then load curated tables. This reduces direct pressure on business-facing databases.
Make retries intelligent
Retries help with transient issues, but aggressive retry settings can worsen throttling. Use backoff patterns and cap retries for known capacity errors.
Track baselines
Store expected run duration, queue duration, rows copied, and throughput for key activities. Baselines make abnormal queueing easier to detect.
Run post-batch reviews
After a slow batch, review actual bottlenecks instead of only increasing parallelism. More parallel work often makes the weakest dependency fail faster.
The practical takeaway
When an ADF Copy activity stays queued, treat it as an architecture signal. Start with Integration Runtime capacity, then examine fan-out, SHIR health, sink throughput, source limits, and network controls. Most enterprise incidents trace back to one of these shared dependencies.
The best fix is not always a larger runtime or a higher concurrency setting. Often, the lasting fix is better workload shaping, cleaner SHIR design, stronger sink loading patterns, and Azure Monitor alerts that catch queue growth early.
Build the batch platform so it fails visibly, scales predictably, and gives every critical pipeline a fair path to execution.
Comments