Azure Data Factory

Overview

Azure Data Factory is Microsoft's managed integration service for hybrid ETL and ELT, and it is the one to understand before Fabric, because Microsoft now describes Fabric Data Factory as its next generation.

Its model is six concepts: pipelines, activities, datasets, linked services, data flows and integration runtimes. Datasets describe the shape of data, linked services the connections, and activities do the work.

The integration runtime is the compute abstraction and the thing that makes it genuinely hybrid. Azure runtimes are serverless, the self-hosted runtime is an agent you install for on-premises and network-isolated sources, and the SSIS runtime is a managed cluster that runs existing SSIS packages unchanged.

That SSIS path is the strongest reason it still exists. Organisations with years of packages can lift them into a managed cluster with full compatibility rather than rewriting.

Transformation comes as mapping data flows, which run on a Spark cluster Microsoft spins up and down for you, now on Spark 3.4. Hand-coded work is delegated to Databricks, HDInsight, Synapse or Azure Functions through activities.

Incremental loading has four documented patterns, including a change data capture resource that needs no pipeline design and, notably, can run continuously, where pipelines themselves are batch only.

Check the connector lifecycle table before committing. Microsoft publishes dated end-of-support and removal milestones, and a long list of connectors, including Redshift v1, Oracle v1 and many SaaS sources, is already removed or scheduled.

Two structural gaps against Fabric: there is no Copilot or AI assistance for Data Factory, and Purview lineage covers only three activities, Copy, Data Flow and Execute SSIS Package.

Its compensating strength is reach. Pricing is published across 60 region entries including three Azure Government regions, which Fabric does not currently match.

Key features and capabilities

The same headings are used for every elt & ingestion entry, so two tools can be read side by side.

Connectors and coverage

• No total is published; the nearest first-party figures are over 80 sources and sinks, and 100 plus connectors
• Categories cover Azure, databases, NoSQL, files, generic protocols, Fabric, and services and apps
• The self-hosted integration runtime handles on-premises sources, as a Windows service or a container
• It scales to eight nodes, is shareable across 120 factories, auto-updates, and supports Key Vault and Private Link
• A managed virtual network with auto-created private endpoints covers network isolation

How pipelines are built

• Authored visually in Data Factory Studio over a JSON model, or through ARM, REST, .NET, Python and PowerShell
• A wide activity library including copy, data flow, lookup, control flow, and calls out to Databricks and Synapse
• A maximum of 80 activities per pipeline
• Schedule, tumbling window and event-based triggers, with parameterised datasets and linked services
• Continuous integration through ARM template export and import, with Azure DevOps and GitHub

Loading and sync behaviour

• Four documented incremental patterns, including a change data capture resource needing no pipeline design
• That resource can run continuously, whereas pipelines themselves are batch only
• Native change capture for SAP, Azure SQL, SQL Server, managed instance, Cosmos DB and Snowflake
• Copy throughput is measured in data integration units, up to 256 per copy activity
• Mapping data flows run transformation on a managed Spark cluster, in three compute tiers

Destinations

• Azure sinks including Blob, Data Lake Storage, SQL Database, Synapse Analytics, Cosmos DB and AI Search
• Fabric Lakehouse and Fabric Warehouse are both supported as sinks
• Third-party sinks including Snowflake, Oracle, SQL Server, SAP HANA, MongoDB, Salesforce and Dataverse
• Both Delta and Iceberg formats are published
• Loading into Synapse can use the COPY command or PolyBase

Security, monitoring and lineage

• Microsoft Entra ID with role-based access control, and system or user-assigned managed identities
• Data encrypted at rest by default, with customer-managed keys in RSA or RSA-HSM
• Customer-managed keys can only be set on an empty factory and cannot be disabled afterwards
• Managed virtual network and Private Link for self-hosted runtime control flow
• Purview lineage covers only Copy, Data Flow and Execute SSIS Package, and only a subset of stores

Extending it yourself

• A custom activity runs arbitrary code or executables on Azure Batch
• SDKs for Python and .NET alongside REST, PowerShell, the CLI and ARM templates
• Notebooks through Databricks and Synapse activities, plus Spark job definitions
• Script and stored procedure activities for arbitrary SQL, with an expression language
• Managed Airflow is offered as a capability with its own billing meters

Where it runs and what it costs

• Priced across 60 region entries, including three Azure Government regions
• Broader sovereign coverage than Fabric Data Factory, which lists no government regions
• A 99.9% service level commitment
• Billed across several meters rather than one, by activity run, unit-hour and vCore-hour

Pricing

Usage-based$1.00 per 1,000 activity runs

Several meters rather than one. In East US, cloud orchestration is $1.00 per thousand activity runs and data movement $0.25 a unit-hour, rising to $1.50 and $0.10 respectively for self-hosted runtimes, with pipeline activity at $0.005 an hour. Mapping data flows cost $0.27325 a vCore-hour general purpose, $0.34275 memory-optimised and $0.19225 compute-optimised. Read and write operations are $0.50 per 50,000 and an inactive pipeline $0.80 a month. Government regions charge $0.3125 a unit-hour. No product-specific free tier is published.

Vendor pricing page →

Demos and videos

• What's new in Azure Data Factory Channel or playlist · YouTube
• Azure Data Factory introduction Tutorial · learn.microsoft.com

About Microsoft

Azure Data Factory is a first-party Microsoft platform service, proprietary and closed source, with no self-hosted distribution; the self-hosted integration runtime is an agent of the managed service rather than the service itself. Version 2 reached general availability on 27 June 2018, with version 1 meters dating to 2015. It is sold pay-as-you-go under the standard Azure agreement with no licence to buy, and carries a 99.9% service level commitment covering both request processing and activity start times. Microsoft's published direction for it is migration, with tooling to assess and upgrade pipelines into Fabric.

azure.microsoft.com