Data Factory in Microsoft Fabric

Overview

Data Factory in Microsoft Fabric is, in Microsoft's own repeated words, the next generation of Azure Data Factory. It is a software-as-a-service workload inside a Fabric workspace rather than a platform service in an Azure subscription, and the two are bought and billed separately.

The practical consequence is that there is no integration runtime to size or manage. Fabric handles the compute, and on-premises access uses the on-premises data gateway instead.

Data movement is not one item but three: Copy job for simplified movement with bulk, incremental and change capture in a single item, Copy activity inside a pipeline, and Mirroring for a near real-time replica in OneLake.

Copy job is where the incremental thinking now lives, and the documentation is refreshingly direct about when to use which: change capture when the source supports it, for inserts, updates and deletes with type 2 history, and watermarking otherwise, which catches only inserts and updates.

Transformation is Dataflow Gen2, a low-code surface with over 300 transformations built on Power Query, alongside notebooks, Spark job definitions, stored procedures, SQL scripts and a dbt job item.

Everything lands on OneLake, and shortcuts let pipelines work on live data in S3, Google Cloud Storage or on-premises sources without first copying it into the workspace.

Copilot is a headline feature rather than an afterthought, designing and editing pipelines from natural language, summarising existing dataflows and explaining errors, with its own capacity meter.

What does not carry across matters when migrating: datasets are gone, debug mode is gone, a list of connectors and many connector-level options are unsupported, and managed virtual networks and private endpoints are still listed as to be determined.

One more gap to weigh for regulated work: Fabric lists 42 regions with all workloads and no government cloud, against Azure Data Factory's 60 region entries including three government regions.

Key features and capabilities

The same headings are used for every elt & ingestion entry, so two tools can be read side by side.

Connectors and coverage

• More than 170 data sources published, spanning cloud, multicloud and hybrid
• Fabric-only connectors include the KQL database, SQL database in Fabric and SharePoint Online files
• On-premises access uses the on-premises data gateway, Windows-only with no container support
• The gateway scales to ten nodes with query-level balancing but has no auto-update or Key Vault integration
• Private networking uses a virtual network gateway in your own network, up to seven nodes, billed separately

How pipelines are built

• A low-code designer in the workspace, with nearly the same expression language as Azure Data Factory
• Fabric-only activities include Dataflow Gen2, Fabric notebooks, Outlook, Teams and semantic model refresh
• About 90% of Azure Data Factory's activities are available; SSIS is in preview and Synapse activities are not
• Schedule, event and tumbling window triggers, integrated into the Activator framework
• Continuous delivery without ARM templates, with item-level cherry-picking and built-in deployment pipelines

Loading and sync behaviour

• Copy job handles full and incremental copy, tracking the state of the last successful run automatically
• Change-capture incremental copy covers inserts, updates and deletes, with continuous sync and type 2 history
• Watermark-based incremental covers only inserts and updates, on a rowversion, date or integer column
• Update methods are append, merge, overwrite and slowly changing dimension type 2
• Throughput is intelligent optimisation rather than a unit you size, each unit consuming 1.5 capacity hours

Destinations

• Native sinks for Fabric Lakehouse, Warehouse, KQL database, SQL database and OneLake
• Fabric Lakehouse supports upsert to table, which the Azure Data Factory connector does not
• Destination support Azure Data Factory lacks, including S3, Google Cloud Storage, BigQuery, MySQL and Teradata
• Automatic table creation and write behaviour for Oracle and Snowflake, which its predecessor does not have
• OneLake is Delta-backed, with change data feed available on tables Copy job creates

Security, monitoring and lineage

• Workspace role-based access control with OneLake security integration
• Workspace identity replaces user-assigned managed identity for most Azure connectors
• Managed virtual networks and private endpoints are listed as still to be determined
• A monitoring hub gives cross-workspace views and stage-by-stage copy duration breakdowns
• Cross-tenant sharing through OneLake shortcuts has no Azure Data Factory equivalent; Purview lineage is not published

Extending it yourself

• Fabric notebook and Spark job definition activities, plus Databricks and HDInsight
• Azure Batch activity for arbitrary code, and function, web and webhook activities for HTTP
• A dbt job item runs SQL transformations on the Fabric warehouse with no-code setup
• An Apache Airflow job item orchestrates Python directed acyclic graphs
• Copilot authors, summarises and debugs pipelines from natural language

Where it runs and what it costs

• 42 Azure regions with all Fabric workloads, out of 54 listed; the other 12 are Power BI only
• No government or sovereign cloud region is listed, unlike Azure Data Factory
• A 99.9% service level commitment, matching its predecessor
• Billed in capacity units consumed per second, with no charge for external or pipeline activities

Pricing

Subscription$0.18 per capacity unit hour

Two layers. Capacity costs $0.18 per capacity unit hour pay-as-you-go, with overage at $0.54, and a one or three-year reservation saving around 41%. On top, each operation has a consumption rate: 1.5 capacity hours per unit of data movement, 0.0056 per non-copy activity run, and Dataflow Gen2 at 12 capacity units a second for the first ten minutes then 1.5. Microsoft's own worked example loads a terabyte of Parquet for about $0.20. Unlike its predecessor, pipeline and external activities are not charged. The 60-day trial excludes Copilot and Private Link.

Vendor pricing page →

Demos and videos

• Get started with Copy job in Fabric Data Factory Tutorial · YouTube
• Introduction to Dataflows Gen2 Product demo · YouTube

About Microsoft

Data Factory in Fabric is a first-party Microsoft service, proprietary and closed source, sold as part of Microsoft Fabric rather than separately. Fabric and this workload reached general availability in November 2023, bringing together Power BI, Synapse Analytics and Azure Data Factory on one software-as-a-service platform. Capacity is bought as Azure F SKUs, pay-as-you-go and pausable or reserved, with a free per-user licence plus a capacity as the minimum. Microsoft reserves the right to change consumption rates, with cancellation options if a change materially increases the capacity a workload needs.

microsoft.com