Dataform

Overview

Dataform is Google Cloud's transformation service for BigQuery, and the single most important thing about it is the last three words. It targets BigQuery exclusively, with no adapters and no other destinations.

If you are on BigQuery, that focus is the appeal: it is a first-party service, it needs no separate vendor relationship, and it costs nothing beyond the BigQuery it runs.

Transformations are written in SQLX, plain SQL extended with an optional configuration block, which adds dependency management and lets Dataform build and visualise the dependency tree for you.

Reuse comes through JavaScript rather than Jinja: inline blocks, shared includes across a repository, and packages across repositories. Anyone expecting dbt's macro idiom should know that up front.

Execution is straightforward. Dataform compiles your code in real time into a SQL compilation result, then runs the actions in dependency order in BigQuery, syncing metadata to Knowledge Catalog afterwards.

Development happens in workspaces, editable virtual copies of a repository where you edit, compile and test before pushing to a Git branch, which is how development and production stay separated.

Promotion uses release configurations, which are templates of compilation settings, paired with workflow configurations that schedule runs of a compilation result.

Data quality is handled by assertions, checking uniqueness, null values or any custom condition, alongside the automated testing that the open-source core provides.

Licensing is split cleanly: Dataform core, the meta-language, is Apache 2.0 open source, while the managed service is a Google Cloud product offered at no charge.

Read the pricing note carefully, because free means no licence and no service fee, not no cost. You still pay for BigQuery compute, and Cloud Logging is required for every workflow run rather than optional.

Key features and capabilities

The same headings are used for every transformation & modelling entry, so two tools can be read side by side.

How transformations are written

• SQLX, plain SQL plus an optional configuration block, is the primary format
• JavaScript provides reuse, as inline blocks, repository includes and cross-repository packages
• Dataform core supplies dependency management, automated data quality testing and documentation
• Dependencies are declared in SQLX, and the dependency tree is built and visualised for you
• Jinja is not used, and macros, seeds and snapshots are not published as named features

Execution and materialisation

• Code compiles in real time into a SQL compilation result, then runs in dependency order
• All compute runs in BigQuery; Dataform has no engine of its own
• Tables, views and other SQL commands are configured through SQLX
• A workflow settings file sets the processing region, which must match the BigQuery dataset location
• A named incremental strategy catalogue and state-aware runs are not published

Testing and data quality

• Assertions are the built-in test type, checking uniqueness, null values or a custom condition
• Automated data quality testing is one of Dataform core's three stated capabilities
• The open-source command-line tool can unit test user-defined functions
• Data contracts, freshness checks and anomaly detection are not published

Environments and change management

• Development workspaces are editable virtual copies of a repository, compiled and tested before pushing
• Release configurations are templates of compilation settings, the promotion mechanism
• Workflow configurations schedule runs of a compilation result at a set frequency
• Native Git version control, with documented GitHub connection and continuous delivery
• Virtual environments, zero-copy clones and blue-green patterns are not published

Documentation and lineage

• Documentation is one of Dataform core's three core capabilities, alongside dependencies and testing
• The console visualises the dependency tree of a workflow
• Metadata syncs asynchronously to Knowledge Catalog during runs, where discovery lives
• Column-level lineage, exposures and a semantic or metrics layer are not published

Platform coverage

• BigQuery only, exclusively, with no adapters or alternative targets
• The narrowest platform coverage in this category, and the clearest trade-off to weigh
• A repository's region need not match the dataset, but the processing region must
• Available across more than 40 regions spanning the Americas, Europe, Asia Pacific, the Middle East and Africa

Where it runs and what it costs

• An open-source command-line tool for local development
• A managed service in the Google Cloud console, from development through to scheduling
• A REST API and Python client libraries for repositories, workspaces and invocations
• Orchestration through Cloud Scheduler, Cloud Workflows or managed Airflow
• No unit of pricing, because the service itself does not bill

Pricing

FreeFree service; you pay for BigQuery

Dataform is a free service. Google made it free on acquisition in December 2020 and it has stayed that way, with no tiers, seats, model metering or trial period, because there is nothing to buy. What you pay for is what it consumes: BigQuery queries at your own rate, which dominates, plus Cloud Logging, which is required for every workflow run rather than optional, and any orchestration services you add. Free here means no licence or service fee, not free to run, so a heavy workload on on-demand BigQuery pricing can still be expensive.

Vendor pricing page →

Demos and videos

• Manage SQL workflows and transformations in BigQuery Webinar or talk · YouTube
• Dataform quickstart, create and run a workflow Tutorial · cloud.google.com

About Google Cloud

Dataform was acquired by Google in December 2020 and made free to all users at the same time; the price was not disclosed, and Google's own pages publish nothing about the original company's founding, founders, headquarters or funding, which is a genuine gap. It is now a fully managed Google Cloud service, owned by Alphabet and documented alongside BigQuery rather than as a standalone brand. Dataform core, the open-source meta-language, is Apache 2.0 and actively maintained; the managed service sits under Google Cloud terms. There is no paid tier of any kind.

cloud.google.com