AWS Glue

Overview

AWS Glue is Amazon's serverless data integration service, and it is really three things sold together: a Spark ETL engine, a set of crawlers that infer schemas, and the Glue Data Catalog that the rest of AWS analytics reads from.

That catalogue is the part with the widest reach. It is a drop-in replacement for the Hive metastore, one per account per region, and it is what Athena, Redshift Spectrum and EMR query against.

Authoring happens in Glue Studio, a visual canvas that generates PySpark or Scala, or by writing the script yourself. Editing the generated script converts the job to script-only mode permanently.

Jobs come in three commands, Spark for batch, Spark structured streaming for continuous work, and a lightweight Python shell, with streaming sources including Kinesis, Kafka and MSK.

Compute is measured in data processing units of 4 vCPUs and 16 GB, across worker types from a quarter of a unit up to sixteen, with memory-optimised variants for heavier work.

Flex execution is the cost lever for non-urgent jobs, running on spare capacity at roughly a third less, though it is unavailable on the largest and memory-optimised workers.

Glue 6.0, generally available in August 2026, is the significant recent release: Spark 4.1.1, full Iceberg v3 support including the variant type, declarative pipelines, Arrow-native Python functions, and 30% lower pricing than earlier versions.

AI arrives as Amazon Q data integration, generating jobs from natural language, alongside longer-standing machine learning features for fuzzy matching, sensitive data detection and data quality.

One dated notice to plan around: Glue for Ray closed to new customers on 30 April 2026, with AWS recommending EKS and the KubeRay operator and publishing a migration path. Glue itself has no retirement date.

Note also what it does not have: there is no self-hosted gateway like Azure's, so on-premises sources are reached through your own VPC rather than an installed agent.

Key features and capabilities

The same headings are used for every elt & ingestion entry, so two tools can be read side by side.

Connectors and coverage

• More than 70 data sources published, managed as Glue connections with a quota of 1,000 per account
• Coverage includes relational databases, warehouses such as Redshift, Snowflake and BigQuery, and NoSQL stores
• On-premises access is through your own VPC rather than an installed agent, unlike Azure Data Factory
• Streaming sources are Kinesis Data Streams, Apache Kafka and Amazon MSK
• JDBC stores are reached only over TLS when configured for it

How pipelines are built

• Glue Studio's visual canvas generates PySpark or Scala; editing the script converts the job permanently
• Orchestration is jobs, triggers and workflows rather than a general control-flow activity library
• Triggers run on a schedule, on demand or on an event, with jobs and crawlers chained together
• Job run queuing holds runs that cannot start instead of failing them, up to a thousand
• Version control through a Git tab in Glue Studio

Loading and sync behaviour

• Job bookmarks are the incremental mechanism, remembering what has already been processed
• Bookmarks and bounded execution do not work on Lake Formation tables with cell-level filters
• Bounded execution and push-down predicates limit how much each run reads
• Change capture is covered by zero-ETL integrations and streaming rather than a general database CDC feature
• Auto scaling assigns workers only when needed, within account quotas of 500 or 1,000 units

Destinations

• S3 is the primary sink, with Redshift, JDBC stores and OpenSearch also supported
• Open table formats supported as targets are Apache Iceberg, Delta Lake and Apache Hudi
• Glue 6.0 adds full Iceberg v3 support, including the variant type with shredding
• Table maintenance is productised and billed, with meters for lake optimisation and materialised view refresh
• Catalogued data is immediately queryable from Athena, EMR and Redshift Spectrum

Security, monitoring and lineage

• IAM roles per job, with Lake Formation providing fine-grained access down to cell level
• Encryption at rest for jobs, catalogue metadata, bookmarks, connection passwords and logs
• Only symmetric customer-managed keys are supported, configured in three separate places
• CloudWatch metrics, continuous logging, the Spark web interface, job run insights and CloudTrail
• Glue's own pages do not enumerate compliance certifications, pointing to the AWS scope list instead

Extending it yourself

• Custom visual transforms can be added to Glue Studio and reused
• A full Glue API with language SDKs and a CLI
• Serverless job notebooks and interactive sessions, defaulting to five processing units
• PySpark or Scala with Glue extensions, plus configurable Python, JAR and file paths
• Amazon Q data integration generates jobs from natural language

Where it runs and what it costs

• 36 regions including two GovCloud regions, with FIPS endpoints in eight
• The largest and memory-optimised worker types exist in only eight regions
• Glue 6.0 is available in every region where Glue operates
• Billed by the second on data processing unit hours, plus catalogue objects and requests

Pricing

Usage-based$0.308 a unit-hour on Glue 6.0

Billed by the second on data processing unit hours. In US East, ETL jobs cost $0.44 a unit-hour on earlier versions and $0.308 on Glue 6.0, a 30% cut; Flex execution drops that to $0.29 and $0.203 for non-urgent work, and memory-optimised runs $0.52 and $0.364. Crawlers, interactive sessions, lake optimisation and materialised view refresh all bill at $0.44. The Data Catalog is free for the first million objects and million requests, then $1.00 per 100,000 objects a month. Zero-ETL ingestion is $1.50 a gigabyte. No time-limited trial is published.

Vendor pricing page →

Demos and videos

• AWS Glue overview Product demo · YouTube
• Introducing AWS Glue Flex capacity Tutorial · YouTube

About Amazon Web Services

AWS Glue is a first-party Amazon Web Services product, proprietary and closed source, although its helper libraries and samples are published openly. It became generally available in August 2017. It is sold pay as you go under the AWS customer agreement, billed by the second on data processing unit hours with no licence to buy and no reservations or savings plans published. It is available in 36 regions including two GovCloud regions, with FIPS endpoints in eight, though several of the larger worker types exist in only eight regions. No service level percentage is published on its own pages.

aws.amazon.com