AWS Glue Data Catalog

Overview

The AWS Glue Data Catalog is the metadata store the AWS analytics stack is built around. It holds the location, schema and runtime statistics of your data, organised into catalogues, databases and tables much like a relational database catalogue, and Athena, EMR, Redshift Spectrum, Lake Formation and SageMaker all read from it.

It is filled either by crawlers, which discover and catalogue sources automatically, or by defining tables by hand. It records schema inference, evolution and versioning, and column statistics for Parquet, ORC, JSON, ION, CSV and XML.

Unlike most catalogues it also does work on tables. It offers managed compaction for Iceberg tables, turning many small objects into larger ones, and manages Iceberg materialised views including their refresh scheduling.

Iceberg support is native rather than bolted on: it exposes an Iceberg REST endpoint so engines such as Spark can connect with SigV4 signing, though namespaces through that endpoint are single-level only. It also federates outward to remote catalogues, with documented targets including Snowflake and Databricks Unity Catalog, and can connect to an existing Hive metastore without migrating metadata.

Governance is delegated to AWS Lake Formation, which adds column, row and cell-level permissions, tag-based access control, credential vending and CloudTrail audit logs. There are no separate Lake Formation charges.

It is managed and serverless, with no self-hosted option, and priced per metadata object and per request, with a generous free tier. Published quotas are worth checking for large estates: a million tables per account and ten million partitions per table.

Key features and capabilities

The same headings are used for every table & technical catalogues entry, so two tools can be read side by side.

What it catalogues

• Catalogues, databases and tables or views; catalogues can be nested
• Billable objects: tables, table versions, partitions, partition indexes, statistics, databases, catalogues
• Also holds connections, user-defined functions, a schema registry and Iceberg materialised views
• Namespaces through the Iceberg REST endpoint are single-level only

Table formats

• Apache Iceberg v1 and v2, on S3 general-purpose buckets and S3 Tables
• Column statistics for Parquet, ORC, JSON, ION, CSV and XML
• Crawlers cover S3, RDS, Redshift and Apache Hive among others
• Transactional table formats are supported under open-table-format handling

Engines that can use it

• Native consumers: Athena, EMR, Redshift Spectrum, Lake Formation, SageMaker AI, Glue ETL
• Third-party engines through the Iceberg REST endpoint, for example Spark with SigV4 signing
• Federated catalogues readable by Redshift, EMR, Athena, Glue and Spark

Access control and auditing

• Lake Formation enforces column, row and cell-level permissions
• Grant and revoke model, plus tag-based and attribute-based access control
• Scoped credentials vended for table data in S3
• CloudTrail audit logs, and metadata encrypted with KMS

Interoperability

• Iceberg REST endpoint covering the standard namespace and table operations
• Connects to an external Hive metastore with no metadata migration
• Federates to remote Iceberg catalogues, including Snowflake and Databricks Unity Catalog
• Federated catalogues for Redshift, S3 Tables, DynamoDB, BigQuery and MySQL

Table maintenance and operations

• Managed compaction for Iceberg tables, toggled per table
• Manages Iceberg materialised views, including refresh scheduling and incremental updates
• Maintains column statistics and lineage
• Quotas per region: 1m tables per account, 10m partitions per table, 100k versions per table

How it runs

• Managed and serverless; no self-hosted option
• Regional endpoints across commercial regions plus GovCloud, with FIPS and dual-stack variants
• Used through the console, Glue Studio, CLI or API

Pricing

Usage-basedFree up to 1m objects and 1m requests a month

Storage: the first million metadata objects are free, then $1.00 per 100,000 objects a month. Requests: the first million a month are free, then $1.00 per million. Catalogue compute tasks, which cover generating statistics, Iceberg compaction and materialised view refresh, are $0.44 per DPU-hour billed by the second with a one-minute minimum. Rates vary by region. Lake Formation permissions carry no separate charge.

Vendor pricing page →

Demos and videos

• AWS Glue overview Product demo · YouTube (Amazon Web Services)
• Simplify and accelerate data integration with AWS Glue Webinar or talk · YouTube (AWS Events)

About Amazon Web Services

The Data Catalog is a component of AWS Glue, Amazon's serverless data integration service, and is operated as a commercial managed service under the AWS Customer Agreement rather than an open-source licence. There is no published source code and no releases to track; AWS maintains and updates it continuously. Governance of the data it describes is handled by a sibling service, AWS Lake Formation, which acts as the authorisation layer over catalogue resources.

aws.amazon.com