Azure Data Lake Storage

Overview

The first thing to understand about Azure Data Lake Storage is that it is not a separate service. Microsoft's own wording is that it is a set of capabilities used with the Blob Storage service, unlocked by enabling one setting on a storage account.

That setting is the hierarchical namespace, and it is what turns a flat blob container into real directories. Renaming or deleting a directory becomes a single atomic metadata operation rather than a rewrite of every object beneath it, which is why analytics engines care about it.

Access is HDFS-compatible through the ABFS driver, so Spark and similar frameworks treat it as a filesystem, while the Blob APIs continue to work on the same data.

Security combines two models. Azure role-based access control handles coarse permissions, and POSIX ACLs set at directory or file level handle the fine-grained ones, which is closer to a filesystem than to S3's policy model.

Capacity is effectively unbounded. Microsoft publishes no limit on account size, file size or total data, with individual files from kilobytes to hundreds of terabytes.

Check the feature support table before committing, because some Blob features still do not work with hierarchical namespace: blob versioning, change feed, object replication and point-in-time restore are all unavailable.

Pricing follows Blob Storage rather than being a premium, and the request charges are where data lake workloads actually cost money, particularly the iterative write meter at $0.065 per 100 operations.

The strategic direction is Microsoft Fabric. OneLake is built on this storage and stores tables as Delta Parquet or Iceberg, exposing the same ADLS Gen2 APIs so existing tools, Synapse, Databricks and Storage Explorer connect directly.

Immutability is strong where it is supported: time-based retention and legal holds up to 400 years, validated by Cohasset Associates against CFTC, FINRA and SEC 17a-4 rules, though version-level policies need a flat namespace.

Key features and capabilities

The same headings are used for every object storage entry, so two tools can be read side by side.

Storage model

• Not a separate service, but Blob Storage with the hierarchical namespace setting enabled
• Containers, which are the same thing as file systems, holding real directories
• No published limit on account size, file size or total data; block blobs reach about 190 TiB
• Blob versioning is not yet supported with hierarchical namespace; snapshots and index tags are preview only

Performance and scale

• Default 40,000 requests a second in larger regions, 20,000 elsewhere, raisable on request
• Default bandwidth up to 60 Gbps ingress and 200 Gbps egress in larger regions
• Up to 3,000 requests a second against a single block blob
• Block-based uploads, with Microsoft advising blocks above 256 KiB for throughput; no transfer acceleration product

Storage classes and lifecycle

• Hot, cool, cold and archive, plus a Smart tier that moves data automatically based on usage
• Minimum retention, 30 days cool, 90 cold, 180 archive, with early deletion prorated
• Rule-based lifecycle management for transitions and expiry, and Storage Actions for scale
• Archive is offline, rehydration can take up to 15 hours, and it supports only LRS, GRS and RA-GRS

Security and governance

• Azure role-based access control combined with POSIX ACLs at directory and file level
• Encryption at rest with Microsoft-managed or customer-managed keys, including cross-tenant keys
• Time-based retention and legal holds, from one day to 400 years, at container level with hierarchical namespace
• Validated by Cohasset Associates against CFTC 1.31, FINRA 4511 and SEC 17a-4(f)
• Version-level immutability is not yet supported with hierarchical namespace

Analytics features

• HDFS-compatible access through the ABFS driver, with Spark and Presto named by Microsoft
• Fabric's OneLake is built on this storage, holding tables as Delta Parquet or Iceberg
• Iceberg and Delta interoperate through metadata virtualisation, readable in both directions
• Query acceleration is billed separately, at $0.002 a GB scanned and $0.0007 a GB returned

Ecosystem

• Blob REST APIs, the Data Lake Storage APIs and the ABFS driver all operate on the same data
• Also NFS 3.0 and SFTP protocol support on the account, plus Blobfuse for mounting
• No S3-compatible endpoint is published, unlike several competitors
• Tooling includes Azure CLI, PowerShell, AzCopy, Storage Explorer and the OneLake file explorer

Where it runs and what it costs

• More than 70 Azure regions and over 400 datacentres
• Up to 250 storage accounts per region per subscription, 500 on request
• No on-premises deployment, though NFS 3.0 lets on-premises Linux clients mount a container
• Priced at Blob Storage rates rather than at a data lake premium

Pricing

Usage-basedHot storage at $0.0208 a GB a month

Billed at Blob Storage rates. In East US on locally redundant storage, hot costs $0.0208 a GB a month for the first 50 TB, cool $0.0152, cold $0.0036 and archive $0.00099, with retrieval at $0.01, $0.03 and $0.02 a GB respectively. Requests are where lake workloads bite: hot writes $0.065 per 10,000 and reads $0.0052, but iterative writes $0.065 per 100, and archive reads $6.50 per 10,000. Deletes are free and immutability adds no capacity charge. A free account includes 5 GB of hot storage for 12 months.

Vendor pricing page →

Demos and videos

• Azure Data Lake Storage Gen2 overview Product demo · YouTube
• Microsoft Azure channel Channel or playlist · YouTube

About Microsoft

Azure Data Lake Storage is a first-party Microsoft capability of Azure Storage rather than a standalone product, and is the successor to Data Lake Storage Gen1, combining its capabilities with Azure Blob Storage; a specific general-availability date is not published on the overview page. It is proprietary, with no open-source or self-hosted edition, governed by the Microsoft online services terms and the storage account service level agreement. Microsoft publishes a footprint of more than 70 Azure regions and over 400 datacentres. One terminology note Microsoft itself flags: container and file system mean the same thing here, as do blob and file.

azure.microsoft.com