Apache Spark

Overview

Apache Spark is the general-purpose distributed engine of the data world: one system for data engineering, data science and machine learning, on a single machine or a cluster of thousands of nodes.

The execution model is a driver program coordinating executor processes on workers. Each application gets its own executors, which stay up for the whole application and run tasks in multiple threads, so applications are isolated from one another.

Work is decomposed into jobs, then stages, then tasks, a task being the unit sent to a single executor. Cluster management is pluggable across standalone, Hadoop YARN and Kubernetes.

Adaptive query execution is the feature that quietly changed Spark's reputation for needing hand-tuning. It uses runtime statistics to re-plan queries mid-flight, and has been on by default since 3.2.

Its breadth is the reason it persists. The same engine handles batch and streaming, distributed ANSI SQL, machine learning through MLlib, and a pandas API, in Python, SQL, Scala, Java or R.

Spark Connect is the newer architectural shift: a thin client sends unresolved logical plans over gRPC and gets Arrow-encoded results back, so Spark can be embedded in applications, IDEs and notebooks without shipping a driver.

Be aware of the trade-off there: the RDD API is not supported through Spark Connect, so code reaching below the DataFrame level still needs classic mode.

Scale claims are concrete rather than marketing. The project states many organisations run clusters of thousands of nodes, the largest known being 8,000, and that it works well up to petabytes.

What to watch: GraphX is marked deprecated on the project's own home page, and from 4.0 brokers, Connect and tools require Java 17 while clients and Streams need Java 11.

Versioning is now predictable, with quarterly feature releases from 4.3.0 onward, annual major releases, and the final release of a major line supported as LTS for 18 months.

Key features and capabilities

The same headings are used for every query & processing engines entry, so two tools can be read side by side.

Execution model

• A driver program coordinating long-lived executor processes on worker nodes
• Work decomposed into jobs, stages and tasks, with task-level parallelism inside each executor
• Adaptive query execution re-plans from runtime statistics, on by default since 3.2
• Spark Connect adds a client-server model over gRPC, returning Arrow-encoded batches

Languages and APIs

• Python, SQL, Scala, Java and R
• DataFrame and Dataset APIs alongside SQL, plus a pandas API on PySpark
• The RDD API remains JVM and classic-mode only, and is unsupported through Spark Connect
• Spark Connect covers most PySpark APIs since 3.4 and most Scala Dataset APIs since 3.5

Data sources and formats

• Built-in file formats, Parquet, ORC, JSON, CSV, text, XML, Avro, Protobuf and binary files
• Hive tables and JDBC databases, plus HBase, Cassandra and any Hadoop InputFormat
• Data Source V2 is the extension point for third-party connectors
• Delta Lake, Iceberg and Hudi plug in through connectors maintained by those projects

Performance and scale

• Adaptive execution coalesces shuffle partitions, splits skew, and converts sort-merge joins to broadcast joins
• Parallelism tuned through max partition bytes and shuffle partition settings
• In-memory columnar caching, join hints, and storage partition joins that avoid shuffles
• The project publishes no benchmark results of its own

What it suits

• Batch and streaming ETL, distributed SQL, large-scale exploration and machine learning training
• The site claims SQL analytics runs faster than most data warehouses, without publishing a benchmark
• GraphX is deprecated on the project's own home page
• No statement of unsuitable workloads is published, though it is not a transactional database

Ecosystem

• Libraries shipped with the project, Spark SQL, Structured Streaming, MLlib and the pandas API
• Table format connectors maintained by Delta Lake, Iceberg and Hudi rather than by Spark
• Claimed adoption of 80% of the Fortune 500
• Issues tracked publicly in Apache JIRA, with development on GitHub

Where it runs and what it costs

• Install with pip for a single node, or use the official container images
• Clusters on standalone, Hadoop YARN or Kubernetes
• Spark Connect embeds clients in applications, IDEs and notebooks
• Managed commercially by Databricks and by every major cloud provider

Pricing

Open sourceFree, Apache 2.0

Spark itself is free under Apache 2.0, with no paid edition and nothing sold by the foundation. Cost comes from the platform you run it on. Databricks, the vendor founded by its creators, prices in Databricks Units billed per second with committed-use discounts, and every major cloud sells a managed Spark of its own. When comparing, note that self-managed Spark trades licence cost for the operational work of cluster sizing, upgrades and tuning, which the managed services exist to absorb.

Vendor pricing page →

Demos and videos

• Apache Spark video playlists Channel or playlist · YouTube
• Introducing Apache Spark 3.0, keynote with Matei Zaharia Webinar or talk · YouTube

About Apache Software Foundation

Spark started at the UC Berkeley AMPLab in 2009, was open-sourced in 2010 and donated to the Apache Software Foundation in 2013, where it is licensed under Apache 2.0 and governed by its project management committee. The project describes itself as developed by hundreds of developers from hundreds of organisations, with over 2,000 contributors, and says 80% of the Fortune 500 use it. The principal commercial vendor is Databricks, founded in 2013 by Spark's creators, and every major cloud packages a managed Spark, but releases are controlled by the foundation rather than by any vendor.

Founded 2013 · Wilmington, Delaware · spark.apache.org