A Drop-In Execution Engine for Your Analytics Workloads

Development on Apache Spark started at Berkeley in 2009, and the first production release shipped on May 30, 2014. In the twelve years since, it has become the analytics workhorse for most of the large corporations in the world, across industries and scale, from seed-stage startups to Fortune 10 enterprises. Every year or so someone declares it old, past its peak, saddled with the JVM, and generally "legacy." And every year there is more of it. What accounts for the disconnect? In this post we'll walk through what we see across customer deployments and why we expect that in ten years there will still be a whole lot of Spark, and probably much more than there is today.

Infinite Scale

Spark scales very well. It’s not rare to see customers running workloads reading dozens of TB, while at the same time other customers process a few GB per workload. The result: for data engineering teams who don’t know how much data they’ll need to process, it’s a clean and easy decision to adopt Spark.

Network Effects

While it’s quite easy to use Spark, especially with PySpark, it’s not easy to deploy it and maintain it. But once the data platform adopts the tooling and learns how to maintain Spark, it is rarely motivated to migrate a piece of critical infrastructure to an unproven alternative, and instead is motivated to push more people to use Spark.

The Challenge of Migrating

Large companies can have thousands of jobs running at any given time, spread across the entire organization. The idea of pushing the various teams to migrate to a new platform is usually a complete non-starter. Oftentimes even gradually moving to systems like Ray is unwelcome due to the cost of maintaining multiple data platforms.

A First Class Citizen in the Data Lake

Delta Lake, Iceberg, and Hudi were each born with Spark as the reference implementation. The result is that Spark works well out-of-the-box with all three, while other systems are gradually adding support. Engineering teams want the best and most recent lakehouse technology and generally Spark supports it.

Extensibility

Spark is easy to extend without forking. Catalyst exposes optimizer rules, planning strategies, and catalog plugins. DataSource V2 lets anyone teach Spark to read a new system. User defined functions (UDFs) let teams introduce Python or Scala logic into the middle of a pipeline without leaving the framework. Plug-ins allow the introduction of new libraries into the system. The result is that the thing people would otherwise leave Spark to get, a new connector, a custom optimization, a domain-specific function library, usually shows up inside Spark instead.

The Competition

Flink is used for some streaming use cases, Ray for AI use cases, Trino for interactive SQL, DuckDB and Polars for data that fits on one machine. Data warehouses with proprietary engines are taking some share. But at this point nobody is really trying to invent a new full-fledged system to replace Spark. The competition is either specializing in a lane or building underneath it.

Improved Engines

In Spark, the underlying engine is not static. The API hasn’t changed much since DataFrames arrived, but adding Tungsten improved performance with whole stage code generation that’s close to the hardware, while query optimizations, fast paths, and new operators also make the same workload faster without code changes. Databricks added Photon, we produced Flarion, and open source brought Gluten and Comet. It’s possible to stay on Spark and get modern performance, similar to how PostgreSQL keeps getting better and adding functionality without the API changing.

Summing Up

The Spark API is likely going to be with us for a long time, but under the hood a lot is going to change. Piece by piece the engine is being replaced, and it's plausible that in ten years none of the original execution code will be left, while every job still runs and every DataFrame still looks the same. It's the Ship of Theseus, except in this version the ship gets faster with every plank. 

‍

A few weeks ago AWS shipped the Spark Upgrade Agent, an AI agent that migrates Spark jobs to Spark 4.0. You point it at a repo, it rewrites deprecated APIs, adjusts for behavioral changes, updates the build for Scala 2.13, submits the result to an EMR cluster, and iterates on failures until the job runs. It handles both Scala and PySpark, and it works the way you'd hope an agent would: plan, transform, validate, repeat. The potential payoff is large - newer Spark versions have better performance and years of accumulated bug fixes.

It looks like a good tool and data teams looking into a Spark migration should consider it, but what we’ve found is that in the enterprise, rewriting code isn’t the main impediment to upgrading Spark workloads. Spark programs are part of complex pipelines, parts of which are poorly understood or maintained, and making changes to a sensitive system is inherently risky. There’s no guarantee that the output data will actually remain the same, and data integrity is the fundamental challenge of completing such a migration.

Several companies have written in detail about major Spark upgrades, and the data integrity challenge is a recurring theme.

Slack

Slack's migration from Spark 2 to Spark 3 took about a year across 60+ EMR clusters and 40+ teams. They saw some code-level breakage: `RAND()` in join keys became an `AnalysisException`, some casts that Spark 2 tolerated started failing, the `Greatest` function handled NULLs differently than its Hive counterpart. Validation was a much larger effort. For billing pipelines, Slack required exact matches, building test tables from production data on Spark 3 and running `EXCEPT` and `COUNT` comparisons in Trino against the Spark 2 outputs, with a Python framework for digging into every discrepancy. The discrepancies weren't all bugs. Non-deterministic row ordering, timestamp variations, and genuine semantic differences between Hive and Spark implementations all produce diffs that need to be investigated. Some are noise, some are real regressions.

Uber

Uber's version of this is bigger and more instructive. They migrated from Spark 2.4 to 3.3 with over two million Spark applications running daily. The code transformation was automated with Polyglot Piranha, their structural rewrite tool. It parses the source code into an AST, matches patterns, and applies transformation rules, including inserting legacy flags like `spark.sql.legacy.allowUntypedScalaUDF` where old behavior had to be preserved. This scaled well. The problem that shaped the whole project was stated plainly: "We had over 40,000 Spark apps, so we couldn't decentralize the data validation." No staging environment, no test cases, no way to ask every team to eyeball their own outputs.

So the flagship engineering artifact of Uber's Spark upgrade wasn't actually a code migrator but Iron Dome: a shadow-testing framework which runs the migrated job against production inputs, rewrites output paths at runtime so results land in staging instead of production, puts guardrails at the Hadoop FileSystem interface so a misrouted write can't touch real data, then compares the shadow output against the production run and only marks the job migrated when they agree. 

Facebook

None of this is specific to the Spark 2-to-3 transition, or even to Spark versions. When Facebook moved Hive workloads onto Spark SQL back in 2017, they ran shadow pipelines writing to tables suffixed `_spark_shadow` so downstream jobs were never exposed, used count checks as a cheap first filter, and reached for full hash validation of outputs only reluctantly (because, as they put it, the hash validation was "sometimes even heavier than the query itself.") Funny enough, proving the new engine produced the same answer could cost more compute than producing the answer. They note that non-deterministic UDFs made validation hard, the same diff-adjudication problem Slack hit eight years later.

Takeaway

In the enterprise, migrations are rightfully considered risky projects that take time and incur risk. This is especially true in the age of AI given that the things that AI doesn’t necessarily deliver are also the riskiest parts of the migration - edge cases, data integrity, the long tail, etc. This isn’t to say that AI can’t help with building tooling for a migration, it clearly can, but usually an agent isn’t going to do the trick alone.

At Flarion, a major goal of ours is to give users the best possible performance and access to modern features without requiring a code migration. If you can get the benefits of Spark 4.2 while staying on Spark 3.4 then that’s a huge time save and reduction in risk. Of course, the data integrity problem doesn’t disappear, it’s now Flarion’s responsibility. One we’re happy to shoulder.

‍