Blog

The Data Processing Honey Badger: Why Spark Persists

Every year someone calls Spark legacy, and every year there is more of it. What keeps it growing, and why its engine is being rebuilt piece by piece.
By
Ran Reichman
•
read time
•
September 10, 2026
Heading 2

Development on Apache Spark started at Berkeley in 2009, and the first production release shipped on May 30, 2014. In the twelve years since, it has become the analytics workhorse for most of the large corporations in the world, across industries and scale, from seed-stage startups to Fortune 10 enterprises. Every year or so someone declares it old, past its peak, saddled with the JVM, and generally "legacy." And every year there is more of it. What accounts for the disconnect? In this post we'll walk through what we see across customer deployments and why we expect that in ten years there will still be a whole lot of Spark, and probably much more than there is today.

Infinite Scale

Spark scales very well. It’s not rare to see customers running workloads reading dozens of TB, while at the same time other customers process a few GB per workload. The result: for data engineering teams who don’t know how much data they’ll need to process, it’s a clean and easy decision to adopt Spark.

Network Effects

While it’s quite easy to use Spark, especially with PySpark, it’s not easy to deploy it and maintain it. But once the data platform adopts the tooling and learns how to maintain Spark, it is rarely motivated to migrate a piece of critical infrastructure to an unproven alternative, and instead is motivated to push more people to use Spark.

The Challenge of Migrating

Large companies can have thousands of jobs running at any given time, spread across the entire organization. The idea of pushing the various teams to migrate to a new platform is usually a complete non-starter. Oftentimes even gradually moving to systems like Ray is unwelcome due to the cost of maintaining multiple data platforms.

A First Class Citizen in the Data Lake

Delta Lake, Iceberg, and Hudi were each born with Spark as the reference implementation. The result is that Spark works well out-of-the-box with all three, while other systems are gradually adding support. Engineering teams want the best and most recent lakehouse technology and generally Spark supports it.

Extensibility

Spark is easy to extend without forking. Catalyst exposes optimizer rules, planning strategies, and catalog plugins. DataSource V2 lets anyone teach Spark to read a new system. User defined functions (UDFs) let teams introduce Python or Scala logic into the middle of a pipeline without leaving the framework. Plug-ins allow the introduction of new libraries into the system. The result is that the thing people would otherwise leave Spark to get, a new connector, a custom optimization, a domain-specific function library, usually shows up inside Spark instead.

The Competition

Flink is used for some streaming use cases, Ray for AI use cases, Trino for interactive SQL, DuckDB and Polars for data that fits on one machine. Data warehouses with proprietary engines are taking some share. But at this point nobody is really trying to invent a new full-fledged system to replace Spark. The competition is either specializing in a lane or building underneath it.

Improved Engines

In Spark, the underlying engine is not static. The API hasn’t changed much since DataFrames arrived, but adding Tungsten improved performance with whole stage code generation that’s close to the hardware, while query optimizations, fast paths, and new operators also make the same workload faster without code changes. Databricks added Photon, we produced Flarion, and open source brought Gluten and Comet. It’s possible to stay on Spark and get modern performance, similar to how PostgreSQL keeps getting better and adding functionality without the API changing.

Summing Up

The Spark API is likely going to be with us for a long time, but under the hood a lot is going to change. Piece by piece the engine is being replaced, and it's plausible that in ten years none of the original execution code will be left, while every job still runs and every DataFrame still looks the same. It's the Ship of Theseus, except in this version the ship gets faster with every plank. 

‍

Resources

Related Posts

Blog
This is some text inside of a div block.
The Challenge of Garbage Collection in Java-based Data Engines

Why a Spark job stalls with executors up and CPUs idle, what the JVM is doing when everything stops, and what you can do about garbage collection.

read time
-
June 23, 2026

If you've run Spark for any length of time, you might have seen a job stall for no obvious reason. The data is loaded, the executors are up, the CPUs are barely ticking over, and the stage just sits there. The culprit is usually garbage collection. In this post we'll look at why it happens and what can be done about it.

‍

What the JVM Is Doing When Everything Stops

Spark runs on the JVM, and the JVM is responsible for memory management. Your code creates objects as it works through the data, the dead ones pile up, and a background process called the garbage collector comes along and clears them out. Most of the time this happens seamlessly and without any user impact.

The trouble is in how it clears them. Before the collector can safely reclaim memory, it needs a moment where nothing is changing underneath it, so it can tell which objects are still in use. To get that moment, it stops the program. Every thread on the executor freezes at the same instant and waits. These are called stop-the-world pauses, and the name is honest about it. Nothing runs until the collector is finished.

Sometimes people ask, does the program really just wait for garbage collection? The answer is yes, that's exactly what happens. It's worth being fair to the modern collectors, because they're good. They do most of their bookkeeping concurrently, alongside the running job, precisely to keep these freezes short, and any single pause today is usually quick. But quick isn't free. The pauses never go away entirely, and the collectors tuned for data throughput will take longer pauses on purpose, because it lets them get more done in between. Add up enough short freezes across a big fleet and a long job, and you're looking at real idle time, which hits both performance and cost.

Can’t I Solve This Problem By Tuning?

The natural fix is to tune. Switch collectors, give it a bigger heap, change how the memory is split. This helps, but only up to a point.

Spark processes data a row at a time, and represents each value as an object on the heap. A single stage over a large dataset creates a huge number of short-lived objects: a value gets boxed, wrapped in an iterator, passed through an operator, and discarded. Every one is work the collector has to do later, and the faster they pile up, the more often it runs. No setting changes that. You can make collection faster and the pauses shorter, but you can't make the collector do less while the engine above it produces garbage by design. That's the ceiling.

Pauses aren't the whole cost either. The objects carry overhead of their own: headers, boxing and unboxing, and a scattered memory layout the CPU can't read efficiently. You pay for it whether the collector is running or not, and it usually stays hidden until someone profiles the job.

Not Making the Garbage in the First Place

If the volume of short-lived objects is the real problem, then the way out is an engine that doesn't create them to begin with, and that's a change you make well below anything a config file can reach. Flarion replaces Spark's row-by-row execution with a native engine written in Rust, built on Apache Arrow and DataFusion. Two things about that design do most of the work.

First, it's columnar. Instead of a parade of individual row objects, data moves through the engine in large contiguous blocks, one per column. The operation that used to allocate an object per value now chews through a whole block at once. The torrent of short-lived objects that kept the collector busy is never generated, so there's nothing to collect.

Second, there's no garbage collector in the picture because Rust doesn't use one. It tracks the ownership of memory as part of the language, so a piece of memory is freed at an exact, known point in the program, the instant it's no longer needed, and nothing ever has to stop the world to go hunting for what's dead. The data also lives off-heap, outside the JVM's managed memory entirely. There's no mechanism left to produce the idle time you were watching, so it's largely gone.

This doesn’t mean we’ve left the JVM behind. It still runs there, and it reaches the native engine through JNI, the standard bridge between Java and native code. What crosses that bridge is control and a handle to where the data lives, not the data itself, which stays off-heap on the native side. So handing work across to the engine doesn't pull everything back onto the heap for the collector to find, and the garbage stays uncreated.

Where It Doesn't Reach

Native execution covers most of a job, not all of it. The driver, the query planning, and any operation the engine doesn't support yet still run on the heap and still make objects for the collector. So garbage collection doesn't disappear from a real workload, it shrinks, and whatever is left tends to gather in the stages that haven't been converted. On more than one workload we've watched it settle into the final write, the last step still handing data back through the JVM. How much you're left with simply tracks how much of the job still runs on Spark.

None of this is something you have to manage. Flarion goes in as a plugin inside the Spark job you already have. What it supports runs natively, with no collector and no pause, and what it doesn't falls back to Spark and runs the way it always did. The acceleration happens underneath, and for the parts that run natively, the pause you used to wait on isn't there.
What This Adds Up To

Garbage collection sets a floor on how well a JVM-based engine can perform at scale. Tuning lowers that floor but never removes it, because the engine keeps producing the work the collector has to clear. Getting under the floor means not creating the garbage at all, which is why the same jobs, unchanged, run faster on a native engine. They finish sooner, and the bill for all that idle time goes with them.

‍

Featured Blog
Blog
The Data Processing Honey Badger: Why Spark Persists

Every year someone calls Spark legacy, and every year there is more of it. What keeps it growing, and why its engine is being rebuilt piece by piece.

read time
-
September 10, 2026

Development on Apache Spark started at Berkeley in 2009, and the first production release shipped on May 30, 2014. In the twelve years since, it has become the analytics workhorse for most of the large corporations in the world, across industries and scale, from seed-stage startups to Fortune 10 enterprises. Every year or so someone declares it old, past its peak, saddled with the JVM, and generally "legacy." And every year there is more of it. What accounts for the disconnect? In this post we'll walk through what we see across customer deployments and why we expect that in ten years there will still be a whole lot of Spark, and probably much more than there is today.

Infinite Scale

Spark scales very well. It’s not rare to see customers running workloads reading dozens of TB, while at the same time other customers process a few GB per workload. The result: for data engineering teams who don’t know how much data they’ll need to process, it’s a clean and easy decision to adopt Spark.

Network Effects

While it’s quite easy to use Spark, especially with PySpark, it’s not easy to deploy it and maintain it. But once the data platform adopts the tooling and learns how to maintain Spark, it is rarely motivated to migrate a piece of critical infrastructure to an unproven alternative, and instead is motivated to push more people to use Spark.

The Challenge of Migrating

Large companies can have thousands of jobs running at any given time, spread across the entire organization. The idea of pushing the various teams to migrate to a new platform is usually a complete non-starter. Oftentimes even gradually moving to systems like Ray is unwelcome due to the cost of maintaining multiple data platforms.

A First Class Citizen in the Data Lake

Delta Lake, Iceberg, and Hudi were each born with Spark as the reference implementation. The result is that Spark works well out-of-the-box with all three, while other systems are gradually adding support. Engineering teams want the best and most recent lakehouse technology and generally Spark supports it.

Extensibility

Spark is easy to extend without forking. Catalyst exposes optimizer rules, planning strategies, and catalog plugins. DataSource V2 lets anyone teach Spark to read a new system. User defined functions (UDFs) let teams introduce Python or Scala logic into the middle of a pipeline without leaving the framework. Plug-ins allow the introduction of new libraries into the system. The result is that the thing people would otherwise leave Spark to get, a new connector, a custom optimization, a domain-specific function library, usually shows up inside Spark instead.

The Competition

Flink is used for some streaming use cases, Ray for AI use cases, Trino for interactive SQL, DuckDB and Polars for data that fits on one machine. Data warehouses with proprietary engines are taking some share. But at this point nobody is really trying to invent a new full-fledged system to replace Spark. The competition is either specializing in a lane or building underneath it.

Improved Engines

In Spark, the underlying engine is not static. The API hasn’t changed much since DataFrames arrived, but adding Tungsten improved performance with whole stage code generation that’s close to the hardware, while query optimizations, fast paths, and new operators also make the same workload faster without code changes. Databricks added Photon, we produced Flarion, and open source brought Gluten and Comet. It’s possible to stay on Spark and get modern performance, similar to how PostgreSQL keeps getting better and adding functionality without the API changing.

Summing Up

The Spark API is likely going to be with us for a long time, but under the hood a lot is going to change. Piece by piece the engine is being replaced, and it's plausible that in ten years none of the original execution code will be left, while every job still runs and every DataFrame still looks the same. It's the Ship of Theseus, except in this version the ship gets faster with every plank. 

‍

Featured Blog
Blog
The Hard Part of Upgrading Spark Versions Isn’t the Code

AI agents can now rewrite Spark jobs for upgrades. Slack, Uber, and Facebook's migrations show the real work: proving the output data didn't change.

read time
-
August 18, 2026

A few weeks ago AWS shipped the Spark Upgrade Agent, an AI agent that migrates Spark jobs to Spark 4.0. You point it at a repo, it rewrites deprecated APIs, adjusts for behavioral changes, updates the build for Scala 2.13, submits the result to an EMR cluster, and iterates on failures until the job runs. It handles both Scala and PySpark, and it works the way you'd hope an agent would: plan, transform, validate, repeat. The potential payoff is large - newer Spark versions have better performance and years of accumulated bug fixes.

It looks like a good tool and data teams looking into a Spark migration should consider it, but what we’ve found is that in the enterprise, rewriting code isn’t the main impediment to upgrading Spark workloads. Spark programs are part of complex pipelines, parts of which are poorly understood or maintained, and making changes to a sensitive system is inherently risky. There’s no guarantee that the output data will actually remain the same, and data integrity is the fundamental challenge of completing such a migration.

Several companies have written in detail about major Spark upgrades, and the data integrity challenge is a recurring theme.

Slack

Slack's migration from Spark 2 to Spark 3 took about a year across 60+ EMR clusters and 40+ teams. They saw some code-level breakage: `RAND()` in join keys became an `AnalysisException`, some casts that Spark 2 tolerated started failing, the `Greatest` function handled NULLs differently than its Hive counterpart. Validation was a much larger effort. For billing pipelines, Slack required exact matches, building test tables from production data on Spark 3 and running `EXCEPT` and `COUNT` comparisons in Trino against the Spark 2 outputs, with a Python framework for digging into every discrepancy. The discrepancies weren't all bugs. Non-deterministic row ordering, timestamp variations, and genuine semantic differences between Hive and Spark implementations all produce diffs that need to be investigated. Some are noise, some are real regressions.

Uber

Uber's version of this is bigger and more instructive. They migrated from Spark 2.4 to 3.3 with over two million Spark applications running daily. The code transformation was automated with Polyglot Piranha, their structural rewrite tool. It parses the source code into an AST, matches patterns, and applies transformation rules, including inserting legacy flags like `spark.sql.legacy.allowUntypedScalaUDF` where old behavior had to be preserved. This scaled well. The problem that shaped the whole project was stated plainly: "We had over 40,000 Spark apps, so we couldn't decentralize the data validation." No staging environment, no test cases, no way to ask every team to eyeball their own outputs.

So the flagship engineering artifact of Uber's Spark upgrade wasn't actually a code migrator but Iron Dome: a shadow-testing framework which runs the migrated job against production inputs, rewrites output paths at runtime so results land in staging instead of production, puts guardrails at the Hadoop FileSystem interface so a misrouted write can't touch real data, then compares the shadow output against the production run and only marks the job migrated when they agree. 

Facebook

None of this is specific to the Spark 2-to-3 transition, or even to Spark versions. When Facebook moved Hive workloads onto Spark SQL back in 2017, they ran shadow pipelines writing to tables suffixed `_spark_shadow` so downstream jobs were never exposed, used count checks as a cheap first filter, and reached for full hash validation of outputs only reluctantly (because, as they put it, the hash validation was "sometimes even heavier than the query itself.") Funny enough, proving the new engine produced the same answer could cost more compute than producing the answer. They note that non-deterministic UDFs made validation hard, the same diff-adjudication problem Slack hit eight years later.

Takeaway

In the enterprise, migrations are rightfully considered risky projects that take time and incur risk. This is especially true in the age of AI given that the things that AI doesn’t necessarily deliver are also the riskiest parts of the migration - edge cases, data integrity, the long tail, etc. This isn’t to say that AI can’t help with building tooling for a migration, it clearly can, but usually an agent isn’t going to do the trick alone.

At Flarion, a major goal of ours is to give users the best possible performance and access to modern features without requiring a code migration. If you can get the benefits of Spark 4.2 while staying on Spark 3.4 then that’s a huge time save and reduction in risk. Of course, the data integrity problem doesn’t disappear, it’s now Flarion’s responsibility. One we’re happy to shoulder.

‍

Oops! Something went wrong while submitting the form.