Spark 4.0's Columnar Journey: Real Progress, Real Limitations

If you've run Spark for any length of time, you might have seen a job stall for no obvious reason. The data is loaded, the executors are up, the CPUs are barely ticking over, and the stage just sits there. The culprit is usually garbage collection. In this post we'll look at why it happens and what can be done about it.

‍

What the JVM Is Doing When Everything Stops

Spark runs on the JVM, and the JVM is responsible for memory management. Your code creates objects as it works through the data, the dead ones pile up, and a background process called the garbage collector comes along and clears them out. Most of the time this happens seamlessly and without any user impact.

The trouble is in how it clears them. Before the collector can safely reclaim memory, it needs a moment where nothing is changing underneath it, so it can tell which objects are still in use. To get that moment, it stops the program. Every thread on the executor freezes at the same instant and waits. These are called stop-the-world pauses, and the name is honest about it. Nothing runs until the collector is finished.

Sometimes people ask, does the program really just wait for garbage collection? The answer is yes, that's exactly what happens. It's worth being fair to the modern collectors, because they're good. They do most of their bookkeeping concurrently, alongside the running job, precisely to keep these freezes short, and any single pause today is usually quick. But quick isn't free. The pauses never go away entirely, and the collectors tuned for data throughput will take longer pauses on purpose, because it lets them get more done in between. Add up enough short freezes across a big fleet and a long job, and you're looking at real idle time, which hits both performance and cost.

Can’t I Solve This Problem By Tuning?

The natural fix is to tune. Switch collectors, give it a bigger heap, change how the memory is split. This helps, but only up to a point.

Spark processes data a row at a time, and represents each value as an object on the heap. A single stage over a large dataset creates a huge number of short-lived objects: a value gets boxed, wrapped in an iterator, passed through an operator, and discarded. Every one is work the collector has to do later, and the faster they pile up, the more often it runs. No setting changes that. You can make collection faster and the pauses shorter, but you can't make the collector do less while the engine above it produces garbage by design. That's the ceiling.

Pauses aren't the whole cost either. The objects carry overhead of their own: headers, boxing and unboxing, and a scattered memory layout the CPU can't read efficiently. You pay for it whether the collector is running or not, and it usually stays hidden until someone profiles the job.

Not Making the Garbage in the First Place

If the volume of short-lived objects is the real problem, then the way out is an engine that doesn't create them to begin with, and that's a change you make well below anything a config file can reach. Flarion replaces Spark's row-by-row execution with a native engine written in Rust, built on Apache Arrow and DataFusion. Two things about that design do most of the work.

First, it's columnar. Instead of a parade of individual row objects, data moves through the engine in large contiguous blocks, one per column. The operation that used to allocate an object per value now chews through a whole block at once. The torrent of short-lived objects that kept the collector busy is never generated, so there's nothing to collect.

Second, there's no garbage collector in the picture because Rust doesn't use one. It tracks the ownership of memory as part of the language, so a piece of memory is freed at an exact, known point in the program, the instant it's no longer needed, and nothing ever has to stop the world to go hunting for what's dead. The data also lives off-heap, outside the JVM's managed memory entirely. There's no mechanism left to produce the idle time you were watching, so it's largely gone.

This doesn’t mean we’ve left the JVM behind. It still runs there, and it reaches the native engine through JNI, the standard bridge between Java and native code. What crosses that bridge is control and a handle to where the data lives, not the data itself, which stays off-heap on the native side. So handing work across to the engine doesn't pull everything back onto the heap for the collector to find, and the garbage stays uncreated.

Where It Doesn't Reach

Native execution covers most of a job, not all of it. The driver, the query planning, and any operation the engine doesn't support yet still run on the heap and still make objects for the collector. So garbage collection doesn't disappear from a real workload, it shrinks, and whatever is left tends to gather in the stages that haven't been converted. On more than one workload we've watched it settle into the final write, the last step still handing data back through the JVM. How much you're left with simply tracks how much of the job still runs on Spark.

None of this is something you have to manage. Flarion goes in as a plugin inside the Spark job you already have. What it supports runs natively, with no collector and no pause, and what it doesn't falls back to Spark and runs the way it always did. The acceleration happens underneath, and for the parts that run natively, the pause you used to wait on isn't there.
What This Adds Up To

Garbage collection sets a floor on how well a JVM-based engine can perform at scale. Tuning lowers that floor but never removes it, because the engine keeps producing the work the collector has to clear. Getting under the floor means not creating the garbage at all, which is why the same jobs, unchanged, run faster on a native engine. They finish sooner, and the bill for all that idle time goes with them.

‍

The consumer of data infrastructure is changing. For two decades, the primary user of platforms like Spark, Ray, and Trino has been a human engineer - someone who writes queries, tunes configurations, debugs failures, and builds intuition about their cluster over months of operation. Now, AI agents are increasingly the ones writing SQL, submitting Spark jobs, orchestrating ETL pipelines, and running analytical workloads. Enterprises are increasingly adopting agentic data engineering workflows. At Flarion, we see a few key dynamics at play.

‍

Tuning Will Converge - Then What?

Configuration tuning is getting automated. Agents can already follow runbooks, detect common failure patterns, and iterate on Spark configurations across runs. Auto-tuning systems are getting smarter. Over time, the combination of agents, automated tooling, and better platform defaults will squeeze most of the available performance out of configuration space. Executor sizing, partition counts, memory fractions, shuffle parameters - these will increasingly be handled by software rather than tribal knowledge.

Once configurations are well-tuned, the remaining performance gap is bounded by the execution engine itself. Spark's JVM-based, row-oriented architecture has fundamental overhead - garbage collection pauses, serialization costs, per-row function call overhead - that no amount of configuration tuning can eliminate. You can tune a Spark job perfectly and still leave 30-60% of potential performance on the table because of the engine's underlying execution model.

Flarion operates below this ceiling. Its native execution engine, built in Rust on Apache Arrow and DataFusion, replaces Spark's physical operators with vectorized execution. The gains come from a fundamentally different execution model. This means Flarion's acceleration stacks on top of whatever tuning an agent or auto-tuner has already done. Even the most optimized jobs get meaningfully faster.

For agents, this is the critical property. After exhausting configuration changes, an agent has two levers: continue building workload-specific tuning rules (low marginal benefit), or enable Flarion in a single configuration line and get 30-60% improvement immediately. In practice, it will do both - but the plugin delivers value on day one while tuning accumulates value over months.

‍

Separation of Concerns

There's also a trust dimension. Even as agents become capable of modifying code and infrastructure, most organizations won't want them to. Separation of concerns applies to agents the same way it applies to engineering teams: the system writing analytical queries should not also be the system reconfiguring the cluster, rewriting pipeline code, or modifying deployment infrastructure. Giving an agent broad permissions to change Spark configurations, refactor ETL logic, or resize clusters introduces risk that most platform teams won't accept.

This means agents will operate within boundaries. They'll submit workloads, not redesign them. They'll use tools, not modify the toolchain. The performance improvements available to an agent are limited to what can be achieved without crossing those boundaries.

Flarion fits precisely within these boundaries - a single configuration change that delivers material performance gains without touching application code, cluster settings, or deployment infrastructure. The agent stays in its lane. The acceleration happens underneath.

‍

Agents Are Measurement-Driven

Agents evaluate tools the way engineers wish they could: purely on observed results. An agent running a Spark job with Flarion enabled will observe that the job completed 40% faster at the same resource cost. It will observe this consistently across runs. Same input, same output, less time, less money. The signal is clean and unambiguous.

This matters because agents will increasingly be the ones deciding which tools to use within their permitted boundaries. They'll A/B test configurations, track cost-per-query over time, and converge on whatever delivers the best results. Tools that provide clear, repeatable value survive this selection process. Tools that require persuasion or organizational buy-in to demonstrate value don't - because agents aren't amenable to persuasion.

Flarion's 30-60% acceleration on real production workloads, validated across companies processing billions of records daily, is exactly the kind of signal that agents optimize for.

‍

The Zero-Config Threshold

The combination of bounded permissions and measurement-driven evaluation creates clear selection pressure on infrastructure. The platforms that agents will adopt are the ones that cross what might be called the zero-config threshold: the point where a tool can be activated and deliver value without requiring expertise.

Consider what an agent requires of data infrastructure. First, activation must be trivial - a single parameter, a plugin, a flag. Something that's easily testable and verifiable. Second, failure modes must be graceful. If something isn't supported, the system should fall back silently rather than throw an error the agent must handle. Agents are poor at diagnosing infrastructure-specific failures; they need systems that degrade predictably rather than fail unexpectedly. Third, the cost model must be transparent. In cloud environments, wall-clock time is cost. An agent optimizing for efficiency needs tools where faster execution directly equals lower spend, without requiring hardware-specific provisioning decisions.

There's a broader principle at work here: infrastructure designed to be usable by an agent will also be easier for a human. Every property that makes a tool agent-friendly - trivial activation, graceful fallback, transparent cost - also makes it friendlier to the human engineer who doesn't have time to read a tuning guide. Building for agents raises the floor for everyone.

Flarion crosses this threshold by design. Its native execution engine intercepts Spark's physical execution plan and replaces supported operators with vectorized native execution. Unsupported operations fall back transparently to Spark. The agent never sees a Flarion-specific error. It never needs to know which operations are accelerated and which aren't. The entire acceleration layer is invisible to the caller, which is precisely what makes it usable by an agent.

‍

Why Battle-Tested Ecosystems Win

Agents will prefer established ecosystems for the same reasons enterprises do: proven reliability at scale, broad connector support, extensive documentation that language models can reason about, and operational patterns that are well-understood. Spark processes petabytes daily across thousands of organizations. Ray orchestrates ML workloads at companies pushing the boundaries of model training. These platforms have accumulated years of production hardening that no new system can replicate quickly.

Making these ecosystems perform better without requiring expertise is where the real leverage lies. Flarion takes this approach across engines. Today it accelerates Spark workloads across every major deployment - open-source Spark, EMR, Dataproc, Databricks, and Spark on Kubernetes. The same Rust-based execution engine extends to Ray Data pipelines and Trino. An agent building a multi-engine analytical workflow gets consistent acceleration everywhere, through the same mechanism: enable the plugin, get faster results. No engine-specific optimization logic. No architectural trade-offs to evaluate.

This also means no new attack surface. Flarion runs as an in-process plugin inside the existing environment. No data leaves the perimeter. An agent can enable acceleration without triggering security reviews or compliance concerns - a friction reducer that matters enormously for enterprise adoption of agentic workflows.

‍

Where This Goes

The logical endpoint of this trend is outcome-oriented infrastructure - systems where agents submit workloads with constraints like "as cheap as possible, under 20 minutes" and the platform figures out the rest. The infrastructure handles resource allocation, configuration tuning, hardware routing, and failure recovery autonomously.

Flarion is building toward this future. The vision is autonomous execution where workloads are submitted with SLA targets and the system handles everything else - auto-tuning, auto-scaling, auto-recovery. The interface an agent actually wants: declare the outcome, let the infrastructure deliver it.

The building blocks are here today. A native execution engine that eliminates JVM overhead. Vectorized processing that leverages modern hardware. Transparent fallback that guarantees compatibility. Cross-engine support that works wherever the workload runs.

The agents are already here. The question is which platforms are ready for them.

‍