Blog
Why The World Needs Flarion. Read More
Blog

Apache Spark Resource Configuration

How to size Spark executors, memory and partitions in production, the failure patterns that show up when the numbers are wrong, and how to read them.
By
Ran Reichman
•
read time
•
February 5, 2025
Heading 2

Apache Spark's resource configuration remains one of the most challenging aspects of operating data pipelines at scale. Theoretical best practices are widely available, but production deployments often require adjustments to accommodate real-world constraints. This guide bridges that gap, exploring how to properly size Spark resources—from executors to partitions—while identifying common failure patterns and strategies to address them in production.

The Baseline Configuration

Consider a typical Spark job processing 1TB of data. A standard recommended setup might include:

  • A cluster of 20 nodes, each with 32 cores and 256GB RAM
  • Effective capacity of 28 cores and 240GB RAM per node after system overhead
  • 4 executors per node (80 total executors)
  • 7 cores per executor (with 1 core reserved for overhead)
  • 56GB RAM per executor
  • ~128MB partition sizes for optimal parallelism

While this configuration serves as a solid starting point, production workloads rarely conform to such clean boundaries. Let's examine some common failure patterns and mitigation strategies.When Reality Hits: Failure Patterns and Solutions

‍Failure Pattern #1: Workload Evolution Requiring Infrastructure Changes

‍A typical scenario: A job that previously ran efficiently on 20 nodes begins to experience increasing memory pressure or extended runtimes, despite configuration adjustments. Signs of resource constraints include:

  • Consistently high GC time across executors (>15% of executor runtime)
  • Storage fraction frequently dropping below 0.3
  • Executor memory usage consistently above 85%
  • Stage attempts failing despite conservative memory settings

Root cause analysis approach:

  1. Analyze growth patterns in your data volume and complexity.
  2. Profile representative jobs to understand resource bottlenecks.

Key scaling triggers:

  • CPU-bound: When average CPU utilization stays above 80% for most of the job duration.
  • Memory-bound: When GC time exceeds 15% or OOM errors occur despite tuning.
  • I/O-bound: When shuffle spill exceeds 20% of executor memory.

If CPU-bound (high CPU utilization, low wait times):

  • First try increasing cores per executor.
  • If insufficient, add nodes while maintaining a similar cores/node ratio.

If memory-bound (Out Of Memory - OOM):

  • First try reducing executors per node to allocate more memory per executor.
  • If insufficient, add nodes with higher memory configurations.

‍Failure Pattern #2: Memory Exhaustion In Compute Heavy Operations

‍A typical scenario: Your job runs fine for many days but then suddenly fails with Out Of Memory (OOM) errors. Investigation reveals that during month-end processing, certain joins produce intermediate results 5-10x larger than your input data. The executor memory gets exhausted trying to handle these large shuffles.A possible solution would be to update the configuration to:

  • spark.executor.memoryOverhead: 25% (increased from default 10%)
  • spark.memory.fraction: 0.75 (decreased from default 0.6)

These settings help because they:- Reserve more memory for off-heap operations (shuffles, network buffers)- Reduce the fraction of memory used for caching, giving more to execution- Allow GC to reclaim memory more aggressively‍

Failure Pattern #3: Data Skew, The Silent Killer

‍A typical scenario: Your daily aggregation job suddenly takes 4 hours instead of 1 hour. Investigation shows that 90% of the data is going to 10% of the partitions. Common culprits:- Timestamp-based keys clustering around business hours- Geographic data concentrated in major cities- Business IDs with vastly different activity levelsBefore implementing solutions, quantify your skew:

  1. Monitor partition sizes through the Spark UI
  2. Track duration variation across tasks within the same stage
  3. Look for orders of magnitude differences in partition sizes

A possible solution would be to analyze your key distribution and for known skewed keys, implement pre-processing like so:// For timestamp skewval smoothed_key = concat(date_col, hash(minute_col) % 10)// For business ID skewval salted_key = concat(business_id, hash(row_number) % 5)Using Spark’s built-in skew handling helps, but understanding the specific skew of your data is more robust and lasting. Spark’s skew handling configurations:

  • spark.sql.adaptive.enabled: true
  • spark.sql.adaptive.skewJoin.enabled: true

‍Failure Pattern #4: Resource Starvation in Mixed Workloads

‍A typical scenario: A seemingly well-configured job starts showing erratic behavior—some stages complete quickly while others seem stuck, executors appear underutilized despite high load, and the overall job progress becomes unpredictable. This is a typical case of resource starvation occurring within a single application.

  1. Late stages in complex DAGs struggle to get resources
  2. Shuffle operations become bottlenecks
  3. Some executors are overwhelmed while others sit idle
  4. Task attempts timeout and retry repeatedly

The root cause often lies in complex transformation chains: sqlCopydata.join(lookup1).groupBy("key1").agg(...).join(lookup2).groupBy("key2").agg(...)Each transformation creates intermediate results that compete for resources. Without proper management, earlier stages can hog resources, starving later stages.Possible solutions include:

  1. Dividing compute-intensive jobs into smaller jobs that use resources more predictably.
  2. If splitting a large job isn’t possible, using checkpoints and persist methods to better divide a single job into distinct parts. (expect a future blog post on these methods)
  3. Applying Spark Shuffle management - setting spark.dynamicAllocation.shuffleTracking.enabled and spark.shuffle.service.enabled to true.

Conclusions & The Path Forward

We've found that most Spark issues manifest first as performance degradation before becoming outright failures. The goal of a data engineering team isn't to prevent all issues but to catch and address them before they impact production stability. While adding resources can sometimes help, precise optimization and proper monitoring often provide more sustainable solutions. Spark offers a robust set of job management tools and settings, but addressing problems through standard Spark configurations alone often proves insufficient.The Flarion platform transforms this landscape in two key ways: through significant workload acceleration that reduces resource requirements and minimizes garbage collection overhead, and by providing enhanced visibility into Spark deployments. This combination of speed and improved observability enables engineering teams to identify potential issues before they escalate into failures, shifting from reactive troubleshooting to proactive optimization. As a result, data engineering teams experience both reduced failure rates and decreased operational burden, creating a more stable and efficient production environment.

Resources

Related Posts

Blog
This is some text inside of a div block.
Why the World Needs Flarion

Why we built Flarion: the cost of processing large data volumes, the pressure for faster results, and the approach we took to both.

read time
-
November 27, 2024

Did you know that data-driven organizations spend up to 40% of their IT budgets on data processing alone?

As organizations scale their data processing capabilities, two critical challenges emerge: the mounting costs of processing big data and the pressing need for faster performance. Today, we're sharing our journey and explaining why Flarion is transforming how organizations leverage their data assets while staying competitive in an increasingly data-driven world.

The Journey to Better Data Processing

Through years of experience across diverse industries, Flarion’s co-founders witnessed the universal struggle of escalating data processing costs and performance bottlenecks.

During his years building data processing systems for mass-scale consumer applications and autonomous vehicles, Ran experienced firsthand how organizations struggled with the growing costs and performance demands of expanding datasets. In consumer applications, better insights can help create great experiences for hundreds of millions of people, but the high computational costs and processing limitations often make this prohibitively expensive. In autonomous vehicles, data processing at scale allows us to understand and tackle the toughest "long tail" challenges, but technical limitations can make this slow and cost-inefficient.

Through his extensive work with enterprises across various industries, Udi observed a consistent pattern: organizations were hitting both a performance and cost ceiling in their data processing capabilities. Despite significant investments in infrastructure and talent, companies found themselves constrained by processing limitations that held back their ability to launch new features or products while managing escalating infrastructure costs.

The Evolution of Data Processing Needs

The landscape of data processing has evolved dramatically. What started as simple analytics has transformed into complex data pipelines processing hundreds of terabytes daily. These diverse challenges underline the pressing need for solutions that address both speed and cost at scale.

In automotive, processing speed directly impacts vehicle safety and performance, while processing costs affect vehicle affordability and market competitiveness. In financial services, faster data processing enables real-time decision-making and better risk assessment, but the infrastructure costs of high-frequency trading and real-time analytics can quickly erode profit margins. For e-commerce companies, efficient data processing means better customer recommendations and inventory management, yet the cost of processing massive customer datasets across global markets can be prohibitive. Almost every industry relies heavily on efficient data processing and analytics, making both speed and cost optimization critical factors in maintaining competitive advantage.

A New Approach to Performance

Traditional approaches to improving data processing often involve extensive code changes, specialized expertise, or specific deployment requirements. For enterprises with massive legacy codebases, these solutions are often impractical or impossible to implement, creating additional complexity without solving the fundamental challenges of performance and cost efficiency.

We built Flarion with a different vision: what if organizations could dramatically improve their data processing performance without changing their code or disrupting existing workflows? With new Spark, Hadoop and Ray execution engines, we've created a solution that delivers up to 3x performance improvement while maintaining robust reliability and full compatibility. Most importantly, Flarion can be implemented in just 5 minutes, requiring minimal effort from organizations looking to modernize their data stack.

Enabling Innovation Through Efficiency

The impact of accelerated data processing extends far beyond just faster completion times. When organizations can process their data more efficiently and cost-effectively, they can explore new use cases, launch innovative features, and focus on extracting value from their data rather than managing infrastructure costs.

For AI and machine learning applications, efficient data processing is becoming increasingly crucial. The ability to process large datasets quickly and reliably can mean the difference between a successful model deployment and a missed opportunity. With Flarion, organizations can focus on innovation rather than infrastructure optimization, all while maintaining their existing codebase and operations.

The Future of Data Processing

As we enter an era where data drives competitive advantage, organizations need solutions that enable them to process more data, faster and more cost-effectively. The future of data processing isn't just about handling today's workloads - it's about being ready for tomorrow's challenges while managing costs sustainably.

With Flarion, organizations are not just keeping pace—they’re leading the charge into a data-driven future. Our solution enables organizations to unlock the full potential of their data assets, whether they're running data processing in the cloud or on-premises. By delivering significant performance improvements through advanced optimization techniques, we're helping organizations process their data more efficiently while reducing their infrastructure costs. Most importantly, we're doing this in a way that respects the reality of enterprise systems - with a solution that can be implemented in minutes, not months.

The future of data processing should empower organizations to focus on innovation and value creation without being held back by legacy infrastructure or rising costs.

At Flarion, we're making that future a reality.

Featured Blog
Blog
The Data Processing Honey Badger: Why Spark Persists

read time
-
September 10, 2026

Development on Apache Spark started at Berkeley in 2009, and the first production release shipped on May 30, 2014. In the twelve years since, it has become the analytics workhorse for most of the large corporations in the world, across industries and scale, from seed-stage startups to Fortune 10 enterprises. Every year or so someone declares it old, past its peak, saddled with the JVM, and generally "legacy." And every year there is more of it. What accounts for the disconnect? In this post we'll walk through what we see across customer deployments and why we expect that in ten years there will still be a whole lot of Spark, and probably much more than there is today.

Infinite Scale

Spark scales very well. It’s not rare to see customers running workloads reading dozens of TB, while at the same time other customers process a few GB per workload. The result: for data engineering teams who don’t know how much data they’ll need to process, it’s a clean and easy decision to adopt Spark.

Network Effects

While it’s quite easy to use Spark, especially with PySpark, it’s not easy to deploy it and maintain it. But once the data platform adopts the tooling and learns how to maintain Spark, it is rarely motivated to migrate a piece of critical infrastructure to an unproven alternative, and instead are motivated to push more people to use Spark.

The Challenge of Migrating

Large companies can have thousands of jobs running at any given time, spread across the entire organization. The idea of pushing the various teams to migrate to a new platform is usually a complete non-starter. Oftentimes even gradually moving to systems like Ray is unwelcome due to the cost of maintaining multipledata platforms.

A First Class Citizen in the Data Lake

Delta Lake, Iceberg, and Hudi were each born with Spark as the reference implementation. The result is that Spark works well out-of-the-box with all three, while other systems are gradually adding support. Engineering teams want the best and most recent lakehouse technology and generally Spark supports it. .

Extensibility

Spark is easy to extend without forking. Catalyst exposes optimizer rules, planning strategies, and catalog plugins. DataSource V2 lets anyone teach Spark to read a new system. User defined functions (UDFs) let teams introduce Python or Scala logic into the middle of a pipeline without leaving the framework. Plug-ins allow the introduction of new libraries into the system. The result is that the thing people would otherwise leave Spark to get, a new connector, a custom optimization, a domain-specific function library, usually shows up inside Spark instead.

The Competition

Flink is used for some streaming use cases, Ray for AI use cases, Trino for interactive SQL, DuckDB and Polars for data that fits on one machine. Data warehouses with proprietary engines are taking some share. But at this point nobody is really trying to invent a new full-fledged system to replace Spark. The competition is either specializing in a lane or building underneath it.

Improved Engines

In Spark, the underlying engine is not static. The API hasn’t changed much since DataFrames arrived, but adding Tungsten improved performance with whole stage code generation that’s close to the hardware, while query optimizations, fast paths, and new operators also make the same workload faster without code changes. Databricks added Photon, we produced Flarion, and open source brought Gluten and Comet. It’s possible to stay on Spark and get modern performance, similar to how PostgreSQL keeps getting better and adding functionality without the API changing.

Summing Up

The Spark API is likely going to be with us for a long time, but under the hood a lot is going to change. Piece by piece the engine is being replaced, and it's plausible that in ten years none of the original execution code will be left, while every job still runs and every DataFrame still looks the same. It's the Ship of Theseus, except in this version the ship gets faster with every plank. 

‍

Featured Blog
Blog
The Hard Part of Upgrading Spark Versions Isn’t the Code

AI agents can now rewrite Spark jobs for upgrades. Slack, Uber, and Facebook's migrations show the real work: proving the output data didn't change.

read time
-
August 18, 2026

A few weeks ago AWS shipped the Spark Upgrade Agent, an AI agent that migrates Spark jobs to Spark 4.0. You point it at a repo, it rewrites deprecated APIs, adjusts for behavioral changes, updates the build for Scala 2.13, submits the result to an EMR cluster, and iterates on failures until the job runs. It handles both Scala and PySpark, and it works the way you'd hope an agent would: plan, transform, validate, repeat. The potential payoff is large - newer Spark versions have better performance and years of accumulated bug fixes.

It looks like a good tool and data teams looking into a Spark migration should consider it, but what we’ve found is that in the enterprise, rewriting code isn’t the main impediment to upgrading Spark workloads. Spark programs are part of complex pipelines, parts of which are poorly understood or maintained, and making changes to a sensitive system is inherently risky. There’s no guarantee that the output data will actually remain the same, and data integrity is the fundamental challenge of completing such a migration.

Several companies have written in detail about major Spark upgrades, and the data integrity challenge is a recurring theme.

Slack

Slack's migration from Spark 2 to Spark 3 took about a year across 60+ EMR clusters and 40+ teams. They saw some code-level breakage: `RAND()` in join keys became an `AnalysisException`, some casts that Spark 2 tolerated started failing, the `Greatest` function handled NULLs differently than its Hive counterpart. Validation was a much larger effort. For billing pipelines, Slack required exact matches, building test tables from production data on Spark 3 and running `EXCEPT` and `COUNT` comparisons in Trino against the Spark 2 outputs, with a Python framework for digging into every discrepancy. The discrepancies weren't all bugs. Non-deterministic row ordering, timestamp variations, and genuine semantic differences between Hive and Spark implementations all produce diffs that need to be investigated. Some are noise, some are real regressions.

Uber

Uber's version of this is bigger and more instructive. They migrated from Spark 2.4 to 3.3 with over two million Spark applications running daily. The code transformation was automated with Polyglot Piranha, their structural rewrite tool. It parses the source code into an AST, matches patterns, and applies transformation rules, including inserting legacy flags like `spark.sql.legacy.allowUntypedScalaUDF` where old behavior had to be preserved. This scaled well. The problem that shaped the whole project was stated plainly: "We had over 40,000 Spark apps, so we couldn't decentralize the data validation." No staging environment, no test cases, no way to ask every team to eyeball their own outputs.

So the flagship engineering artifact of Uber's Spark upgrade wasn't actually a code migrator but Iron Dome: a shadow-testing framework which runs the migrated job against production inputs, rewrites output paths at runtime so results land in staging instead of production, puts guardrails at the Hadoop FileSystem interface so a misrouted write can't touch real data, then compares the shadow output against the production run and only marks the job migrated when they agree. 

Facebook

None of this is specific to the Spark 2-to-3 transition, or even to Spark versions. When Facebook moved Hive workloads onto Spark SQL back in 2017, they ran shadow pipelines writing to tables suffixed `_spark_shadow` so downstream jobs were never exposed, used count checks as a cheap first filter, and reached for full hash validation of outputs only reluctantly (because, as they put it, the hash validation was "sometimes even heavier than the query itself.") Funny enough, proving the new engine produced the same answer could cost more compute than producing the answer. They note that non-deterministic UDFs made validation hard, the same diff-adjudication problem Slack hit eight years later.

Takeaway

In the enterprise, migrations are rightfully considered risky projects that take time and incur risk. This is especially true in the age of AI given that the things that AI doesn’t necessarily deliver are also the riskiest parts of the migration - edge cases, data integrity, the long tail, etc. This isn’t to say that AI can’t help with building tooling for a migration, it clearly can, but usually an agent isn’t going to do the trick alone.

At Flarion, a major goal of ours is to give users the best possible performance and access to modern features without requiring a code migration. If you can get the benefits of Spark 4.2 while staying on Spark 3.4 then that’s a huge time save and reduction in risk. Of course, the data integrity problem doesn’t disappear, it’s now Flarion’s responsibility. One we’re happy to shoulder.

‍

Oops! Something went wrong while submitting the form.