Blog
Why The World Needs Flarion. Read More
Blog

Vectorized Processing

Vectorized execution processes batches of rows instead of one row at a time. Where the idea came from, how it works, and why data systems depend on it.
By
Ran Reichman
•
read time
•
March 18, 2025
Heading 2

Vectorization has emerged as the most critical performance innovation in modern data platforms. At its core, the concept is straightforward: process entire batches of data simultaneously rather than one row at a time. This approach unlocks substantial efficiency gains and has become fundamental to high-performance data systems.

The Birth of Vectorized Processing

The database community first embraced vectorization through pioneering systems like MonetDB and VectorWise in the mid-2000s. These systems addressed the observation that traditional row-by-row processing created significant CPU bottlenecks. Their solution involved processing data in batches small enough to fit in CPU caches, dramatically improving query performance by eliminating per-row function call overhead.

In parallel, the scientific Python ecosystem built NumPy and Pandas around vectorized operations, allowing data scientists to perform bulk calculations orders of magnitude faster than Python loops. These early implementations demonstrated that vectorization represented a fundamental paradigm shift in data processing.

How Vectorization Transforms Performance

Vectorization aligns with modern hardware capabilities through multiple mechanisms:

  • CPU Vector Instructions (SIMD): Modern CPUs include SIMD (Single Instruction Multiple Data) units that can perform the same operation on multiple values simultaneously. These specialized processor features have evolved significantly:


    • SIMD Evolution: From early MMX and SSE instructions processing 128 bits (4 integers) at once, to AVX-256 handling 8 integers, and modern AVX-512 capable of processing 16 integers or floats in a single instruction

    • Hardware Implementation: SIMD registers are wider than standard registers—256 or 512 bits versus 64 bits—allowing a single instruction to operate on multiple data elements

    • Operation Types: Common SIMD operations in data processing include vectorized comparison (generating bitmasks for filtering), arithmetic (sum, multiply, divide entire arrays), and specialized operations like shuffle and gather/scatter

    • Compiler Support: Modern compilers can auto-vectorize simple loops, while high-performance systems use intrinsics (specialized C functions that map directly to SIMD instructions) for maximum control

    • Performance Impact: SIMD instructions can provide theoretical speedups proportional to the vector width—up to 16x for certain operations on AVX-512 systems

  • Memory Efficiency: Columnar data layouts enable sequential memory access, maximizing cache efficiency and minimizing memory stalls.

  • Reduced Overhead: With vectorization, the cost of function calls and interpretation is amortized across hundreds or thousands of values.

A simple example illustrates the difference. Consider summing a column with a million values:

  • Traditional approach: Loop through one million values, with function call overhead for each
  • Vectorized approach: Process 1,024 values at once in a tight loop, leveraging SIMD instructions

The Role of Apache Arrow

Apache Arrow has become the central enabling technology for the vectorization ecosystem. It provides:

  1. Zero-copy columnar memory format: Arrow defines a standardized in-memory columnar representation that allows data to be processed without serialization or deserialization when moving between systems.

  2. SIMD-optimized compute kernels: Arrow includes a library of vectorized operations optimized for modern CPUs, ensuring that as new vector instruction sets emerge (AVX-512, ARM SVE), all Arrow-based systems can benefit.

  3. Cross-language compatibility: Arrow implementations exist across multiple programming languages (C++, Rust, Python, Java, etc.), enabling efficient data exchange between different environments.

  4. Integration across the ecosystem: Major platforms including Spark, DataFusion, Polars, and Velox have adopted Arrow as their interchange format.

  5. Flight protocol: Arrow Flight provides high-performance data transfer between systems using the Arrow format, offering substantial improvements over traditional protocols.

The significance of Arrow lies in its ability to break down silos between previously isolated data systems. A dataset in Arrow format can move seamlessly between a Spark cluster, Python analysis environment, and GPU-accelerated visualization tool with minimal overhead.

The Vectorization Landscape Today

This approach has permeated virtually every corner of the data ecosystem:

Analytical Databases

  • ClickHouse processes data in batches, routinely scanning billions of records per second on a single server
  • DuckDB processes fixed-size batches of 1,024 values, matching dedicated database servers for medium-sized datasets
  • Apache DataFusion operates natively on columnar RecordBatches, performing highly efficient SIMD-enabled computations

Big Data Systems

  • Apache Spark now leverages Pandas UDFs with Arrow as a zero-copy data interchange format, though it still does not use vectorization in its primary flows
  • Databricks Photon replaces row-wise processing with a native columnar engine
  • Meta's Velox provides a unified C++ execution engine with vectorized expression evaluation

Data Science and ML

  • Polars combines Apache Arrow's memory-efficient format with multi-threaded, SIMD-accelerated operations
  • TensorFlow and PyTorch leverage optimized libraries like Intel's oneAPI Math Kernel Library and NVIDIA CUDA
  • Scientific computing applications depend on vectorization to achieve performance at scale

Real-World Impact: Quantifiable Improvements

The performance gains from vectorization translate to measurable improvements:

  • Databricks Photon achieves over 10× speedups on some SQL and DataFrame operations
  • Meta's Velox delivers 6-7× faster performance on heavy analytical queries in production at Facebook
  • CockroachDB's vectorized OLAP engine yields up to 4× speedups in standard analytics benchmarks
  • In machine learning, GPU-accelerated vectorized operations can be 10-100× faster than CPU-based sequential processing

These improvements enable interactive queries on terabytes of data, ML models trained in minutes instead of hours, and scientific simulations at previously impossible resolutions.

The Future of Vectorized Processing

As hardware continues to evolve with wider vector units, more cores, and specialized accelerators, vectorization remains the foundation of high-performance data systems. The convergence between database technology, data science tools, and ML frameworks demonstrates that vectorization has become a fundamental paradigm for modern computing.

Embracing vectorized processing is now essential for delivering the performance required by data-intensive applications across industries and domains.

‍

Resources

Related Posts

Blog
This is some text inside of a div block.
Why the World Needs Flarion

Why we built Flarion: the cost of processing large data volumes, the pressure for faster results, and the approach we took to both.

read time
-
November 27, 2024

Did you know that data-driven organizations spend up to 40% of their IT budgets on data processing alone?

As organizations scale their data processing capabilities, two critical challenges emerge: the mounting costs of processing big data and the pressing need for faster performance. Today, we're sharing our journey and explaining why Flarion is transforming how organizations leverage their data assets while staying competitive in an increasingly data-driven world.

The Journey to Better Data Processing

Through years of experience across diverse industries, Flarion’s co-founders witnessed the universal struggle of escalating data processing costs and performance bottlenecks.

During his years building data processing systems for mass-scale consumer applications and autonomous vehicles, Ran experienced firsthand how organizations struggled with the growing costs and performance demands of expanding datasets. In consumer applications, better insights can help create great experiences for hundreds of millions of people, but the high computational costs and processing limitations often make this prohibitively expensive. In autonomous vehicles, data processing at scale allows us to understand and tackle the toughest "long tail" challenges, but technical limitations can make this slow and cost-inefficient.

Through his extensive work with enterprises across various industries, Udi observed a consistent pattern: organizations were hitting both a performance and cost ceiling in their data processing capabilities. Despite significant investments in infrastructure and talent, companies found themselves constrained by processing limitations that held back their ability to launch new features or products while managing escalating infrastructure costs.

The Evolution of Data Processing Needs

The landscape of data processing has evolved dramatically. What started as simple analytics has transformed into complex data pipelines processing hundreds of terabytes daily. These diverse challenges underline the pressing need for solutions that address both speed and cost at scale.

In automotive, processing speed directly impacts vehicle safety and performance, while processing costs affect vehicle affordability and market competitiveness. In financial services, faster data processing enables real-time decision-making and better risk assessment, but the infrastructure costs of high-frequency trading and real-time analytics can quickly erode profit margins. For e-commerce companies, efficient data processing means better customer recommendations and inventory management, yet the cost of processing massive customer datasets across global markets can be prohibitive. Almost every industry relies heavily on efficient data processing and analytics, making both speed and cost optimization critical factors in maintaining competitive advantage.

A New Approach to Performance

Traditional approaches to improving data processing often involve extensive code changes, specialized expertise, or specific deployment requirements. For enterprises with massive legacy codebases, these solutions are often impractical or impossible to implement, creating additional complexity without solving the fundamental challenges of performance and cost efficiency.

We built Flarion with a different vision: what if organizations could dramatically improve their data processing performance without changing their code or disrupting existing workflows? With new Spark, Hadoop and Ray execution engines, we've created a solution that delivers up to 3x performance improvement while maintaining robust reliability and full compatibility. Most importantly, Flarion can be implemented in just 5 minutes, requiring minimal effort from organizations looking to modernize their data stack.

Enabling Innovation Through Efficiency

The impact of accelerated data processing extends far beyond just faster completion times. When organizations can process their data more efficiently and cost-effectively, they can explore new use cases, launch innovative features, and focus on extracting value from their data rather than managing infrastructure costs.

For AI and machine learning applications, efficient data processing is becoming increasingly crucial. The ability to process large datasets quickly and reliably can mean the difference between a successful model deployment and a missed opportunity. With Flarion, organizations can focus on innovation rather than infrastructure optimization, all while maintaining their existing codebase and operations.

The Future of Data Processing

As we enter an era where data drives competitive advantage, organizations need solutions that enable them to process more data, faster and more cost-effectively. The future of data processing isn't just about handling today's workloads - it's about being ready for tomorrow's challenges while managing costs sustainably.

With Flarion, organizations are not just keeping pace—they’re leading the charge into a data-driven future. Our solution enables organizations to unlock the full potential of their data assets, whether they're running data processing in the cloud or on-premises. By delivering significant performance improvements through advanced optimization techniques, we're helping organizations process their data more efficiently while reducing their infrastructure costs. Most importantly, we're doing this in a way that respects the reality of enterprise systems - with a solution that can be implemented in minutes, not months.

The future of data processing should empower organizations to focus on innovation and value creation without being held back by legacy infrastructure or rising costs.

At Flarion, we're making that future a reality.

Featured Blog
Blog
The Hard Part of Upgrading Spark Versions Isn’t the Code

AI agents can now rewrite Spark jobs for upgrades. Slack, Uber, and Facebook's migrations show the real work: proving the output data didn't change.

read time
-
August 18, 2026

A few weeks ago AWS shipped the Spark Upgrade Agent, an AI agent that migrates Spark jobs to Spark 4.0. You point it at a repo, it rewrites deprecated APIs, adjusts for behavioral changes, updates the build for Scala 2.13, submits the result to an EMR cluster, and iterates on failures until the job runs. It handles both Scala and PySpark, and it works the way you'd hope an agent would: plan, transform, validate, repeat. The potential payoff is large - newer Spark versions have better performance and years of accumulated bug fixes.

It looks like a good tool and data teams looking into a Spark migration should consider it, but what we’ve found is that in the enterprise, rewriting code isn’t the main impediment to upgrading Spark workloads. Spark programs are part of complex pipelines, parts of which are poorly understood or maintained, and making changes to a sensitive system is inherently risky. There’s no guarantee that the output data will actually remain the same, and data integrity is the fundamental challenge of completing such a migration.

Several companies have written in detail about major Spark upgrades, and the data integrity challenge is a recurring theme.

Slack

Slack's migration from Spark 2 to Spark 3 took about a year across 60+ EMR clusters and 40+ teams. They saw some code-level breakage: `RAND()` in join keys became an `AnalysisException`, some casts that Spark 2 tolerated started failing, the `Greatest` function handled NULLs differently than its Hive counterpart. Validation was a much larger effort. For billing pipelines, Slack required exact matches, building test tables from production data on Spark 3 and running `EXCEPT` and `COUNT` comparisons in Trino against the Spark 2 outputs, with a Python framework for digging into every discrepancy. The discrepancies weren't all bugs. Non-deterministic row ordering, timestamp variations, and genuine semantic differences between Hive and Spark implementations all produce diffs that need to be investigated. Some are noise, some are real regressions.

Uber

Uber's version of this is bigger and more instructive. They migrated from Spark 2.4 to 3.3 with over two million Spark applications running daily. The code transformation was automated with Polyglot Piranha, their structural rewrite tool. It parses the source code into an AST, matches patterns, and applies transformation rules, including inserting legacy flags like `spark.sql.legacy.allowUntypedScalaUDF` where old behavior had to be preserved. This scaled well. The problem that shaped the whole project was stated plainly: "We had over 40,000 Spark apps, so we couldn't decentralize the data validation." No staging environment, no test cases, no way to ask every team to eyeball their own outputs.

So the flagship engineering artifact of Uber's Spark upgrade wasn't actually a code migrator but Iron Dome: a shadow-testing framework which runs the migrated job against production inputs, rewrites output paths at runtime so results land in staging instead of production, puts guardrails at the Hadoop FileSystem interface so a misrouted write can't touch real data, then compares the shadow output against the production run and only marks the job migrated when they agree. 

Facebook

None of this is specific to the Spark 2-to-3 transition, or even to Spark versions. When Facebook moved Hive workloads onto Spark SQL back in 2017, they ran shadow pipelines writing to tables suffixed `_spark_shadow` so downstream jobs were never exposed, used count checks as a cheap first filter, and reached for full hash validation of outputs only reluctantly (because, as they put it, the hash validation was "sometimes even heavier than the query itself.") Funny enough, proving the new engine produced the same answer could cost more compute than producing the answer. They note that non-deterministic UDFs made validation hard, the same diff-adjudication problem Slack hit eight years later.

Takeaway

In the enterprise, migrations are rightfully considered risky projects that take time and incur risk. This is especially true in the age of AI given that the things that AI doesn’t necessarily deliver are also the riskiest parts of the migration - edge cases, data integrity, the long tail, etc. This isn’t to say that AI can’t help with building tooling for a migration, it clearly can, but usually an agent isn’t going to do the trick alone.

At Flarion, a major goal of ours is to give users the best possible performance and access to modern features without requiring a code migration. If you can get the benefits of Spark 4.2 while staying on Spark 3.4 then that’s a huge time save and reduction in risk. Of course, the data integrity problem doesn’t disappear, it’s now Flarion’s responsibility. One we’re happy to shoulder.

‍

Featured Blog
Blog
Arrow by Default: What Spark 4.2's Python UDF Change Actually Means

Spark 4.2 turns on Arrow-optimized Python UDFs and Arrow data exchange by default. What that config flip says about where Spark execution is heading.

read time
-
July 27, 2026

Spark 4.2 was released in mid-July, and a lot of the attention went to the headline features: geospatial types, change data capture, vector search. Tucked into the performance section was a smaller item that we found particularly interesting. Arrow-optimized Python UDFs, and Arrow-based data exchange with Python in general, are now on by default.

One can consider this a mere config flip. The feature has existed since Spark 3.5. But defaults are how a platform tells you what it considers normal, and Spark just declared that the normal way to move data between the JVM and Python is Apache Arrow. It's worth walking through why that boundary was slow in the first place, what Arrow does about it, and what it changes for everything sitting underneath.

‍

What a Python UDF Actually Costs

PySpark has a split identity. The engine that executes your job runs on the JVM, and your UDF runs in a separate Python process, because that's where Python code has to run. Every batch of rows that passes through the UDF makes a round trip: out of the JVM, across a socket into the Python worker, through your function, and back.

Until now, the default way to make that trip was pickle. Each row was converted from Spark's internal representation into a Python object, serialized, sent across, deserialized, processed, and then the whole sequence ran again in reverse. One row at a time, one object at a time. The work is pure overhead. It exists because the two sides of the boundary represented the same data differently, and translation was the only way across.

For UDF-heavy jobs the translation regularly cost more than the function being called. It's one of the oldest pieces of PySpark folklore: keep your logic in built-in expressions if you can, because the moment you write a Python UDF, you pay a tax that has nothing to do with what the UDF does.

‍

What Arrow Brings to the Table

Apache Arrow is a specification for how tabular data is laid out in memory: columnar, in large contiguous buffers, with a defined binary layout for every type. The layout is the same regardless of which language or engine produced it. If two systems both hold data in Arrow format, one can hand the other a batch without converting anything, given that the bytes are already in the shape the receiver expects.

Applied to the Python boundary, this removes most of the tax. Instead of serializing rows into Python objects, the JVM sends Arrow batches, and the Python side reads them directly as columnar data. There is still a process boundary and still a copy across the socket, but the expensive part - turning every value into an object and back - is gone. Spark's own benchmarks for Arrow-optimized UDFs showed roughly 2x speedups on chained UDFs when the feature shipped in 3.5, with larger gains the more the workload was dominated by the boundary rather than the function.

The history of this feature tells you something about defaults. Arrow UDFs arrived as an opt-in in Spark 3.5. Spark 4.1 added Arrow-native UDF decorators that skip the pandas conversion entirely. With 4.2, Arrow is the default and pickle is the fallback. Everyone gets the faster boundary, including the large majority of users who never knew there was a flag.

‍

The Ecosystem Keeps Converging on Arrow

The UDF change is one instance of a pattern that has been running for years. `toPandas` and `createDataFrame` now use Arrow by default too, in the same release. Spark Connect streams query results to clients as Arrow batches. Outside of Spark: pandas can be backed by Arrow, Polars is built on it, DuckDB reads and writes it natively, DataFusion uses it as its internal memory model. When these systems exchange data with each other, more and more often no conversion happens, because both ends already speak the same format.

This is what a de facto standard looks like while it's forming. Nobody mandated Arrow; each project adopted it because interoperating through a shared memory layout is cheaper than maintaining pairwise converters. Every Spark release for the past several years has replaced another row-based boundary with an Arrow one, and there's no reason to expect the direction to reverse.

‍

What the Default Doesn't Change

It's worth being precise about what got faster. The boundary between the JVM and Python is now columnar. The engine on the JVM side of that boundary is the same one it was before — predominantly row-oriented, executing on the heap, one row at a time through most operators.

That produces a slightly odd shape for a typical PySpark job. The scan reads Parquet, which is columnar on disk. Spark turns it into rows to execute the joins and aggregations. At the UDF boundary, those rows are batched back into Arrow's columnar form, shipped to Python, processed, returned, and turned back into rows for whatever comes next. The data changes representation multiple times, and the fast columnar format only exists at the edges. Spark 4.2 made the edges cheap. The middle is where most of the job's time goes, and the middle didn't change.

Rewriting the executor is a different scale of undertaking than adopting Arrow at the boundaries, and the boundaries were a reasonable place to start. But the release defines the remaining gap fairly precisely: the format Spark now uses to talk to Python is not the format it uses to compute.

‍

Running the Middle on Arrow Too

That gap is where Flarion sits. Our engine executes Spark's operators - scans, joins, aggregations, and the rest - in native Rust code built on Apache Arrow and DataFusion, replacing the row-at-a-time JVM path for the parts of the plan it supports. Inside the engine, data stays in Arrow's columnar layout the whole way through. It goes in as a plugin on the Spark job you already have; unsupported operations fall back to Spark and run the way they always did.

Spark standardizing its boundaries on Arrow makes this arrangement steadily cleaner. When the engine hands data back to Spark, or Spark hands data to a Python worker, both sides increasingly agree on the memory layout, so the crossings that used to require translation become handoffs. Data moves between Spark's JVM and our engine through Arrow's C Data Interface without copying at all — a pointer to the buffers crosses the boundary, and the data stays where it is.

The UDF change is a preview of what that feels like, applied to one boundary. The excitement around it comes from removing translation overhead at a single crossing point. An Arrow-native engine applies the same idea to the execution itself: the scan produces Arrow, the join consumes Arrow, and the representation never changes because there's nothing to change it into.

‍

Summing Up

Spark 4.2's UDF change is a nice speedup, but the reason it caught our attention is what it says about direction. Spark is a conservative project and it doesn't change defaults lightly, because millions of jobs run on whatever the defaults are. When a project like that decides Arrow is how data should cross the Python boundary, it's acknowledging what the rest of the ecosystem already settled on. In short, the future is Arrow.

‍

Oops! Something went wrong while submitting the form.