Rust for data processing: a hands-on guide for engineers

use polars::prelude::*;

fn main() -> Result<(), Box> {
    // Read CSV file
    let df = CsvReader::from_path("transactions.csv")?
        .infer_schema(Some(1000))
        .has_header(true)
        .finish()?;

    // Filter rows where amount > 100
    let filtered = df.lazy()
        .filter(col("amount").gt(lit(100)))
        .collect()?;

    // Group by user_id, sum amounts
    let grouped = filtered.lazy()
        .groupby(["user_id"])?
        .agg([col("amount").sum().alias("total_spent")])
        .sort("total_spent", SortOptions::default().with_descending(true))
        .collect()?;

    println!("{}", grouped);
    Ok(())
}
```<p></p>

<p>Notice the <code>.lazy()</code> call. Polars uses lazy evaluation—it builds a query plan, optimizes it, and only executes when you call <code>.collect()</code>. This means the engine can push down filters, combine operations, and avoid unnecessary allocations. In production, this is the difference between scalability and pain.</p>

<p>For most data workloads—CSV ingestion, Parquet transformations, joining tables—Polars handles it natively and faster than you'd expect. If you're processing gigabytes of structured data, this should be your first reach.</p>

<h2 id="rayon-parallel-iterators-with-one-change">Rayon — Parallel iterators with one change</h2>

<p>Not every data task fits neatly into a DataFrame. Sometimes you're processing records row-by-row, transforming structs, or applying custom business logic. That's where Rayon shines: it replaces standard Rust iterators with parallel versions, often with a single method name change.</p>

<p>In a trading risk system we built, we needed to calculate Greeks (delta, gamma, vega) for 50,000 option positions every minute. Sequential iteration was too slow; we couldn't afford a thread pool library. Rayon gave us parallel computation with minimal refactoring:</p>

<pre><code class="language-rust">use rayon::prelude::*;

#[derive(Clone)]
struct Position {
    id: String,
    underlying_price: f64,
    strike: f64,
    volatility: f64,
}

#[derive(Clone)]
struct Greeks {
    delta: f64,
    gamma: f64,
    vega: f64,
}

fn calculate_greeks(pos: &Position) -> Greeks {
    // Simplified Black-Scholes Greeks calculation
    let d1 = ((pos.underlying_price / pos.strike).ln() + 0.5 * pos.volatility.powi(2))
        / pos.volatility;
    let delta = d1.tanh();
    let gamma = 1.0 / (pos.strike * pos.volatility * d1.abs());
    let vega = pos.strike * d1.exp() * pos.volatility;
    
    Greeks { delta, gamma, vega }
}

fn main() {
    let positions: Vec = vec![
        // Load positions from database or file
    ];

    // Single change: .iter().par_bridge() instead of .iter()
    let greeks: Vec = positions
        .par_iter()
        .map(|pos| calculate_greeks(pos))
        .collect();

    println!("Calculated Greeks for {} positions", greeks.len());
}
```<p></p>

<p>That's it. Swap <code>.iter()</code> for <code>.par_iter()</code>, and Rayon distributes work across CPU cores. No manual thread spawning, no channel management. For compute-bound work on large collections, this is one of Rust's best productivity wins.</p>

<p>One caveat: Rayon assumes your closures are thread-safe (they must implement <code>Send + Sync</code>). If you're mutating shared state, you'll need <code>Arc<Mutex<T>></code> or similar. But in most data scenarios—transformations, calculations, filtering—you're not mutating shared data anyway.</p>

<h2 id="arrow-columnar-data-formats">Arrow — Columnar data formats</h2>

<p>Apache Arrow is a memory format standard for analytics. Instead of storing data row-by-row, Arrow stores it column-by-column, which is far more cache-friendly for aggregations and transformations. Most modern data tools—Polars, DuckDB, Parquet writers—build on Arrow internally.</p>

<p>When you're building data infrastructure that needs to interop with Python, Java, or C++ systems, Arrow is your lingua franca. You can write Rust, export to Arrow format, and consume it downstream without serialization overhead.</p>

<p>Here's how you'd create and inspect Arrow arrays:</p>

<pre><code class="language-rust">use arrow::array::{Float64Array, StringArray, RecordBatch};
use arrow::datatypes::{DataType, Field, Schema};
use std::sync::Arc;

fn main() -> Result<(), Box> {
    // Create Arrow arrays (columns)
    let names = StringArray::from(vec!["Alice", "Bob", "Charlie"]);
    let scores = Float64Array::from(vec![95.5, 87.2, 92.1]);

    // Define schema
    let schema = Schema::new(vec![
        Field::new("name", DataType::Utf8, false),
        Field::new("score", DataType::Float64, false),
    ]);

    // Create record batch (columnar table)
    let batch = RecordBatch::try_new(
        Arc::new(schema),
        vec![Arc::new(names), Arc::new(scores)],
    )?;

    // Access columns
    println!("Batch has {} rows and {} columns", 
             batch.num_rows(), 
             batch.num_columns());

    Ok(())
}
```<p></p>

<p>Arrow excels when you're moving data between systems or need to maintain strict schema compatibility. If you're calling out to Python analytics or feeding data into a data warehouse, Arrow Parquet files are your default choice—they're fast to write, compress well, and every tool understands them.</p>

<h2 id="python-bindings-pyo3-exposes-rust-to-python">Python bindings — PyO3 exposes Rust to Python</h2>

<p>You've built a fast Rust data processor. Now your data science team wants to call it from Python. Enter PyO3: a framework that lets you write Rust functions and expose them as Python modules with minimal boilerplate.</p>

<p>On a recent SaaS analytics platform, we had CPU-bound clustering logic that was slowing down the Python backend. Moving the core algorithm to Rust and exposing it via PyO3 cut execution time from 12 seconds to 800ms per request.</p>

<p>Here's a simple example: a Rust function that normalizes vectors, exposed to Python:</p>

<pre><code class="language-rust">use pyo3::prelude::*;

#[pyfunction]
fn normalize_vector(values: Vec) -> PyResult> {
    let sum_of_squares: f64 = values.iter().map(|x| x * x).sum();
    let magnitude = sum_of_squares.sqrt();
    
    if magnitude == 0.0 {
        return Err(PyErr::new::(
            "Cannot normalize zero vector"
        ));
    }

    Ok(values.iter().map(|x| x / magnitude).collect())
}

#[pymodule]
fn vector_ops(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(normalize_vector, m)?)?;
    Ok(())
}

[package]
name = "vector_ops"
version = "0.1.0"
edition = "2021"

[dependencies]
pyo3 = { version = "0.21", features = ["extension-module"] }

[lib]
crate-type = ["cdylib"]

import vector_ops

result = vector_ops.normalize_vector([3.0, 4.0])
print(result)  # [0.6, 0.8]