skip to content

2 min

  • tags
  • PyO3
  • maturin
  • Rust
  • Python

do list comprehensions dream of type safety?

sneaking a crab into my Python backend

Logan Voss / Unsplash

There's a class of problem in Python that no amount of async or clever caching will fix: CPU-bound work. Your server isn't waiting on a database or an API, it's just grinding through data, row by row.

I've hit this a couple of times building data exports on Python microservices. On my pet project, users can export their financial data as CSV, Excel or PDF, which means pulling transactions from the db and serialising potentially thousands of rows into formatted files.

"Potentially" is doing a lot of work in that sentence, because my DAU sits at ~2: girlfriend, myself, stop counting. We are still far, boys. It was just the perfect excuse to finally get hands-on with Rust bindings after reading about them for a while.

TL;DR I wrote the serialisation layer in Rust, and I was surprised by how incredibly simple it was.

The magic is PyO3, a crate that lets you write native Python modules in Rust, paired with maturin, a build tool that compiles that Rust into a Python wheel. For a language with such a scary reputation, the setup is tiny: a Cargo.toml that pulls in PyO3 with the extension-module feature, and a pyproject.toml that tells Python to use maturin as its build backend. Crab bindings… sorted.

On the Rust side, you annotate functions with #[pyfunction] and register them in a #[pymodule]. You pull values out of Python dicts, work with native Rust types inside, and hand back bytes. PyO3 does the conversions for you.

rust15 lines
#[pyfunction]
fn export_csv<'py>(
    py: Python<'py>,
    rows: Vec<HashMap<String, String>>,
    currency: &str,
) -> PyResult<Bound<'py, PyBytes>> {
    let csv = build_csv(&rows, currency)?;
    Ok(PyBytes::new(py, &csv))
}

#[pymodule]
fn scribe(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(export_csv, m)?)?;
    Ok(())
}

From Python, the module is invisible. You import scribe, call scribe.export_csv(rows, currency), and get bytes back. It's just a function call, and the rest of the codebase has no idea there's a crab in there.

python3 lines
import scribe

payload = scribe.export_csv(rows, currency)  # plain bytes

Kept the boundary clean on purpose. Python does what it's good at: auth, db queries, request routing. It collects the data, turns db records into plain dicts, and hands them to the crate for the compute-heavy part.

The performance difference is real, but the bigger win is having Rust's type system and memory safety wrapped around the serialisation logic. A compiler that enforces exhaustive pattern matching on export scopes, and refuses to build when I forget one is worth the setup.

The elephant in the room is that writing Rust and actually understanding Rust are two very different worlds. Couple of obscure concepts to us, the uninitiated.

If you ask me, this is one of the best uses of developing in the age of Artificial Intelligence. I've got a wealth of knowledge on tap, and I can ask why the borrow checker hates me as many times as I need.