Out of bad habit, my first thought was a status endpoint the client could poll (known in the hood as short-polling). I've built countless of these, and they seem to be the favourite way for dev teams to rack up tech debt they never pay off.

There are a few well-known options here, and it's mostly about picking the right one for your case. And if you ask what’s the purpose then, it’s because every time I do one of these light-hearted write-ups, I end up in deep and fantastic conversations with fellow nerds.

So, onward. Short-polling takes a minute to set up: you ask the job, on an interval, whether it's finished. The server answers, the connection closes, and you repeat until it fails or succeeds.

It's the developer's version of "Are we there yet?". Most of those requests are worthless. They burn server cycles and bandwidth, and open the door to state management bugs you didn't need.

Interval is basically impossible to get right - too slow and the updates feel choppy, too eager and you're bloating the network. That said, for long-running jobs where nobody needs an instant update, a well-spaced interval is perfectly fine.

Long-polling is the same idea, slightly more sophisticated. Plain HTTP but the server holds the connection open until there's an actual update or it times out after ~30s.

Handling reconnections gracefully and timing them with a margin before the timeout can be a bit of a pain. Keeps threads captive on synchronous servers, which isn't ideal. Still, unless you have high-frequency updates, I'd pick it over short-polling. It has served devs well for a long time.

Now, the real deal: WebSockets. They start life as an HTTP request then upgrade to their own protocol over TCP. That's why proxies and firewall configs can be a bit of a PIA (pardon my french), but in return you get proper bidirectional streaming.

Either side sends messages whenever it wants, with only a few bytes of framing on each one. It's quick, but trade-offs are extra complexity on the infrastructure side.

Unless you really need low-latency, two-way streaming, think collaboration tools or multiplayer anything, they're overkill.

For my simple stuff, I went with Server-Sent Events. Same push-from-the-server idea, way easier to set up, and no load balancer drama.

It's still plain HTTP. The client opens a connection with the EventSource API, and the server streams text events the moment they happen. The browser handles reconnection for you and keeps track of the last event it received.

On the server, you return a streaming response with the Content-Type set to text/event-stream, and send each event as your generator yields it. The client gets them in real time instead of waiting for the function to finish.

@app.get("/exports/{job_id}/events")
async def export_events(job_id: str):
    async def stream():
        async for status in watch_export(job_id):
            yield f"id: {status.seq}\nevent: status\ndata: {status.json()}\n\n"

    return StreamingResponse(stream(), media_type="text/event-stream")

Then…

const events = new EventSource(`/exports/${jobId}/events`);

events.addEventListener("status", (e) => {
  const status = JSON.parse(e.data);
  if (status.done) events.close();
});

If you're on a Python back-end, it's dead simple, as most frameworks have you covered out of the box. On FastAPI, look at StreamingResponse (thx Starlette), on Flask, stream_with_context, and there are plenty of others.

My events are just plain text with data, event and id fields. The id is what makes reconnection easy: the browser sends it back as Last-Event-ID when it reconnects, so the server can resume from where it left off.

Know your options, you could open an envelope with a chainsaw too but it's prlly not the most efficient way to do it.

I've hit this a couple of times building data exports on Python microservices. On my pet project, users can export their financial data as CSV, Excel or PDF, which means pulling transactions from the db and serialising potentially thousands of rows into formatted files.

"Potentially" is doing a lot of work in that sentence, because my DAU sits at ~2: girlfriend, myself, stop counting. We are still far, boys. It was just the perfect excuse to finally get hands-on with Rust bindings after reading about them for a while.

TL;DR I wrote the serialisation layer in Rust, and I was surprised by how incredibly simple it was.

The magic is PyO3, a crate that lets you write native Python modules in Rust, paired with maturin, a build tool that compiles that Rust into a Python wheel. For a language with such a scary reputation, the setup is tiny: a Cargo.toml that pulls in PyO3 with the extension-module feature, and a pyproject.toml that tells Python to use maturin as its build backend. Crab bindings… sorted.

On the Rust side, you annotate functions with #[pyfunction] and register them in a #[pymodule]. You pull values out of Python dicts, work with native Rust types inside, and hand back bytes. PyO3 does the conversions for you.

#[pyfunction]
fn export_csv<'py>(
    py: Python<'py>,
    rows: Vec<HashMap<String, String>>,
    currency: &str,
) -> PyResult<Bound<'py, PyBytes>> {
    let csv = build_csv(&rows, currency)?;
    Ok(PyBytes::new(py, &csv))
}

#[pymodule]
fn scribe(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(export_csv, m)?)?;
    Ok(())
}

From Python, the module is invisible. You import scribe, call scribe.export_csv(rows, currency), and get bytes back. It's just a function call, and the rest of the codebase has no idea there's a crab in there.

import scribe

payload = scribe.export_csv(rows, currency)  # plain bytes

Kept the boundary clean on purpose. Python does what it's good at: auth, db queries, request routing. It collects the data, turns db records into plain dicts, and hands them to the crate for the compute-heavy part.

The performance difference is real, but the bigger win is having Rust's type system and memory safety wrapped around the serialisation logic. A compiler that enforces exhaustive pattern matching on export scopes, and refuses to build when I forget one is worth the setup.

The elephant in the room is that writing Rust and actually understanding Rust are two very different worlds. Couple of obscure concepts to us, the uninitiated.

If you ask me, this is one of the best uses of developing in the age of Artificial Intelligence. I've got a wealth of knowledge on tap, and I can ask why the borrow checker hates me as many times as I need.

I swapped my whole email provider from AWS SES to Resend in under an hour. The provider change itself was about 20 lines of code, because everything email-related sits behind a small EmailProvider protocol.

Verification emails, welcome emails, suppression checks: all of it talks to that interface, and none of it knows what's behind it. So I wrote a ResendProvider that exposes the same methods on top of a different SDK, deleted the SESProvider, and rewrote the webhook handler for Resend's payload format.

Then I got to delete a good chunk of Terraform I no longer needed, which is my favourite kind of commit. Templates, rate limiting, the suppression list and token generation didn't change at all. Zero modifications to business logic.

The pattern is simple, and very tempting to break. You define a protocol for what your application needs from the dependency, not what the dependency happens to offer. Send an email. Check the suppression list. Handle a bounce.

class EmailProvider(Protocol):
    def send(self, message: EmailMessage) -> None: ...
    def is_suppressed(self, address: str) -> bool: ...
    def handle_bounce(self, payload: dict) -> None: ...

Then you write a thin adapter that maps the provider's SDK onto that protocol, and the rest of the app only ever imports the protocol.

The hard part is discipline. Keep the adapter thin, forever. No business logic, no special cases, no magic. The moment decisions creep into the adapter, you're coupled to the provider again, just with extra steps.

This goes well beyond email: payment processors, notification services, storage backends, anything where a third party owns the implementation. If swapping it means writing one new class and deleting the old one, you did the work right.