Regulatory intelligence pipeline

LLM-backed classification and enrichment at document scale, in a production SaaS.

Year: 2023 — present
Role: Senior engineer on the data platform team
Stack: Python FastAPI Airflow PostgreSQL MongoDB LLMs AWS

Problem

Enterprise customers need to know which of thousands of regulatory documents and news items actually affect them. The classification that answered that question was a large body of hand-maintained rules: expensive to change, impossible to reason about, and quietly drifting away from how analysts actually thought about the problem.

Approach

I replaced targeted stages of that rule set with LLM-backed classification, built around evaluation rather than vibes: a labelled set drawn from real analyst decisions, a harness that scores each candidate prompt and model against it, and a cost column next to the accuracy column. Several stages ended up on a smaller, cheaper model once the numbers showed it matched the larger one on the task that mattered.

Outcome

Classification stages that used to take weeks of rule-writing now take a prompt change and an evaluation run. Batch report processing moved onto Airflow in the same period, turning an opaque multi-hour job into observable, individually retryable tasks.

Evaluation is the product

The model choice is the least interesting decision. What makes an LLM stage shippable is having a way to answer “is this better than what we had?” before it reaches a customer. Building the evaluation harness first meant that swapping models later became a one-line experiment rather than a negotiation.

What I can and cannot show

This work is under NDA, so this page describes shape and outcome rather than internals: no customer data, no proprietary prompts, no internal architecture diagrams. Happy to talk through the engineering reasoning in an interview.