What I Learned Building 20+ AI Automations at a Fortune 500 Data Company

Most AI projects fail the same way. Not because the model was wrong. Not because the idea was bad. They fail because the team built something that works in a notebook, showed it to leadership, got a green light, and then spent the next six months discovering that the real world doesn't cooperate with demos.

I've been building production AI systems at LexisNexis Risk Solutions for several years. Legal data, insurance records, compliance documents: the kind of material that has real consequences when something goes wrong. Over 20 automations shipped. Some elegant, some scrappy, all of them educational in ways I didn't expect.

What I want to share here isn't a success story. It's a field report.

The Demo Is Not the Product

The hardest thing to communicate to a senior stakeholder is that a working demo and a working product are two entirely different things, separated by a gap that can take months to close.

A demo runs on clean data you curated yourself. It handles the happy path. It runs once, in a controlled environment, while you narrate. Nobody is watching it at 2am when a batch job throws a malformed record at it.

A production system runs on data you didn't curate. It has to handle edge cases nobody thought to mention during the requirements phase. It has to degrade gracefully when the upstream system sends garbage. It has to log enough information that when something goes wrong, and something will go wrong, you can figure out what happened without replaying the incident.

I've seen teams nail the demo, get budget approval, and then spend four months in purgatory trying to get the thing to actually work. The model wasn't the problem. The gap between "it works on the examples we showed" and "it works on everything your business actually sends through it" was.

Data Quality Is the Real Work

Here's something that surprised me the first time and now surprises me every time I have to explain it to a client: the AI model is almost never the hard part.

The hard part is the data.

In legal and insurance workflows, you're dealing with documents that have been scanned, faxed, renamed by a paralegal in 2018, stored in a system that uses a different schema than the one that replaced it, and occasionally corrupted in ways that nobody noticed because a human could still read them fine. The model can't.

I've pulled insurance claim forms where the field labeled "date of loss" contained a social security number because someone had mistyped it into the wrong field fifteen years ago and the record was never corrected. I've seen legal filings where the party names were split across two fields because the original form only had one line and the intake clerk improvised. I've seen compliance documents where mandatory checkboxes were left blank because the system marked them as optional in a 2019 update that nobody documented.

None of this shows up in the sample data you get handed at the start of a project. It shows up three weeks after go-live.

Expect to spend 40% to 60% of your project time on data. If someone tells you the data is clean, treat that as an optimistic estimate and plan accordingly.

The Last Mile Is Where Projects Die

Integration is where most AI projects quietly stop moving.

You've done the hard work. The model extracts clauses accurately. The classification is solid. The output looks exactly like what the business asked for. Then you hit the integration phase and discover that the case management system your client has been using since 2011 has an API that was documented in 2013 and last updated in 2016. Or it doesn't have an API at all. Or it has an API the IT team hasn't approved for external use. Or the vendor charges a separate licensing fee to enable API access.

I've had projects where the AI component took three weeks and the integration took three months. Not because anyone was being difficult. Enterprise systems accumulate complexity over time, and that complexity doesn't announce itself upfront.

The way I handle this now: integration discovery happens in week one. Before I write a single line of model code, I want to know exactly how outputs are going to flow into the existing system. If that path is unclear, I treat it as a project risk and say so explicitly.

The organizations that move fastest on AI are the ones that approach integration honestly, not the ones that assume it'll be fine.

Human-in-the-Loop Is a Feature, Not a Concession

When I tell clients that the system I'm building will route certain decisions to a human reviewer rather than processing them automatically, I sometimes get pushback. The implicit question is: aren't we paying for AI so we don't have to involve humans?

That framing is backwards.

Human-in-the-loop isn't a limitation you tolerate until the AI gets better. It's the thing that makes the AI trustworthy enough to use in the first place. Especially in legal, insurance, and compliance work, where a wrong output doesn't just create rework. It creates liability.

The systems that work best in production aren't the ones that automate everything. They're the ones that automate the right things and know their own boundaries. A model that accurately processes 85% of incoming documents automatically, flags the ambiguous 12%, and routes the genuinely complex 3% to a senior reviewer is not a failure. That's a win. The team that used to spend eight hours processing a week's worth of documents now spends ninety minutes reviewing the ones that actually need human judgment.

The automation handles volume. The human handles judgment. That division of labor is sustainable in a way that pure automation almost never is, because pure automation is brittle in exactly the cases that matter most.

Start Small Enough That You Can Actually Succeed

Every AI project I've seen fail in a blaze of organizational frustration had the same origin story: someone approved a large, ambitious scope, the project ran long, the budget ran out, and the business had moved on before anything shipped.

The projects that succeed start with a scope small enough that you can deliver real value in four to six weeks. Not a demo. Actual value. Something a real user is doing differently on a Monday morning because of what you built.

That might mean automating one document type instead of twelve. It might mean handling one step in a five-step workflow instead of the whole thing. It might feel modest compared to the full vision. But modest things that ship are worth more than ambitious things that don't.

Once you've shipped something that works and that people are using, the conversation about expanding scope is completely different. You're not asking anyone to take a leap of faith. You're showing them what works and asking if they'd like more of it.

What "Production-Grade" Actually Means

Most people use the phrase "production-ready" to mean "it worked when we tested it." That's not what it means.

A production-grade AI system has a few properties that don't exist in prototypes.

It has structured logging. When something goes wrong, wrong output, unexpected input, a timeout, a failed call to an external service, there's a record of what happened, what the model received, and what it returned. Without this, debugging is archaeology.

It has defined behavior for every failure mode. What happens when the upstream document is missing a required field? What happens when the model's confidence score falls below a threshold? What happens when an API call fails on the third retry? These paths need to exist in the code, not in the incident post-mortem.

It has a way to measure whether it's still working. Not just "is the service up" monitoring, but accuracy monitoring. Models drift. Data distributions shift. The set of documents coming through in March may look meaningfully different from the set you validated against in November. If you don't have a way to detect that drift, you'll find out about it when someone in the business notices the outputs have gotten worse.

It has a clear path for human correction. When an automated output is wrong, someone needs to be able to correct it, and that correction needs to feed back into the system in some form. Otherwise you're accumulating errors with no mechanism to improve.

None of this is glamorous. All of it matters.

Domain Expertise Beats AI Expertise

If I could give myself one piece of advice before my first production AI project, it would be this: learn the domain before you build anything.

Not at the surface level. Not "insurance claims are documents that describe losses." Actually learn it. Learn what adjusters do all day. Learn which fields matter for subrogation and which are just administrative housekeeping. Learn why a date of loss matters differently than a date of notice. Learn what the underwriter cares about that the claims team doesn't, and vice versa.

The model doesn't know any of this. You have to build that knowledge into how you frame the problem, how you structure your prompts, how you design your validation logic, what you flag for human review.

I've seen technically strong AI teams build systems that were accurate on the metrics they measured and useless to the people they built them for, because they optimized for the wrong things. They built what they were asked for rather than what was needed, because they never understood the difference.

Domain expertise doesn't replace technical skill. It makes technical skill useful.

A Note on Where This Comes From

Everything above I learned the hard way, usually on someone else's dime and timeline. Building AI at scale inside a Fortune 500 data company means you don't get to hand-wave the hard parts. If something breaks, the consequences are real.

When I started Grayhaven, it was specifically because I wanted to bring that production discipline to mid-market firms that are getting pitched AI constantly but haven't yet seen what it looks like when it actually works. Not a demo. Not a proof of concept that quietly retires after the engagement ends. Something that runs reliably, gets measured, and makes someone's actual job better.

If you're looking at AI and trying to figure out what's real and what's noise, the question I'd encourage you to ask any vendor, including me, is: what does this look like six months after go-live? How will we know if it's working? What happens when it's not?

Those questions have answers when you've built things that have to work every day.

If you want to talk through what that looks like for your team, I offer a free readiness assessment that usually runs about thirty minutes. You can reach me through the contact form on this site.