1. Organizations that have stalled on AI pilots often go back and buy better models or different tools rather than fix the data layer underneath. Who in the organization is positioned to make the most accurate diagnosis, though, when the data team and the AI team are separate functions with separate incentives?
There are teams with expertise in AI and in data platform technologies, but the AI pilot is not about technology, it’s about driving business outcomes. AI results can be a simple question on pristine data that is already in place, but that is not how the world operates. The right data must be searched, assembled, cleaned and then the final business logic applied. The user must be the person who understands the business, its data and can verify if the AI produced the results correctly. The pilot should be driven by a CDO or the leader of a business function such as finance that owns the business outcome with assistance from technology teams.
2. To that point, data prep problems in enterprise AI rollouts can be invisible until they cause a production failure. What are some signals that tell you a pilot is sitting on a data foundation that’ll break at scale? When does fixing it become more expensive than it would have been at the start?
Start with “did the demo run on a file someone cleaned by hand?” Who prepared the input data? How long did it take? If analysts spend two weeks getting it ready, the pilot is testing the model on data the production system won’t see. Another tell I see is when business definitions (like “revenue”) oscillated between living inside a prompt or one person’s head, and thus every team gets a slightly different number.
Fixing adds up quickly when folks start making decisions on the output. Before that, it’s an engineering project with a known scope. But when it’s beyond that stage, now you have to figure out which decisions relied on bad or outdated data and then try winning back the trust of the people who made them.
3. A CFO approving an AI budget is evaluating expected returns against visible cost. Data prep can be somewhat hidden, absorbed by engineering time, analyst hours, and downstream cost of those decisions made on bad data. How do you make the true cost of underinvesting in data prep visible?
Hours might be the best place to start, since CFOs already trust that unit, more or less. Ask analysts how much of their week goes into getting data ready. Within finance and ops teams, that’s often more than half their time. Multiply that out by headcount and loaded cost and the case makes itself.
Faulty decisions based on incomplete/outdated/etc. data are generally harder to price. I’d recommend tracing one real miss, maybe it was a forecast that was off. After you walk it back to the root case, there’s a good chance it was a spreadsheet join or a definition that drifted between teams.
AI also adds its own hidden costs, because if a monthly workflow goes back through a model on every run, you pay for inference again on work that was already approved.
4. The risk of an AI agent producing a plausible-but-wrong answer is higher when the person evaluating the output doesn’t have the technical foundation to interrogate the logic.
What failure modes does that create and how have you seen it go wrong in practice?
There’s actually a failure that doesn’t look like one. Say you have a business analyst uploading a spreadsheet to an AI chat and asking it to reconcile a quarter’s worth of transactions. The assistant writes and runs a script, but the analyst can’t read the code so they check that the total looks about right and send it along. But maybe a join dropped rows where customer IDs didn’t match, and revenue is off two percent because of it. Once that conversation closes, the logic used for the answer goes with it.
Software developers got AI that works with them all the way to production, presented in code they can review. Business analysts, meanwhile, were essentially handed question-and-answering and left to trust it. That’s why we built and recently launched Prophecy DataGPT. Business analysts work in ChatGPT, Claude, Copilot, or our own app, and the answer results come with a visual workflow where they can check things step by step before approving it. That way you get approval workflows that run as governed code the same way each time.
5. Explainability in AI for data is different from explainability in AI for decisions. How do you build that kind of explanation into an agentic system without making it so verbose that nobody reads it?
Explaining why a model made the decision it did can be relatively hard, while data work is more forgiving. Transformations are deterministic and you can, ideally, show each one. You just need to avoid explaining in prose, because no one is going to read three paragraphs about what an agent did (and those paragraphs might be wrong anyway). Instead, we show the work as a visual workflow where each step opens up to show the data going in and coming out. For example, an analyst checking a discount rule opens that singular step, looks at the row it touched and moves on. In other words, the workflow becomes the explanation so that it can no longer drift away from what actually ran. The open code is still right there underneath for anyone who wants it.
6. The analyst who catches an AI agent’s mistake before it reaches a decision is doing something valuable that most organizations don’t have a clear process for. How should enterprises be designing human review into agentic data workflows?
Good question. Start with putting the review at approval, prior to anything getting scheduled or shared, and give it to the person who knows the business rules. In finance, that’d be the analyst who owns the report, plus a data engineer for anything going into production. But even after initial approval, you need to set up monitoring to make sure the work reruns exactly as approved month after month.
Reviews should be pretty fast, otherwise people stop doing them. If and when an analyst catches something, the fix should go into the shared definitions so every future workflow picks it up, versus a correction that just lives in one file.
7. The audit trail question is significant in regulated industries, so when an AI agent has transformed data and produced an output that informed a decision, what does the documentation of that process need to look like to satisfy a regulator, an internal audit team, or a legal inquiry?
The question to answer is, can someone rebuild the output from the record alone? AI chat transcripts alone will fail that, since they’re built to show the question and the answer without much in between. Records need precise inputs and timelines of when they were pulled. They need transformation logic as code anyone can rerun as needed and version control. They also need to include run histories proving every scheduled run used the approved version. Audits and other legal questions tend to come down to what the system did on a specific date, so if the AI generated something slightly different on every run, you can’t say for certain what it did.
8. Just about every data modernization initiative in the last decade has talked of reducing spreadsheet dependency. What is it about how enterprises work that keeps producing this outcome and what would change it?
They’re popular because the person who understands the business can change the logic in them right now, versus filing a ticket and waiting around. Modernization programs have focused on moving the data somewhere better governed and handing the logic to an engineering team with a backlog. But the analyst still has a report due at the end of the week and a pricing exception the cloud data platform doesn’t know about, so they export it to Excel. For them, it’s a rational choice.
The change is giving analysts the same speed on a fully-governed platform. They can describe what they need and see the logic. They can adjust it and push it to production without a handoff. Make that path faster than Excel, and they’ll take it.
9. Last question: as agentic AI takes on data prep work, the human role in that process shifts from doing the work to defining what needs to be done and verifying that it was done correctly. What does that shift require from the people who currently do data preparation?
It boils down to asking them to document their subject matter expertise, and keeping them in the loop for decision making when building with AI. A good analyst knows that their company’s EMEA revenue excludes intercompany transfers, or that one vendor’s invoices always come in with the day and month swapped. Agents can’t apply that knowledge until someone turns it into definitions and rules, and doing that well is a valuable skill. As is verification. You need to read a workflow and ask whether each step does what the business needs. Analysts are best suited for it because they already know which numbers to distrust. They won’t need to code, and they’ll spend less time on VLOOKUPs and more on judgment, which is what they were hired for.
Bio:
Raj Bains is the CEO and co-founder of Prophecy, an AI data prep platform. Previously, Raj held engineering and product management roles at NVIDIA, Microsoft, Hortonworks (merged with Cloudera), Clustrix (acquired by MariaDB), and RMS.
