What an AI analyst can't do yet (an honest list, from someone who uses them daily)
The 60-second version
- Eight limits, and only some are shrinking. Causation and accountability are structural — no bigger model fixes them.
- Consistency comes from architecture (fixed metric definitions), not from model size.
- Data the tool can't see is the #1 source of confident wrong answers — and that one is on us, not the model.
a fast junior who has read the whole schema and knows nothing about the business
A lot of AI-analytics marketing hints that the analyst job is done: ask a question, get an answer, the data team can go home. I use these tools every working day, and that isn't the experience. They're genuinely useful — and the space between what they do well and what they're sold as doing is exactly where trust gets burned.
So here's the list I wish I'd had before relying on one: eight things an AI analyst still can't do, why each limit exists, and whether it's getting smaller.
1. It doesn't know what your columns mean
Give a model your schema and it sees table names, column names, types and — if anyone wrote them — descriptions. What it doesn't see is the context that decides whether an answer is right.
Say a sales table has amount, amount_net, paid_value and invoice_total. Which one is revenue? That depends on whether tax is in, whether amount was abandoned after a migration, and whether paid_value stays empty for cash-on-delivery orders. None of that is visible in the schema.
2. It can't tell you the data is broken
It queries what's there. If the spend pipeline quietly stopped loading on the 14th, it will report — confidently — that spend fell 60%, because in the data, it did.
People catch this with a gut feel built from months of watching the same numbers. A model has a sense of what businesses look like in general, not yours. Anomaly alerts help a bit, and bring their own false alarms: a real festive-season spike looks just like a data error to a statistical test.
3. It can't prove cause and effect
Ask "did the campaign work?" and you get a correlation in careful language. Proving causation needs a counterfactual — what would have happened without the campaign — and that never happened, so it isn't in your data.
| Kind of question | Example | Can an AI analyst answer it? |
|---|---|---|
| Descriptive | Revenue by channel last quarter | Yes — this is what SQL is for |
| Diagnostic | Why did CAC rise? | Partly — it can break the change down, not rank true causes |
| Causal | Would sales have risen anyway? | No — needs a holdout designed before the fact |
4. It can't decide what matters
"Should we cut this campaign?" depends on cash, growth targets, what was promised to the board, whether the channel is strategic beyond its ROAS, and whether the team has time for the alternative. None of that lives in a warehouse.
The tool can tell you the campaign returns 0.8x on contribution. Whether that means "cut it" or "a planned investment we agreed to fund for two quarters" is a business call with inputs it can't see.
5. It can't stay consistent on its own
Ask the same question in two chats and you can get two queries with slightly different filters. "Revenue" on Monday includes tax; on Thursday, after a differently-worded question, it doesn't.
That's the biggest practical objection to putting an AI analyst in front of leadership. A dashboard is at least consistently wrong. An AI analyst can be inconsistently right — two people quote two numbers for one metric in the same meeting, and trust in the whole setup evaporates.
6. It loses the thread on long investigations
"Why did CAC go up last month?" isn't one query. It's: did spend rise or did customers fall? Which channel? Did that channel's CPM jump or its conversion rate drop? Did that line up with a creative change or an audience change? Each step depends on the previous answer.
Today's tools manage some of this and drift on longer chains — an early wrong turn compounds, because nothing in the loop stops to ask whether the whole line of inquiry still makes sense. Shrinking? Noticeably — this is where capability has moved most. Long chains still need a person checking the forks.
7. It can't see data you never connected
Obvious, and forgotten constantly. If offline sales live in a spreadsheet, influencer payments in someone's inbox and return-to-origin data in a courier portal nobody exports, then its answer on "total marketing efficiency" is wrong — and it won't say so, because it can't know those things exist.
In my experience this causes more confidently wrong answers than anything else on this list, and the model has nothing to do with it. A human analyst asks "is this everything?" A tool assumes what it can see is all there is.
8. It can't be accountable
When a number in a board deck is wrong, someone owns it — and that changes behaviour. An analyst who'll be asked about a number checks it differently.
A good tool can show its SQL, its assumptions and its row counts so that a person can take responsibility. The checking still has to happen; the tool's job is to make it cheap, not to remove it.
How to evaluate one in an afternoon
- Does it show the SQL?Ask three questions. Reject any answer that arrives without its query — you can't check a join, filter or grain you can't see.
- Does it admit when it doesn't know?Ask about a metric nobody defined. A tool that quietly guesses here will quietly guess where it matters. "Revenue is ambiguous — gross or net?" is the right behaviour.
- Is it consistent?Same question, three phrasings, three separate sessions. Compare the SQL. Variance here becomes variance in your deck.
- Does it show the cost before running?A warehouse dry run returns the bytes a query will scan, for free. A tool that doesn't surface that is choosing not to.
The honest upside
The ceiling is real, and so is the floor — which is higher than the sceptics say. The biggest win is that an AI analyst removes the queue. Not that it replaces an analyst, but that small questions finally get asked instead of dropped because they weren't worth interrupting someone for. Those small, "might be nothing" questions are where a surprising share of useful findings come from.
It's also genuinely good at the mechanical parts: remembering a schema across dozens of tables, writing valid SQL, applying a stated definition consistently within a conversation, and doing it in seconds. That's most of the volume of analytics work, even if it isn't most of the value.
Quick gut-check
No holdout was ever run. The AI says the campaign caused a 30% rise in sales. What's wrong?
Nothing — the warehouse data proves it
No. The warehouse holds what happened, not what would have happened without the campaign.
Causation needs a counterfactual; without a holdout this is correlation
Exactly. Incrementality needs a control group created before the fact.
The model is too small to judge campaigns
No. Model size is irrelevant — the limit is structural.
Questions people ask
Will better models fix these?
Some. Long multi-step analysis (6) is improving fast. Consistency (5) is being solved by architecture. Causation (3) and accountability (8) aren't model problems. Broken data (2) and unconnected data (7) are ours to fix.
So should I avoid them?
No — use one with the ceiling in view. The real failure is treating its output as the answer instead of a draft with its working shown.
What's the single best thing I can do to make one work better?
Write column descriptions and define your core metrics in one fixed place. Unglamorous, and it tackles limits 1 and 5 — the two behind most wrong answers.
Takeaways
- Eight limits; causation and accountability are structural, not model-sized.
- Consistency comes from fixed metric definitions, not bigger models.
- Unconnected data is the most common cause of confidently wrong answers — and it's on us.
- Test any tool on four things: shows SQL, admits ignorance, stays consistent, previews cost.