AI Pilot Stalled? Five Questions to Diagnose It and Restart in a Real Workflow

Arturo Tapia · 2026-11-05

An AI pilot stalled at the chatbot stage can have several causes, and design is one worth checking first: the pilot may have been built to answer questions, not to act inside a workflow. Test five things before blaming the model: whether it writes to systems, approval load, ownership, data access and exception handling.

AI pilot stalled? Five questions to diagnose it and restart in a real workflow

This is for two readers. The first is the department head or operations lead who ran, or inherited, an AI pilot that never got past a chatbot or assistant. The second is the founder or executive deciding whether to fund a second attempt. The article covers a diagnosis, how to pick a first process, how to design the pilot, how to choose between building, buying and hiring help, and how to size the cost of waiting. All numbers in the worked examples are assumptions for illustration. They are not results from any client or study.

A note on evidence. We have not found a primary source that measures how often each cause of a stalled pilot occurs, so this article makes no claim about which cause is most common. The design gap described below is a working hypothesis based on how these pilots are structured. The five questions and the scorecard are diagnostic aids to help you test it against your own pilot, not proven explanations.

Why do AI pilots stall?

AI pilots stall for several reasons, including model accuracy on your own data, poor data quality, security or legal blockers, unclear ownership, running cost and weak change management. One cause worth checking early, because it is fixable by design, is a pilot built to answer questions rather than to do work inside a process. Whether it applies to you is something to test, not assume.

In that design gap, the pilot produces useful answers in a chat window, but the real work, such as updating a record, sending a document or closing a ticket, still sits with a person. Nothing in the operation changes, so there is little evidence to justify the next phase of funding. It tends to show up in one of two forms:

• The agent never touched a real workflow. It lived in a chat interface next to the process, not inside it.
• The review load cancelled the benefit. The agent proposed or took actions, but every one needed a person's approval, and checking took about as long as doing the task, or approvals queued up behind one reviewer.

None of this rules out a model problem. If the agent's proposals were often wrong on your real cases, that is a separate finding, and the shadow-mode step described below is how you measure it. The five questions below help you separate these possibilities.

What is the difference between an AI agent and a chatbot?

The terms describe a spectrum of capability and design, not rigid categories. A chatbot is typically built to answer questions in a conversation. An AI agent is typically built to pursue a defined task using tools, within permissions and rules you set, and to hand off to a person when a case falls outside them. The practical difference is how much work happens in your systems and under what supervision.

The boundary is blurry. A chatbot can be given tools, such as looking up an order status. An agent can complete a useful task without writing to any system of record, such as researching a supplier and drafting a summary for a person to use. What matters for a pilot is the design: which tools it can use, what it is allowed to change, and how much it does without a person in the loop.

Three terms used in this article. A system of record is the system that holds the official version of a business fact. Examples are an enterprise resource planning (ERP) system, the software that holds finance and operations records such as invoices and purchase orders, and a customer relationship management (CRM) system, the software that holds customer accounts and interactions. An exception is a case that does not fit the process rules and needs a person.

Is the chatbot why our AI project stalled?

You cannot know until you check, but five questions will tell you whether the chatbot-shaped design is a plausible cause. If you answer no to several of them, a design gap is likely contributing. Write down the answer to each with evidence from the pilot, not impressions. These are diagnostic aids, and a clean sweep does not rule out model, data or governance problems.

1. Does it write to a system of record, or only answer questions?

If the agent only produces text for a person to copy elsewhere, it is working as an assistant. That can be valuable, but it will not change process metrics on its own. Check whether it can create, update or close something in the system where the work lives, and whether it should be allowed to.

2. Does a person have to approve every single action?

Approval on every action can still save time when checking a proposed action is much quicker than doing the work. It saves little when review takes about as long as the task, or when one reviewer becomes the bottleneck. Measure review time against doing-it-yourself time. Review effort should also scale with risk, as covered in the pilot design section.

3. Is there one owner and one measurable process outcome?

A pilot with a goal like "explore AI" is hard to judge. You need a named process owner and one measure, such as handling time per invoice or share of tickets resolved without rework.

4. Does the agent have access to the tools and data the human uses?

If the person uses five systems and the agent sees one export, it will fail on cases the person handles easily. List every system, document and shared mailbox the human touches for this task and compare.

5. Was there a plan for exceptions and handoffs?

Every process has cases that do not fit. If the pilot had no defined route for them, the agent either guessed or stopped, and staff lost trust. Look for a written list of exception types and who receives each one.

Which operations should we automate with AI agents first?

Start with a process that has high enough volume to matter, clear rules, a low exception rate, accessible systems and a modest, reversible cost of error. Those traits make the agent's work checkable and the result measurable. A process that is rare, vague or high-stakes is a poor first candidate.

The selection scorecard

Score each candidate process from 1 (poor) to 5 (strong) on each criterion. For exception rate and cost of error, a higher score means fewer exceptions and a cheaper mistake.

This scorecard is an editorial heuristic from Factorial Growth, not a validated benchmark. The equal weighting of the five criteria and the volume guide of "hundreds a month" are rules of thumb. Change the weights to fit your situation. A low-volume process can be worth automating if each case is expensive, and in a regulated area you may want a low score on cost of error to rule a process out regardless of its total.

Worked example: scoring three candidate processes (assumed numbers)

Everything in this example is assumed for illustration. These are not client results or industry benchmarks. Suppose an operations team scores three processes:

Process A scores highest under these assumptions. Process C scores low because each case needs legal judgement and a mistake is costly. That does not mean AI cannot help there, only that it is a poor first process, and a person would still decide on every case.

Now size process A in three separate layers. Assumptions: 1,200 invoices a month; 6 minutes of manual handling each on average; a loaded labour cost of $35 an hour; the agent fully handles 50 percent (low case) or 70 percent (high case) of invoices; and, importantly, the invoices it handles take the same 6 minutes on average as any other, which may not be true because easy cases may be quicker. Review is assumed at 1 minute per agent-handled invoice, and routing each non-automated invoice to a person adds an assumed 1 minute of handoff overhead. The agent's running cost (model usage, licence, maintenance) is assumed at $500 a month.

Two cautions. Freed hours only become savings if the time is redeployed, or reduces overtime, backlog or future hiring. And the automated share, review time and running cost are exactly what shadow mode measures. Replace each number with your own before you rely on the result.

How do you design a pilot that does not stall?

Design the pilot around one process, run it in shadow mode first, set approval gates by risk and reversibility, and launch with supervision. The aim is an agent that acts inside the workflow with human review proportional to the risk of each action, not one that waits for approval on everything or acts on everything unchecked.

Shadow mode, approval gates by risk tier, supervised launch

1. Shadow mode. The agent runs on live cases but does not act. Its proposed actions are compared with what staff actually did. This gives you an accuracy picture, the exception types and a measured review time before anything is changed.
2. Approval gates by risk and reversibility. Sort each action into a tier and let reversibility decide the review timing:
   • Low risk and easily reversible: can run automatically, with sampled review afterwards.
   • Medium risk but reversible: can run before review only if the action can be undone cleanly and no policy or legal rule requires prior sign-off. Review these on a schedule, such as daily, and tighten it if errors appear.
   • High risk, irreversible, or governed by policy, regulation or contract: a named person approves before the action happens, whatever its apparent size. Examples include payments released outside normal terms, messages to customers that cannot be recalled, and changes with legal effect. Confirm the list with your finance, compliance or legal owners.
3. Supervised launch. Go live on a share of volume with the process owner watching the measure agreed upfront. Widen only when exception handling holds up.
4. Ongoing review. A person reviews a sample of the agent's work on a schedule and owns changes to the rules.

What your team must provide

• A process owner who can make decisions about how the process runs and who accepts the measure.
• System access with permissions scoped to the task, set up by your IT or security team.
• Exception examples: a set of real past cases that did not fit the rules, and who handled them.
• Reviewer time during shadow mode and the first weeks live.

If you want help turning your chosen process into roles, permissions and handoffs, we offer an agent org chart service. We sell this service. Design your agent team and you get a mapped agent org chart for your first workflow: which agents do what, which systems each touches, and where a person reviews or decides.

Should we build, buy, or hire an agency for AI agents?

Build in-house if you have engineers with spare capacity and a long horizon. Buy a platform if your process fits what the product already does. Hire an agency or consultant if you need the design and supervision set up without growing a team. Each path still needs your process owner and system access.

Comparison: in-house build, agent platform, agency or consultant

Time to value varies widely with your systems and how quickly access is granted, so ask any provider for a range and the assumptions behind it.

When Factorial Growth is not the right fit

We sell AI agent team design and setup, so weigh this with that in mind. We are probably not the right fit if:

• You have a strong engineering team that wants to own the build and only needs time to do it.
• Your candidate process scores low on the scorecard, for example rare, high-stakes decisions with no clear rules.
• You cannot name a process owner or grant system access. No provider can fix that for you.
• You are looking for a fully hands-off result. A person still owns the process, reviews samples and decides on exceptions.

If a platform you already license covers the process, start there. See our AI for business page for what we do.

What does it cost the business to stay stalled?

The cost of staying stalled is not the full cost of the process today. It is the net benefit you would forgo by not fixing it, minus what the second attempt would cost you. Money already spent on the first pilot is a sunk cost and should not count in that comparison. This section is for the person who signs.

Cost of waiting, separated into baseline, gross and net

Using the same assumed invoice process as above, keep three numbers apart:

1. Baseline process cost (what the process costs today, not what waiting costs): handling of $4,200 a month (120 hours at $35), plus assumed errors: 1.5 percent of 1,200 invoices is 18, at an assumed $150 each to fix or absorb, which is $2,700. Total about $6,900 a month. This is the size of the pool, and most of it would remain with or without an agent.
2. Potential gross hours displaced: 60 to 84 hours a month, before review, handoffs and running cost.
3. Estimated net benefit: about $900 to $1,740 a month under the assumptions above, or roughly $10,800 to $20,880 a year, if the freed hours are actually redeployed or avoid cost.

The error cost is deliberately left out of the net benefit. We have no basis for assuming an agent would reduce errors on this process; it might lower them, leave them unchanged or add new kinds. Measure it in shadow mode before counting it either way.

Now subtract the future cost of trying again: design time, access setup, reviewer time during shadow mode and launch, and ongoing supervision. If that outlay is large relative to the net benefit, or the net benefit is small, a rational answer may be to wait, pick a different process, or stop. What you spent on the first pilot (licences, staff time) should not change that answer. If you choose to fund a second attempt, the reputational drag of a pilot that did not ship is a real organisational factor, but it is difficult to put a number on, so treat it as a judgement and not a line item.

What is the next step to get an agent team into a real workflow?

The next step is to run the five questions, score two or three candidate processes, pick one, and map the roles, permissions and handoffs before you build anything. We sell that mapping as a service: Design your agent team and you leave with an agent org chart for your first workflow that your process owner and IT can review.