You may already pay for ChatGPT or Copilot, and your engineers may write code with an agent every day. Nothing on the operations side of the company is wired to it yet, and the board wants to know what you're doing with AI.
The first thing I ask is whether someone on your team can explain one process out loud, start to finish, including the exceptions. If they can, that process is a candidate. If the answer is "it depends, ask Fran", it isn't one yet.
Agents follow instructions about as well as the instructions are written
At Uber I owned the knowledge base behind R2D2, an internal Slackbot for the fraud investigations team, and I sponsored the bot; a support agent built it. Before LLMs, it answered about 85% of agents' questions from the knowledge base, visibly in Slack, so support staff saw where each answer came from. They learned to use the knowledge base themselves, or to ask for an update when an answer was out of date. Language models today can do far more than that Slackbot could, and the same rule applies to them. If the steps live in one person's head, an agent has nothing to follow.
Good first candidates
- Sorting the support inbox into the handful of request types you already get.
- Pulling deal details out of email and into HubSpot, with a person approving each one at first.
- Answering the same five questions customers ask at 10pm on a Sunday, from answers you've already written.
- Drafting the weekly pilot report from numbers that already exist somewhere.
Poor first candidates
Work where the rules change every time, or where your team can't agree what a good result looks like. Money decisions can wait until you've watched an agent on easier work. When I wrote the logic for automated cancellation refunds at Uber, most of the work was setting the thresholds, so the automation handled the normal curve of refunds and the outliers went to a person.
Some decisions stay with people. Never hand the adjudication of a safety investigation to an agent, least of all on a system that's new to your team. A person has to be responsible for that call.
Keep a person on it until the numbers say otherwise
For the first weeks, the agent drafts and a person approves. Count how often the person changes the draft, and tag the ones the agent got wrong. At Uber I tagged false positives to train the models, and each tag went into the next round of training. When that number is low and steady, let the agent send on its own for that one kind of request. At iAdvize, for Samsung's Galaxy S20 launch, the sentiment analysis in live chat flagged at-risk customers in real time and alerted senior executives.