AI Assistant Design
Scoping an assistant that does one job well, instead of one that does everything approximately.
An assistant that can do anything will be trusted with nothing.
The common failure isn't technical. It's scope. Businesses build a general-purpose assistant, get inconsistent results across every use, and can't tell whether the problem is the model, the data or the question.
A narrow assistant that handles one job reliably is worth more than a broad one that handles ten jobs at seventy percent.
Read about human-in-the-loop design · Request Your Free Systems Audit
Start with the job, not the capability
The wrong question: what could an AI do for us?
The right one: which specific, recurring task costs us time and has a knowable right answer?
Good candidates share three properties:
Recurring. It happens often enough to matter.
Bounded. The range of inputs is predictable.
Checkable. You can tell whether the output was right.
The third is the one most often missing. A task whose output nobody can evaluate is a task where errors accumulate invisibly.
Four shapes of assistant
Most useful assistants are one of these. Pick one deliberately rather than building something that drifts between them.
The answerer
Responds to questions from a defined body of knowledge. Internal policy questions, product details, procedural queries.
Needs: a clean knowledge base, and a defined behaviour for "I don't know."
The drafter
Produces a first version a human finishes. Replies, summaries, descriptions, reports.
Needs: examples of good output, and a human who actually edits rather than forwards.
The classifier
Sorts things into categories. Enquiry types, priority, routing, sentiment.
Needs: clearly defined categories, and a rule for ambiguous cases.
The extractor
Pulls structured data out of unstructured input. Details from emails, fields from documents.
Needs: a defined output format, and validation on what it produces.
The classifier and extractor are the most reliable and the least glamorous. They're where most of the real value sits in a small business.
Define the boundaries explicitly
Write these down before building:
What it handles. Specifically. Not "customer questions" — which customer questions.
What it refuses. The cases where it should decline rather than attempt. Pricing exceptions, complaints, anything legal, anything about an individual's account.
What it escalates. When it hands to a person, how, and with what context attached.
What it never does. Sends anything externally, changes a record, commits to anything.
The refusal behaviour matters more than the capability. An assistant that confidently answers a question it shouldn't have touched causes more damage than one that handles fewer cases.
Design the "I don't know" path first
Most assistants are designed around the cases they handle. The cases they can't handle are where trust is won or lost.
It should say so plainly. Not produce a plausible answer from adjacent material.
It should hand over cleanly. To a named person or queue, with the original question and what it found attached.
It should log it. Unanswered questions are your documentation backlog, generated by real demand.
If you can't make the assistant reliably recognise its own limits, narrow the scope until you can.
Keep the human in the right place
Match the review to the risk:
| Output | Review |
|---|---|
| Internal answer to an internal question | None, log and sample |
| Draft for a person to send | The person editing is the review |
| Classification that routes work | Sample weekly |
| Anything reaching a customer directly | Per item, before sending |
| Anything changing a record or moving money | Never autonomous |
The mistake in both directions: reviewing everything until review becomes a formality, or reviewing nothing because the early results looked good.
Give it the right inputs
An assistant performs on what it can read. Three things matter:
Scope the sources deliberately. Connect what the job needs, not everything available. Broad access makes output harder to predict and problems harder to trace.
Resolve contradictions before connecting. Two documents with different answers produce a confident blend of both.
Prefer live sources for volatile data. Prices, availability and status should be read from the system that owns them, not from a document describing them last quarter.
Write the instructions like a procedure
The instruction set is a document someone else should be able to read and understand.
Include: what the assistant is for, what it has access to, the format of its output, what it must refuse, how it escalates, and the tone it uses.
Keep it in version control, with an owner and a review date, like any other procedure. An instruction set that nobody can find and nobody owns drifts the same way documentation does.
Test before trusting
Build a set of real cases. Twenty to fifty actual examples from your business, with the correct answers agreed in advance.
Include the hard ones. Ambiguous inputs, missing information, questions that should be refused, things just outside scope.
Run them before launch, and again after any change. Instruction changes have non-obvious effects; a test set is how you notice.
Keep the failures. Every real-world mistake becomes a test case.
What to measure
| Measure | Tells you |
|---|---|
| Questions handled without escalation | Coverage |
| Escalations | Where the scope boundary is |
| Corrections to drafted output | Whether drafting saves time |
| Wrong answers found | The thing that matters most |
That last one needs a deliberate process to find — wrong answers that nobody checks look exactly like right ones.
Sample regularly. Weekly at first, monthly once stable.
Signs the scope is wrong
- Results are inconsistent across different kinds of question
- The instruction set keeps growing to handle new exceptions
- People have learned which questions to avoid asking it
- You can't explain in one sentence what it's for
Each of these means narrow the scope rather than refine the configuration.
When not to build one
The underlying process isn't documented. The assistant will need the same knowledge a person would, and it won't ask.
The right answer is genuinely contested inside the business. Fix that first.
Volume is low. The setup and maintenance cost exceeds the saving.
Nobody will own it. An assistant with no owner degrades as the business changes around it.
One job, done reliably, with a clear boundary. Scope is the design decision that determines everything else.
Wondering where AI would actually help?
A free systems audit identifies the tasks worth it, and the ones that need fixing first.