How to Add AI to an Existing Product (Without Rebuilding It)

There’s a particular kind of pressure that lands on product teams with an existing, working product: the board, the competitors, or the customers all want to know what you’re doing with AI. The instinct is to reach for something big — a rebuild, a platform migration, an “AI-first” repositioning. That’s almost always the wrong first move.

The useful version of this work is narrower and much less disruptive. You pick one workflow where a language model genuinely helps, you build a thin layer that calls a model provider, you measure whether the output is good enough, and you ship it behind a flag. Done properly, the change touches a handful of files and a new service boundary. Done badly, you get a feature that demos well, costs more than it earns, and quietly hallucinates at customers.

This guide is for product and engineering leaders at companies with a product already in production and paying customers. It assumes you’re not building foundation models, you’re not staffing a research team, and you need the result to be maintainable by the engineers you already have.

Picking the Right First Use Case

Most failed AI features fail at this step, before any code is written.

What makes a good first use case?

Look for work that is currently done by a human, is language-shaped, is repetitive, and has a tolerant failure mode. Summarising a long support thread, drafting a first-pass reply, extracting structured fields from a messy document, tagging and routing inbound requests, converting a natural-language question into a filter your existing search already supports — these all share the same property: a wrong answer is a mild annoyance a human catches, not a financial or legal event.

Avoid, for a first project, anything where the model is the last line of defence. Pricing decisions, compliance determinations, irreversible actions on customer data, or medical and legal advice all need a much heavier engineering investment than a first feature should carry.

A quick filter:

Question	If the answer is no
Is there a human doing this today?	You may be inventing demand, not serving it
Can you show a “good” and “bad” output?	You can’t evaluate it, so you can’t ship it safely
Does a wrong answer get caught cheaply?	Push it later, behind approval workflows
Do you have the data it needs?	Fix retrieval before touching models
Would users pay attention if it saved them ten minutes?	Pick a more painful workflow

Start where the value is measurable

If you can’t say what the feature is supposed to move — handling time, conversion, time-to-first-draft, tickets deflected — you’ll have no way to defend it when someone asks about the invoice. Pick a metric before you start, and capture the baseline before you launch.

Build a Thin Integration Layer

The single most important architectural decision is to keep the model at arm’s length.

Put all model calls behind one internal module or service with your own interface: a function that takes a task and typed inputs, and returns typed outputs. Your application code calls summariseThread(thread), not a provider SDK directly. Behind that boundary you own prompt construction, retries, timeouts, token accounting, caching, logging and the choice of provider.

This gets you several things at once:

Provider portability. Model quality, pricing and availability move quickly. If swapping providers means editing one adapter rather than forty call sites, you can respond to that in an afternoon.
A place to put guardrails. Validation, redaction and rate limiting live in one place instead of being re-implemented per feature.
Testability. You can stub the boundary and keep the rest of your test suite fast and deterministic.
Observability. Every call flows through code you control, so logging and cost attribution are free rather than retrofitted.

Keep prompts in version control as files or typed templates, not glued together inline. Prompts are behaviour; they deserve the same review and history as code.

Treat the model as an unreliable network dependency, because that’s what it is. Set aggressive timeouts, decide what happens on failure (degrade to the non-AI path, don’t hang the page), and never make a synchronous model call sit in the critical path of something that must not fail.

RAG or Fine-Tuning?

This is the question teams get stuck on, and for most products the answer is clear.

Retrieval-augmented generation

RAG means fetching relevant content from your own data at request time and putting it into the prompt as context, so the model answers from your material rather than from its training. It’s the right default when the answer depends on facts you own — your documentation, your tickets, your catalogue, your customer’s own records — especially when those facts change.

The hard part of RAG is retrieval, not generation. If you hand the model the wrong three paragraphs, no amount of prompt engineering rescues the answer. That means:

Chunk documents along meaningful boundaries (sections, headings) rather than fixed character counts where you can.
Combine keyword and vector search. Pure semantic search is notoriously weak on exact identifiers, product codes and names.
Apply your existing permissions to retrieval. If a user can’t read a document in your app, it must not reach their prompt. This is the most common serious bug in RAG systems.
Return sources with the answer. Citations make the feature verifiable and dramatically reduce the harm of a wrong answer.
Measure retrieval separately from generation. “Did the right chunk make it into context?” is a question with a clean answer.

Fine-tuning

Fine-tuning adjusts a model’s weights on your examples. It’s good at teaching form — a consistent tone, a house output format, a domain-specific classification style — and poor at teaching facts, because the facts get frozen into the model and go stale.

For a first AI feature, you almost certainly want RAG plus careful prompting. Consider fine-tuning later, when you have a meaningful volume of reviewed examples, a stable task, and a clear reason: usually cost or latency, where a smaller fine-tuned model matches a larger general one at a fraction of the price.

Evaluate Before You Launch

“It looked good when I tried it” is not evaluation. You need a way to tell whether a prompt change made things better or worse, because you will change prompts constantly.

Build an evaluation set: 50 to 200 real inputs drawn from production data, each with a note on what a good answer contains. It doesn’t need to be elaborate. A spreadsheet and a script that runs the current pipeline over every case and writes the outputs to a file will carry you a long way.

Then score. Different tasks need different scoring:

Extraction and classification can be scored exactly against labelled answers.
Retrieval can be scored on whether the correct source appeared in the top results.
Open-ended text needs human review, or a model-graded rubric that you have spot-checked against human judgement.

Run the evaluation in CI on prompt and model changes, and keep the results. The benefit compounds: six months in, the difference between teams that can ship AI changes confidently and teams that can’t is almost entirely whether they built this early.

Guardrails and Human-in-the-Loop

Constrain the output

Ask for structured output — JSON against a schema — wherever the result feeds another system, and validate it before use. If validation fails, retry once with the error, then fall back. Never let raw model text reach a database write, a shell, a query, or a rendered HTML surface without passing through something that checks it.

Keep a human on consequential actions

For anything with a real-world effect — sending an email, issuing a refund, changing a record, publishing content — the strong default is draft-and-approve. The model produces a proposal, a person accepts, edits or rejects it. This preserves most of the time saving while capping the downside, and it generates exactly the review data you’ll want later.

As confidence grows, you can automate the narrow cases where evaluation shows the model is reliable and the action is reversible. Move that line deliberately, with data.

Be honest in the interface

Label AI-generated content. Show sources. Give users an obvious way to report a bad output, and route those reports somewhere a human reads. Users forgive a system that is visibly uncertain far more readily than one that is confidently wrong.

Cost and Latency

Both are design constraints, not afterthoughts.

On cost: you pay per token, so context size is your main lever. Retrieve three good chunks instead of twenty mediocre ones. Cache aggressively — identical or near-identical requests are common in real products. Route by difficulty: use a small, cheap model for classification and routing, and reserve the expensive model for the tasks that need it. Set hard per-user and per-tenant limits before launch, not after the first surprising bill. Log token counts per feature so you can attribute spend.

On latency: model calls are slow compared with everything else in your stack. Stream responses so the user sees progress. Move anything that doesn’t need to be interactive into a background job. Do retrieval in parallel with whatever else the request needs. If a task requires several chained model calls, ask whether one well-constructed call would do — chains multiply both latency and failure modes.

Privacy and Data Boundaries

Before the first call goes out, answer these in writing: what data leaves your infrastructure, which provider receives it, where is it processed, is it retained, and is it used for training? Providers differ, and their enterprise terms usually differ from their default terms.

Then work through the implications:

Update your privacy notice and sub-processor list. If you process personal data on behalf of customers, adding an AI provider is a sub-processor change and may carry notification obligations under your contracts.
Redact or tokenise what the model doesn’t need. Most tasks don’t require real names, card details or full identifiers in context.
Enforce tenant isolation in retrieval, caching and logging. A shared cache keyed only on prompt text can leak one customer’s data to another.
Be careful with prompt logs. They’re invaluable for debugging and they contain everything the user typed, so treat them with the same retention and access controls as your production database.
Give enterprise customers a way to opt out, and know whether a self-hosted or regional deployment option exists if they insist.

Monitoring in Production

Ship behind a feature flag to a small cohort first. Then watch:

Quality signals: thumbs-up/down rates, edit distance between draft and sent version, how often users fall back to the manual path.
Failure signals: schema validation failures, empty retrievals, provider errors, timeouts.
Cost signals: tokens and spend per feature, per tenant, per day, with an alert on unusual growth.
Latency: at the 95th percentile, not the mean. Averages hide the experience people complain about.

Sample real conversations and read them weekly. Nothing else tells you as quickly that the feature is being used for something you never designed it for — which is usually where the next version comes from.

Common Mistakes

Starting with a chatbot. A general assistant is the hardest possible first project: unbounded input, no clear success criterion, and high user expectations. A narrow, embedded feature is easier to build and easier to justify.

Skipping the baseline. If you don’t know how long the task took before, you can’t show it takes less time now.

Letting the model do arithmetic or lookups. Call your existing code for anything deterministic. Models are for language, not for maths or for being a database.

Treating prompts as untracked configuration. Undocumented prompt edits are the leading cause of “it used to work last week”.

How Ten Peaks Tech Can Help

We help teams add AI features to products that already exist and already have customers — choosing the use case, building the integration layer, and putting evaluation and guardrails in place before anything reaches users. See our AI integration services, or our software development work if the feature needs product engineering around it.

Related reading: Prompt Injection: How to Protect LLM-Powered Apps on securing these features, Application Security Checklist for SaaS Startups, and What Does a Fractional CTO Do? if you need help deciding what to build at all.