Almost all the companies we meet have already 'tested AI'. Someone demonstrated a stunning result in a meeting, enthusiasm rose, and then nothing. Six months later, the tool is used by three people, without a common rule, and no one knows if it saves time.
This is not a model issue. It is a scoping issue: a test is not a pilot, and a pilot is not an implementation.
Why AI demos nearly always look successful
A demonstration is conducted on a selected case, with proprietary data, by someone who can articulate the request, and judged by someone who is discovering it. These four conditions are never found in production.
The right question is not 'does it work?', but 'does it still work when the data is dirty, the request is poorly formulated, and the result is checked by an exacting person?'
Five safeguards for a serious pilot
A successful pilot always hinges on five key points. Missing one leads to an extended demonstration.
1. A task, not a technology
Using AI is not a project. Reducing by half the time spent preparing a sales proposal is. The task must be named, frequent, and already carried out by someone today: this gives a point of comparison.
2. An Explicitly Authorised Data
What sources is the system allowed to access? Who decided this? Can customer data leave the company? Until these answers are written, the pilot will advance on unstable legal ground, and no one will dare to expand it.
3. Human Validation Placed in the Right Spot
Human control should not be everywhere: it should be where an error could impact the company. A meeting summary can be casually reviewed. A quote, a contract, or a message sent to a client cannot be sent without a recheck. Placing this validation point is a design decision, not an interface detail.
4. A Observable Business KPI
Not 'the quality of responses,' too vague, but metrics that can be tracked weekly: average time per file, rework rate after review, number of files processed, number of errors detected downstream. An indicator that no one tracks does not exist.
5. Known Cost in Advance
The cost per call in the model, multiplied by the actual volume, gives a monthly budget. Many pilots work perfectly and turn out to be economically absurd at scale. It’s better to know this beforehand.
The questions to ask before any pilot
- Which specific task is performed how many times per month?
- What data, authorised by whom, and to what extent?
- Who validates, when, and on what criteria?
- What key performance indicator will we track, and how often?
- What is the actual monthly cost per volume expected?
Measure quality before expanding
The simplest method is often the best: compile a set of thirty to fifty real cases, representative ones, including the difficult cases, and have the person doing the work today evaluate the results. Two scores suffice: usable as is, or not.
This evaluation takes half a day and is worth more than three months of debate. It also provides a basis: when changing the model or formulation, we will know if we are progressing or regressing.
How privacy changes the approach
Privacy is not a hurdle to add at the end: it determines the architecture. Depending on the sensitivity of the data, we will choose a cloud service with a suitable contract, European hosting, or a model executed in your infrastructure. These three options have neither the same cost nor the same level of performance: early arbitration avoids rebuilding everything.
In all cases, a few simple rules are sufficient to avoid most incidents: never share personal or contractual data without clear legal basis, log what has been sent, and train teams to recognise what should not be shared.
Scale up or stop
At the end of a pilot, three outcomes are acceptable: extend, adjust, or stop. The fourth, letting it live without deciding, is the most costly because it installs unregulated usage.
Stopping a pilot is not a failure. It is the normal result of an experiment on two or three subjects: the goal was to learn quickly and at low cost, and that is exactly what happened.
To keep in mind
- Start with a frequent and already performed task: it provides the point of comparison.
- Write authorised data before writing the first prompt.
- Place human validation where errors could impact the company, not everywhere.
- Evaluate up to thirty to fifty real cases, including difficult ones.
- Explicitly decide at the end: expand, adjust, or stop.