Most pilots succeed. That is the problem — a trial that cannot produce a negative result has not tested anything.
Start a conversation with the AI Adoption Concierge, already scoped to pilots & proof of value. Pick a starting point, or describe your situation directly.
The standard firm pilot recruits volunteers who are already interested, runs on convenient work, has no agreed measure of success, and concludes with a positive impression. It would have returned that result for almost any product. A pilot worth running has a defined task, a defined comparison, participants who are not all enthusiasts, and a pass mark set before it starts. It should also measure the thing the firm actually cares about — which is rarely raw speed and usually whether the work was good enough to use without redoing it.
Each of these is routinely omitted, and each omission biases the result toward yes.
A specific job, not "try it and see". Broad pilots produce broad impressions and no decision.
Agreed before the trial, so the result can be no rather than reinterpreted afterwards.
Include sceptics and ordinary users, or you have measured enthusiasm rather than the tool.
Against the current way of doing it — otherwise there is no baseline and no claim to make.
Output good enough to use without redoing. Time saved that produces unusable work is not saved.
Including what did not work, which is the part that makes the next pilot better.
How a pilot is run.
Usually not the licence fee. Usually the year that follows it.
A pilot staffed entirely by volunteers measures the volunteers. The colleague who thinks this is overhyped will find the failure modes fastest, and if the tool convinces them it will convince the firm.
Rework rate is the most informative single measure and the least often used: what proportion of the output was good enough to use without substantial redoing. Time saved is worth capturing but misleads on its own, because a tool that halves drafting time and doubles review time has saved nothing. Reviewer-rated quality on a simple scale, applied by someone who did not produce the work, adds the dimension raw speed misses. Adoption during the pilot is also telling — people quietly stopping is a result.
Something repetitive, well-bounded, internally facing, and easy to score. Summarising documents into a defined format, first-pass review against fixed criteria, or drafting routine correspondence all work. The temptation is to pilot on the most valuable and most complex work, because that is where the upside looks largest — but complex work is hard to score, high-risk to get wrong, and produces an ambiguous result that settles nothing.
Set an end date and a decision point at the start, and name who decides. A surprising number of firm pilots simply continue: the trial licences roll over, usage drifts, and no one ever decides whether it worked. That is the worst outcome, because the firm bears the cost and the governance exposure without ever having concluded the tool is worth it. A written result at a fixed date — pass, fail, or extend for a stated reason — avoids it.
This is a question for the firm and its own advisers, and worth resolving before the pilot rather than during it. It touches client confidentiality, outside counsel guidelines that increasingly address AI use, and the communication duties discussed in guidance such as ABA Formal Opinion 512. Many firms structure early pilots on internal or non-confidential material specifically to avoid the question until the governance is settled, which is usually the simpler sequence.
Describe the task and who would take part. The Institute will help you set the measure.