Most evaluations go wrong the same way: someone runs the vendor’s demo, is impressed, buys, and discovers in week three that the limit is minutes rather than meetings, or that the output cannot be exported. The test below takes an afternoon and answers the questions that actually cause regret.
1. Write down the job, in one sentence
Before opening anything, state what you are buying: “transcribe and summarise four client calls a week”, “produce twenty on-brand social images a month”, “draft first versions of support replies”.
That sentence eliminates most of a shortlist immediately, because it exposes the shape you need. A tool that generates images is not a tool that lays them out, and a notetaker that joins calls does nothing with a folder of PDFs.
2. Bring your hardest input
This is the step people skip and the one that decides everything.
For a transcription tool, use a call with two accents, crosstalk and your own jargon, not a clean solo recording. Every tool handles the clean one. For writing, use a full piece at the length you publish, since tools that hold a brief for a paragraph often lose it by 1,500 words. For data, use a real file with the missing values and inconsistent headers your files actually have.
Otter and Descript transcribe near-identically on clean audio and separate visibly on a bad microphone. You will not see that on a demo.
3. Judge the second attempt, not the first
The first result is where vendors compete and where the difference is smallest. The second is where you find out whether you can fix one detail or must start again.
Ask for a change: shorten this paragraph, keep the character but change the background, redo the chart as a percentage. Tools with inpainting, style references or editable output let you converge. Tools without them make you reroll and hope, which is a different cost entirely.
4. Price one finished result, including the failures
Take the meter from the pricing page and multiply by the attempts your test actually took.
If a usable image needed four generations at your quality setting, the plan’s credit count divides by four. If a shot took five tries, an allowance of forty clips is eight usable seconds. Midjourney at $10 gives 3.3 fast GPU hours, which sounds generous until you count exploration.
Then check the ceiling behaviour: Zapier switches to pay-as-you-go so automations keep running, others pause. One of those can produce a surprise invoice and the other can stop a workflow at three in the morning. Decide which you prefer before you meet it.
5. Read four lines of the terms
Is customer content used to train models, and does that depend on the plan. How long is it retained. Is there human review. Where is it stored.
For anything confidential, a setting is not the same protection as a contractual exclusion on a business plan. There is more on this in does AI train on your data.
6. Test the exit before the entry
The question nobody asks in month one: can I get my work out.
Can transcripts, designs and generated assets be exported in a format something else can open. Is there an API, or is this a place data goes in and never comes out. Does cancelling delete the archive.
A tool you cannot leave is a more expensive commitment than its price suggests, and the answer takes two minutes to find.
What not to weigh
Benchmarks published by the vendor. Feature counts. Which underlying model it runs, since the serious products all offer several and switch them every few months.
What survives contact with a real week is the meter, the workflow fit, the licence and the exit. Every listing in this directory states those four for exactly that reason, with the date its pricing page was checked.
