Skip to content

How I test AI tools

I follow the same rules in every test: I write the verdict rule before the test, keep both sides identical except for one difference, check every item one by one instead of trusting the model when it says "done", and have the outputs judged in a blind vote.

The verdict rule is written before the test

The test's question, the sides being compared, the measurements and the verdict threshold are written down and locked before the test starts. Anything that changes later is noted with its date. That way I can't interpret the result afterwards in whatever way suits me.

Thresholds are written as numbers. In the Fable test, it was green if Fable spent less than 1.5 times what Opus spent, and red if it spent 2.5 times as much or more. In the Argon test, it was green if Argon finished no more than 5 points behind and cost at most half as much. If a test is expanded later, the new threshold is also written down before the new tasks are run.

Each task is run only once. That's why I say "one run, so luck plays a part" in every video.

Green, red and split

  • Green: it workedThe tool finished the job, passed the checks and stayed on the right side of the threshold.
  • Red: it brokeThe claim didn't hold, the job wasn't finished, or the threshold was crossed.
  • SplitOne eye green, one eye red: it works, but there's a price.

The two eyes can look at two separate questions. In the Fable test, the left eye looked at quality and the right eye at the usage limit. In the Argon test, the left eye looked at the work and the right eye at the money.

A tool that gets a red has the right to appeal and can be judged again. For Stageshow, against the objection "the prompt was bad", I ran an appeal round with a detailed brief, then one more run following the skill author's own instructions. There are no sponsors; I pay for the tools I use myself.

One variable

  • The sides run with the same tool, the same account, the same settings and word for word the same prompt. The only difference is the thing I'm measuring: in Stageshow, whether the skill is installed; in the Fable test, the model.
  • The prompt doesn't name the thing being measured. In the Stageshow test, Claude decided on its own whether to use the skill.
  • Test folders are kept outside my own project notes, so my instructions don't leak into either side.
  • The sides run one after the other. Each side starts in its own clean folder with a new session.
  • Settings that would skew the measurement are turned off and noted. In the Fable test, fast mode was off and both models' effort was fixed at the same level.

The intervention rule

The session is sent only the task, and I wait until it says "done". Apart from that, the sentences I can write are fixed in advance, and each one is counted:

  • If the model asks a question, there is one answer: "You decide, do whatever is best." This counts as an intervention.
  • If it goes in circles on the same error for more than 10 minutes, or stops before the task is done, "Continue" is sent. This counts as an intervention too.
  • Approval prompts don't count as interventions, but they are counted in a separate column.
  • If the quota runs out, the session stops there and is judged as it stands at that point.

The acceptance list

Each task's acceptance list is written before the run. There are usually five items, each worth one point. The items are measurable. For example: "when a subscription is deleted in the app, the widget updates within 30 seconds".

Test materials and answer keys are kept in a folder the sides can't see. In second-round tasks, the first round's items are checked again. The condition "don't break anything that works" is part of the score too.

How the checks are done

I don't trust the model when it says "done". Every item on the acceptance list is tried one by one:

  • Web tasks are checked in headless Chrome at phone size, with real touch, mouse and keyboard input. No eyeballing.
  • iPhone tasks are built and tested with Xcode UI tests in a clean simulator.
  • For video outputs, format, length, loudness, and frozen and black frames are measured.
  • A glitch caused by the checking method or the setup isn't counted against the model; the run is redone from scratch and the reason is noted.

Measurements

MeasurementHow
TimeFrom the prompt to "done", taken from the session log's timestamps. Time spent waiting for approval is noted separately.
CostThe tokens in the session log are converted to dollars at the API list price on the day of the test. On a subscription you don't pay this amount separately, but it is what comes out of your limit. If the price is about to change, a second calculation is given too.
InterventionsThe number of cases covered by the rule; approvals in a separate column.
Does it really workWhat the model claimed when it finished ("I tested it", "it works on the phone") and whether the claim held up in the checks. A claim that turns out to be false affects the verdict.

No other session runs during a test. If one did, that is written down and the measurement is chosen accordingly.

Blind test

  • Outputs are recorded the same way, at the same size, and randomly numbered "1" and "2". The key isn't revealed until the votes are in.
  • The judges are me and two AI models. The models don't see the folders or the session logs, only the recordings or the frames. A judge who learns an earlier result is replaced.
  • The questions are written in advance. The tested tool's own checklist isn't used, because it would favor its own side.
  • If the blinding is weak, that is stated openly. In Stageshow, I had watched the sessions, so I didn't count my own vote as fully blind; both model judges picked the same side, so the verdict didn't change.
  • If a recording is unfair to one side, it's thrown out and recorded again.

These rules are reviewed with every new test. If every side gets full marks, the test isn't telling quality apart; so new tests include at least one task that separates the sides from the start.