How we test software
We give every tool the same fixed tasks and publish everything: the prompts, the input files, the pass rules, each full answer and a screenshot. You can check every result, or run the test yourself.
What we test
Each test procedure is a list of everyday tasks with exact prompts and written pass rules. The procedures have version numbers. Current procedures: AI assistants: 10 everyday tasks.
How a run works
- We use the tool's free version only and never enter a card. If the tool works signed out, we test it signed out and say so.
- Runs are done by the SoftwareGradebook test lab (scripted runs): an automated Chrome browser on Windows 11, with a clean profile and a desktop-size window.
- Every task starts in a new chat. The exact prompt is pasted. There is one try and no follow-up question.
- The full answer is saved as text, and a screenshot is taken. Both are published on the review.
How a task passes
A task passes only when every pass rule holds. Word counts, reading level, JSON, the spreadsheet formula and required words are checked by the grading script, which we publish. A few rules need a reading of the answer, for example that a cited link really states the fact. The test agent checks those, and the answers are published in full so anyone can check them.
If a tool cannot do a task on its free version (for example, no file upload when signed out), the task fails and the reason is written next to it.
How the score works
The score is the number of tasks passed, scaled to 10. It is computed from the saved results only. Nobody types a score. Answer time is shown when we could measure it, but it is not scored, because timing in an automated browser is not exact.
Not yet tested
A tool we could not run stays "Not yet tested", with the reason and the date. We never estimate a result.
Retests and changes
We retest when a tool's free plan or a procedure changes, and at least every three months. When a rule changes, the procedure gets a new version. Older results stay visible but stop counting in scores. Every change to a published result is listed on the corrections page.