A minimum reporting standard for AI evaluations
A short proposal for the least an evaluation result has to disclose before it can be compared with another result or repeated by anyone else.
What this is
Independent research on how AI systems actually behave
Dingal AI Research is a one-person practice. It runs evaluations of AI systems, writes up what the results do and do not support, and publishes the methods and code so anyone can check them.
Featured
All notesFailure compounds faster than per-step accuracy suggests
Measuring how language-model agents fail over long tasks. Per-step accuracy barely separates the systems; end-to-end success separates them a lot.
The work
AboutWhat I work on
Measuring what AI systems can and cannot do reliably, and how those measurements should be reported so they mean something.
The researchHow the work is published
Methods in full, code and data where I can release it, and the limitations stated plainly. Nothing is held back because it turned out inconvenient.
Data and codeWho this is
A single researcher working independently. No institution behind it, no funding conditioned on a result, no products to sell.
AboutLatest
All notesThe evaluation harness is now public
The code I use to run and record my own evaluations, released so the results here can be reproduced.
A year of interpretability work, and what held up
Which methods for looking inside a model produced findings I would still defend, and which did not survive being checked.
Why this exists
What this practice is for, what it will publish, and the things it will not do.