Every AI company has impressive numbers. Very few explain how those numbers could be wrong. If we are going to ask organisations to trust Kybern near consequential decisions, we owe them the opposite: a test we could fail, run in a way they can check.
We call it the Reality Challenge. This article explains where it stands today, what the current result does and does not show, and what happens next.
Where it stands today: an internal harness
Kybern runs the same engine across 19 synthetic demonstration worlds, from naval readiness and grid operations to banking settlement and hospital capacity. Inside each world we built test cases that probe eight behaviours we consider essential for a system that advises on real decisions, including real attempts to make it act outside its authority.
On that internal harness, Kybern passed 259 of 259 cases. A naive, dashboard-style baseline given the same cases passed 74.
Why that is not proof
We designed the harness ourselves, on our own synthetic data. Many of the baseline’s failures are failures by construction: a dashboard was never designed to notice a missing report, keep two conflicting sources, or refuse an action it has no authority to take. Beating it shows that Kybern does what it was built to do. It does not show that Kybern is right about the real world.
A test you cannot fail is a demonstration, not evidence. The internal harness is a demonstration. We are publishing it as one.
What happens next: the real test
The next stage is run with partners, on their terms rather than ours:
- Expert cases, scored blind. Cases come from people who know the domain, and the scoring is done without knowing which answers came from Kybern.
- Success agreed before the run. What counts as a good result is fixed in advance, so nobody can move the goalposts afterwards.
- Failures published. Where Kybern gets it wrong, we say so, and we say what we changed.
Keeping score, all the time
The Reality Challenge is a milestone, but the habit behind it runs through the product. Kybern writes every forecast down before the outcome is known, scores it against what actually happened, misses included, and compares it with a naive forecast. A miss is treated as a warning that its picture of the world may be wrong, not as noise to be hidden.
That record grows with every pilot. Over time it becomes the most honest answer we can give to the only question that matters: can you trust it with a real decision?
Take part
We are looking for pilot and research partners who want to shape the real test: bring the cases, agree the yardstick, and see the results, good and bad.
Results above are from Kybern’s internal test harness on synthetic data. Kybern is in working prototype, with pilots open.