Evaluation Protocol
We do not publish benchmark claims before the cases are reproducible. FineSchema will be evaluated through reproducible case sets across emotion inference, historical claim verification, and AI action verification.
100 authored cases · reproducible JSON
Deterministic prototype · npm run eval
Naive classifier · rule-only gate · LLM judge · human reviewer
Withheld until the case set is reproducible
Nothing hidden — every case, every miss.
All 100 cases with the prototype engine's actual output: what it matched, what it missed, and why each case is hard. Failures are shown, not filtered. Aggregate scores stay unpublished until the set is stable.
35 Emotion cases · 35 Historical Truth cases · 30 AI Action Verification cases — 100 authored cases across emotion, truth, and action verification.
A case can have the correct truth state while still missing important reasoning findings. Verdict match and finding coverage are reported separately, never as a single score.
Permission with withdrawal
I'm fine. Do whatever you want.ctx: Romantic conflict after a delayed response.
Surface expression and context conflict; emotion hypotheses retained, none asserted.
Surface calm, latent withdrawal. Reducing it to one emotion is the failure.
One protocol, three kinds of judgment.
Each track has its own case set, its own findings vocabulary, and its own failure modes — scored by the same runner.
Affective Cognition
Whether the engine separates surface emotion from latent emotion hypotheses, using text and context.
- Surface emotion
- Latent emotion hypotheses
- Contradiction detection
- Uncertainty retention
- Safe response quality
Historical Truth Verification
Whether a single historical claim, compared against multiple evidence sources, yields a justified truth state.
- Claim decomposition
- Source reliability scoring
- Contradiction detection
- Verified / uncertain separation
- Final truth state consistency
AI Action Verification
Whether a proposed agent action is sufficiently evidenced — and whether overgeneralized or unsafe action logic is caught.
- Evidence sufficiency
- Missing evidence detection
- Overgeneralization detection
- Action severity judgment
- Recommended safer action
Baseline comparison planned: naive classifier, rule-only gate, LLM judge, and human reviewer. Actual numbers follow once the case set and runner are stable.
Built in the open, in this order.
No public benchmark score yet. The protocol is being built before claims are published.
Reproducible protocol
100 authored cases across three tracks; deterministic runner scores the prototype every build.
Baseline adapters
Naive classifier and rule-only gate run on the same cases; LLM-judge and human-reviewer adapters stubbed, ready to connect.
Coverage & calibration
Deepen controls, contradictions, and designed failures per track, and tighten finding vocabulary before any comparison is shown.
Published comparison
Once the set is stable, publish per-track results with failures — never a single cherry-picked number.
The same cases, run four other ways.
FineSchema earns nothing by grading its own homework. Each baseline runs on the identical case set so the comparison is honest. Two are scaffolded; two are stubs awaiting an external key or a human.
Naive classifier
scaffoldedSurface signal only — never preserves uncertainty. The floor any real engine must beat.
Rule-only gate
scaffoldedHard keyword gates, no evidence-reliability weighting. Strong on blatant defects, blind to nuance.
LLM judge
stub · needs API keyAdapter interface + placeholder result. Requires an external LLM key, not wired in this phase.
Human reviewer
stub · needs reviewerAnnotation-queue adapter. Requires a human adjudicator, folded into the same harness later.
A case is a contract, not an anecdote.
Finding vocabulary is used to make evaluation reproducible instead of relying on loose sentence matching. Every case states its expected truth state, the canonical finding keys the engine must surface, and the outputs that are unacceptable. The runner is deterministic — same cases, same verdicts, every run.
FineSchema is not evaluated only by final verdict. It is evaluated by whether the reasoning trace preserves the right uncertainty, contradiction, and missing-evidence findings.
type EvaluationCase = {
id: string;
track: "emotion" | "truth" | "action";
title: string;
input: string;
context?: string;
evidence?: EvidenceItem[];
expectedTruthState: TruthState;
expectedFindingKeys: string[]; // canonical keys
expectedFindings: string[]; // human-readable note
unacceptableOutputs: string[];
difficulty: string;
whyThisCaseIsHard: string;
};
type EvaluationResult = {
caseId: string;
predictedTruthState: TruthState;
predictedFindingKeys: string[];
matchedFindingKeys: string[];
missedFindingKeys: string[];
falseFindingKeys: string[];
findingKeyCoverage: { matched: number; expected: number };
uncertaintyRetained: boolean;
verdictMatched: boolean;
};No benchmark numbers will be published until the case set is reproducible.
FineSchema is evaluated not only by correctness, but by whether it preserves uncertainty when certainty is not justified.
The goal is not to always answer. The goal is to know when not to answer.
A strong engine should sometimes say UNKNOWN, UNCERTAIN, or CONTESTED.
Current set: 100 authored cases — 35 Emotion cases · 35 Historical Truth cases · 30 AI Action Verification cases. Deliberately ambiguous; keyword-guessable cases are excluded, and designed-failure cases are kept and labelled.
Why UNKNOWN, UNCERTAIN, and CONTESTED matter.
Strong systems must know when not to answer. These three states are the product, not a weakness — they are where most AI quietly guesses.
When the evidence cannot decide, inventing a verdict is the failure — not the restraint.
Holding several plausible readings is a judgment, not the absence of one.
A real dispute should read as disputed — collapsing it to one side erases the disagreement.
Nine states — when each is used, and what it does not mean.
A binary true/false destroys information. Each state has a job; each has a boundary it must not overstep.
What makes a result we would trust.
The protocol is being built before claims are published. This is the bar a published number must clear.
- 01Every case declares its expected state, required findings, and unacceptable outputs.
- 02The runner is deterministic — same cases in, same verdicts out, every run.
- 03Cases are merged from per-track source files with a fixed naming rule.
- 04Designed-failure cases are kept and labelled, not removed to flatter the score.
- 05Control cases guard against over-blocking; thin-signal cases force UNKNOWN.
- 06No aggregate score is published until the set is stable.
Failure Cases Matter
FineSchema is not designed to hide uncertainty. Every failed or contested case will be used to improve schema coverage, contradiction detection, and uncertainty calibration.
A defect reported where none exists — manufactured doubt.
A real defect the schema failed to catch.
A legitimate, policy-scoped action blocked by a coarse gate. One such case is already in the set, by design.
An unsafe action allowed because its rationale used the right words.
Asserting a verdict where the evidence only supports UNCERTAIN — or the reverse.
Failing to name what evidence would have changed the verdict.
Judgment you can inspect.
We are opening early conversations with researchers, institutions, and teams who need inspectable judgment — in history, in emotion, in AI action.
Current demos are illustrative prototypes. Production FineSchema will connect to domain evidence, policy cartridges, and live verification APIs.
See how FineSchema will be evaluated →