So this document stands on its own: a frontier success judge, asked whether a manipulation succeeded, returned false passes at a flat 0.95 confidence, right or wrong. The cause is representational. On a public corpus, an object's learned box moved 2.4% while its true visible extent moved 67.7%. The judge reasons over the box, so it never sees the change. The full account is the companion paper, Confidence Without Correctness. This document is the proposed response and the study that would settle it.
The architecture, and the pieces of it that already stand on evidence
The principle: separate who grounds from who decides
A governance rule, not a new algorithm
The diagnostic finding is that a learned judge cannot be trusted with the final metric call, because the representation it reasons over has already smoothed away the evidence. The response is not a better model. It is a division of authority.
The architecture assigns each function to the part that can be held accountable for it:
- The learned model grounds symbols: it names objects and points to roughly where they are.
- A system-owned policy defines the permitted change: what the task is allowed to alter.
- An independent measurement audits everything protected: the preservation complement.
- A validity layer decides whether that measurement deserves authority at all.
- Missing or compromised evidence produces abstain, never a learned guess.
The single load-bearing rule: the learned model may ground task symbols, but it may not author the system invariant, choose the protected set, set the unseen-task tolerance, or convert invalid measurement into a PASS. That rule is what makes the failure in the diagnostic paper unreachable by the policy: there is no learned surface left in the seat where the false pass was manufactured.
The evidence that already stands
Four results, each with its primary source file
| Component | Result | Status | Source |
|---|---|---|---|
| Existence proof. The firewall removes the failure class in a controlled setting. | Direct Gemini: 9 false passes in 20. Gold cascade: 0 false passes, 0 false fails, 16/20 correct, 4 abstentions. | established | RESULTS_SPEC_ EXECUTION_V2.2.md |
| Authority projection. The learned layer cannot widen the protected set. | Semantic IR compiles 24/28 contracts (from 22/28); 0/28 non-target authority widening; malformed and role-swapped contracts fail closed. | established | RESULTS_CONTRACT_ INTEROP_V2.4.md |
| Boundary map. Where the current implementation loses its evidence. | DROID endpoints: 0/12 complete coverage (arm occludes / containment hides). Multi-surface: 1 vision-temporal, 2 multi-surface, 9 richer-state-required. | established | RESULTS_DROID100_ GROUNDING_V2.5.md …_MULTISURFACE_V2.6.md |
| External surface. The firewall runs on a real public corpus with honest yields. | Registration 48/48; blind cross-check 293/336 (87.2%); dense-support 2 PASS / 0 FAIL / 3 ABSTAIN; 0/6 held-out false alarms. | established | RESULTS_LANGUAGE_ TABLE_*_V2.7–V3.2.md |
Read together, these say something narrow and true: the architecture is coherent, the failure it targets is real, one component (authority projection) is provably complete, and the whole runs on real public pixels while abstaining honestly where its evidence fails. That is a working skeleton with one bone fully set.
What is not established, and why a study is required
The headline efficacy claim is not yet supported by the evidence
The distinction that matters: the diagnostic paper was done and merely unwritten. This architecture is not done. Its central claim, that the firewall removes the confident-false-pass error class on untouched data at a usable yield, rests on controlled captures and small development sets, not on a clean held-out population.
Each gap is a thing the study below is built to close. None of them requires a new idea; each requires more subjects, independent ground truth, or a corpus with a property the current ones lack. That is precisely what a single well-designed study can supply, and it is the reason the honest next artifact is a study specification rather than a finished paper.
One pre-registered experiment that would settle whether this is a useful auditor or a careful apparatus
Objective and question
The one question the study answers
On untouched data with independent ground truth, does the authority-separated firewall remove the confident-false-pass error class relative to direct-VLM judgment and ordinary baselines, and at what characterized yield?
The study is designed to be able to return "no." A result that shows the firewall is safe but scores too few episodes to matter is a valid, publishable outcome, and the design below reports it plainly rather than rescuing it.
Pre-registered hypotheses
Committed before any measurement
- H1 · safetyThe firewall's false-pass rate is below the direct VLM's, and near zero in absolute terms, on held-out episodes.
- H2 · no bad tradeThe removed false passes become abstentions, not false fails. The firewall does not simply swap one unsafe error for another.
- H3 · yieldUsable coverage (the non-abstain fraction) at a fixed false-pass ceiling is characterized as a curve against capture discipline, not a single point.
- H4 · valueReviewer-yield is above zero: a measurable fraction of the firewall's flags and abstentions route a human to a genuine label error.
Arms compared
Four judgment surfaces on identical inputs, plus one fallback for out-of-envelope strata
| Arm | What it is | Role |
|---|---|---|
| A · direct VLM | the learned judge's own success verdict + confidence | the incumbent being audited |
| B · global change | ordinary whole-frame change detector | cheap baseline, raises the bar |
| C · firewall | frozen v3.2 role-local support + validity gate | the proposed instrument |
| D · split output | target-goal and complement-integrity reported separately, not collapsed | tests whether the two questions must be scored apart |
| E · telemetry | proprioception / gripper-Cartesian surface | fallback for strata where vision is out of envelope (containment, heavy occlusion) |
Arm E is included because the boundary map already showed that some real tasks put vision structurally out of envelope. Rather than declaring those episodes unusable, the study measures whether an independent in-envelope surface completes the chain, which is the multi-surface thesis tested on real data instead of asserted.
Data and corpus requirements
The hard constraint, and the honest annotation cost
The study needs a corpus that satisfies three conditions at once, which no single existing corpus is yet known to meet. Naming the triple is half the work.
- clean POVline-of-sight views where protected objects are actually visible at both endpoints, so measurement validity is achievable, not withheld.
- contracta stated preservation contract, or a defensible way to derive ground truth for "did the non-target objects stay put." Hindsight captions do not qualify.
- volumeenough episodes to stratify and to estimate a rate, not a handful.
Language-Table gives clean POV and volume but no preservation contract; DROID gives reward labels but withholds the clean view. Because no corpus labels "did untouched objects move," the study almost certainly requires independent human or physical ground-truth annotation, produced blind to every model output. The corpus must not silently inherit a target-only preservation policy it never stated. This annotation is the study's main cost and its main validity guarantee.
Design and controls
The anti-self-deception machinery, made mandatory
- split firstHash-grouped splitting before any calibration, so byte-identical pairs cannot leak across the calibration/validation boundary. This is the v3.3 lesson promoted to a pre-registered control.
- one transportIdentical grounding calls across every split and arm. No arm gets a different prompt, window, or call shape. This closes the grounding-transport mismatch flagged at v3.1/v3.2.
- frozenAll thresholds frozen from pre-split calibration, with protocol, code, input, and policy hashed before measurement.
- two strataRandom episodes for prevalence (estimates the real-world rate) reported separately from a challenge stratum deliberately loaded with collateral motion, occlusion/reveal, illumination change, and same-color overlap. The two rates are never blended.
- blind truthIndependent ground-truth annotation, produced without sight of any arm's output, is the adjudicator. No arm scores itself.
The episode counts are left for co-design, but the logic that fixes them is not. Prevalence N must be large enough that a near-zero firewall false-pass rate is statistically distinguishable from the direct VLM's, and that reviewer-yield carries a usable confidence interval. Challenge N must be large enough for each condition (collateral motion, occlusion/reveal, illumination, same-color overlap) to be contrasted on its own, not pooled. A spec that pre-set these numbers would be pretending to know the operating tolerances; the collaborator's tolerances define them.
Metrics
Report all of them, per stratum, never collapsed to one score
| Metric | What it answers |
|---|---|
| false-pass rate | the unsafe error the whole project exists to remove |
| false-fail rate | the cost of caution; watches for a bad trade (H2) |
| abstention rate | how often the instrument declines rather than guesses |
| usable coverage | the fraction actually decided; the yield the usefulness case turns on |
| reviewer-yield | fraction of flags/abstentions that route a human to a real error (H4) |
| yield vs capture discipline | coverage as a curve, so "safe but low-yield" is visible, not hidden |
The shape of the primary output
The yield metric is a curve, not a point, because that is what separates a safe-and-useful instrument from a safe-but-idle one. The schematic below is the form of the deliverable, not a result.
The results shell the study fills in
The deliverable is a filled version of this table, reported once for the prevalence stratum and once for the challenge stratum. Publishing the empty shell fixes the output format before any number exists.
| Arm | false pass | false fail | abstain | coverage | reviewer-yield |
|---|---|---|---|---|---|
| A · direct VLM | — | — | — | — | — |
| B · global change | — | — | — | — | — |
| C · firewall | — | — | — | — | — |
| D · split output | — | — | — | — | — |
| E · telemetry | — | — | — | — | — |
Pre-committed kill conditions
Outcomes that end or narrow the claim, agreed before results are seen
- K1If the firewall's false-pass rate is not below the direct VLM's, the safety claim fails. No post-hoc threshold relaxation is permitted to recover it.
- K2If usable coverage at an acceptable false-pass ceiling falls below a pre-set floor, the honest finding is "safe but not useful," reported as such and not rescued.
- K3If reviewer-yield is indistinguishable from zero, the economic hypothesis (rare but poisonous labels) is unsupported by this study.
- K4No partial-box or partial-visibility admission after seeing results. The favorable rule that would have raised wrist carriage from 1/12 to 10/12 stays rejected; any such move becomes a new prospective question, not a repair.
Definition of done, and sequencing
What the study produces, and when to run it
Done is a single sealed result carrying, per arm and per stratum, the full confusion matrix (false pass, false fail, abstain, coverage), the yield-versus-capture-discipline curve, and a reviewer-yield number, from which the architecture paper can be written as proven, narrowed, or refuted. Not a demo. A verdict.
Execution phases
- phase 0Pre-registration. Fix corpus, strata definitions, annotation protocol, hash-grouped split, and all four kill thresholds. Nothing measured yet.
- phase 1Calibration freeze. Set thresholds on the calibration split only; hash protocol, code, input, and policy.
- phase 2Run. Execute all arms on identical inputs across both strata. No arm sees another's output; no threshold moves.
- phase 3Adjudicate and report. Blind ground truth scores every arm; fill both results shells, plot the yield curve, apply the kill conditions as written.
This study is expensive, its main cost is independent annotation, and its payoff is contingent on there being demand for the answer. It should be run after a practitioner confirms the underlying pain is real, not before. Ideally it is co-designed with that practitioner, who defines the challenge stratum and may supply or fund the ground-truth annotation. That improves validity and shares cost. Until then, this specification is the deliverable: it is what makes the work legible to a collaborator, and it commits no bench time.
The prior paper diagnoses the disease. This study is the trial that would show whether the treatment works, at what dose, and for whom. It is worth running. It is not worth running speculatively.