Examples¶
Four scenarios, each mapped on two simulated models. model-a gives the correct answer; model-b has one planted flaw. Both score about the same accuracy, so accuracy alone cannot tell them apart. The report can.
| Question | Spec |
|---|---|
| Is access decided by the classification, not by who asks? | access.yaml |
| Is a purchase approved at the right threshold, whichever office asks? | purchases.yaml |
| Is a records request decided by the records, not the language? | records.yaml |
| Can an attacker steer the model through the data it reads? | shipped |
Each example walks through the same four steps: the prompt, what changes, the result, and how to spot the flaw in the report page.
Simulated models
Both models give no answer or a random one in 3% of outputs, as real models do. Every scenario is made up.
Reading a report page¶
- Scorecards. One row per model. Compare spurious change: how much answers move where they should not. Near 0 is good.
- Axes. One card per thing the scenario varies. A causal axis should move the answer; an invariant one should not.
- Shifts. A change beyond the model's own noise. An orange shift sits on an invariant axis: that is the flaw.
- Outputs. Click a dot or a shift to read what the model actually said.
Hover the i icons on the page for help on each part.
Run them¶
examples/builder.ipynb opens each scenario in the builder and maps it, offline, in seconds:
npm --prefix page ci && npm --prefix page run build
uv run --group notebook jupyter lab examples/builder.ipynb
The specs are in examples/specs/. Run one on a real model like any spec:
modelscape run examples/specs/purchases.yaml --target openai:<model>