Scenarios¶

A scenario states the expected behavior of a model on one task: the variants it asks, and the correct answer of each. It is fixed, so every model is mapped the same way.

Where scenarios come from¶

Shipped scenarios¶

The specs are in modelscape/scenarios/. Open one in the builder with Builder.load("pressure"). evidence is written in Python because its correct answer weighs every report by its reliability, which a spec cannot compute.

modelscape run takes a shipped name, a spec file, or a Python module that defines SCENARIO. To make your own scenario runnable by name, register it in your package:

[project.entry-points."modelscape.scenarios"]
refunds = "my_package.refunds:SCENARIO"

Scorecard¶

Every run gives one scorecard per model:

On a nominal axis, a shift compares a value with the baseline, such as role=director. On an ordinal or numeric axis, it compares neighbors, such as pressure 4 → 5. An unreadable answer counts as a wrong one, so read every number with answered.

Comparing models¶

Pass several targets. The first is the reference:

modelscape run pressure --target openai:<a> --target anthropic:<b>

The report then lists where each model differs from the reference beyond the noise of both. The same works for a new release (--target <old> --target <new>) or for drift over time: label a target, as in --target openai:<model>#2026-10, and run it again later next to a new label. Stored outputs are reused.

To choose between models across several scenarios, save a report per scenario and compare them:

modelscape compare runs/*.json --require "invariant_shifts==0" --require "answered>=0.95"

Gating CI¶

--require turns a run into a test that exits with status 1 when a requirement is missed:

modelscape run pressure --target openai:<model> --require "accuracy>=0.95" --require "invariant_shifts==0"
modelscape check pressure.json --require "invariant_shifts==0"

Requirements bound accuracy, answered, spurious_change, missed_change, invariant_shifts (the number of shifts on invariant axes), or divergences (where a model differs from the reference). Prefix a case group to check only that group, as in sensitive:invariant_shifts==0, and use --gate-target to check only some models.

- run: pip install "modelscape[openai]"
- run: modelscape run pressure --target openai:<model> --require "invariant_shifts==0"
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

From Python¶

import asyncio

from modelscape import CallableTarget, Runner
from modelscape.scenarios import load


def always_grant(request):
    return '{"decision": "grant"}'


async def main():
    scenario = load("pressure")
    target = CallableTarget(always_grant, name="always-grant")
    results = await Runner().run(scenario.experiment([target]))
    print(scenario.report(results).summary())


asyncio.run(main())

CallableTarget wraps any function, so you can map your own application or try modelscape without a model. In a notebook, await the run instead of asyncio.run, and show ReportPage(report, results) to explore it. modelscape.gate.check(report, requirements) checks requirements.

Every prompt¶

scenario.prompts() returns every variant as a pandas DataFrame, with the pandas extra:

from modelscape.scenarios import load

prompts = load("pressure").prompts()
prompts[prompts["changed"] == "requester"]

Provenance¶

Every report records the modelscape version, each model's full settings, the model versions the provider reported serving, failures, token use, and when the outputs were made.