API reference¶
Controlled behavioral mapping of large language models.
structured task -> controlled changes -> repeated observations -> behavioral profile
__all__
module-attribute
¶
__all__ = ['Axis', 'AxisScore', 'AxisSensitivity', 'BehaviorProfile', 'BoundaryCandidate', 'BoundaryCriterion', 'CallableDifference', 'CallableMetric', 'CallableTarget', 'Coordinate', 'Divergence', 'Entry', 'ExecutionConfig', 'Experiment', 'Grid', 'Inference', 'Lexicon', 'MeasureAgreement', 'MeasureValue', 'Message', 'Node', 'Prompt', 'Provenance', 'Report', 'ReportPage', 'Requirement', 'Response', 'Results', 'RetryPolicy', 'Runner', 'Scale', 'Scenario', 'ScenarioPage', 'Scorecard', 'Spec', 'Sweep', 'TargetOutput', 'TargetProvenance', 'TargetRequest', 'Task', 'TaskSchema', 'TaskSpace', 'Template', 'Variant', '__version__', 'attribute', 'json_field', 'json_reply', 'perturbation', 'realizer', 'text_match']
AxisScore ¶
Bases: FrozenModel
One axis of a scorecard.
scale decides which steps are tested. role is causal when changing this axis
alone, between any two evaluated coordinates, changes the correct answer of some case, and
invariant otherwise. accuracy follows the axis with every other axis at its
baseline. shifts are the steps where answers changed beyond run-to-run variation:
adjacent steps on an ordinal or numeric axis, comparisons with the baseline on a nominal
one. threshold compares where the answer flips with where it should, on a causal
numeric axis when the answers could be read.
AxisSensitivity ¶
Bases: FrozenModel
How strongly one target responds to one axis under one measure.
peak is the transition with the largest excess change: a local step on an ordinal or
numeric axis, or a baseline comparison on a nominal one.
BehaviorProfile ¶
Bases: FrozenModel
How one target's behavior changes along one axis under one difference measure.
baseline compares every axis value with the experiment's baseline; at the baseline
itself this is the run-to-run variation. local compares each pair of adjacent axis
values. Other axes stay at fixed. Point metrics at the same coordinates are in
metrics. candidate_boundaries are selected from steps.
steps
property
¶
steps: tuple[Transition, ...]
The transitions tested for boundaries: neighbors, or the baseline if nominal.
BoundaryCandidate ¶
Bases: FrozenModel
A transition where behavior changed more than run-to-run variation explains.
It shows where to look. It does not prove a capability threshold, a failure point, or a mechanism.
BoundaryCriterion ¶
Bases: FrozenModel
The rule that decides which transitions are candidate boundaries.
A transition qualifies when its permutation p-value, corrected with Holm's method for the
number of transitions tested together, is at most alpha, and its estimated excess
change is larger than min_excess.
Divergence ¶
Bases: FrozenModel
Where two targets answer differently beyond the run-to-run variation of each.
contrasts holds one comparison per evaluated coordinate; candidates are the ones
criterion selects.
Inference ¶
Bases: FrozenModel
Settings for bootstrap intervals and permutation tests.
The same settings always give the same results. Set permutations to zero to skip the
permutation tests.
MeasureAgreement ¶
Bases: FrozenModel
Whether two difference measures tell the same story about the same transitions.
rank_correlation compares how both measures order the transitions by excess change.
boundary_overlap is the share of candidate boundaries that both measures find; two
measures that find no boundary at all agree fully. Both are None when there is nothing
to compare.
Results ¶
Results(experiment: ExperimentRecord, variants: Sequence[Variant], observations: Sequence[Observation], point_measurements: Sequence[PointMeasurement], difference_measurements: Sequence[DifferenceMeasurement])
All records of one experiment run, with the analyses defined over them.
Every aggregate returned here carries measurement_ids; trace recovers the
underlying measurements, observations, and variants.
load
classmethod
¶
load(store: Store, experiment_id: str) -> Results
Rebuild the results of a completed run from a store.
select ¶
select(*, target: str | None = None, case: str | None = None, coordinate: Mapping[str, AxisValue] | None = None) -> tuple[Observation, ...]
Observations filtered by target name, case id, and coordinate.
A partial coordinate is completed with baseline values, as everywhere else.
table ¶
table() -> list[dict[str, JsonValue]]
One tidy row per observation, with its point metrics and execution timing.
profile ¶
profile(axis: str, measure: str, *, target: str | None = None, fixed: Mapping[str, AxisValue] | None = None, cases: Sequence[str] | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> BehaviorProfile
Behavior of one target along one axis under one difference measure.
profiles ¶
profiles(axis: str, measure: str, *, fixed: Mapping[str, AxisValue] | None = None, cases: Sequence[str] | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> tuple[BehaviorProfile, ...]
One profile per target, computed identically for direct comparison.
transitions ¶
transitions(measure: str, *, target: str | None = None, cases: Sequence[str] | None = None, inference: Inference | None = None) -> tuple[Transition, ...]
Every local transition between evaluated neighbors on ordinal and numeric axes.
variability ¶
variability(measure: str, *, target: str | None = None, inference: Inference | None = None) -> tuple[VariantVariability, ...]
Within-variant variation of every variant under one difference measure.
sensitivity ¶
sensitivity(measure: str, *, target: str | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> tuple[AxisSensitivity, ...]
Every axis ranked by its largest excess change around the baseline.
agreement ¶
agreement(measures: Sequence[str] | None = None, *, target: str | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> tuple[MeasureAgreement, ...]
Pairwise agreement of difference measures over every tested transition.
These are the local transitions between evaluated neighbors and, on nominal axes, the comparisons with the baseline.
scorecard ¶
scorecard(metric: str, measure: str, *, target: str | None = None, group: str = 'all', answered: str | None = None, answer: Answer | None = None, cases: Sequence[str] | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> Scorecard
Accuracy, spurious and missed change, and per-axis shifts of one target.
metric scores correctness and measure compares two answers; every axis is read
along its line through the baseline. answered, a metric that is 1 when an answer
could be read, gives the share of readable outputs. answer, which reads the answer
from an output, adds a threshold to every causal numeric axis.
divergence ¶
divergence(measure: str, first: str, second: str, *, cases: Sequence[str] | None = None, criterion: BoundaryCriterion | None = None, inference: Inference | None = None) -> Divergence
Where two targets of this experiment differ beyond the run-to-run variation of each.
first must come before second in the experiment's targets.
trace ¶
trace(item: Traceable) -> Trace
Recover the measurements, observations, and variants behind an aggregate.
Scorecard ¶
Bases: FrozenModel
Headline numbers for one target on one case group.
answered is the share of outputs, over every coordinate, from which an answer could be
read. spurious_change is answer change beyond noise where the correct answer stays the
same; missed_change is the share of answers that stay the same where it moves. Both are
ideally zero, and None when nothing could be measured.
invariant_shifts
property
¶
invariant_shifts: int
The number of shifts on invariant axes, where answers should not change.
Coordinate ¶
Coordinate(values: Mapping[str, AxisValue] | None = None)
Bases: Mapping[str, AxisValue]
Immutable location in a task space, mapping every axis name to one declared value.
Grid ¶
Bases: FrozenModel
Evaluate the full Cartesian product of the selected axes, others held at baseline.
Message ¶
Bases: FrozenModel
One turn of a conversation.
Prompt ¶
Bases: FrozenModel
The exact text a target receives: an optional system instruction and a conversation.
Sweep ¶
Bases: FrozenModel
Evaluate the baseline and every one-axis-at-a-time variation around it.
TaskSchema ¶
Bases: FrozenModel
A machine-readable description of a task language.
structure is the JSON Schema of valid tasks, and attributes lists the properties
computed from every task: the dimensions a task space can vary.
Variant ¶
Bases: FrozenModel
A task at one coordinate, the exact prompt for it, and the steps that produced it.
ExecutionConfig ¶
Bases: FrozenModel
How observations are executed. Does not affect observation identity.
Experiment
dataclass
¶
Experiment(*, task_space: TaskSpace[Any], targets: Sequence[Target], repetitions: int = 1, point_metrics: Sequence[PointMetric] = (), difference_measures: Sequence[DifferenceMeasure] = (), design: Sweep | Grid = Sweep(), execution: ExecutionConfig = ExecutionConfig(), seed: int = 0, scenario: Identity | None = None)
A task space evaluated against targets with repetitions and explicit measures.
design selects which coordinates are evaluated: a one-axis-at-a-time Sweep around
the baseline by default, or a Grid when interactions between axes matter. seed is
part of every observation identity, fixes the randomized execution schedule, and derives
the seed hint passed to targets.
RetryPolicy ¶
Bases: FrozenModel
Exponential backoff for target invocations that raise or time out.
Runner ¶
Runner(*, store: Store | None = None, on_progress: ProgressCallback | None = None, redact: Redactor | None = None)
Runs experiments, saving each output as it arrives and reusing stored ones on reruns.
Observations run in a fresh random order per repetition. redact cleans every output
before it is stored.
Requirement ¶
Bases: FrozenModel
A bound on one report quantity, such as accuracy>=0.9 or control:accuracy>=0.9.
violations ¶
violations(report: Report, targets: Collection[str] | None = None) -> list[str]
One line for every scorecard or divergence in scope that misses this requirement.
targets limits the requirement to those targets; a divergence belongs to the target
compared with the first.
Node ¶
Bases: BaseModel
An immutable component of a task, such as a piece of evidence or a hypothesis.
Task ¶
Bases: BaseModel
A task, written in a task language you define.
A subclass defines the language: typed fields and validators decide which tasks are
valid, @attribute methods compute the properties that can be varied, and reference
gives the correct answer when there is one. The language is named after its class, not its
module, so moving the code keeps every stored record valid.
CallableDifference
dataclass
¶
CallableDifference(function: DifferenceFunction, *, name: str, version: str = '1', parameters: Mapping[str, JsonValue] = dict[str, JsonValue]())
A difference measure defined by a sync or async function of two responses.
Each Response exposes variant and output, so the function can compare
decisions, confidence, cited evidence, or anything else relative to the task it answered.
CallableMetric
dataclass
¶
CallableMetric(function: MetricFunction, *, name: str, version: str = '1', parameters: Mapping[str, JsonValue] = dict[str, JsonValue]())
A point metric defined by a sync or async function of a variant and its output.
The function returns a number, or a MeasureValue with structured details.
MeasureValue ¶
Bases: FrozenModel
A scalar produced by a measure, with optional structured auxiliary data.
Response ¶
Bases: NamedTuple
One output together with the variant that produced it.
The variant carries the task, its attributes, reference, prompt, and coordinate, so a measure can judge behavior in context rather than from text alone.
ReportPage ¶
ReportPage(report: Report | str | PathLike[str], outputs: Outputs | None = None)
Bases: Page
A report as one HTML page; click any number to read the outputs behind it.
outputs is the run's Results, a store, or a store file. The page embeds only the
outputs behind the report's numbers.
ScenarioPage ¶
ScenarioPage(scenario: Scenario)
Bases: Page
A scenario as one HTML page, to review before anything runs.
Shows every axis with its role, and every variant of every case with its correct answer and the words it changes in the prompt.
Provenance ¶
Bases: FrozenModel
The software, times, and targets behind a report.
Report ¶
Bases: FrozenModel
Scorecards per target and group, divergences from the first target, and provenance.
format is the version of the report file layout, raised when a release changes it.
TargetProvenance ¶
Bases: FrozenModel
One target's identity, the model versions its provider reported, and its usage.
Scenario
dataclass
¶
Scenario(*, name: str, version: str, description: str, task_space: TaskSpace[Any], answer: Answer, repetitions: int = 5, measures: Sequence[DifferenceMeasure] = (), design: Sweep | Grid = Sweep(), groups: Mapping[str, Sequence[str]] = dict[str, Sequence[str]](), criterion: BoundaryCriterion = BoundaryCriterion(), seed: int = 0)
A task space with a fixed protocol, so every target is mapped the same way.
answer reads the answer from an output, or None when it cannot; compared with each
task's reference() it scores every target. groups name case subsets with their own
scorecards. Bump version whenever results could change.
describe ¶
describe() -> str
The scenario's cases, axes, and variants, checked, and the calls it makes per target.
example ¶
example() -> str
Every variant of the first case: what it changes, its correct answer, and its prompt.
variants ¶
variants() -> Iterator[Variant]
Every variant the scenario asks each target, case by case.
prompts ¶
prompts() -> DataFrame
Every variant as one row: its case, coordinate, changed axes, prompt, and answer.
Requires the pandas extra: uv add "modelscape[pandas]".
experiment ¶
experiment(targets: Sequence[Target], execution: ExecutionConfig | None = None) -> Experiment
The experiment that maps targets with this scenario.
report ¶
report(results: Results) -> Report
Scorecards, divergences, and provenance of a run of this scenario.
Axis ¶
Axis(attribute: Attribute[TaskT, ValueT] | str, values: Iterable[ValueT], via: Perturbation[TaskT, ValueT], *, scale: Scale, baseline: ValueT | None = None)
Bases: Generic[TaskT, ValueT]
One attribute of a task language, varied over discrete values by one perturbation.
scale is nominal, ordinal, or numeric; see AxisSpec. The baseline
defaults to the first value.
TaskSpace ¶
TaskSpace(cases: Mapping[str, TaskT], axes: Sequence[Axis[TaskT, Any]], realizer: Realizer[TaskT] | None = None)
Bases: Generic[TaskT]
Seed tasks of one task language, the axes that vary them, and how they are realized.
The task at a coordinate is made from a seed by applying, in axis order, the perturbation of every axis whose attribute differs from the coordinate. Each step is checked, and the final task must have exactly the coordinate's attribute values.
task ¶
task(case: str, coordinate: Mapping[str, AxisValue] | None = None) -> TaskT
The task of one seed at one coordinate; axes you leave out stay at their baseline.
variant ¶
variant(case: str, coordinate: Mapping[str, AxisValue] | None = None) -> Variant
The task at a coordinate, with its prompt and the record of every step.
Spec ¶
Bases: FrozenModel
A scenario written as data, read from and written to YAML.
prompt is one text, or versions of the whole prompt under one attribute name, such as
{friendliness: {neutral: ..., polite: ...}}. slots fill the prompt's
{placeholders}; each is a shipped lexicon by name, a list of values, a mapping of labels
to texts, a list of rows, or a Range of numbers. A text may be one text per prompt
version. vary maps each attribute to vary to its scale, or to a Varied. groups
name the cases whose attributes match their conditions, each with its own scorecard.
CallableTarget ¶
CallableTarget(function: TargetFunction, *, name: str, version: str = '1', parameters: Mapping[str, JsonValue] | None = None)
Any function from a TargetRequest to text, JSON, or a TargetOutput, as a target.
Synchronous functions run in worker threads. Put behavior-relevant settings in
parameters, never credentials.
TargetOutput ¶
Bases: FrozenModel
One invocation result. Model output is untrusted data and is never executed or rendered.
TargetRequest ¶
Bases: FrozenModel
What a target receives: the prompt only, never the task, coordinate, or step history.
seed is a hint derived from the observation's id. Providers may ignore it or honor it
only partly, so no analysis assumes a seed makes outputs reproducible. Repetitions measure
run-to-run variation either way.
Entry ¶
Bases: FrozenModel
One way to fill a slot: its label, the text it puts in the prompt, and its properties.
text is one text, or one text per prompt version, such as {en: ..., fr: ...}.
text_in ¶
text_in(version: AxisValue | None) -> str
The text this entry puts in prompt version version.
Lexicon ¶
Bases: FrozenModel
The entries a slot can take.
Labels are unique and of one type, and every entry has the same properties, each of one type.
of
classmethod
¶
of(values: Lexical) -> Lexicon
A lexicon from values, a mapping of labels to texts, rows, or a lexicon.
A row is a mapping with a label, a text, or both; its other keys are properties.
load
classmethod
¶
load(path: str | PathLike[str]) -> Lexicon
A lexicon from a YAML file holding any form of takes.
data ¶
data() -> list[AxisValue] | list[dict[str, Cell]] | dict[AxisValue, Texts]
The lexicon in the shortest form of takes back: values, a mapping, or rows.
values ¶
values(property: str | None = None) -> tuple[AxisValue, ...]
Every label, or every value of one property, in order of first appearance.
swap ¶
swap(entry: Entry, property: str, value: AxisValue) -> Entry
The counterpart of entry with property set to value.
It agrees with entry on every other property and has the same rank among such
entries.
Template ¶
Template(text: str | Versions, slots: Mapping[str, Lexical], *, name: str = 'Template', system: str | None = None, reference: Reference | None = None, version: str = '1')
A prompt with {slot} placeholders, each filled from a lexicon.
Every slot is an attribute valued by its entry's label, and every property is an attribute
named slot.property. Varying a property swaps each entry for its counterpart, so
name.gender swaps each name for one of the other gender. reference computes the
correct answer from the attributes.
The prompt may come in versions, such as {"friendliness": {"neutral": ..., "polite":
...}}; friendliness is then an attribute like a slot.
values ¶
values(attribute: str) -> tuple[AxisValue, ...]
Every value of attribute in its lexicon, in order of first appearance.
axis ¶
axis(attribute: str, scale: Scale, values: Iterable[AxisValue] | None = None, *, baseline: AxisValue | None = None) -> Axis[Filling, Any]
An axis over a slot, a slot property, or the prompt versions.
By default it takes every value of the attribute.
space ¶
space(axes: Sequence[Axis[Filling, Any]], *, cases: int | None = None, seed: int = 0) -> TaskSpace[Filling]
A task space over axes, with up to cases distinct seeds drawn at the baseline.
Slots and properties that are not axes vary between seeds. Without cases, every
combination is a seed. The same seed always draws the same seeds.
attribute ¶
attribute(function: Callable[[TaskT], ValueT]) -> Attribute[TaskT, ValueT]
Declare a task property that axes can vary and perturbations must keep fixed.
realizer ¶
realizer(function: Callable[[TaskT], Prompt | str]) -> FunctionRealizer[TaskT]
realizer(*, name: str | None = None, version: str = '1', system: str | None = None, parameters: Mapping[str, JsonValue] | None = None) -> Callable[[Callable[[TaskT], Prompt | str]], FunctionRealizer[TaskT]]
realizer(function: Callable[[TaskT], Prompt | str] | None = None, /, *, name: str | None = None, version: str = '1', system: str | None = None, parameters: Mapping[str, JsonValue] | None = None) -> FunctionRealizer[TaskT] | Callable[[Callable[[TaskT], Prompt | str]], FunctionRealizer[TaskT]]
Declare a function task -> prompt as a realizer.
Change version whenever the wording changes. system is sent as the system
instruction when the function returns plain text.
json_field ¶
json_field(name: str, choices: Iterable[AxisValue] | None = None) -> Answer
An answer that reads field name of a JSON reply, such as {"verdict": "close"}.
With choices, any other value counts as no answer.
json_reply ¶
json_reply(output: TargetOutput) -> dict[str, object]
The JSON object in an output, read from its first { to its last }.
Empty when the output holds no JSON object.
text_match ¶
text_match(pattern: str, choices: Iterable[AxisValue]) -> Answer
An answer that reads the first match of pattern, or of its first group.
A match that spells a number or a boolean becomes one; any value not in choices counts
as no answer.
modelscape.lexicons ¶
Lexicons that ship with modelscape, ready to fill template slots by name.
names holds 70 full names, five for each of two genders in seven regions, so gender and
region swaps pair every name with a counterpart of the same rank. roles, pressure,
injections, and tones each start with an empty entry, so a slot filled from them leaves
the baseline prompt as it was.
modelscape.builder ¶
A notebook widget that writes a scenario spec: mark what varies, set the answer, save YAML.
Requires the widget extra: uv add "modelscape[widget]".
Builder ¶
Builder(prompt: str = '', *, name: str = 'scenario')
Bases: AnyWidget
Write a scenario spec from an example prompt, with a live preview.
Show it in a notebook, select text to make a variable, choose the scale of what varies and
which answer is correct, and save the YAML. Every edit is also a method, so the widget
is optional. prompts lists every variant the spec asks.
prompts
property
¶
prompts: DataFrame
Every variant of the spec as one row, as Scenario.prompts gives it.
load
classmethod
¶
load(path: str | PathLike[str]) -> Builder
A builder editing a spec: a YAML file, or a shipped spec by name, such as pressure.
save ¶
save(path: str | PathLike[str]) -> Path
Write the spec as YAML, complete or not, and return its path.
mark ¶
mark(piece: int, start: int, end: int, slot: str | None = None, *, version: AxisValue | None = None, lexicon: str | None = None) -> None
Make characters start to end of prompt piece piece a slot.
The pieces are those of prompt version version, the first by default, and only that
version changes. An existing slot is placed there; a new one takes the selection as
its value, which becomes a number or a boolean if it spells one, or takes the values of
the shipped lexicon.
remove ¶
remove(slot: str) -> None
Turn slot back into the text of its first value, in every prompt version.
per_version ¶
per_version(slot: str, enabled: bool = True) -> None
Give every value of slot one text per prompt version, or one text again.
rows ¶
rows(slot: str, rows: Sequence[Mapping[str, JsonValue]]) -> None
Replace the values of slot; cells that spell numbers or booleans become them.
A text may map each prompt version to its text.
range ¶
range(slot: str, start: Number, stop: Number, step: Number) -> None
Make the values of slot the numbers from start up to stop, step apart.
move ¶
move(slot: str, row: int) -> None
Move value row of slot one place up; at the top, it is the baseline.
extend ¶
extend(slot: str, property: str) -> None
Give every value of slot a new property, empty until edited.
vary ¶
vary(attribute: str, scale: Scale | None) -> None
Vary attribute on scale, or stop varying it with None.
versions ¶
versions(name: str | None, labels: Sequence[AxisValue] = ()) -> None
Write the whole prompt in versions: one per label, all values of attribute name.
The first label is the baseline. A new version starts as a copy of the first. Without a name or labels, the first version becomes the one prompt again.
version ¶
version(label: AxisValue, text: str) -> None
Set the text of prompt version label, written like edit("prompt", ...).
answer ¶
answer(field: str | None, choices: Sequence[AxisValue], correct: AxisValue | Mapping[str, Mapping[AxisValue, AxisValue]] | Sequence[Mapping[str, JsonValue]] | None = None, *, pattern: str | None = None) -> None
Read field of a JSON reply, or the first match of pattern, one of choices.
correct is as in a spec; a table may also be given as pairs of a value and its answer.
group ¶
group(name: str, when: Mapping[str, JsonValue] | None) -> None
Name the cases whose attributes match when; None removes group name.
edit ¶
edit(key: str, value: JsonValue) -> None
Set a top-level field such as name, system, or cases; None clears it.
A prompt is template text: slots in braces, and literal braces doubled. It sets the
first version of a prompt in versions.
instruct ¶
instruct() -> None
End the system prompt with an instruction to reply in the answer's JSON format.