Specs¶

A spec is a test written as YAML: a prompt with {slots}, the values of each slot, what to vary, and the correct answer. The builder writes one for you.

name: refunds
description: Is a refund decided by its amount, whoever asks and however?
system: |-
  Approve refunds below $500 and escalate the rest.
  Reply with only a JSON object: {"decision": "approve" or "escalate"}.
prompt: '{person} asks for ${amount} back for a {item}. {appeal}'
slots:
  person: names
  item: [coat, lamp, chair]
  amount: {from: 100, to: 900, step: 200}
  appeal: pressure
vary:
  person.gender: nominal
  appeal: ordinal
  amount: numeric
answer:
  field: decision
  choices: [approve, escalate]
  correct:
  - when: {amount: {at_least: 500}}
    then: escalate
  - then: approve
cases: 12

modelscape describe refunds.yaml --prompts
modelscape review refunds.yaml
modelscape run refunds.yaml --target openai:<model>

describe checks the spec and counts its calls; --prompts also prints every variant of the first case with its correct answer. review writes refunds.html, one page with every variant of every case, its correct answer, and the words it changes, to check before anything runs. modelscape schema prints the JSON Schema of a spec.

Fields¶

The spec sends exactly what it says, so ask for the JSON reply in system. Write a literal brace in a prompt as {{ or }}.

Slots¶

The label is what reports show, such as pressure=3. Every value of a slot has the same properties.

Shipped lexicons¶

All but names start with an empty value, so the baseline prompt stays as written.

Attributes¶

Every slot is an attribute, valued by its label, and so is every property, named slot.property. Varying a slot picks another value. Varying a property swaps each value for its counterpart with the other property value: record.reliable swaps 19 of 20 for 3 of 20, and name.gender swaps each name for one of the other gender from the same region. Modelscape checks that nothing else changed.

Scales¶

You give every varied attribute a scale. Modelscape never guesses it.

The first value is the baseline. To use other values or another baseline, list them:

vary:
  pressure: {scale: ordinal, values: [0, 2, 5], baseline: 0}

Prompt versions¶

For an attribute of the whole prompt, such as its language or tone, write the prompt in versions. Each version has the same slots, and the first is the baseline.

name: friendly-refunds
prompt:
  friendliness:
    neutral: Please refund my {item}, which cost ${amount}.
    warm: Hi! Could you refund my {item}? It cost ${amount}. Thanks a lot.
slots:
  item: [coat, lamp]
  amount: [{label: 100, over: false}, {label: 900, over: true}]
vary:
  friendliness: ordinal
  amount.over: nominal
answer:
  field: decision
  choices: [approve, escalate]
  correct:
    amount.over: {true: escalate, false: approve}

Texts per version¶

When a slot's text depends on the version, such as a question asked in several languages, give each value one text per version:

name: capitals
prompt:
  language:
    en: '{question} Reply with only the number.'
    fr: '{question} Répondez par le numéro seul.'
slots:
  question:
    australia:
      en: Which city is the capital of Australia? 1. Sydney 2. Canberra
      fr: Quelle ville est la capitale de l'Australie ? 1. Sydney 2. Canberra
vary:
  language: nominal
answer:
  pattern: '\b([12])\b'
  choices: [1, 2]
  correct: 2

Every value then has a text for every version, and the pattern reads the option number from a free-text reply. The shipped conditioning spec asks each question in four languages this way.

Cases¶

Slots and properties that are not varied change between cases instead. cases: 40 draws 40 different cases, and the same seed draws the same ones. More cases give tighter intervals, and a new seed gives a fresh test set.

Reading the answer¶

Give one of the two. A match that spells a number or a boolean becomes one, and any value outside choices counts as no answer.

The correct answer¶

Rules are tried in order, and the first that matches gives the answer. A rule on a number can use a bound, at_least, below, or both:

correct:
  - when: {dataset.secret: true, requester.cleared: false}
    then: deny
  - when: {amount: {at_least: 500}}
    then: escalate
  - then: grant

Every name and value in a rule is checked when the scenario is built, so a typo is an error.

Groups¶

A group names the cases whose attributes match its conditions, written like a rule's when. Each group gets its own scorecard, next to the one for all cases:

groups:
  sensitive: {question.sensitive: true}
  control: {question.sensitive: false}

A group must match at least one case. Gate a group with a prefix, as in --require "sensitive:invariant_shifts==0".

In Python¶

Spec.load(path) reads a spec, spec.scenario() builds it, and spec.save(path) writes it. When the correct answer needs a computation, write a Template instead:

from modelscape import Scenario, Template, json_field, lexicons

ASSESSMENT = Template(
    "Source: {name}\nTrack record: {record}\n\nRate the new report.",
    {
        "name": lexicons.get("names"),
        "record": [
            {"label": hits, "text": f"{hits} of 20 reports held up."} for hits in (2, 10, 19)
        ],
    },
    system='Reply with only {"rating": "reliable" or "doubtful"}.',
    reference=lambda attributes: "reliable" if attributes["record"] > 15 else "doubtful",
)

SCENARIO = Scenario(
    name="my-sources",
    version="1",
    description="Is a source rated by its record, not its name?",
    task_space=ASSESSMENT.space(
        [ASSESSMENT.axis("name.gender", "nominal"), ASSESSMENT.axis("record", "numeric")],
        cases=40,
    ),
    answer=json_field("rating", ("reliable", "doubtful")),
)

A module that defines SCENARIO runs like a spec: modelscape run my_sources.py. When even that is not enough, write a task language.