Compare commits

...

27 Commits

Author SHA1 Message Date
ctan c454a67b84 docs(phase2): record the P5.2 few-shot prompts and repair loop 2026-09-25 23:24:29 +02:00
ctan dc890fc8fd feat(phase2): few-shot prompts + bounded repair loop (P5.2) 2026-09-25 23:24:27 +02:00
ctan d4db4f6d33 docs(phase2): record the P5.1 provider client and plan status 2026-09-25 22:22:30 +02:00
ctan 42802f7bf5 feat(phase2): OpenAI-compatible provider client (P5.1) 2026-09-25 22:22:03 +02:00
ctan 790df17623 docs(phase2): record the P4 robustness measurements and plan status 2026-09-25 21:34:13 +02:00
ctan cd69312ff4 feat(phase2): adversary and robustness cartography (P4) 2026-09-25 21:34:11 +02:00
ctan 016d6e1459 docs(phase2): record the P3 optimizer playbook and plan status 2026-09-22 15:44:06 +02:00
ctan dd64c2ebd6 feat(phase2): optimizer and scenario compiler (P3) 2026-09-22 15:23:05 +02:00
ctan 115f54a7e5 docs(phase2): record the P2 sweep measurements and plan status 2026-09-22 13:48:01 +02:00
ctan dbd1a5822e feat(phase2): strategy compiler and sweep runner (P2) 2026-09-22 13:46:05 +02:00
ctan 15d24be638 feat(phase2): P1c 2026-09-22 12:21:23 +02:00
ctan de4645838b feat(phase2): provider-agnostic LLM client + composite cache (P1b) 2026-09-22 09:45:18 +02:00
ctan f0617a0de6 feat(phase2): external config + versioned prompt pack (P1a)
config.py centralizes phase-2 path/env reads (no hard-coded data paths,
section 15). prompt.py loads the EXTERNAL, versioned prompt pack (section
14): pack version + per-prompt content hash feed prompt_version for the
response-cache composite key (D6); missing/invalid packs raise a clear
PromptPackError, never a silent fallback. Minimal fixture pack under
tests/fixtures/prompt_pack/ pins CI to a local, versioned pack.
2026-09-22 09:21:28 +02:00
ctan 26c5b51b98 docs(phase2): record the runner entry points (P0.4) and replay (P0.5)
run_encounter gains llm_policy and per-run policy_factory (mutually
exclusive with policy/llm_policy); replay.py logs every (observation,
action) keyed by call_seq to JSONL and replay() reproduces the recorded
battle bit-identically (wins/draws/rounds/transcript) with mismatch,
exhausted, duplicate and out-of-order guards. P0 frontier complete: the
recorded run replays identically.
2026-09-22 08:45:52 +02:00
ctan 6c6de16e60 feat(phase2): replay log and replay policy (P0.5)
Add replay.py: a ReplayLog records every (observation, action) pair a
policy produced, keyed by monotonic call_seq, plus the EncounterSpec,
seed and round_cap, and serializes the whole battle to JSONL. replay()
turns a log back into an observation-only LLMPolicy that returns each
recorded action, raising ReplayMismatchError on a differing observation
and ReplayExhaustedError past the last call.

Observations are matched by their canonical JSON (observation_key), not
raw dataclass equality, because JSON collapses tuples and lists and the
snapshot's inner dict[str, object] holds both. Action and the spec are
dumped through Pydantic TypeAdapters (mode=json) so tuple fields survive
as arrays and are coerced back on load.

Move TrickCondition/ReadyTrigger out of the TYPE_CHECKING block in
combat.py: TypeAdapter(Action) must resolve them at runtime to serialize
the dataclass, so the Literal aliases now live at module level.
2026-09-21 15:08:34 +02:00
ctan 334f8f9d05 feat(phase2): runner llm_policy and policy_factory entry points 2026-09-21 14:46:25 +02:00
ctan ac7fe20054 docs(phase2): record the compact view measurements (P0.2) and candidates (P0.3)
Compact render is 15,691 bytes (~3.9k tokens) = 24.4% of the raw 64,204-byte
to_dict() snapshot on the representative 7-combatant encounter, hitting the
~2-4k token target; grids are sparse and determinism is pinned in tests.
candidates() derives plausible actions from the pairs with no second legality
engine (swift/free/full_round excluded).
2026-09-21 14:36:24 +02:00
ctan 8b5043072a feat(phase2): candidate actions 2026-09-21 13:46:05 +02:00
ctan 9da9537995 feat(phase2): compact observation view for LLM prompts
Build a decision-sufficient ObservationView from the raw snapshot and
render it as a deterministic line-based prompt string. The raw to_dict()
is ~64 KB (~16k tokens) for a 7-combatant encounter; the compact view
renders at ~15.7 KB (~3.9k tokens, 24% of raw), under the 4k-token
ceiling of the phase-2 architecture doc.

Compaction: sparse terrain/elevation/threatened grids, per-combatant
key selection (occupied_cells, abilities, skills, initiative_mod/total,
temp_hp_rounds_left, features, carried_items, weapon_damage, aware,
alignment, armor_spell_failure, arcane_caster, run_rounds dropped),
attacks compacted to formula+type damage strings, spells to 7
decision-relevant fields, pair target_index replaced by the stable
combatant id, condition_definitions dropped. Attacks and spells render
as positional tuples declared by attack_fields/spell_fields legend
lines, with '-' for missing values.

Tests pin the size bounds (compact < 0.25 x raw and < 16000 bytes), the
sparse-grid contract, the dropped-key contract and the deterministic
id/render contract on a caster-heavy 3v4 representative battle.
2026-09-21 13:21:33 +02:00
ctan 25418f164f docs(phase2): record the executed spike (measurements + first danger map)
Measures: observation 64 KB (~16k tokens) for 7 combatants / 22 KB for a 2v2;
124 policy calls per battle, i.e. 124k calls per 1000-run sweep cell -> a
per-turn LLM in a sweep is infeasible (D9 validated) and a compact view is
mandatory. Danger map (default vs focus-fire monsters, 100 runs): the default
policy understates danger (PC deaths 13%->24% and 0%->19%, TPK 0%->2%) and the
win rate alone misleads (rises to 98% while deaths rise) -> D12/D13 validated.
Notes the spike limits and the remaining P0.2-P0.5 items.
2026-09-21 12:15:16 +02:00
ctan de88543c5c feat(phase2): danger-cartography spike - per-run outcomes, objectives, focus-fire adversary
run_encounter gains an opt-in collect_runs flag returning one RunOutcome
per battle (winner, rounds, per-combatant id/side/hp/hp_max/dead) next to
the EncounterReport; default behavior is unchanged. objectives.py provides
the pure optimizer primitives: P(survives all), P(TPK), P(>=1 death),
expected hp lost, expected rounds and a worst-decile (CVaR-style) tail
metric. phase2_spike.py holds the spike-level strengthened monster policy
(focus fire on the lowest-AC enemy, ties to the wounded; party side stays
on default_policy). tests/test_phase2_danger_map.py prints the danger-map
sketch: warband and lich scenarios measured default vs focus-fire at
100 runs / seed 11, pinning the delta deterministically.
2026-09-21 12:07:23 +02:00
ctan e5c4892a57 test(phase2): measurement spike - observation size and policy calls per battle
Representative 7-combatant caster-heavy encounter on sample_arena,
measured through the real engine path (engine._build_observation at each
policy call, delegating to default_policy):
- observation: 64,204 bytes at first call, 67,712 max mid-battle (~16.9k tokens)
- policy calls/battle: 124 (7 combatants, round_cap=20), 12 for a 2v2 duel
- projected calls per sweep cell: 124 / 12,400 / 124,000 at runs 1/100/1000
Loose bounds (obs < 200 KB, calls/battle < 200) + seed-determinism pinned.
2026-09-21 11:26:24 +02:00
ctan 3c3e30634e feat(phase2): observation-only LLMPolicy entry (D1) + pinned schema_version (P0.1)
- LLMPolicy is now Callable[[Observation], tuple[Action, ...]]: the snapshot
  is the policy's sole information source; the live CombatantState is no
  longer passed to new-style policies.
- CombatEngine._llm_adapter keeps a compatibility path for pre-D1
  two-argument callables: the signature is inspected ONCE at wrap time
  (_accepts_state) and legacy policies still receive (observation, state).
- Observation.to_dict() carries schema_version (OBSERVATION_SCHEMA_VERSION=1)
  as its first pinned key; bump only on removal/meaning change.
- Pinned keyset tests updated (test_observation, test_observation_enriched);
  new tests: observation-only e2e, legacy 2-arg e2e through the adapter,
  schema_version pin + JSON-serializability.
- README: documents the new entry signature and the schema_version key.
2026-09-21 11:10:28 +02:00
ctan 3fd9860b5c docs(phase2): record the arbitrated decisions (external prompts, cartography-first)
All recommendations accepted by the user except prompt location: prompts live
outside this repo (separate pack, configurable path, versioned; prompt_version
feeds the cache key and the replay record; a pinned pack for integration, a
minimal local fixture for unit tests, no silent fallback). Also records the
repo-scope decision (code only: calculation + LLM + report) with the final
detachment of data/sheets/prompts as a backlog item and the design constraint
to avoid hardcoded data paths in the new code. Priority: cartography first.
2026-09-21 10:49:00 +02:00
ctan d794828c7e docs(phase2): add the optimizer and adversarial-cartography layers
The execution tiers answer 'how to run a tactic', not 'which tactic is best'
nor 'where is the danger'. Adds two decision layers: an optimizer over
StrategySpec (objectives incl. tail risk, coarse-to-fine search, LLM as
candidate seeder) and an adversarial cartography (AdversarySpec + alternating
min-max + scenario sweeps + danger frontier + sensitivity), giving the danger
view for BOTH sides. Reframes the LLM as hypothesis generator / red team /
explicator / reporter, never the optimizer or per-turn executor in sweeps.
Adds D12-D14 and phases P3 (optimizer) / P4 (robustness).
2026-09-21 10:27:11 +02:00
ctan 630f5321e0 docs(phase2): revise the LLM architecture after critique
Key correction: the LLM must not sit on the per-turn hot path of a Monte
Carlo sweep (runs x rounds x combatants calls, no cross-run cache reuse).
Adopts a two-tier design - a compiled, parameterised deterministic Policy for
quantitative sweeps (tier A) and a per-turn NLPolicy for a handful of showcase
battles (tier B). Also fixes: 'engine unchanged' -> rules unchanged/additive
boundary; replay keyed by a monotonic call_seq (delay re-queues, summons);
composite response-cache key incl. model id; ObservationView as a structured
object; candidates derived from pairs instead of a second legality engine;
per-run PolicyFactory; cost guardrails in P1; concurrency; D1 to settle in P0.
2026-09-21 09:55:53 +02:00
ctan be64451ba4 docs(phase2): architecture and implementation plan for the LLM decision layer
Design-only document (no code): target architecture (compact versioned
observation view, legal-actions API, LLM client + cache, NL policy with
parse/validate/repair/fallback, strategy sweep, replay/eval harness), the
concrete gaps in the current runner/CLI wiring, the decisions to settle and a
phased implementation plan (P0 foundations, P1 client+policy, P2 strategy
sweep, P3 provider+report) with exit criteria and risks.
2026-09-21 09:51:05 +02:00
51 changed files with 10717 additions and 60 deletions
+21 -5
View File
@@ -1951,8 +1951,11 @@ Règles modélisées :
retour : la politique peut les muter sans affecter le combat ; l'objet est un
dataclass figé au premier niveau, les dictionnaires internes sont des copies
mutables, non deep-freezés). Une politique LLM
se branche via `CombatEngine(..., llm_policy=...)` (`LLMPolicy(observation,
state) -> tuple[Action, ...]`). Schéma enrichi (E1-E5) :
se branche via `CombatEngine(..., llm_policy=...)` (`LLMPolicy(observation)
-> tuple[Action, ...]` : la politique ne reçoit QUE le snapshot, jamais
l'état vivant ; un callable hérité à deux arguments `(observation, state)`
reste accepté via l'adaptateur de compatibilité D1, signature inspectée
une fois à la construction). Schéma enrichi (E1-E5) :
- Par combattant : id, side, pos, occupied_cells, hp/hp_max, active/dead,
dying/stabilized, conditions (noms + rounds_left), resources_left (+ `spell_slot_{level}`),
weapon_broken/weapon_destroyed/weapon_stolen/weapon_dropped, carried_items
@@ -2050,7 +2053,10 @@ Règles modélisées :
charge_reason (str | None — None si `can_charge` est vrai ; sinon la
raison de `ChargeResult` verbatim (`no_line_of_effect` / `no_charge_path`)
ou `no_weapon` si aucune arme de mêlée utilisable).
- Top-level : round, acting_combatant, combatants, pairs, terrain, elevation,
- Top-level : schema_version (P0.1 : version épinglée du schéma d'observation,
constante `OBSERVATION_SCHEMA_VERSION` = 1 ; incrémentée seulement si une
clé est retirée ou change de sens), round, acting_combatant, combatants,
pairs, terrain, elevation,
threatened (grille bool des cases menacées au contact par les ennemis —
AoO), ranged_threatened (U6d : grille bool des cases sous tir ennemi à
distance — champ de tir des armes à distance utilisables des ennemis
@@ -2315,7 +2321,17 @@ Limites de sorts (conformance D6-D9, documentées plutôt que bricolées) :
`win_rate_band` (bande 3σ bornée à [0, 1]).
- `runner.py` — `EncounterSpec`/`Side`, `build_states` (placement en zone +
désambiguïsation des ids), `run_encounter` (un `SeededRng` par run) et
agrégation en `EncounterReport` avec attrition par combattant.
agrégation en `EncounterReport` avec attrition par combattant. Option
`collect_runs=True` (Phase 2) : renvoie en plus un `RunOutcome` par combat
(vainqueur, rounds, pv/état de mort par combattant) — comportement par
défaut inchangé.
- `objectives.py` — primitives d'objectif Phase 2 (pur, sans moteur) :
P(survie totale), P(TPK), P(≥1 mort), pv perdus attendus, rounds attendus
et métrique de queue (dcile pire des pv restants) sur les `RunOutcome`.
- `phase2_spike.py` — spike de cartographie du danger : politique monstre
« focus fire » (concentration sur la cible la plus faible en CA, départage
aux pv) contre laquelle la carte de danger mesure le groupe ; la face PJ
reste sur `default_policy`.
- `cli.py` — front-end argparse `pf1e-sim`, rapport français, code de sortie 2
en cas d'erreur.
- `loaders/` — `combatant.py` (chargeur unifié PJ + monstres avec résolution
@@ -2329,7 +2345,7 @@ Limites de sorts (conformance D6-D9, documentées plutôt que bricolées) :
La gate de validation complète (tests + lint + types) :
```bash
uv run pytest -q # 1981 tests
uv run pytest -q # 2000 tests
uv run ruff check src tests
uv run basedpyright src # mode strict
```
+701
View File
@@ -0,0 +1,701 @@
# Phase 2 — Architecture et plan d'implémentation de la couche de décision LLM
> Document de conception (révisé). **Aucun code n'est encore écrit.**
> Révision intégrant deux demandes : **optimiser la meilleure stratégie de
> survie** et **cartographier tous les cas où un camp est en difficulté**.
## 1. Objectif et périmètre
Le simulateur doit répondre à **trois questions distinctes**, souvent confondues :
| Question | Nature | Réponse apportée par |
|---|---|---|
| « Comment *exécuter* une tactique donnée ? » | Exécution | Étage A (compilé) / Étage B (LLM par tour) |
| « Quelle est la *meilleure* stratégie de survie ? » | **Optimisation** | Couche **Optimiseur** (recherche sur `StrategySpec`) |
| « *Où* les PJ (ou les monstres) sont-ils en difficulté ? » | **Robustesse adversariale** | Couche **Robustesse** (min-max + cartographie de scénarios) |
**Hors périmètre** : la rédaction finale du rapport d'équilibrage (phase 3).
**Principe directeur** : les **règles** du moteur restent inchangées ; la couche
LLM et les couches d'analyse sont **additives**. La LLM n'est ni l'optimiseur ni
l'exécutant des sweeps.
## 2. Exécution — les deux étages (rappel)
Mettre la LLM à chaque tour dans une boucle Monte Carlo est prohibitif
(`runs × rounds × combattants` appels, caches non mutualisables). D'où :
- **Étage A — stratégie compilée (quantitatif).** La LLM traduit une stratégie NL
en un **objet de paramètres** (`StrategySpec`) ; un **compilateur déterministe**
en fait une `Policy` Python exécutée **sans appel LLM**. → bandes valides, coût
maîtrisé, **déterminisme bit-à-bit**.
- **Étage B — LLM par tour (qualitatif, vitrine).** `NLPolicy(observation)` à
chaque tour, sur **≤ 20 runs**. → démonstration du raisonnement, transcripts
exploitables ; **pas** de conclusion statistique.
`default_policy` devient un cas particulier : `compile_strategy(SPEC_PAR_DEFAUT)`.
## 3. Décider — les deux nouvelles couches
L'étage A fournit une **fonction d'évaluation déterministe** : c'est le bon
substrat pour *chercher* et pour *éprouver*.
### 3.1 Optimiseur — « meilleure stratégie de survie »
- **Espace** : les leviers de `StrategySpec` (par PJ ou par rôle).
- **Oracle** : `evaluate(spec, scenario, n, seeds) → métriques` (runs parallèles).
- **Objectifs** : `P(survivre tous)`, `P(TPK)`, `P(≥1 mort)`, espérance de PV
perdus, ressources consommées, rounds, **risque de queue (CVaR / pire décile)**.
- **Algorithmes** : grille grossière / échantillonnage latin → affinage
(descente coordonnée, **CMA-ES**, **optimisation bayésienne**) ; la LLM peut
**amorcer** avec des stratégies candidates (NL → `StrategySpec`).
- **Sortie** : un **playbook** classé (« si groupés : Buff → Contrôle → Focus
lanceur ; reculer sous 30 % PV ») avec incertitude.
### 3.2 Robustesse et cartographie — « où est le danger »
Un danger n'existe que **contre un adversaire qui cherche à gagner** : évaluer
les PJ contre la politique par défaut **sous-estime** le risque. Il faut :
- **`AdversarySpec`** (symétrique de `StrategySpec`) : priorité de cible, focus
fire, usage des capacités, embuscade, positionnement.
- **Min-max** : pour chaque scénario, `max_politique_PJ · min_politique_monstres`
(optimisation alternée = meilleures réponses) → pire cas réaliste. La même
mesure donne **les deux sens** : difficulté pour les PJ *et* pour les monstres
(rencontre trop facile).
- **Cartographie** (au lieu d'une énumération impossible) :
1. grille grossière sur `ScenarioSpec` → cellules d'échec ;
2. **affinage près de la frontière** (bissection / apprentissage actif) ;
3. **frontière de danger** : pour chaque mix, le seuil (ex. « < 45 % PV
cumulés et sans soigneur à portée ⇒ `P(TPK) > 20 %` ») ;
4. **analyse de sensibilité** : facteurs dominants (surprise, terrain,
épuisement des ressources, initiative).
## 4. État des lieux (socle vérifié)
| Élément | Emplacement | Vérifié |
|---|---|---|
| `Observation` figée, `to_dict()` JSON, sans `engine.*` | `combat.py:1262` / `1310` | oui |
| `Policy` / `LLMPolicy` / `ReactionPolicy` | `combat.py:10573/10575/10577` | oui |
| Câblage `policy`/`llm_policy` exclusifs | `combat.py:1615-1623` | oui |
| `ActionResult` + `ACTION_RESULT_REASONS` + `last_turn_results` | `combat.py:1483`, `1528` | oui |
| Démo « observation seule » | `tests/test_llm_engine_capabilities.py` | oui |
| Déterminisme `SeededRng(seed+i)` + métriques 3σ + attrition | `runner.py`, `metrics.py` | oui |
| Base de règles conforme | `docs/conformance-srd.md` (gate 1981) | oui |
**Manques confirmés** : `run_encounter` sans `llm_policy` (`runner.py:177`) ;
CLI sans politique ; pas de `schema_version` ; pas de vue compacte ; pas d'API
d'actions légales ; pas de mémoire inter-tours ; pas de journal de replay ;
`LLMPolicy(obs, state)` transmet l'état vivant ; **aucune couche d'objectifs,
de recherche, d'adversaire ou de scénarios**.
## 5. Architecture cible
```
Moteur (règles inchangées ; frontière additive)
|
|-- EXÉCUTION
| |- Étage A : compile_strategy(StrategySpec) -> Policy (sweeps)
| \-- Étage B : NLPolicy(observation) (vitrine <=20 runs)
|
|-- DÉCISION / ANALYSE
| |- objectives.py : métriques de survie/danger (dont queue)
| |- scenarios.py : ScenarioSpec + échantillonnage
| |- search.py : optimiseur (objectif = objectives), oracle = simulateur
| |- adversary.py : AdversarySpec + politiques de monstres fortes (red team)
| \-- robustness.py : min-max, cartographie de danger, frontière, sensibilité
|
\-- SUPPORT
|- view.py : ObservationView (objet structuré, versionné) + render()
|- candidates.py : actions plausibles dérivées des pairs
|- prompt.py : chargement EXTERNE (pack séparé) + versionnage
|- client.py : LLMClient + cache composite + FakeLLMClient scripté + coût
|- replay.py : JSONL clé call_seq + replay()
\-- sweep.py : exécution parallèle des runs (substrat commun)
```
**Interfaces clés** : `ObservationView` (objet + `render()`), `candidates()`
(dérivées des `pairs`, pas de 2ᵉ moteur de légalité), `ResponseCache` (clé
composite), `ReplayRecord` (clé `call_seq`), `PolicyFactory` (par run),
`evaluate(...)` (oracle d'optimisation), `danger_map(...)` (cartographie).
## 6. Espaces de recherche
| Espace | Contenu | Utilisé par |
|---|---|---|
| `StrategySpec` | priorité de cible, style d'engagement, distance, esquive d'AoO, seuil de repli, buffs d'ouverture, politique de sorts, ressources, focus fire | Optimiseur / Étage A |
| `AdversarySpec` | mêmes leviers côté monstres (focus, ciblage du soigneur/lanceur, contrôle, embuscade) | Robustesse |
| `ScenarioSpec` | composition/niveaux, **PV et ressources initiaux**, déploiement/positionnement, carte/terrain, **surprise**, distance, mix et nombre de monstres, CR | Cartographie |
## 7. Objectifs et métriques
| Métrique | Pourquoi |
|---|---|
| `P(survivre tous)` / `P(TPK)` / `P(≥ 1 mort)` | lecture directe de la survie |
| Espérance de PV perdus (groupe / PJ) | gravité |
| Ressources consommées (sorts, potions, canal) | soutenabilité |
| Rounds | tempo |
| **CVaR / pire décile** | **risque de queue** — la moyenne cache les TPK |
| Bandes 3σ (n suffisant) | incertitude — **étage A uniquement** |
## 8. Rôle de la LLM (recadré)
| Usage | Pourquoi la LLM |
|---|---|
| NL → `StrategySpec` / `AdversarySpec` | traduction structurée (petit, robuste) |
| **Générateur d'hypothèses** (candidats variés à amorcer) | créativité, pas de gradient |
| **Red Team** (tactiques de monstres fortes) | révéler les faiblesses |
| **Explicateur** (lire les transcripts dangereux) | causalité en langage naturel |
| **Rapporteur** (playbook + carte de risque) | rédaction (phase 3) |
**Non-usage** : optimisation numérique (faite par `search.py`) et exécution par
tour en sweep (faite par l'étage A).
## 9. Décisions
> Toutes les décisions ci-dessous sont **arbitrées** (voir §13) ; le tableau
> conserve la recommandation validée. Les prompts sont traités au §14.
| # | Décision | Recommandation |
|---|---|---|
| D1 | Entrée : `(observation, state)` ou `(observation)` ? | **`(observation)`** + adaptateur. **À trancher en P0.0.** |
| D2 | Vue dédiée vs `to_dict()` brut | Vue compacte, structurée, versionnée |
| D3 | Mémoire inter-tours | **Retirée de la v1** |
| D4 | Sortie d'actions | JSON strict, **schéma généré depuis `Action`**, `swift/free/full_round` exclus |
| D5 | Légalité | **Pas de 2ᵉ moteur** : candidats dérivés des `pairs` |
| D6 | Reproductibilité | Cache composite `(model, params, prompt_version, schema_version, hash)` |
| D7 | Runner/CLI | `llm_policy` + `policy_factory` sur `run_encounter` ; entrée de sweep dédiée |
| D8 | Périmètre v1 | Sous-ensemble curé ; `ReactionPolicy` par défaut conservée |
| D9 | Économie | **Deux étages** (exécution) |
| D10 | Parallélisme | Runs indépendants → pool de threads, cache shardé |
| D11 | Réactions | v1 par défaut ; LLM via `ready` |
| D12 | Danger = **adversarial** | Mesurer contre des monstres **optimisés**, sinon sous-estimation |
| D13 | Objectifs | Inclure le **risque de queue**, pas seulement la moyenne |
| D14 | Cartographie | **Non exhaustive** : coarse-to-fine + frontière + sensibilité |
## 10. Plan d'implémentation
### P0 — Frontière durcie (sans LLM ; débloque tout)
P0.0 trancher D1 (obs seule + adaptateur) · P0.1 `schema_version` + clés pinnées ·
P0.2 `ObservationView` + mesure de taille · P0.3 `candidates()` (depuis `pairs`) ·
P0.4 `run_encounter(..., llm_policy=, policy_factory=)` · P0.5 `ReplayLog`
(`call_seq`) + `replay()` identique.
**Sortie** : gate verte ; un run enregistré se rejoue à l'identique.
### P1 — Client et politiques
P1.1 `LLMClient` + `FakeLLMClient` scripté + cache composite · P1.2 **garde-fous
coût/tokens + retry/backoff** · P1.3 `NLPolicy` (étage B : parse, valide, 1
re-essai, repli) · P1.4 `PromptBuilder` versionné + schéma d'action généré.
**Sortie** : bataille complète pilotée par le Fake, hors ligne.
### P2 — Étage A : compilation, objectifs, sweep
P2.1 compilateur NL → `Policy` paramétrée · P2.2 `objectives.py` · P2.3
`SweepRunner` **parallèle**.
**Sortie** : sweep ≥ 2×2 avec métriques, borné en coût.
### P3 — Optimiseur (« meilleure stratégie »)
P3.1 `scenarios.py` (`ScenarioSpec` + échantillonnage) · P3.2 `search.py`
(grille → affinage) · P3.3 amorçage par candidats LLM.
**Sortie** : playbook classé avec incertitude pour ≥ 1 rencontre type.
### P4 — Adversaire et cartographie (« où est le danger »)
P4.1 `AdversarySpec` + politiques de monstres fortes · P4.2 `robustness.py`
(min-max alterné) · P4.3 `danger_map` (coarse-to-fine, frontière, sensibilité).
**Sortie** : carte de danger (PJ **et** monstres) + frontière seuils.
### P5 — Fournisseur réel et rapport (frontière phase 3)
P5.1 `ProviderClient` (clé env, coût/observabilité) · P5.2 few-shot + réparation
complète · P5.3 rapport FR (phase 3).
## 11. Risques
1. **Économie per-tour** → résolue par les deux étages.
2. **Adversaire trop faible** : danger sous-estimé → monstres **optimisés** (D12).
3. **Replay sous-spécifié** (`delay`, invocations) → clé `call_seq`.
4. **Cache sans identité de modèle** → clé composite.
5. **Divergence de légalité** → candidats dérivés des `pairs`.
6. **Bascule tardive de D1** → à trancher en P0.0.
7. **Bandes 3σ trompeuses** à petit n → étage A pour le quantitatif.
8. **Explosion combinatoire** → coarse-to-fine, pas d'énumération (D14).
9. **Événements rares (TPK)** : n énorme en Monte Carlo naïf → ciblage par
scénarios, ou annoncer la limite.
## 12. Première action recommandée
Un **spike** (sans LLM) qui :
(a) tranche **D1** + `schema_version` + test de clés pinné ;
(b) **mesure** taille de prompt et appels par cellule de sweep (décide per-tour
vs compilé) ;
(c) esquisse `ScenarioSpec` + `objectives.py` et **valide la chaîne de
cartographie** sur une rencontre type, en comparant `default_policy` contre une
politique de monstres renforcée — ce dernier point prouve immédiatement la valeur
« prédire où sont les difficultés », avant toute LLM.
## 13. Décisions arbitrées
| # | Décision | Choix retenu |
|---|---|---|
| D1 | Entrée de la politique | **`(observation)` seule** + adaptateur de compatibilité |
| D9 | Exécution | **Deux étages** (compilé pour les sweeps, per-tour pour la vitrine) |
| D12 | Mesure du danger | **Contre un adversaire optimisé** (`AdversarySpec` + meilleure réponse) |
| D14 | Cartographie | **Non exhaustive** : région d'échec + frontière + sensibilité |
| — | Priorité v1 | **Cartographie d'abord**, puis optimiseur (mêmes briques) |
| — | Fournisseur/modèle | **Client agnostique** ; implémentation distante d'abord, locale ensuite ; **mode JSON/schéma requis** |
| — | Budget | **Plafonds petits** (≤ 50 appels de compilation/campagne, ≤ 20 combats en vitrine) + coupe-circuit dans le client |
| — | Reproductibilité | **Cache obligatoire** ; clé composite incluant le modèle ; replay `call_seq` |
| — | Prompts | **Externes / dépôt séparé** (voir §14) |
Reste à préciser au fil des lots (non bloquant) : forme exacte des leviers de
`StrategySpec`/`AdversarySpec`, choix de l'algorithme d'optimisation et seuils
statistiques par défaut (le fournisseur distant est **arbitré en P5.1** : un
gateway OpenAI-compatible, §16).
## 14. Prompts externes (décision utilisateur)
Les prompts **ne vivent pas dans ce dépôt** : ce dépôt ne doit contenir que le
**code du simulateur** (calcul + couche LLM + rapport). Conséquences :
- **Chargement configurable** : le code lit les prompts depuis un **chemin
externe** (variable d'environnement / fichier de configuration), jamais codés
en dur.
- **Versionnage côté pack** : le dépôt de prompts porte sa propre version
(semver ou hash de commit) ; chaque prompt a un hash individuel.
- **Traçabilité** : le `prompt_version` (version du pack + hash du prompt)
entre dans la **clé du cache**, dans le `ReplayRecord` et dans les rapports de
sweep — sinon une campagne n'est pas reproductible après modification d'un
prompt.
- **Tests** : le pack externe est **épinglé** pour l'intégration (version fixée) ;
les tests unitaires utilisent un **fixture de prompt minimal** local pour que
la CI ne dépende pas du dépôt externe.
- **Validation** : un prompt manquant/invalide doit produire une **erreur
claire** au démarrage (pas de fallback silencieux).
## 15. Périmètre du dépôt et backlog de détachement
Décision utilisateur : à terme, **seul le code** doit rester ici. Sont
concernés par un détachement (données et contenu) : fiches de personnages
(`fiches_personnages/`), données de monstres/PJ/sorts/cartes (`data/`), et les
fichiers de prompts.
**Pour l'instant : laissé en place** (rattrapage prévu en fin de projet).
Contrainte de conception à respecter dès maintenant pour ne pas se peindre dans
un coin :
- **Aucun chemin de données en dur** dans le nouveau code (couche phase 2) :
centraliser la résolution des chemins (config/env) pour préparer le
déplacement.
- Les prompts sont **déjà** conçus comme externes (§14) — pas de régression.
- Le détachement final devra fournir un **jeu de données d'exemple** et une
documentation de configuration (chemins) pour que le dépôt code reste utilisable
seul.
## 16. Résultats du spike (exécuté)
**Livré** : D1 (`LLMPolicy(Observation)` + `LegacyLLMPolicy`), `schema_version`
pinné, `objectives.py`, `run_encounter(..., collect_runs=True)`, une politique
de monstres **focus fire** (spike) et une **carte de danger** de démonstration.
### Mesures (décision d'architecture)
| Mesure | Valeur |
|---|---|
| Observation, 7 combattants | **64 Ko ≈ 16 000 tokens** (max 67 Ko) |
| Observation, duel 2v2 | 22 Ko ≈ 5 500 tokens |
| Appels de politique / combat | **124** (7 combattants), 12 (duel) |
| Appels par cellule de sweep (1 000 runs) | **124 000** |
→ **Conclusion 1** : un LLM **par tour** en sweep est infaisable (~2 milliards de
tokens par cellule) — **D9 validé**, et la **vue compacte est obligatoire**
(cible ~2-4 k tokens, contre 16 k pour le snapshot brut).
### Carte de danger (RUNS=100, seed=11)
| Scénario | Monstres | Victoire PJ | Survivent | TPK | ≥1 mort | PV perdus | Rounds |
|---|---|---|---|---|---|---|---|
| warband 4v4 | défaut | 90,0 % | 87,0 % | 0,0 % | 13,0 % | 66,1 | 12,3 |
| warband 4v4 | focus fire | 98,0 % | 76,0 % | 0,0 % | **24,0 %** | 45,0 | 9,6 |
| liche 2v2 | défaut | 0,0 % | 100,0 % | 0,0 % | 0,0 % | 44,0 | 3,7 |
| liche 2v2 | focus fire | 0,0 % | 81,0 % | **2,0 %** | **19,0 %** | 44,0 | 5,8 |
→ **Conclusion 2** : mesuré contre la politique par défaut, le danger est
**sous-estimé** (morts 13 %→24 % et 0 %→19 %, TPK 0 %→2 %) — **D12 validé**.
→ **Conclusion 3** : le **taux de victoire trompe** (il monte à 98 % pendant que
les morts augmentent) → les métriques de mort/PV/**queue** sont nécessaires —
**D13 validé**.
**Limites du spike** : n=100 (bandes larges ±9 %), adversaire de niveau spike
(focus fire seulement, sans positionnement ni sorts), 2 scénarios. La
cartographie réelle (P4) demandera plus de runs, un adversaire complet
(`AdversarySpec`) et un balayage de `ScenarioSpec`.
### Vue compacte (P0.2, mesures réelles — commit 9da9537)
`llm_view.py` : `ObservationView` (structuré, déterministe) + `build_view()`
+ `render()` vers texte pour prompt. Règles : **grilles sparsifiées** (terrain
et élévation = cellules non défaut ; `threatened`/`ranged_threatened` = liste
de cellules, omis si vide), **clés décisionnelles seules** par combattant et
par paire, `schema_version` inclus, ordre déterministe trié. Mesure sur
l'affrontement représentatif (oni/misty/nairda vs lich/thrune_agent/orc/goblin,
seed 1) :
| Vue | Octets | Tokens | % du raw |
|---|---|---|---|
| brute `to_dict()` | 64 204 | ~16 051 | 100 % |
| **compacte `render()`** | **15 691** | **~3 922** | **24,4 %** |
Ratio 24,4 % = **cible ~2-4k tokens atteinte**. Grilles : terrain 78 cellules
sparses, élévation 0, menacées 15, menacées à distance 116 (au lieu de matrices
pleines). Contrats de test : réduction < 25 % et < 16 000 octets, grilles
sparses, clés supprimées (condition_definitions, etc. — voir le test), ids/
render déterministes. Test : `tests/test_phase2_view.py`.
### Candidats d'action (P0.3 — commit 8b50430)
`candidates.py` : `candidates(observation) -> tuple[Action, ...]` — actions
**plausibles** dérivées des `pairs` et des données par combattant (attack/
full_attack si `weapon_for`, charge si `can_charge`, pas de 5 ft + attaque si
`can_step5_to_attack`, mouvement vers une cible, sort connu avec slot, canal/
hex/deed si ressource présente, défense totale, ready/delay) ; `swift`/`free`/
`full_round` exclus ; liste bornée et triée. **Pas de second moteur de
légalité** : le moteur reste l'arbitre. Test : `tests/test_phase2_candidates.py`.
### Runner et rejeu (P0.4 + P0.5 — commits 334f8f9, 6c6de16)
`run_encounter` accepte désormais `llm_policy=` (observation-only ou legacy,
transmise à chaque `CombatEngine`) et `policy_factory=` (constructeur de
`Policy` **par run**, appelé une fois par indice dans l'ordre ; `None` pour un
run = repli `default_policy`). Les trois entrées (`policy`, `llm_policy`,
`policy_factory`) sont mutuellement exclusives (erreur claire). Test :
`tests/test_phase2_runner.py` (factory 1 appel/run dans l'ordre, repli,
exclusivité, une bataille complète pilotée par `llm_policy`).
`replay.py` : `ReplayRecord(call_seq, observation, action)` + `ReplayLog`
(spec + seed + round_cap + appels) sérialisé en **JSONL** (clé `call_seq`),
`observation_key()` (digest déterministe, insensible à la collapse
tuple/liste de JSON), et `replay(log) -> LLMPolicy` qui renvoie les actions
enregistrées dans l'ordre des `call_seq`, avec gardes : observation
divergente (`ReplayMismatchError`), log épuisé (`ReplayExhaustedError`),
`call_seq` dupliqué ou hors ordre, JSONL invalide. **Un run enregistré se
rejoue à l'identique** (wins/draws/rounds/transcript — aussi depuis un log
dégelé du JSONL). Test : `tests/test_phase2_replay.py`. Le rejet de replay
par défaut est documenté (D3 : pas de mémoire inter-tours ; `delay`/ready
rejoués par les actions enregistrées elles-mêmes, sans ré-invocation).
### Étage A (P2 — commit dbd1a58)
`strategy.py` : `StrategySpec` (cible `nearest`/`weakest_ac`/`lowest_hp`,
`retreat_hp_fraction`, `spell_policy` `default`/`none`, `use_charges`,
`use_class_abilities`) et `compile_strategy(spec) -> Policy` —
`compile_strategy(StrategySpec()) is default_policy` (identité) ; les autres
specs compilent vers des politiques paramétrées (échelle de mêlée, charge
quand la ligne droite existe, retraite sous le seuil de pv, sorts / charges /
capacités de classe désactivables), cible choisie par `select_target`
(CA la plus basse, pv les plus bas, ou le plus proche). Test :
`tests/test_phase2_strategy.py`.
`sweep.py` : `SweepCell(scenario, strategy, spec, policy, runs, seed)` +
`SweepRunner(party=, workers=)` (registry des sorts chargée **une fois**,
lecture seule partagée entre threads — D10), `run(cells, max_total_runs=)`
(erreur claire nommant `runs`/`budget`, cellules ≥ 1 run) et `run_grid(...)`
en ordre **scénario-majeur** — chaque cellule : `run_encounter(collect_runs=True)`
+ `danger_metrics`. Une politique `None` = `default_policy` (le runner
n'importe pas `strategy.py`). `format_sweep` : rendu texte aligné (ascii).
Tests : `tests/test_phase2_sweep.py` (déterminisme bit-à-bit, workers=1 ≡
workers=4, cellule `None` ≡ `run_encounter` direct, borne de coût).
**Mesure représentative (RUNS=100, seed=7, partie `players` = gobelins,
adversaire `focus fire` = orcs)** :
| Scénario | Monstres | Victoire PJ | Survivent | TPK | ≥1 mort | PV perdus | Rounds |
|---|---|---|---|---|---|---|---|
| 3 gobelins vs 2 orcs | défaut | 93,0 % | 96,0 % | 0,0 % | 4,0 % | 5,0 | 6,5 |
| 3 gobelins vs 2 orcs | focus fire | 98,0 % | 100,0 % | 0,0 % | 0,0 % | 2,4 | 5,6 |
| 2 gobelins vs 3 orcs | défaut | 50,0 % | 96,0 % | 0,0 % | 4,0 % | 7,5 | 10,6 |
| 2 gobelins vs 3 orcs | focus fire | 54,0 % | 96,0 % | 0,0 % | 4,0 % | 7,3 | 10,1 |
→ la sortie P2 (« sweep ≥ 2×2 avec métriques, borné en coût ») est atteinte :
2×2 = 400 combats en quelques secondes sur 4 workers, métriques de danger
complètes par cellule, déterminisme vérifié. Ce 2×2 gobelins/orcs est un
**smoke test du runner** (n=100, bande 3σ ±6 % sur les taux) : à ce n, le
delta focus fire est dans le bruit sur les morts (4→0 en 3v2, plat en 2v3) —
contrairement au spike warband/liche où il sortait nettement de la bande. Le
signal le plus net ici est la **queue** (worst-decile HP 1,4 → 6,8 en 3v2 ;
0,0 dans les deux cellules 2v3) et les rounds moyens (6,5 → 5,6 en 3v2). La
cartographie du danger à grande n, PJ **et** monstres, reste **P4**.
### Optimiseur (P3 — commit dd64c2e)
`scenarios.py` : `SideSpec(name, zone, combatant_files, hp_fraction=1.0)` et
`ScenarioSpec(name, map_file, sides, round_cap=100)` — `build_encounter`
compile un scénario en `EncounterSpec` déterministe (aucun chemin de données
en dur dans le src, §15), `scenario_variants(base, hp_fractions=...)` étend
grossier le scénario par fractions de PV des côtés (grille de robustesse, sans
RNG). Test : `tests/test_phase2_scenarios.py`.
`search.py` : `evaluate(strategy, scenario, *, party, runs, seed, workers=4)`
(`StrategyBenchmark` : métriques de danger du camp `party` + bande 3σ sur le
taux de victoire), `PlaybookEntry(rank, strategy, benchmark)`,
`strategy_name` (nom canonique de `StrategySpec`) et
`optimize(scenario, *, party, runs, seed, workers=4, max_total_runs=None,
candidates=())` — recherche en 3 étages : grille grossière de 24 cellules
(cible × politique de sorts × charges × capacités de classe, repli à 0),
affinage des 3 meilleures cellules sur `retreat_hp_fraction ∈ {0.25, 0.5,
0.75}` (9 cellules ; cellules déjà mesurées dédupliquées), puis les candidats
LLM (`candidates`, P3.3) dédupliqués et évalués. Classement :
`(-P(survive), PV perdus, -pire dcile PV, rounds, nom)`. Le budget
`max_total_runs` est vérifié **avant** tout combat (`ValueError` nommant
`budget`/`max_total_runs`). `format_playbook` rend le classement en colonnes
ASCII avec la bande 3σ. Test : `tests/test_phase2_search.py`.
**Playbook représentatif (RUNS=200, seed=7, workers=4, camp mesuré =
`players` = 3 gobelins vs 2 orcs)** — 33 cellules (grille 24 + affinage 9 +
candidats 0) ; les rangs non listés sont des variantes à leviers inertes ou
de repli strictement identiques à leur cellule de tête :
| Rang | Stratégie | Victoire PJ (3σ) | Survivent | ≥1 mort | PV perdus | Pire dcile | Rounds |
|---|---|---|---|---|---|---|---|
| 1 | `lowest_hp` (+ inertes, rangs 2-8) | 97,0 % [93,4 ; 100,0] | 99,0 % | 1,0 % | 3,0 | 5,2 | 5,8 |
| 9 | `weakest_ac` (+ inertes, rangs 10-16) | 97,0 % [93,4 ; 100,0] | 99,0 % | 1,0 % | 3,0 | 5,2 | 5,8 |
| 17 | `lowest_hp, retreat=0.25` (+ inertes) | 96,0 % [91,8 ; 100,0] | 97,5 % | 2,5 % | 3,1 | 3,9 | 6,1 |
| 20 | `lowest_hp, retreat=0.5` (+ inertes) | 95,0 % [90,4 ; 99,6] | 97,5 % | 2,5 % | 3,5 | 3,0 | 6,4 |
| 23 | `default` (nearest, repli 0) | 95,0 % [90,4 ; 99,6] | 97,0 % | 3,0 % | 5,0 | 2,8 | 6,9 |
| 31 | `lowest_hp, retreat=0.75` (+ inertes) | 94,0 % [89,0 ; 99,0] | 95,5 % | 4,5 % | 3,7 | 2,7 | 7,6 |
TPK 0,0 % dans les 33 cellules. → la sortie P3 (« meilleur candidat du grid +
affinage, playbook classé ») est atteinte : 6600 combats en quelques secondes
sur 4 workers, classement déterministe (rejoué bit-à-bit), budget vérifié
avant exécution.
→ **Le ciblage est le levier dominant** : `lowest_hp` et `weakest_ac` battent
`default` sur toutes les métriques du camp — PV perdus 5,0 → 3,0, pire dcile
2,8 → 5,2, rounds 6,9 → 5,8, survive 97,0 → 99,0 %. Égalité bit-à-bit entre
les deux ordres de cibles : les adversaires adverses étant statistiquement
identiques (gobelins entre eux, orcs entre eux), les deux heuristiques
sélectionnent la même cible au même rang.
→ **La retraite agressive est strictement dominée** : seuil 0,25 → 0,5 → 0,75
dégrade la survie (99,0 → 97,5 → 97,5 → 95,5 %), le pire dcile (5,2 → 3,9 →
3,0 → 2,7) et allonge le combat (5,8 → 6,1 → 6,4 → 7,6 rounds) — une retraite
est une action à round complet qui paie le départ sans compenser.
→ **Leviers inertes sur cette rencontre** : `spell_policy`, `use_charges` et
`use_class_abilities` ne changent rien (gobelins et orcs n'ont ni sorts, ni
ressources, ni capacités de classe) — leurs variantes font exactement égalité
avec leur cellule de grille (rangs 2-8, 10-16, 24-30). Un scénario avec
lanceur est nécessaire pour discriminer ces leviers (rencontres riches en
P4/P5).
→ **Sémantique de mesure — symétrique** : une politique par cellule, appliquée
aux **deux** camps (`CombatEngine` exécute la politique passée au moteur, sans
routage par côté ; `compile_strategy` est côté-agnostic) ; le classement porte
sur les `DangerMetrics` du camp `party`. Contrairement au spike P2
(`focus_fire_policy` routait par côté : orcs uniquement), le playbook mesure
donc « la meilleure stratégie quand tout le monde la joue » — l'asymétrie PJ
contre adversaire optimisé arrive en P4 (`AdversarySpec`).
→ Cohérence : la cellule `default` (95,0 % [90,4 ; 99,6]) contient la ligne P2
défaut (93,0 %) dans sa bande 3σ (mêmes seeds, n = 100 → 200).
→ Limite : P3.3 (candidats LLM) est câblé et testé (déduplication,
0 candidat ici) mais non exercé contre un vrai LLM — aucun fournisseur avant
P5. Bande 3σ ±4,6 points à p = 0,97, n = 200 : au sein d'une famille à
leviers inertes les écarts sont nuls (bit-identiques), mais un départage plus
fin (n plus grand, carte plus riche) reste ouvert.
### Adversaire et robustesse (P4)
`adversary.py` : `AdversarySpec(target_priority="nearest" | "lowest_hp" |
"weakest_ac" | "healer", spell_policy="default" | "none",
retreat_hp_fraction=0.0 | 0.25 | 0.5, use_charges=True | False)` — les
leviers adverses (D12 : l'adversaire joue pour gagner) ; un seul
`AdversarySpec()` (défaut = `nearest`, la politique focus fire du spike P2)
est composé avec la stratégie du camp mesuré en une seule politique de
combat par `compile_matchup(party, adversary, party_side=)`. `adversary_name`
: nom canonique d'un levier. Test : `tests/test_phase2_adversary.py`.
`robustness.py` : `matchup_benchmark(scenario, party, adversary, *,
party_side, runs, seed) -> MatchupBenchmark` (mesure le danger des DEUX
camps — même mesure, les deux sens du §3.2), puis les trois couches :
- `adversarial_analysis` (P4.2, min-max alterné) : itérations successives —
la partie maximise sa survie parmi `party_candidates`, puis l'adversaire
minimise le danger de la partie parmi `adversary_candidates` ; chaque
itération est rendue par `format_robustness`. Le budget `max_total_runs=`
est vérifié **avant** tout combat (`ValueError` nommant
`budget`/`max_total_runs`).
- `danger_map` (P4.3) : grille `SideSpec.hp_fraction` PJ × adversaire, une
cellule par paire de fractions = un `MatchupBenchmark` ; `format_danger_map`
rend la grille (survie des deux camps par cellule).
- `frontier` (P4.3) : bissection de la fraction de PV PJ où `criterion`
croise `target` (convention §3.2), adversaire fixé — le retour est plafonné
au pas `precision` (`ceil`, jamais arrondi, pour rester du côté croisant du
seuil) ; None quand même `max_fraction` ne croise pas. `format_frontier`
rend « no frontier ».
- `sensitivity` (P4.3) : classe les 7 leviers adverses par |delta| du
critère de la partie vs baseline `AdversarySpec()` (budget exact 8×runs,
vérifié avant combat) ; `format_sensitivity` affiche les probabilités en %,
`expected_hp_lost` à 1 décimale.
Tests : `tests/test_phase2_robustness.py`.
**Mesures représentatives (RUNS=100, seed=7, camp mesuré = `players` =
3 gobelins vs 2 orcs, `sample_arena.yaml`)** :
P4.1 — équilibre du min-max alterné (2 itérations ; l'adversaire best-
response converge vers `AdversarySpec()` dès la première itération, les deux
itérations sont bit-identiques) :
| Stratégie PJ | P(survie PJ) | PV perdus PJ | P(survie adv) | PV perdus adv | TPK |
|---|---|---|---|---|---|
| `default` (nearest) | 96 % | 5,0 | 98 % | 11,6 | 0 % |
| `lowest_hp` | 95 % | 6,3 | 99 % | 11,5 | 0 % |
| `retreat=0.5` | 97 % | 5,3 | 99 % | 11,4 | 0 % |
→ la meilleure réponse adverse est le ciblage `nearest` par défaut (le plus
dangereux pour la partie, voir sensitivity) ; face à cet équilibre les trois
stratégies PJ sont dans le bruit (bande 3σ ±6 points à n=100). Le camp
adverse encaisse ~11,5 PV moyens par combat (il perd presque toutes les
runs) mais ne **meurt** (hp ≤ −CON) que dans ~2 % des runs (98-99 % de
survie adv) : la rencontre bat l'adversaire en le mettant à ≤ 0 pv sans
souvent le tuer — cohérent avec la règle de mort `hp ≤ −CON` du moteur.
P4.2 — danger_map (PJ `default` × adversaire `weakest_ac`, fractions
0.5/0.7/1.0, 9 cellules ; grille `Party\Adv` : survie partie/survie
adversaire par cellule) :
| | adv=0.5 | adv=0.7 | adv=1.0 |
|---|---|---|---|
| party=0.5 | 100 %/100 % | 100 %/99 % | 99 %/99 % |
| party=0.7 | 99 %/100 % | 100 %/99 % | 100 %/99 % |
| party=1.0 | 100 %/100 % | 100 %/99 % | 100 %/99 % |
→ la grille est plate (99-100 %) et le TPK nul partout : la mort est un
événement rare sur cette rencontre (au plus 1 run sur 100 par cellule) et
les écarts observés restent sous la bande 3σ (±6 points) — même décimée à
50 % de PV, la partie survit. Une carte discriminante exige une rencontre
plus équilibrée (le 2 gobelins vs 3 orcs de P2, où ≥ 1 mort monte à 4 %).
P4.3 — frontier (`p_survives`, target=0.5, fractions [0.10, 1.00],
adversaire `healer`) : **`no frontier`** — la survie de la partie ne
descend jamais sous 50 %, même à 10 % de PV initiaux : le plancher de
robustesse de la partie est sous la barre des 10 %.
P4.3 — sensitivity (criterion=`expected_hp_lost`, baseline `default` :
PV perdus 4,97, survie 96 %) :
| Rang | Levier | Value | Delta |
|---|---|---|---|
| 1 | `healer` | 2,4 | −2,6 |
| 2 | `lowest_hp` | 2,4 | −2,6 |
| 3 | `weakest_ac` | 2,4 | −2,6 |
| 4 | `retreat=0.5` | 3,1 | −1,9 |
| 5 | `retreat=0.25` | 4,0 | −1,0 |
| 6 | `spell_policy=none` | 5,0 | +0,0 |
| 7 | `use_charges=false` | 5,0 | +0,0 |
→ le ciblage est le levier adverse le plus sensible : quitter `nearest`
pour n'importe quel ciblage explicite (healer/lowest_hp/weakest_ac, ex æquo
à −2,6) fait tomber l'attrition de la partie de 5,0 à 2,4 PV et remonte sa
survie à 100 % — le ciblage de la cible la plus proche (la plus engagée en
mêlée) est donc le plus dangereux pour la partie ; des adversaires
statistiquement identiques rendent les trois ciblages ex æquo (même cible au
même rang, déjà observé en P3). La retraite orc (combat écourté) réduit
aussi le danger (−1,9 / −1,0). Leviers inertes : `spell_policy` et
`use_charges` (les orcs n'ont ni sorts ni ressources) — delta 0,000.
Budget : `adversarial_analysis(max_total_runs=1)` → `ValueError`
(« robustness budget exceeded: 800 runs > 1 max_total_runs ») avant tout
combat. Déterminisme : deux passes de `matchup_benchmark` bit-identiques
(p_survives 1,000 / p_tpk 0,000 / hp_lost 2,4).
→ la sortie P4 (« adversaire `AdversarySpec` + cartographie de danger
complète, bornée en coût ») est atteinte : min-max alterné convergeant,
grille de danger 3×3, frontière de survie, classement des leviers adverses,
déterminisme et budget vérifiés. Les mesures placent la rencontre de
référence 3v2 du côté sûr de l'équilibre : TPK nul, plancher de survie sous
10 % de PV, et le seul levier adverse qui y compte est le ciblage par
défaut.
### Fournisseur réel OpenAI-compatible (P5.1)
Le « ProviderClient » du §10 est **`OpenAILLMClient`** (`provider.py`) : il
parle le format wire **chat-completions** d'OpenAI sur la bibliothèque
standard (`urllib.request`) — aucune dépendance HTTP dans le dépôt.
L'endpoint, le modèle et la clé viennent de l'environnement (§15) :
`PF1E_LLM_BASE_URL` (défaut `https://api.openai.com/v1`), `PF1E_LLM_MODEL`
(défaut `gpt-4o-mini`) et `PF1E_LLM_API_KEY` (jamais codée en dur) ; un
gateway vLLM / llama.cpp / compatible se branche par variables d'environnement,
sans toucher au code.
Taxonomie d'erreurs (partagée avec `client.py`) :
- `ProviderError` (transitoire) : erreur de transport, HTTP 429 et tout 5xx —
rejoués par la boucle de backoff de `LLMClient` et comptés par le
coupe-circuit ;
- `ProviderRejectedError` (permanent) : tout autre 4xx (clé invalide, modèle
inconnu, quota) — levée immédiatement, jamais rejouée ;
- `ProviderResponseError` (permanent) : un corps 2xx qui n'est pas un payload
chat-completions valide (JSON invalide, `choices`/`message`/`content`
absents, `usage` malformé) — levée immédiatement.
Le corps de requête demande `response_format` JSON (`json_object`, mode
JSON/schéma requis, D arbitrées §13) ; l'usage est lu du bloc `usage` du
payload et estimé au caractère (`estimate_tokens`) quand le fournisseur
l'omet. Les 17 cas de `tests/test_phase2_provider.py` pinent le format wire,
la taxonomie, la résolution de clé par environnement, le ledger de tokens et
le cache — à travers un transport `urlopen` injectable, jamais le réseau.
Gate verte : **2186 tests**.
### Few-shot et réparation complète (P5.2)
**Few-shot externe** (`prompt.py` + `prompt_builder.py`) : un pack peut
porter, à côté de chaque `{name}.md`, un fichier `{name}.examples.json` —
une **liste JSON** de paires d'enseignement `{"observation": {...},
"actions": [...]}` (clés strictement validées, toute clé inconnue est un
`PromptPackError`). La découverte est optionnelle et triée ; un fichier
orphelin (sans `{name}.md` correspondant) est une erreur de pack ; un
contenu non-JSON, un non-tableau, un item non-objet, une `observation`
non-objet ou des `actions` non-tableau/non-objets sont rejetés à la même
taxonomie (`_pack_error`, message citant le chemin et l'index). Le paquet
expose les paires via `PromptPack.examples(name)` (tuple de
`FewShotExample` frozen) ; le **hash de version couvre désormais aussi le
contenu des exemples** : `prompt_version = sha256(content + "\0" +
examples_text)[:12]`, **byte-identique** quand aucun fichier d'exemples
n'existe (le cache/ledger D6 existant n'est pas invalidé). `PromptBuilder`
insère entre le template et l'observation une section `## few-shot examples`
dont chaque bloc `### example {n}` rend `observation:` puis `response:` en
JSON trié (`sort_keys=True`) — bit-identique sans exemples (garde de
régression testée sur l'ordre des sections).
**Réparation complète** (`nl_policy.py`) : la boucle de réparation de P1.3
devient **bornée et tracée**. `NLPolicy` porte `max_repairs` (défaut `1` —
comportement d'origine) : chaque réponse invalide est suivie d'un retry dont
le prompt accumule le **trail chronologique** des échecs précédents
(`-- previous reply invalid (attempt N) --` + message), refermé par
`Return a valid JSON action sequence.` ; au-delà de `max_repairs` tentatives
échouées, le fallback scripté prend le relais (`fallbacks`). `max_repairs=0`
désactive la réparation (fallback immédiat) ; une valeur négative est
rejetée au `__post_init__` (`ValueError`). Les compteurs vivent dans
`NLPolicyStats` (dataclass mutable, `policy.stats`) : `calls`,
`repairs_attempted`, `repairs_succeeded` et `fallbacks` — la matière des
rapports de P5.3 et du sweep.
Fixture `tests/fixtures/prompt_pack_fs/` (pack « few-shot » : version
`0.1.0-test`, même prompt texte que le pack source, un micro-exemple
goblin→orc). Les 25 nouveaux cas pinent : la validation de
`tests/test_phase2_prompt.py` (12), le rendu et l'ordre des sections de
`tests/test_phase2_prompt_builder.py` (5, dont le tableau vide → pas de
section) et la boucle bornée de `tests/test_phase2_nl_policy.py` (8, via un
client enregistreur dérivé de `FakeLLMClient` qui capture chaque prompt
envoyé — trail vérifié tentative par tentative). Gate verte : **2211 tests**.
### État du plan
- **P0** ✅ · **P1** ✅ (`LLMClient` + `FakeLLMClient` scripté + cache
composite, garde-fous coût/tokens, `NLPolicy`, `PromptBuilder` versionné) ·
**P2** ✅ (`strategy.py` + `SweepRunner`, sortie mesurée) · **P3** ✅
(`scenarios.py` + `search.py`, playbook représentatif mesuré) · **P4** ✅
(`adversary.py` + `robustness.py`, adversaire `AdversarySpec` + carte de
danger complète) · **P5.1** ✅ (`provider.py`, fournisseur OpenAI-compatible
HTTP au format chat-completions) · **P5.2** ✅ (few-shot externe
`*.examples.json` hashé dans `prompt_version` + boucle de réparation bornée
`max_repairs` tracée par `NLPolicyStats`) — gate verte (**2211 tests**).
- Détail P0 : **P0.0 (D1)** ✅ · **P0.1 (`schema_version`)** ✅ · mesures ✅ ·
`objectives.py` ✅ · `collect_runs` ✅ · **P0.2** (vue compacte) ✅ ·
**P0.3** (`candidates`) ✅ · **P0.4** (`llm_policy` + `policy_factory`) ✅ ·
**P0.5** (ReplayLog `call_seq` + `replay()` identique) ✅.
- **Frontière P0 atteinte** : gate verte (2022 tests) ; un run enregistré se
rejoue à l'identique.
- Suite : **P5.3** (rapport FR).
+11
View File
@@ -56,6 +56,17 @@ allowed-confusables = ["σ"]
"src/pf1e_simulator/combat.py" = ["PLR0913"]
# runner.py: run_encounter needs 6 params (spec, runs, seed, policy, spell_registry, capture_first_transcript)
"src/pf1e_simulator/runner.py" = ["PLR0913"]
# client.py: LLMClient/FakeLLMClient need the full safeguard knobs (cache, budget, pricing, retry, breaker)
"src/pf1e_simulator/client.py" = ["PLR0913"]
# provider.py: OpenAILLMClient.__init__ needs the full knob set (model, key,
# base url, timeout, transport, schema, cache, budget, pricing, retry, breaker)
# and the chat-completions URL is built from an env-gated base URL defaulting
# to HTTPS (section 15), so the S310 audit and the 14-param constructor are
# deliberate
"src/pf1e_simulator/provider.py" = ["PLR0913", "S310"]
# robustness.py: the cartography API takes the full knob set (scenario, specs,
# side, runs, seed, fractions, target, precision, budget) as keyword parameters
"src/pf1e_simulator/robustness.py" = ["PLR0913"]
# ── Basedpyright ──────────────────────────────────────────────────────────────
[tool.basedpyright]
+124
View File
@@ -0,0 +1,124 @@
"""Adversary spec and compilation (docs/phase-2-llm-architecture.md section 10, P4.1).
The adversary-facing mirror of strategy.py: ``AdversarySpec`` is a
field-for-field copy of ``StrategySpec`` (same knobs, same validation) so the
enemy side of a matchup carries its own spec type without touching the party
search space. ``compile_adversary`` has identity semantics — it rebuilds the
mirrored ``StrategySpec`` and delegates to ``compile_strategy``, so the
all-default spec compiles to the module-level ``default_policy`` object
itself. ``compile_matchup`` glues the two sides into ONE ``Policy`` that
routes per combatant side: the engine invokes a single policy for every
turn (``CombatEngine._take_turn`` → ``_policy(self, state)``), so the policy
branches on ``state.side``, following the ``phase2_spike.focus_fire_policy``
pattern. Deterministic throughout: no RNG, no LLM.
"""
from __future__ import annotations
from dataclasses import dataclass
from pf1e_simulator.combat import (
Action,
CombatantState,
CombatEngine,
Policy,
default_policy,
)
from pf1e_simulator.strategy import (
SpellPolicy,
StrategySpec,
TargetPriority,
compile_strategy,
)
@dataclass(frozen=True)
class AdversarySpec:
"""Field-for-field mirror of ``StrategySpec`` (same defaults, same validation)."""
name: str = "default"
target_priority: TargetPriority = "nearest"
retreat_hp_fraction: float = 0.0 # 0.0 = never retreat
spell_policy: SpellPolicy = "default"
use_charges: bool = True
use_class_abilities: bool = True
def __post_init__(self) -> None:
if not self.name:
msg = "name must be a non-empty string"
raise ValueError(msg)
if not 0.0 <= self.retreat_hp_fraction <= 1.0:
msg = "retreat_hp_fraction must be within [0, 1]"
raise ValueError(msg)
def adversary_name(spec: AdversarySpec) -> str:
"""A deterministic label for `spec` from its non-default levers.
Local mirror of ``search.strategy_name`` (same field order: target_priority,
spell_policy, use_charges, use_class_abilities, retreat_hp_fraction; joined
with ", "; the `name` field never appears; all-default is "default"), kept
local so this module never imports search.py.
"""
levers: list[str] = []
if spec.target_priority != "nearest":
levers.append(spec.target_priority)
if spec.spell_policy != "default":
levers.append(f"spell_policy={spec.spell_policy}")
if not spec.use_charges:
levers.append("use_charges=false")
if not spec.use_class_abilities:
levers.append("use_class_abilities=false")
if spec.retreat_hp_fraction != 0.0:
levers.append(f"retreat={spec.retreat_hp_fraction}")
if not levers:
return "default"
return ", ".join(levers)
def compile_adversary(spec: AdversarySpec) -> Policy:
"""Compile an ``AdversarySpec`` into a ``Policy`` (identity semantics).
Rebuilds the mirrored ``StrategySpec`` and delegates to
``compile_strategy``, so ``compile_adversary(AdversarySpec()) is
default_policy`` holds exactly like the strategy-side identity.
"""
return compile_strategy(
StrategySpec(
name=spec.name,
target_priority=spec.target_priority,
retreat_hp_fraction=spec.retreat_hp_fraction,
spell_policy=spec.spell_policy,
use_charges=spec.use_charges,
use_class_abilities=spec.use_class_abilities,
)
)
def compile_matchup(
party: StrategySpec, adversary: AdversarySpec, *, party_side: str = "players"
) -> Policy:
"""One ``Policy`` that routes every turn by ``state.side`` (P4.1).
Turns of ``party_side`` delegate to ``compile_strategy(party)``; every
other side delegates to ``compile_adversary(adversary)`` — the engine
invokes ONE policy for all combatants, so side routing lives inside the
policy (the ``phase2_spike.focus_fire_policy`` pattern, generalised to
per-side compiled strategies). When both specs compile to
``default_policy``, the default object itself is returned (identity, no
wrapper). An empty ``party_side`` is a ValueError.
"""
if not party_side:
msg = "party_side must be a non-empty string"
raise ValueError(msg)
party_policy = compile_strategy(party)
adversary_policy = compile_adversary(adversary)
if party_policy is default_policy and adversary_policy is default_policy:
return default_policy
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
if state.side == party_side:
return party_policy(engine, state)
return adversary_policy(engine, state)
return policy
+384
View File
@@ -0,0 +1,384 @@
"""Plausible candidate actions derived from the observation pairs (P0.3).
The engine stays the arbiter: ``candidates`` is a pure function of the
``Observation`` snapshot (no ``engine.*`` access) that proposes a
deterministic menu of plausible actions for the acting combatant. Every
candidate is derived from the per-pair decision helpers (``weapon_for``,
``can_charge``, ``can_step5_to_attack``, ``distance_ft``,
``has_line_of_effect``) and the combatant snapshots (``resources_left``
including ``spell_slot_{level}``, the enriched ``spells`` list). A candidate
that fails a legality gate at execution is rejected with explicit feedback
(``last_turn_results``) — never silently dropped here.
Candidate categories (docs ``phase-2-llm-architecture.md`` P0.3, D4/D5)
-----------------------------------------------------------------------
- ``attack`` / ``full_attack`` for every enemy the acting combatant can
already hit (``weapon_for`` is set).
- ``charge`` for every enemy whose pair says a straight-line charge is
possible (``can_charge``).
- ``5foot_step`` for every enemy one step brings into reach
(``can_step5_to_attack``).
- ``move`` toward every enemy (auto-pathed by the engine); skipped while
grappled (``grappled_with`` — the engine forbids movement then).
- ``cast_spell`` for every known spell with an available slot
(``spell_slot_{level}``, cantrips free), a castable ``cast_time`` (not
``long``), at every enemy it reaches (``distance_ft <= range_ft`` and
``has_line_of_effect`` — the same gate ``_cast_spell`` applies). Harmless
spells (cure, buffs) target the caster: a beneficial spell on a hostile
target would heal or buff the enemy.
- ``channel_energy`` (heal and harm modes) / ``hex`` (evil_eye, slumber per
enemy) / ``deed`` (up_close_and_deadly, quick_clear) when the matching
resource (``channel_energy`` / ``hexes`` / ``grit``) is present.
- ``total_defense`` and ``delay`` (always plausible, no target).
- ``ready`` with each trigger, readied attack on the nearest enemy.
Kinds emitted (D4): ``attack``, ``full_attack``, ``charge``, ``5foot_step``,
``move``, ``cast_spell``, ``channel_energy``, ``hex``, ``deed``,
``total_defense``, ``ready``, ``delay``. ``swift``/``free``/``full_round``
are excluded (``full_attack`` is the supported full-round kind). The menu is
capped at ``MAX_CANDIDATES`` (32), trimmed from the tail of the deterministic
order so the core options always survive.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, TypeGuard
from .combat import Action, Observation
if TYPE_CHECKING:
from .combat import ReadyTrigger
# Ready triggers offered for the ``ready`` candidate (the engine's
# ``ReadyTrigger`` literal).
_READY_TRIGGERS: tuple[ReadyTrigger, ...] = (
"enemy_enters_reach",
"enemy_casts",
"enemy_attacks",
"enemy_hits_ally",
)
# Class-ability resource keys (in ``resources_left``) that gate the
# channel/hex/deed candidates.
_CHANNEL_RESOURCE = "channel_energy"
_HEX_RESOURCE = "hexes"
_DEED_RESOURCE = "grit"
# Candidate kinds the generator may emit (D4). The generic minor kinds
# (``swift``/``free``/``full_round``) are excluded: a policy proposes real
# actions, not placeholders.
_CANDIDATE_KINDS: frozenset[str] = frozenset(
{
"attack",
"full_attack",
"charge",
"5foot_step",
"move",
"cast_spell",
"channel_energy",
"hex",
"deed",
"total_defense",
"ready",
"delay",
}
)
# Deterministic menu order: most decisive first, so the cap trims the least
# useful tail (extra ready triggers, delay) rather than the core options.
_KIND_PRIORITY: dict[str, int] = {
"full_attack": 0,
"attack": 1,
"charge": 2,
"5foot_step": 3,
"move": 4,
"cast_spell": 5,
"channel_energy": 6,
"hex": 7,
"deed": 8,
"total_defense": 9,
"ready": 10,
"delay": 11,
}
# Upper bound on the returned menu: keeps the LLM prompt compact while
# covering every category.
MAX_CANDIDATES = 32
def candidates(observation: Observation) -> tuple[Action, ...]:
"""Deterministic menu of plausible actions for the acting combatant.
Pure read of the snapshot: no engine access, no RNG, no mutation. The
engine stays the arbiter — a candidate that fails a legality gate is
rejected with explicit feedback, never silently dropped here.
"""
combatants = observation.combatants
if not (0 <= observation.acting_combatant < len(combatants)):
return ()
acting = combatants[observation.acting_combatant]
if acting.get("active") is not True:
return ()
acting_side = acting.get("side")
pairs = tuple(
pair
for pair in observation.pairs
if _pair_target_is_enemy(pair, combatants, acting_side)
)
result: list[Action] = []
_attack_candidates(result, pairs, combatants)
_charge_candidates(result, pairs, combatants)
_step_candidates(result, pairs, combatants)
_move_candidates(result, acting, pairs, combatants)
_spell_candidates(result, acting, pairs, combatants)
_class_ability_candidates(result, acting, pairs, combatants)
result.append(Action(kind="total_defense"))
_ready_candidates(result, pairs, combatants)
result.append(Action(kind="delay"))
result.sort(key=_sort_key)
return tuple(result[:MAX_CANDIDATES])
def _attack_candidates(
result: list[Action],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Standard and full-round attacks for every enemy already in reach."""
for pair in pairs:
if pair.get("weapon_for") is not None:
target = _target_id(pair, combatants)
result.append(Action(kind="attack", target_id=target))
result.append(Action(kind="full_attack", target_id=target))
def _charge_candidates(
result: list[Action],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Charge for every enemy with a clear straight-line path."""
result.extend(
Action(kind="charge", target_id=_target_id(pair, combatants))
for pair in pairs
if pair.get("can_charge") is True
)
def _step_candidates(
result: list[Action],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""5-foot step for every enemy one step brings into reach."""
result.extend(
Action(kind="5foot_step", target_id=_target_id(pair, combatants))
for pair in pairs
if pair.get("can_step5_to_attack") is True
)
def _move_candidates(
result: list[Action],
acting: dict[str, object],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Move toward every enemy; skipped while grappled (movement forbidden)."""
if acting.get("grappled_with") is not None:
return
result.extend(
Action(kind="move", target_id=_target_id(pair, combatants)) for pair in pairs
)
def _spell_candidates(
result: list[Action],
acting: dict[str, object],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Cast every known spell with a slot, at every enemy it reaches.
Harmless spells (cure, buffs) target the caster: a beneficial spell on a
hostile target would heal or buff the enemy. Offensive spells are
proposed per enemy the spell reaches.
"""
spells = acting.get("spells")
resources = acting.get("resources_left")
if not _is_tuple(spells) or not _is_str_object_dict(resources):
return
acting_id = _as_str(acting["id"])
for spell in spells:
if not _is_str_object_dict(spell):
continue
if not _spell_has_slot(spell, resources):
continue
if _as_str(spell.get("cast_time", "standard")) == "long":
continue
name = _as_str(spell["name"])
if spell.get("harmless") is True:
result.append(
Action(kind="cast_spell", target_id=acting_id, spell_name=name)
)
continue
result.extend(
Action(
kind="cast_spell",
target_id=_target_id(pair, combatants),
spell_name=name,
)
for pair in pairs
if _spell_reaches_pair(spell, pair)
)
def _class_ability_candidates(
result: list[Action],
acting: dict[str, object],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Channel/hex/deed candidates, each gated on its resource being present."""
resources = acting.get("resources_left")
if not _is_str_object_dict(resources):
return
if _resource_available(resources, _CHANNEL_RESOURCE):
result.append(Action(kind="channel_energy", channel_mode="heal"))
result.append(Action(kind="channel_energy", channel_mode="harm"))
if _resource_available(resources, _HEX_RESOURCE):
for pair in pairs:
target = _target_id(pair, combatants)
result.append(Action(kind="hex", hex_name="evil_eye", target_id=target))
result.append(Action(kind="hex", hex_name="slumber", target_id=target))
if _resource_available(resources, _DEED_RESOURCE):
result.append(Action(kind="deed", deed_name="up_close_and_deadly"))
result.append(Action(kind="deed", deed_name="quick_clear"))
def _ready_candidates(
result: list[Action],
pairs: tuple[dict[str, object], ...],
combatants: tuple[dict[str, object], ...],
) -> None:
"""Ready with each trigger, readied attack on the nearest enemy."""
nearest = _nearest_enemy(pairs, combatants)
if nearest is None:
return
result.extend(
Action(kind="ready", ready_trigger=trigger, target_id=nearest)
for trigger in _READY_TRIGGERS
)
def _sort_key(action: Action) -> tuple[int, str, str, str, str, str, str]:
"""Deterministic ordering of the candidate menu (decisive first)."""
return (
_KIND_PRIORITY.get(action.kind, 99),
action.target_id or "",
action.spell_name or "",
action.hex_name or "",
action.deed_name or "",
action.ready_trigger or "",
action.channel_mode or "",
)
def _target_id(
pair: dict[str, object], combatants: tuple[dict[str, object], ...]
) -> str:
"""Stable id of the pair's target (falls back to the index)."""
index = pair.get("target_index")
if isinstance(index, int) and 0 <= index < len(combatants):
cid = combatants[index].get("id")
if isinstance(cid, str):
return cid
return str(index)
def _pair_target_is_enemy(
pair: dict[str, object],
combatants: tuple[dict[str, object], ...],
acting_side: object,
) -> bool:
"""True when the pair's target is an active combatant on the other side.
The observation's pairs cover every other combatant (allies included),
so the side check is what keeps the menu hostile-only.
"""
index = pair.get("target_index")
if not isinstance(index, int) or not (0 <= index < len(combatants)):
return False
target = combatants[index]
return target.get("active") is True and target.get("side") != acting_side
def _resource_available(resources: dict[str, object], key: str) -> bool:
"""True when the class-ability resource has at least one use left."""
count = resources.get(key)
return isinstance(count, int) and count > 0
def _spell_has_slot(spell: dict[str, object], resources: dict[str, object]) -> bool:
"""True when the caster still has a slot for the spell's level.
Cantrips (level 0) are cast freely — the snapshot has no level-0 slot
key, mirroring ``_has_slot`` in the engine.
"""
level = spell.get("level")
if not isinstance(level, int):
return False
if level == 0:
return True
count = resources.get(f"spell_slot_{level}")
return isinstance(count, int) and count > 0
def _spell_reaches_pair(spell: dict[str, object], pair: dict[str, object]) -> bool:
"""True when the spell's range covers the pair with line of effect.
Mirrors the engine's ``_spell_reaches`` gate: ``distance_ft`` (the pair's
creature distance) within the snapshot's precomputed ``range_ft``, plus
the pair's ``has_line_of_effect``.
"""
range_ft = spell.get("range_ft")
distance_ft = pair.get("distance_ft")
return (
isinstance(range_ft, int)
and isinstance(distance_ft, int)
and distance_ft <= range_ft
and pair.get("has_line_of_effect") is True
)
def _nearest_enemy(
pairs: tuple[dict[str, object], ...], combatants: tuple[dict[str, object], ...]
) -> str | None:
"""Id of the nearest enemy (lowest ``distance_ft``), or None."""
best: str | None = None
best_distance: int | None = None
for pair in pairs:
distance = pair.get("distance_ft")
if not isinstance(distance, int):
continue
if best_distance is None or distance < best_distance:
best_distance = distance
best = _target_id(pair, combatants)
return best
def _is_tuple(value: object) -> TypeGuard[tuple[object, ...]]:
"""Narrow ``object`` to a tuple (snapshot collections are tuples)."""
return isinstance(value, tuple)
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
"""Narrow ``object`` to the snapshot's ``dict[str, object]`` shape."""
return isinstance(value, dict)
def _as_str(value: object) -> str:
if not isinstance(value, str):
msg = f"expected str, got {type(value).__name__}"
raise TypeError(msg)
return value
+536
View File
@@ -0,0 +1,536 @@
"""Provider-agnostic LLM client for the phase-2 decision layer (P1.1/P1.2).
One abstraction for every LLM call the simulator makes, so sweeps, the
compiler and the per-turn showcase all share the same safeguards:
- **Composite cache** (decision D6): a ``ResponseCache`` keyed by
``(model, params, prompt_version, schema_version, prompt hash)``. The
``prompt_version`` (external pack version + per-prompt content hash, see
``prompt.py``) makes a campaign reproducible after a prompt edit (section
14). The cache is sharded (decision D10) and thread-safe, so parallel
sweep workers can share one instance.
- **Cost/token guards** (P1.2, section 13): a ``CostBudget`` enforces
optional ceilings on provider calls, total tokens and estimated USD cost,
raising a typed ``BudgetExceededError`` instead of silently continuing.
The ledger counts every attempt plus tokens/cost of successful completions.
- **Retry/backoff**: a transient ``ProviderError`` is retried with
exponential backoff (injectable ``sleep`` for tests), then a typed
``RetryExhaustedError`` is raised.
- **Circuit breaker** (section 13): after ``failure_threshold`` consecutive
failures the client refuses further calls (``CircuitOpenError``) for
``open_refusals`` calls, then allows one probe attempt; a success closes
the breaker, a failure reopens it.
``LLMClient`` is abstract and provider-agnostic (section 13): subclasses
implement ``_call_once`` (one attempt, returning text + token usage).
``FakeLLMClient`` serves a scripted sequence of responses offline (P1.1),
with optional simulated transient failures - this is what drives P1.3's
fully offline battle tests.
"""
from __future__ import annotations
import hashlib
import time
import zlib
from abc import ABC, abstractmethod
from dataclasses import dataclass
from threading import RLock
from typing import TYPE_CHECKING
from pf1e_simulator.combat import OBSERVATION_SCHEMA_VERSION
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
class ClientError(RuntimeError):
"""Base error for the LLM client layer."""
class ProviderError(ClientError):
"""A transient provider failure - safe to retry."""
class RetryExhaustedError(ClientError):
"""Every retry attempt failed; the last provider error is the cause."""
class CircuitOpenError(ClientError):
"""The circuit breaker is open: calls are refused, not attempted."""
class BudgetExceededError(ClientError):
"""A cost/token/call ceiling was hit (P1.2, section 13)."""
class ScriptExhaustedError(ClientError):
"""A scripted fake client has no more responses to serve."""
@dataclass(frozen=True)
class ClientParams:
"""Sampling parameters; part of the composite cache key (D6)."""
temperature: float = 0.0
max_tokens: int = 1024
top_p: float = 1.0
@dataclass(frozen=True)
class Pricing:
"""USD per 1000 tokens, used to estimate the cost ledger."""
prompt_per_1k_usd: float = 0.0
completion_per_1k_usd: float = 0.0
def cost_usd(self, usage: TokenUsage) -> float:
return (
usage.prompt_tokens / 1000 * self.prompt_per_1k_usd
+ usage.completion_tokens / 1000 * self.completion_per_1k_usd
)
@dataclass(frozen=True)
class CostBudget:
"""Optional hard ceilings enforced by ``LLMClient`` (section 13).
Each field defaults to ``None`` = unlimited for that resource.
"""
max_calls: int | None = None
max_total_tokens: int | None = None
max_cost_usd: float | None = None
@dataclass(frozen=True)
class TokenUsage:
"""Token counts of one completion (measured by the provider, or estimated)."""
prompt_tokens: int
completion_tokens: int
@property
def total(self) -> int:
return self.prompt_tokens + self.completion_tokens
@dataclass(frozen=True)
class Completion:
"""One successful provider response."""
text: str
usage: TokenUsage
@dataclass(frozen=True)
class ClientStats:
"""Read-only snapshot of the client ledger (for reports and sweeps)."""
calls: int
prompt_tokens: int
completion_tokens: int
cost_usd: float
cache_hits: int
cache_misses: int
def estimate_tokens(text: str) -> int:
"""Rough token estimate (chars / 4) used when the provider reports none."""
return max(1, len(text) // 4)
def response_cache_key(
*,
model: str,
params: ClientParams,
prompt_version: str,
schema_version: int,
prompt: str,
) -> str:
"""Composite cache key (D6): model, params, prompt version, schema, hash.
The prompt hash covers the full rendered prompt text, so two different
observations never collide even when they share every other component.
"""
params_component = (
f"t={params.temperature:g};mt={params.max_tokens};tp={params.top_p:g}"
)
digest = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
return (
f"{model}|{params_component}|{prompt_version}"
f"|schema={schema_version}|{digest}"
)
class ResponseCache:
"""Thread-safe, sharded cache of completion texts (D6 + D10).
The key is the composite string from :func:`response_cache_key`; the
value is the completion text. Shards are selected by ``zlib.crc32`` so
parallel sweep workers (D10) never contend on one dict.
"""
def __init__(self, shard_count: int = 8) -> None:
if shard_count < 1:
msg = f"shard_count must be >= 1, got {shard_count}"
raise ValueError(msg)
self._shards: tuple[dict[str, str], ...] = tuple({} for _ in range(shard_count))
self._locks: tuple[RLock, ...] = tuple(RLock() for _ in range(shard_count))
self._stat_lock = RLock()
self._hits = 0
self._misses = 0
def get(self, key: str) -> str | None:
"""Return the cached completion for ``key``, or ``None``."""
shard, lock = self._shard_for(key)
with lock:
value = shard.get(key)
with self._stat_lock:
if value is None:
self._misses += 1
else:
self._hits += 1
return value
def put(self, key: str, text: str) -> None:
"""Store ``text`` under ``key`` (idempotent)."""
shard, lock = self._shard_for(key)
with lock:
shard[key] = text
def clear(self) -> None:
"""Drop every entry and reset the hit/miss counters (tests only)."""
for shard, lock in zip(self._shards, self._locks, strict=True):
with lock:
shard.clear()
with self._stat_lock:
self._hits = 0
self._misses = 0
@property
def hits(self) -> int:
with self._stat_lock:
return self._hits
@property
def misses(self) -> int:
with self._stat_lock:
return self._misses
def _shard_for(self, key: str) -> tuple[dict[str, str], RLock]:
index = zlib.crc32(key.encode("utf-8")) % len(self._shards)
return self._shards[index], self._locks[index]
class CircuitBreaker:
"""Trips after N consecutive failures; probes after K refusals.
Not thread-safe by itself: ``LLMClient`` serializes all access with its
own lock.
"""
def __init__(self, *, failure_threshold: int = 3, open_refusals: int = 2) -> None:
if failure_threshold < 1:
msg = f"failure_threshold must be >= 1, got {failure_threshold}"
raise ValueError(msg)
if open_refusals < 0:
msg = f"open_refusals must be >= 0, got {open_refusals}"
raise ValueError(msg)
self.failure_threshold = failure_threshold
self.open_refusals = open_refusals
self._consecutive_failures = 0
self._refusals = 0
@property
def is_open(self) -> bool:
return self._consecutive_failures >= self.failure_threshold
@property
def probe_due(self) -> bool:
"""True when an open breaker allows exactly one probe attempt."""
return self.is_open and self._refusals >= self.open_refusals
@property
def consecutive_failures(self) -> int:
return self._consecutive_failures
def record_failure(self) -> None:
self._consecutive_failures += 1
self._refusals = 0
def record_refusal(self) -> None:
self._refusals += 1
def close(self) -> None:
self._consecutive_failures = 0
self._refusals = 0
class LLMClient(ABC):
"""Provider-agnostic client: caching, budgets, retry/backoff, breaker.
Subclasses implement :meth:`_call_once` (one provider attempt). The
public :meth:`complete` adds the shared safeguards around it.
"""
def __init__(
self,
*,
model: str,
prompt_version: str,
schema_version: int = OBSERVATION_SCHEMA_VERSION,
cache: ResponseCache | None = None,
budget: CostBudget | None = None,
pricing: Pricing | None = None,
retries: int = 2,
backoff_base_s: float = 0.25,
sleep: Callable[[float], None] = time.sleep,
breaker: CircuitBreaker | None = None,
) -> None:
if retries < 0:
msg = f"retries must be >= 0, got {retries}"
raise ValueError(msg)
if backoff_base_s < 0:
msg = f"backoff_base_s must be >= 0, got {backoff_base_s}"
raise ValueError(msg)
self.model = model
self.prompt_version = prompt_version
self.schema_version = schema_version
# Cache is mandatory (section 13); a client without an explicit one
# gets its own instance, so caching is never silently off.
self._cache = cache if cache is not None else ResponseCache()
self._budget = budget
self._pricing = pricing
self._retries = retries
self._backoff_base_s = backoff_base_s
self._sleep = sleep
self._breaker = breaker if breaker is not None else CircuitBreaker()
self._lock = RLock()
self._calls = 0
self._prompt_tokens = 0
self._completion_tokens = 0
self._cost_usd = 0.0
self._cache_hits = 0
self._cache_misses = 0
@abstractmethod
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
"""One provider attempt; raise ``ProviderError`` on transient failure."""
def complete(
self,
prompt: str,
*,
params: ClientParams | None = None,
) -> str:
"""Complete ``prompt`` through the cache/budget/breaker safeguards.
Returns the completion text (from the cache when possible). Raises a
typed ``ClientError`` subclass on budget, circuit or retry failure.
"""
parameters = params if params is not None else ClientParams()
key = response_cache_key(
model=self.model,
params=parameters,
prompt_version=self.prompt_version,
schema_version=self.schema_version,
prompt=prompt,
)
with self._lock:
self._check_call_budget()
cached = self._cache.get(key)
if cached is not None:
self._cache_hits += 1
return cached
self._cache_misses += 1
if self._breaker.is_open and not self._breaker.probe_due:
self._breaker.record_refusal()
msg = (
f"circuit breaker open "
f"({self._breaker.consecutive_failures} consecutive "
f"failures); call refused"
)
raise CircuitOpenError(msg)
try:
completion = self._complete_with_retry(prompt, parameters)
except RetryExhaustedError:
with self._lock:
self._breaker.record_failure()
raise
with self._lock:
self._record_usage(completion.usage)
self._breaker.close()
self._cache.put(key, completion.text)
return completion.text
def _complete_with_retry(
self,
prompt: str,
params: ClientParams,
) -> Completion:
"""Attempt ``_call_once`` up to ``retries + 1`` times, backing off."""
last_error: ProviderError | None = None
for attempt in range(self._retries + 1):
with self._lock:
self._calls += 1
try:
return self._call_once(prompt, params=params)
except ProviderError as exc:
last_error = exc
if attempt < self._retries:
self._sleep(self._backoff_base_s * (2**attempt))
msg = (
f"provider failed after {self._retries + 1} attempt(s): {last_error}"
)
raise RetryExhaustedError(msg) from last_error
def _check_call_budget(self) -> None:
if self._budget is None or self._budget.max_calls is None:
return
if self._calls >= self._budget.max_calls:
msg = (
f"call budget exceeded: {self._calls} >= "
f"{self._budget.max_calls} calls"
)
raise BudgetExceededError(msg)
def _record_usage(self, usage: TokenUsage) -> None:
"""Add a completion to the ledger, enforcing the token/cost ceilings."""
if self._budget is not None:
ceiling = self._budget.max_total_tokens
if ceiling is not None:
total_after = (
self._prompt_tokens + self._completion_tokens + usage.total
)
if total_after > ceiling:
msg = (
f"token budget exceeded: {total_after} > {ceiling} "
f"total tokens"
)
raise BudgetExceededError(msg)
self._prompt_tokens += usage.prompt_tokens
self._completion_tokens += usage.completion_tokens
if self._pricing is not None:
cost = self._pricing.cost_usd(usage)
self._cost_usd += cost
ceiling = None if self._budget is None else self._budget.max_cost_usd
if ceiling is not None and self._cost_usd > ceiling:
msg = (
f"cost budget exceeded: {self._cost_usd:.6f} USD > "
f"{ceiling:.6f} USD"
)
raise BudgetExceededError(msg)
@property
def calls(self) -> int:
with self._lock:
return self._calls
@property
def prompt_tokens(self) -> int:
with self._lock:
return self._prompt_tokens
@property
def completion_tokens(self) -> int:
with self._lock:
return self._completion_tokens
@property
def cost_usd(self) -> float:
with self._lock:
return self._cost_usd
@property
def cache_hits(self) -> int:
with self._lock:
return self._cache_hits
@property
def cache_misses(self) -> int:
with self._lock:
return self._cache_misses
@property
def stats(self) -> ClientStats:
with self._lock:
return ClientStats(
calls=self._calls,
prompt_tokens=self._prompt_tokens,
completion_tokens=self._completion_tokens,
cost_usd=self._cost_usd,
cache_hits=self._cache_hits,
cache_misses=self._cache_misses,
)
class FakeLLMClient(LLMClient):
"""Scripted, offline LLM client (P1.1) for tests and the showcase.
Serves ``script`` in order, one response per successful call. With
``failures > 0`` the first ``failures`` attempts raise a simulated
``ProviderError`` (to exercise retry/backoff and the breaker); scripts
are only consumed by successes. A call past the script raises
``ScriptExhaustedError`` (a test-authoring bug, not a transient outage).
Token usage is estimated from character counts.
"""
def __init__(
self,
script: Sequence[str],
*,
model: str = "fake",
prompt_version: str = "fixture:fake:0",
failures: int = 0,
schema_version: int = OBSERVATION_SCHEMA_VERSION,
cache: ResponseCache | None = None,
budget: CostBudget | None = None,
pricing: Pricing | None = None,
retries: int = 2,
backoff_base_s: float = 0.25,
breaker: CircuitBreaker | None = None,
) -> None:
if failures < 0:
msg = f"failures must be >= 0, got {failures}"
raise ValueError(msg)
super().__init__(
model=model,
prompt_version=prompt_version,
schema_version=schema_version,
cache=cache,
budget=budget,
pricing=pricing,
retries=retries,
backoff_base_s=backoff_base_s,
sleep=lambda _: None, # tests must never actually sleep
breaker=breaker,
)
self._script = tuple(script)
self._failures = failures
self._attempts = 0
self._served = 0
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
del params # a scripted fake ignores sampling parameters
with self._lock:
self._attempts += 1
if self._attempts <= self._failures:
msg = f"simulated provider outage (attempt {self._attempts})"
raise ProviderError(msg)
if self._served >= len(self._script):
msg = (
f"fake script exhausted after {self._served} served "
f"response(s) of {len(self._script)}"
)
raise ScriptExhaustedError(msg)
text = self._script[self._served]
self._served += 1
usage = TokenUsage(
prompt_tokens=estimate_tokens(prompt),
completion_tokens=estimate_tokens(text),
)
return Completion(text=text, usage=usage)
+65 -18
View File
@@ -121,6 +121,7 @@ Phase 0 documented deviations from PF1e (conventions):
from __future__ import annotations
import inspect
from collections.abc import Callable, Iterable
from dataclasses import asdict, dataclass, field, replace
from itertools import pairwise
@@ -177,19 +178,21 @@ if TYPE_CHECKING:
from pf1e_simulator.rng import Rng
from pf1e_simulator.spells import SpellSpec
TrickCondition = Literal[
"dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"
]
ReadyTrigger = Literal[
"enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"
]
from pf1e_simulator.los import (
cover_bonus_cells,
has_line_of_effect_cells,
)
# Runtime aliases (not only annotations): Pydantic type adapters over ``Action``
# must resolve them, so they cannot live inside the TYPE_CHECKING block.
TrickCondition = Literal[
"dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"
]
ReadyTrigger = Literal[
"enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"
]
_SQUARE_FT = 5 # Phase 0 maps use 5-ft squares
_NATURAL_ONE = 1 # PF1e: natural 1 always misses
_NATURAL_TWENTY = 20 # PF1e: natural 20 always hits and threatens
@@ -1259,6 +1262,12 @@ class ChargeResult:
reason: str | None
# Observation schema version (P0.1, docs/phase-2-llm-architecture.md section
# 10). Bump only when a top-level key is removed or its meaning changes;
# additive keys keep the version.
OBSERVATION_SCHEMA_VERSION = 1
@dataclass(frozen=True)
class Observation:
"""Serializable snapshot of combat state for a policy/LLM.
@@ -1269,7 +1278,9 @@ class Observation:
collections are copies (tuples/dicts), never read back by the engine, so a
policy may mutate them freely without touching combat; the snapshot is
JSON-serializable (the top level is a frozen dataclass; the inner dicts are
plain mutable copies, not deep-frozen).
plain mutable copies, not deep-frozen). ``to_dict()`` carries
``schema_version`` (``OBSERVATION_SCHEMA_VERSION``) as its first key so a
consumer can detect breaking schema changes.
``combatants`` holds one dict per combatant in ``self._states`` order;
``pairs`` holds one dict per (acting combatant x each other combatant) with
@@ -1310,6 +1321,7 @@ class Observation:
def to_dict(self) -> dict[str, object]:
"""Return a JSON-serializable dict with the pinned top-level keys."""
return {
"schema_version": OBSERVATION_SCHEMA_VERSION,
"round": self.round,
"acting_combatant": self.acting_combatant,
"combatants": self.combatants,
@@ -1514,7 +1526,7 @@ _REASON_CANNOT_COMBINE = "cannot_combine"
# Action kinds that are declared but have no engine dispatch (U1 surfaces
# them as explicit rejections instead of silent no-ops).
_UNSUPPORTED_ACTION_KINDS: frozenset[str] = frozenset({"swift", "free", "full_round"})
UNSUPPORTED_ACTION_KINDS: frozenset[str] = frozenset({"swift", "free", "full_round"})
# Action kinds whose helper is a silent no-op (or worse) without a resolved
# target; _execute rejects them up front with missing_target instead.
# move / 5foot_step / run are exempt when an explicit destination is provided
@@ -1613,7 +1625,7 @@ class CombatEngine:
*,
round_cap: int = 100,
policy: Policy | None = None,
llm_policy: LLMPolicy | None = None,
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
spell_registry: dict[str, SpellSpec] | None = None,
reaction_policy: ReactionPolicy | None = None,
monsters_die_at_zero: bool = False,
@@ -1685,18 +1697,24 @@ class CombatEngine:
if self._monsters_die_at_zero and not _is_pc_combatant(state.combatant):
state.dies_at_zero = True
def _llm_adapter(self, llm_policy: LLMPolicy) -> Policy:
"""Wrap an ``LLMPolicy`` into a ``Policy``.
def _llm_adapter(self, llm_policy: LLMPolicy | LegacyLLMPolicy) -> Policy:
"""Wrap an ``LLMPolicy`` (or a pre-D1 two-argument callable) into a ``Policy``.
At the point of invocation (``actions = self._policy(self, state)`` in
``_take_turn``), the Observation is built for the current state and
delegated to ``llm_policy(observation, state)``. The adapter consumes
no RNG and mutates no state.
delegated. New-style policies get ``llm_policy(observation)`` only;
a legacy ``llm_policy(observation, state)`` callable is detected ONCE
here (signature inspection, not per turn) and keeps receiving the live
state. The adapter consumes no RNG and mutates no state.
"""
two_arg = _accepts_state(llm_policy)
call: Callable[..., tuple[Action, ...]] = llm_policy
def adapter(_engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
observation = self._build_observation(state)
return llm_policy(observation, state)
if two_arg:
return call(observation, state)
return call(observation)
return adapter
@@ -5079,7 +5097,7 @@ class CombatEngine:
reports ok.
"""
kind = action.kind
if kind in _UNSUPPORTED_ACTION_KINDS:
if kind in UNSUPPORTED_ACTION_KINDS:
self._log(
f"round {self._current_round} {state.combatant.id}: "
f"{kind} rejected ({_REASON_UNSUPPORTED_KIND})"
@@ -10572,7 +10590,36 @@ def _ready_payload(action: Action) -> Action:
Policy = Callable[[CombatEngine, CombatantState], tuple[Action, ...]]
LLMPolicy = Callable[[Observation, CombatantState], tuple[Action, ...]]
# D1 (docs/phase-2-llm-architecture.md section 13): the LLM policy entry is
# observation-only -- the snapshot is the policy's sole information source and
# the live CombatantState is never handed to a new-style policy. Pre-D1
# two-argument callables remain accepted via CombatEngine._llm_adapter.
LLMPolicy = Callable[[Observation], tuple[Action, ...]]
LegacyLLMPolicy = Callable[[Observation, CombatantState], tuple[Action, ...]]
_LEGACY_LLM_POLICY_ARITY = 2
def _accepts_state(llm_policy: LLMPolicy | LegacyLLMPolicy) -> bool:
"""True when ``llm_policy`` is a pre-D1 callable taking (observation, state).
Counts REQUIRED positional parameters (a defaulted second parameter is a
new-style policy that tolerates the 1-argument call). Callables whose
signature cannot be inspected default to the new-style 1-argument call.
"""
try:
parameters = list(inspect.signature(llm_policy).parameters.values())
except (TypeError, ValueError):
return False
required_positional = [
p
for p in parameters
if p.default is inspect.Parameter.empty
and p.kind
in (inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)
]
return len(required_positional) >= _LEGACY_LLM_POLICY_ARITY
ReactionPolicy = Callable[
[CombatEngine, CombatantState, ReactionEvent], Action | None
+62
View File
@@ -0,0 +1,62 @@
"""Environment-driven configuration for the phase-2 LLM layer (P1/P5.1).
Architecture doc section 15: the phase-2 code never hard-codes data or
configuration paths -- every externally supplied value is resolved here from
the environment, so the repository can keep shipping pure code once the data
sets and the prompt pack detach. The prompt-pack contract is documented in
section 14 (see :mod:`pf1e_simulator.prompt`).
"""
from __future__ import annotations
import os
from pathlib import Path
# Environment variable names (documented in the architecture doc, sections
# 14-15). ``PF1E_PROMPT_PACK_DIR`` points at the external, versioned prompt
# pack; ``PF1E_LLM_API_KEY`` feeds the remote provider client (P5.1);
# ``PF1E_LLM_BASE_URL`` and ``PF1E_LLM_MODEL`` select the provider endpoint
# and model, so a self-hosted OpenAI-compatible gateway (vLLM, llama.cpp...)
# is configured without touching the code.
ENV_PROMPT_PACK_DIR = "PF1E_PROMPT_PACK_DIR"
ENV_LLM_API_KEY = "PF1E_LLM_API_KEY"
ENV_LLM_BASE_URL = "PF1E_LLM_BASE_URL"
ENV_LLM_MODEL = "PF1E_LLM_MODEL"
# Sensible offline defaults: the OpenAI-compatible chat endpoint and a small,
# cheap model. Nothing here is a secret -- the API key stays environment-only.
DEFAULT_LLM_BASE_URL = "https://api.openai.com/v1"
DEFAULT_LLM_MODEL = "gpt-4o-mini"
def env_str(name: str, *, default: str | None = None) -> str | None:
"""Read a configuration string from the environment, if set."""
return os.environ.get(name, default)
def env_path(name: str) -> Path | None:
"""Read a configuration path from the environment, if set."""
raw = os.environ.get(name)
return Path(raw) if raw else None
def llm_api_key() -> str | None:
"""The provider API key, or None when the caller runs offline."""
return env_str(ENV_LLM_API_KEY)
def llm_base_url() -> str:
"""The OpenAI-compatible base URL (``PF1E_LLM_BASE_URL``, else default).
The returned value has no trailing slash; ``OpenAILLMClient`` appends
``/chat/completions`` to it (P5.1).
"""
raw = env_str(ENV_LLM_BASE_URL)
return raw.rstrip("/") if raw else DEFAULT_LLM_BASE_URL
def llm_model() -> str:
"""The provider model name (``PF1E_LLM_MODEL``, else default)."""
raw = env_str(ENV_LLM_MODEL)
return raw or DEFAULT_LLM_MODEL
+545
View File
@@ -0,0 +1,545 @@
"""Compact, decision-sufficient observation view for LLM policies (P0.2).
The raw ``Observation.to_dict()`` snapshot is ~64 KB (~16k tokens) for a
7-combatant encounter - too large to feed an LLM every turn (docs
``phase-2-llm-architecture.md`` section 16, target ~2-4k tokens). This module
builds a compact ``ObservationView`` that keeps every tactically relevant
fact while dropping duplicates, derivables and verbose expansions, and
renders it as a deterministic, line-based prompt string.
Compaction rules
----------------
- ``terrain``: sparse list of ``(row, col, type)`` for non-default cells
only. A cell is default when its semantics are ``move_cost == 1``,
``blocks_los == False``, ``cover == False`` and ``cover_bonus == 0``.
- ``elevation``: sparse list of ``(row, col, height)`` for non-zero cells.
- ``threatened`` / ``ranged_threatened``: sparse list of ``(row, col)`` for
True cells (empty when no cell is covered).
- per-combatant: only decision-relevant keys are kept (see
``_COMPACT_COMBATANT_KEYS``). Dropped: ``occupied_cells`` (derivable from
``pos`` + ``size``), ``abilities``/``skills`` (raw scores, not used by the
candidate generator), ``initiative_mod``/``initiative_total`` (the
top-level order suffices), ``temp_hp_rounds_left`` (``temp_hp`` is the
current value), ``features``/``feature_details`` (the resources and
``sneak_dice`` already expose the class abilities), ``carried_items``,
``weapon_damage``, ``aware``, ``alignment``, ``armor_spell_failure``,
``arcane_caster`` (all zero in the data) and ``run_rounds``.
- attacks: compacted to ``{id, name, kind, attack_bonus, damage (strings
like "1d6 slashing"), damage_bonus, crit_range, crit_mult, reach_ft,
range_increment_ft, count, firearm}`` - sunder-only object stats
(``hardness``/``hit_points``) and ``splash`` are dropped.
- spells: compacted to ``{name, level, range_ft, save, dc, cast_time,
area}`` - ``summary`` (the bulk of the raw size), ``school``,
``half_on_save``, ``save_negates``, ``harmless`` and ``components`` are
dropped (no component failure in the engine's data).
- per-pair: all decision helpers are kept unchanged, except ``target_index``
is replaced by the stable combatant ``target`` id.
- ``acting_combatant`` is rendered as the combatant id (not the index).
- ``condition_definitions`` is dropped (condition names suffice).
- every collection is deterministically ordered (fixed key order, row-major
or the raw snapshot's own sorted order).
Render format
-------------
The render is line-based ``key=value`` text. The nested ``attacks`` and
``spells`` expansions are rendered as positional tuples (the field order is
declared once in the ``attack_fields`` / ``spell_fields`` legend lines) with
``-`` standing for a missing value (``None`` save/dc/area, no range
increment); this is what brings the view under the ~4k-token ceiling.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import TypeGuard
from .combat import OBSERVATION_SCHEMA_VERSION, Observation
# Per-combatant keys kept in the compact view, in deterministic order.
# Everything else in the raw snapshot is derivable, verbose, or not
# decision-relevant for the candidate generator and the LLM.
_COMPACT_COMBATANT_KEYS: tuple[str, ...] = (
"id",
"side",
"pos",
"hp",
"hp_max",
"nonlethal",
"active",
"dead",
"dying",
"stabilized",
"conditions",
"resources_left",
"weapon_broken",
"weapon_destroyed",
"weapon_stolen",
"weapon_dropped",
"grappled_with",
"hidden_from",
"moved_this_turn",
"power_attack",
"defensive_ac_bonus",
"fight_defensively",
"aid_attack_bonus",
"aid_ac_bonus",
"running",
"run_rest_rounds",
"bleed_damage",
"recurring_damage",
"aoo_remaining",
"name",
"level",
"size",
"natural_reach_ft",
"creature_type",
"bab",
"casting_ability",
"dr",
"metamagic_feats",
"summon_duration",
"immediate_used",
"deed_up_close_and_deadly",
"active_buffs",
"one_shot_bonuses",
"temp_hp",
"speed_land_ft",
"speed_fly_ft",
"speed_effective_ft",
"ac",
"saves",
"cmb",
"cmd",
"attacks",
"spells",
"spell_resistance",
"resistances",
"immunities",
"sneak_dice",
"engaged_in_melee",
"last_turn_results",
"readied",
"pending_cast",
)
# Per-pair keys kept (all decision helpers), in the raw snapshot's order.
# ``target_index`` is replaced by the stable combatant ``target`` id.
_PAIR_KEYS: tuple[str, ...] = (
"distance_ft",
"weapon_for",
"weapon_reason",
"range_penalty",
"can_charge",
"charge_reason",
"can_step5_to_attack",
"has_line_of_effect",
"flanking",
"movement_cost_ft",
"target_threatens_me",
"enemies_threatening_me",
"cover_bonus",
)
# Attack keys kept in the compact expansion (sunder-only object stats and
# the splash marker are dropped).
_ATTACK_KEYS: tuple[str, ...] = (
"id",
"name",
"kind",
"attack_bonus",
"damage",
"damage_bonus",
"crit_range",
"crit_mult",
"reach_ft",
"range_increment_ft",
"count",
"firearm",
)
# Spell keys kept in the compact expansion (the summary is the bulk of the
# raw size; school/components/save flags are not decision-relevant).
_SPELL_KEYS: tuple[str, ...] = (
"name",
"level",
"range_ft",
"save",
"dc",
"cast_time",
"area",
)
@dataclass(frozen=True)
class ObservationView:
"""Compact, decision-sufficient view of one combat observation.
All collections are deterministically ordered; ``terrain``/``elevation``/
``threatened``/``ranged_threatened`` are sparse (see module docstring).
``acting_combatant`` is the acting combatant's id.
"""
schema_version: int
round: int
acting_combatant: str
dimensions: tuple[int, int]
terrain: tuple[tuple[int, int, str], ...]
elevation: tuple[tuple[int, int, int], ...]
threatened: tuple[tuple[int, int], ...]
ranged_threatened: tuple[tuple[int, int], ...]
terrain_semantics: tuple[dict[str, object], ...]
initiative_order: tuple[str, ...]
combatants: tuple[dict[str, object], ...]
pairs: tuple[dict[str, object], ...]
def build_view(observation: Observation) -> ObservationView:
"""Build a compact ``ObservationView`` from a raw ``Observation``."""
height = len(observation.terrain)
width = len(observation.terrain[0]) if height else 0
semantics_by_type = {
str(sem["type"]): sem for sem in observation.terrain_semantics
}
terrain = tuple(
(r, c, cell_type)
for r, row in enumerate(observation.terrain)
for c, cell_type in enumerate(row)
if cell_type is not None
and not _is_default_terrain(cell_type, semantics_by_type)
)
elevation = tuple(
(r, c, height_at)
for r, row in enumerate(observation.elevation)
for c, height_at in enumerate(row)
if height_at != 0
)
combatants = tuple(
_compact_combatant(combatant) for combatant in observation.combatants
)
return ObservationView(
schema_version=OBSERVATION_SCHEMA_VERSION,
round=observation.round,
acting_combatant=_combatant_id(combatants, observation.acting_combatant),
dimensions=(height, width),
terrain=terrain,
elevation=elevation,
threatened=_sparse_bool_grid(observation.threatened),
ranged_threatened=_sparse_bool_grid(observation.ranged_threatened),
terrain_semantics=observation.terrain_semantics,
initiative_order=observation.initiative_order,
combatants=combatants,
pairs=tuple(_compact_pair(pair, combatants) for pair in observation.pairs),
)
def render(view: ObservationView) -> str:
"""Render a compact view as a deterministic, line-based prompt string."""
lines: list[str] = [
f"schema_version={view.schema_version}",
f"round={view.round}",
f"acting={view.acting_combatant}",
f"map={view.dimensions[0]}x{view.dimensions[1]}",
f"terrain={_fmt_cells(view.terrain)}",
f"elevation={_fmt_cells(view.elevation)}",
f"threatened={_fmt_cells2(view.threatened)}",
f"ranged_threatened={_fmt_cells2(view.ranged_threatened)}",
f"terrain_semantics={_fmt_semantics(view.terrain_semantics)}",
f"initiative={','.join(view.initiative_order)}",
f"attack_fields={','.join(_ATTACK_KEYS)}",
f"spell_fields={','.join(_SPELL_KEYS)}",
"combatants:",
]
for index, combatant in enumerate(view.combatants):
lines.append(f" {index} " + _fmt_combatant(combatant))
lines.append("pairs:")
lines.extend(" " + _fmt_kv(pair) for pair in view.pairs)
return "\n".join(lines) + "\n"
def compact_observation(observation: Observation) -> str:
"""One-shot: build the compact view of an observation and render it."""
return render(build_view(observation))
def _is_default_terrain(
cell_type: str,
semantics_by_type: dict[str, dict[str, object]],
) -> bool:
"""True when a cell carries no tactical information worth listing."""
sem = semantics_by_type.get(cell_type)
if sem is None:
return False
return (
sem.get("move_cost") == 1
and not sem.get("blocks_los")
and not sem.get("cover")
and sem.get("cover_bonus") == 0
)
def _sparse_bool_grid(
grid: tuple[tuple[bool, ...], ...],
) -> tuple[tuple[int, int], ...]:
"""Sparse ``(row, col)`` list of True cells, row-major."""
return tuple(
(r, c)
for r, row in enumerate(grid)
for c, value in enumerate(row)
if value
)
def _compact_combatant(combatant: dict[str, object]) -> dict[str, object]:
"""Select the decision-relevant keys and compact nested expansions."""
compact = {
key: combatant[key] for key in _COMPACT_COMBATANT_KEYS if key in combatant
}
attacks = compact.get("attacks")
if _is_tuple(attacks):
compact["attacks"] = tuple(
_compact_attack(attack) for attack in attacks if _is_str_object_dict(attack)
)
spells = compact.get("spells")
if _is_tuple(spells):
compact["spells"] = tuple(
_compact_spell(spell) for spell in spells if _is_str_object_dict(spell)
)
return compact
def _compact_attack(attack: dict[str, object]) -> dict[str, object]:
"""Compact one attack: damage becomes formula+type strings."""
return {
"id": _as_str(attack["id"]),
"name": _as_str(attack["name"]),
"kind": _as_str(attack["kind"]),
"attack_bonus": _as_int(attack["attack_bonus"]),
"damage": _damage_strings(attack.get("damage")),
"damage_bonus": _as_int(attack["damage_bonus"]),
"crit_range": _as_int(attack["crit_range"]),
"crit_mult": _as_int(attack["crit_mult"]),
"reach_ft": _as_int(attack["reach_ft"]),
"range_increment_ft": _as_optional_int(attack["range_increment_ft"]),
"count": _as_int(attack["count"]),
"firearm": _as_bool(attack["firearm"]),
}
def _compact_spell(spell: dict[str, object]) -> dict[str, object]:
"""Compact one spell: keep only the decision-relevant fields."""
return {
"name": _as_str(spell["name"]),
"level": _as_optional_int(spell["level"]),
"range_ft": _as_int(spell["range_ft"]),
"save": _as_str(spell["save"]),
"dc": _as_optional_int(spell["dc"]),
"cast_time": _as_str(spell["cast_time"]),
"area": spell.get("area"),
}
def _compact_pair(
pair: dict[str, object],
combatants: tuple[dict[str, object], ...],
) -> dict[str, object]:
"""Select the decision-helper keys of one pair, in fixed order."""
compact = {key: pair[key] for key in _PAIR_KEYS if key in pair}
compact["target"] = _combatant_id(combatants, _as_int(pair["target_index"]))
return compact
def _combatant_id(combatants: tuple[dict[str, object], ...], index: int) -> str:
"""Stable id of the combatant at ``index`` (falls back to the index)."""
if 0 <= index < len(combatants):
cid = combatants[index].get("id")
if isinstance(cid, str):
return cid
return str(index)
def _damage_strings(value: object) -> tuple[str, ...]:
"""Render a damage component list as ``"1d6 slashing"`` strings."""
if not _is_tuple(value):
return ()
strings: list[str] = []
for component in value:
if not _is_str_object_dict(component):
continue
formula = component.get("formula")
if not isinstance(formula, str):
continue
types = component.get("types")
if _is_tuple(types):
parts = [formula, *(t for t in types if isinstance(t, str))]
else:
parts = [formula]
strings.append(" ".join(parts))
return tuple(strings)
def _fmt_combatant(combatant: dict[str, object]) -> str:
"""Render one compact combatant, attacks/spells positionally."""
parts: list[str] = []
for key, value in combatant.items():
if key == "attacks":
if _is_tuple(value):
parts.append(
"attacks=("
+ ",".join(
_fmt_attack(attack)
for attack in value
if _is_str_object_dict(attack)
)
+ ")"
)
continue
if key == "spells":
if _is_tuple(value):
parts.append(
"spells=("
+ ",".join(
_fmt_spell(spell)
for spell in value
if _is_str_object_dict(spell)
)
+ ")"
)
continue
parts.append(f"{key}={_fmt_value(value)}")
return " ".join(parts)
def _fmt_attack(attack: dict[str, object]) -> str:
"""Positional attack tuple; ``-`` for a missing range increment."""
damage = attack.get("damage")
damage_str = (
"+".join(d for d in damage if isinstance(d, str)) if _is_tuple(damage) else ""
)
if not damage_str:
damage_str = "-"
range_increment = _as_optional_int(attack["range_increment_ft"])
range_str = "-" if range_increment is None else str(range_increment)
return (
f"({_as_str(attack['id'])},{_as_str(attack['name'])},"
f"{_as_str(attack['kind'])},{_as_int(attack['attack_bonus'])},"
f"{damage_str},{_as_int(attack['damage_bonus'])},"
f"{_as_int(attack['crit_range'])},{_as_int(attack['crit_mult'])},"
f"{_as_int(attack['reach_ft'])},{range_str},"
f"{_as_int(attack['count'])},{1 if _as_bool(attack['firearm']) else 0})"
)
def _fmt_spell(spell: dict[str, object]) -> str:
"""Positional spell tuple; ``-`` for a missing save/dc/area."""
save = _as_str(spell["save"])
save_str = "-" if save == "none" else save
dc = _as_optional_int(spell["dc"])
dc_str = "-" if dc is None else str(dc)
area = spell.get("area")
if area is None:
area_str = "-"
elif _is_str_object_dict(area):
shape = _as_str(area["shape"])
radius = _as_int(area["radius_ft"])
centered = _as_bool(area["centered_on_caster"])
area_str = f"{shape}:{radius}" + (":c" if centered else "")
else:
area_str = "-"
level = _as_optional_int(spell["level"])
level_str = "-" if level is None else str(level)
return (
f"({_as_str(spell['name'])},{level_str},"
f"{_as_int(spell['range_ft'])},{save_str},{dc_str},"
f"{_as_str(spell['cast_time'])},{area_str})"
)
def _fmt_cells(cells: tuple[tuple[int, int, object], ...]) -> str:
return "".join(f"({r},{c},{_fmt_value(v)})" for r, c, v in cells)
def _fmt_cells2(cells: tuple[tuple[int, int], ...]) -> str:
return "".join(f"({r},{c})" for r, c in cells)
def _fmt_semantics(semantics: tuple[dict[str, object], ...]) -> str:
parts: list[str] = []
for sem in semantics:
name = str(sem.get("type", "?"))
move_cost = sem.get("move_cost")
blocks_los = sem.get("blocks_los")
cover = sem.get("cover")
cover_bonus = sem.get("cover_bonus")
parts.append(
f"{name}{{mc={move_cost},los={1 if blocks_los else 0},"
f"cov={1 if cover else 0},cb={cover_bonus}}}"
)
return ",".join(parts)
def _fmt_kv(mapping: dict[str, object]) -> str:
return " ".join(f"{key}={_fmt_value(value)}" for key, value in mapping.items())
def _fmt_value(value: object) -> str:
"""Compact, unambiguous rendering of one snapshot value.
``None`` falls through to ``str(value)`` (already ``"None"``).
"""
if isinstance(value, bool):
return "1" if value else "0"
if isinstance(value, (int, float)):
return str(value)
if isinstance(value, str):
return (
value
if value and all(c.isalnum() or c in "_-" for c in value)
else json.dumps(value)
)
if _is_tuple(value):
return "(" + ",".join(_fmt_value(item) for item in value) + ")"
if _is_str_object_dict(value):
return (
"{"
+ ",".join(f"{key}:{_fmt_value(item)}" for key, item in value.items())
+ "}"
)
return str(value)
def _as_str(value: object) -> str:
if not isinstance(value, str):
msg = f"expected str, got {type(value).__name__}"
raise TypeError(msg)
return value
def _as_int(value: object) -> int:
if not isinstance(value, int):
msg = f"expected int, got {type(value).__name__}"
raise TypeError(msg)
return value
def _as_bool(value: object) -> bool:
if not isinstance(value, bool):
msg = f"expected bool, got {type(value).__name__}"
raise TypeError(msg)
return value
def _as_optional_int(value: object) -> int | None:
if value is None:
return None
return _as_int(value)
def _is_tuple(value: object) -> TypeGuard[tuple[object, ...]]:
"""Narrow ``object`` to a tuple (snapshot collections are tuples)."""
return isinstance(value, tuple)
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
"""Narrow ``object`` to the snapshot's ``dict[str, object]`` shape."""
return isinstance(value, dict)
+313
View File
@@ -0,0 +1,313 @@
"""Natural-language policy (P1.3): parses LLM JSON replies into Action sequences.
P5.2 adds a bounded repair loop: on a parse failure the policy re-asks the
model with an accumulated error trail up to ``max_repairs`` times, then
falls back to the scripted policy; ``NLPolicyStats`` exposes the call and
repair counters for the sweep and the phase-3 report.
"""
from __future__ import annotations
import json
from collections.abc import Callable
from dataclasses import dataclass, field, fields
from typing import TYPE_CHECKING, Literal, TypeGuard, get_args
from pf1e_simulator.candidates import candidates
from pf1e_simulator.combat import Action, ReadyTrigger, TrickCondition
if TYPE_CHECKING:
from pf1e_simulator.client import LLMClient
from pf1e_simulator.combat import Observation
from pf1e_simulator.prompt_builder import PromptBuilder
# ---- Type aliases for the 27 supported kinds (filtered from Action.kind) ----
KindValue = Literal[
"attack", "move", "wait", "immediate", "full_attack", "charge", "withdraw",
"5foot_step", "cast_spell", "maneuver", "channel_energy", "hex", "deed",
"spell_combat", "inspire_courage", "retrieve_weapon", "repair_weapon",
"activate_power_attack", "delay", "ready", "total_defense",
"fight_defensively", "stand_up", "drop_prone", "aid_another", "run", "hide",
]
_KIND_VALUES: frozenset[str] = frozenset(get_args(KindValue))
_FIELD_NAMES: frozenset[str] = frozenset(f.name for f in fields(Action))
_PAIR_LEN = 2
_CHANNEL_MODES: frozenset[str] = frozenset(get_args(Literal["heal", "harm"]))
_AID_MODES: frozenset[str] = frozenset(get_args(Literal["attack", "ac"]))
_TRICK_CONDITIONS: frozenset[str] = frozenset(get_args(TrickCondition))
_READY_TRIGGERS: frozenset[str] = frozenset(get_args(ReadyTrigger))
class NLPolicyError(ValueError):
"""The LLM reply is not a valid JSON action sequence."""
# ---- Type guards -------------------------------------------------------------
def _is_kind(value: object) -> TypeGuard[KindValue]:
return value in _KIND_VALUES
def _is_channel_mode(value: object) -> TypeGuard[Literal["heal", "harm"]]:
return value in _CHANNEL_MODES
def _is_aid_mode(value: object) -> TypeGuard[Literal["attack", "ac"]]:
return value in _AID_MODES
def _is_trick_condition(value: object) -> TypeGuard[TrickCondition]:
return value in _TRICK_CONDITIONS
def _is_ready_trigger(value: object) -> TypeGuard[ReadyTrigger]:
return value in _READY_TRIGGERS
def _is_object_list(value: object) -> TypeGuard[list[object]]:
return isinstance(value, list)
def _is_object_seq(value: object) -> TypeGuard[list[object] | tuple[object, ...]]:
return isinstance(value, (list, tuple))
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
return isinstance(value, dict)
# ---- Coercion helpers --------------------------------------------------------
def _opt_str(value: object, name: str) -> str | None:
if value is None:
return None
if isinstance(value, str):
return value
msg = f"{name!r} must be a string"
raise NLPolicyError(msg)
def _str_tuple(value: object, name: str) -> tuple[str, ...]:
if value is None:
return ()
if not _is_object_seq(value):
msg = f"{name!r} must be a list of strings"
raise NLPolicyError(msg)
out: list[str] = []
for item in value:
if not isinstance(item, str):
msg = f"{name!r} must be a list of strings"
raise NLPolicyError(msg)
out.append(item)
return tuple(out)
def _int_pair(value: object, name: str) -> tuple[int, int] | None:
if value is None:
return None
if not _is_object_seq(value):
msg = f"{name!r} must be a [row, col] pair"
raise NLPolicyError(msg)
if len(value) != _PAIR_LEN:
msg = f"{name!r} must be a [row, col] pair"
raise NLPolicyError(msg)
first, second = value[0], value[1]
if (
isinstance(first, bool)
or isinstance(second, bool)
or not isinstance(first, int)
or not isinstance(second, int)
):
msg = f"{name!r} must be a [row, col] pair"
raise NLPolicyError(msg)
return (first, second)
def _opt_bool(value: object, name: str) -> bool:
if value is None:
return False
if isinstance(value, bool):
return value
msg = f"{name!r} must be a boolean"
raise NLPolicyError(msg)
def _opt_channel_mode(value: object) -> Literal["heal", "harm"] | None:
if value is None:
return None
if _is_channel_mode(value):
return value
msg = "'channel_mode' must be one of: heal, harm"
raise NLPolicyError(msg)
def _opt_aid_mode(value: object) -> Literal["attack", "ac"] | None:
if value is None:
return None
if _is_aid_mode(value):
return value
msg = "'aid_mode' must be one of: attack, ac"
raise NLPolicyError(msg)
def _opt_trick_condition(value: object) -> TrickCondition | None:
if value is None:
return None
if _is_trick_condition(value):
return value
msg = "'trick_condition' must be one of: " + ", ".join(get_args(TrickCondition))
raise NLPolicyError(msg)
def _opt_ready_trigger(value: object) -> ReadyTrigger | None:
if value is None:
return None
if _is_ready_trigger(value):
return value
msg = "'ready_trigger' must be one of: " + ", ".join(get_args(ReadyTrigger))
raise NLPolicyError(msg)
# ---- Action construction -----------------------------------------------------
def _action_from_item(item: object) -> Action:
if not _is_object_dict(item):
msg = "each action must be a JSON object"
raise NLPolicyError(msg)
fields_dict = item
unknown = sorted(set(fields_dict) - _FIELD_NAMES)
if unknown:
msg = f"unknown action field(s): {', '.join(unknown)}"
raise NLPolicyError(msg)
kind = fields_dict.get("kind")
if not _is_kind(kind):
msg = "missing or unknown 'kind'"
raise NLPolicyError(msg)
return Action(
kind=kind,
target_id=_opt_str(fields_dict.get("target_id"), "target_id"),
spell_name=_opt_str(fields_dict.get("spell_name"), "spell_name"),
maneuver=_opt_str(fields_dict.get("maneuver"), "maneuver"),
hex_name=_opt_str(fields_dict.get("hex_name"), "hex_name"),
deed_name=_opt_str(fields_dict.get("deed_name"), "deed_name"),
class_ability=_opt_str(fields_dict.get("class_ability"), "class_ability"),
channel_mode=_opt_channel_mode(fields_dict.get("channel_mode")),
channel_exclude=_str_tuple(fields_dict.get("channel_exclude"), "channel_exclude"),
direction=_int_pair(fields_dict.get("direction"), "direction"),
trick_condition=_opt_trick_condition(fields_dict.get("trick_condition")),
weapon_id=_opt_str(fields_dict.get("weapon_id"), "weapon_id"),
ready_trigger=_opt_ready_trigger(fields_dict.get("ready_trigger")),
destination=_int_pair(fields_dict.get("destination"), "destination"),
target_pos=_int_pair(fields_dict.get("target_pos"), "target_pos"),
aid_mode=_opt_aid_mode(fields_dict.get("aid_mode")),
metamagic=_str_tuple(fields_dict.get("metamagic"), "metamagic"),
cast_defensively=_opt_bool(fields_dict.get("cast_defensively"), "cast_defensively"),
tumble=_opt_bool(fields_dict.get("tumble"), "tumble"),
)
# ---- Public parse API --------------------------------------------------------
def parse_actions(reply: str) -> tuple[Action, ...]:
"""Parse an LLM reply into a tuple of Action objects."""
try:
parsed = json.loads(reply)
except json.JSONDecodeError as exc:
msg = f"reply is not valid JSON: {exc}"
raise NLPolicyError(msg) from exc
if not _is_object_list(parsed):
msg = "reply must be a JSON array of action objects"
raise NLPolicyError(msg)
items = parsed
return tuple(_action_from_item(item) for item in items)
# ---- Fallback policy ---------------------------------------------------------
Fallback = Callable[[tuple[Action, ...]], tuple[Action, ...]]
def fallback_default(menu: tuple[Action, ...]) -> tuple[Action, ...]:
"""Deterministic fallback: first decisive action, then a defensive hold."""
for kind in ("full_attack", "attack", "charge", "move", "5foot_step"):
for action in menu:
if action.kind == kind:
return (action,)
if menu:
return (menu[0],)
return (Action(kind="wait"),)
# ---- Repair protocol ---------------------------------------------------------
_REPAIR_INSTRUCTION = "Return a valid JSON action sequence."
_REPAIR_HEADER_TEMPLATE = "-- previous reply invalid (attempt {}) --\n{}"
@dataclass
class NLPolicyStats:
"""Mutable counters for the LLM decision loop (repair observability).
The phase-3 report and the sweep pipeline read these counters to
quantify how often the LLM layer repairs invalid replies or falls back
to the scripted policy.
"""
calls: int = 0
repairs_attempted: int = 0
repairs_succeeded: int = 0
fallbacks: int = 0
# ---- NLPolicy ----------------------------------------------------------------
@dataclass(frozen=True)
class NLPolicy:
"""Turn LLM completions into Action sequences, with a scripted fallback."""
client: LLMClient
builder: PromptBuilder
prompt_name: str = "nl_default"
fallback: Fallback = fallback_default
max_repairs: int = 1
stats: NLPolicyStats = field(default_factory=NLPolicyStats)
def __post_init__(self) -> None:
if self.max_repairs < 0:
msg = f"max_repairs must be >= 0, got {self.max_repairs}"
raise ValueError(msg)
expected = self.builder.prompt_version(self.prompt_name)
if self.client.prompt_version != expected:
msg = (
f"client prompt_version {self.client.prompt_version!r} does not "
f"match pack {expected!r} for {self.prompt_name!r}"
)
raise ValueError(msg)
def __call__(self, observation: Observation) -> tuple[Action, ...]:
menu = candidates(observation)
prompt = self.builder.build(observation, prompt_name=self.prompt_name)
reply = self.client.complete(prompt)
self.stats.calls += 1
errors: list[NLPolicyError] = []
while True:
try:
actions = parse_actions(reply)
break
except NLPolicyError as exc:
if len(errors) >= self.max_repairs:
self.stats.fallbacks += 1
return self.fallback(menu)
errors.append(exc)
self.stats.repairs_attempted += 1
trail = "\n\n".join(
_REPAIR_HEADER_TEMPLATE.format(index, error)
for index, error in enumerate(errors, start=1)
)
retry_prompt = f"{prompt}\n\n{trail}\n{_REPAIR_INSTRUCTION}"
reply = self.client.complete(retry_prompt)
self.stats.calls += 1
if errors:
self.stats.repairs_succeeded += 1
return actions
+135
View File
@@ -0,0 +1,135 @@
"""Objective primitives for the Phase-2 optimizer and danger cartography.
Pure functions over the per-run summaries produced by
``run_encounter(..., collect_runs=True)`` (docs/phase-2-llm-architecture.md
sections 3.1 and 3.2). These are the metrics an optimizer maximizes (party
survival) and a danger map reports per scenario - no engine access, no
state, no RNG: given the same ``RunOutcome`` tuple they always return the
same numbers, which is what makes them safe as a search objective.
Definitions (all probabilities are empirical frequencies over the runs;
``party`` is the side name being measured, e.g. ``"players"``):
- ``p_survives`` : P(party survives all) - no party member dead.
- ``p_tpk`` : P(total party kill) - every party member dead.
- ``p_any_death`` : P(at least one party member death).
- ``expected_hp_lost``: mean total party hit points lost (hp_max - hp,
clamped to [0, hp_max] per member: healing above max never counts
negative, and overkill on a corpse is not counted as extra loss).
- ``expected_rounds`` : mean battle length.
- ``worst_decile_party_hp`` : TAIL metric - the mean party hp *remaining*
over the worst decile of runs (the 10% with the largest hp loss, at
least 1 run): a CVaR-style measure so rare disasters are not averaged
away. A dead or dying member contributes 0 hp remaining.
A side with no members in a run (should not happen in a normal encounter)
is treated vacuously: survives all, no TPK, 0 hp lost.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from pf1e_simulator.runner import CombatantOutcome, RunOutcome
@dataclass(frozen=True)
class DangerMetrics:
"""The full danger profile of one party over a batch of run outcomes."""
runs: int
p_survives: float
p_tpk: float
p_any_death: float
expected_hp_lost: float
expected_rounds: float
worst_decile_party_hp: float
def _party_members(outcome: RunOutcome, party: str) -> tuple[CombatantOutcome, ...]:
return tuple(c for c in outcome.combatants if c.side == party)
def _hp_lost(outcome: RunOutcome, party: str) -> int:
return sum(
min(c.hp_max, max(0, c.hp_max - c.hp))
for c in _party_members(outcome, party)
)
def _hp_remaining(outcome: RunOutcome, party: str) -> int:
return sum(max(0, c.hp) for c in _party_members(outcome, party))
def p_survives(outcomes: Sequence[RunOutcome], party: str) -> float:
"""P(no member of `party` is dead at the end of a run)."""
if not outcomes:
return 0.0
hits = sum(1 for o in outcomes if not any(c.dead for c in _party_members(o, party)))
return hits / len(outcomes)
def p_tpk(outcomes: Sequence[RunOutcome], party: str) -> float:
"""P(every member of `party` is dead at the end of a run)."""
if not outcomes:
return 0.0
hits = 0
for o in outcomes:
members = _party_members(o, party)
if members and all(c.dead for c in members):
hits += 1
return hits / len(outcomes)
def p_any_death(outcomes: Sequence[RunOutcome], party: str) -> float:
"""P(at least one member of `party` is dead at the end of a run)."""
if not outcomes:
return 0.0
hits = sum(1 for o in outcomes if any(c.dead for c in _party_members(o, party)))
return hits / len(outcomes)
def expected_hp_lost(outcomes: Sequence[RunOutcome], party: str) -> float:
"""Mean total `party` hit points lost per run (clamped to [0, hp_max] per member)."""
if not outcomes:
return 0.0
return sum(_hp_lost(o, party) for o in outcomes) / len(outcomes)
def expected_rounds(outcomes: Sequence[RunOutcome]) -> float:
"""Mean number of rounds per run (draws count their full round cap)."""
if not outcomes:
return 0.0
return sum(o.rounds for o in outcomes) / len(outcomes)
def worst_decile_party_hp(outcomes: Sequence[RunOutcome], party: str) -> float:
"""Mean `party` hp remaining over the worst-decile runs (tail/CVaR metric).
The worst decile is the ``max(1, ceil(10%))`` runs with the largest hp
loss; ties keep list order. Complements ``expected_hp_lost``: the mean
hides tail disasters, this number does not.
"""
if not outcomes:
return 0.0
tail = max(1, math.ceil(len(outcomes) / 10))
worst = sorted(outcomes, key=lambda o: _hp_lost(o, party), reverse=True)[:tail]
return sum(_hp_remaining(o, party) for o in worst) / tail
def danger_metrics(outcomes: Sequence[RunOutcome], party: str) -> DangerMetrics:
"""Compute the whole danger profile in one call (the optimizer's objective)."""
return DangerMetrics(
runs=len(outcomes),
p_survives=p_survives(outcomes, party),
p_tpk=p_tpk(outcomes, party),
p_any_death=p_any_death(outcomes, party),
expected_hp_lost=expected_hp_lost(outcomes, party),
expected_rounds=expected_rounds(outcomes),
worst_decile_party_hp=worst_decile_party_hp(outcomes, party),
)
+86
View File
@@ -0,0 +1,86 @@
"""Phase-2 spike: a strengthened monster policy for danger cartography.
Spike-level, not the final ``adversary.py`` (docs/phase-2-llm-architecture.md
sections 3.2 and 10, P4.1): the danger a party faces only exists against an
adversary that plays to win, and ``default_policy`` is a weak baseline - each
monster attacks its own nearest enemy, so the party never concentrates
punishment on one member. This module provides the minimal upgrade that
proves the point: FOCUS FIRE. Every monster on the designated side
concentrates on the same party member - the active enemy with the lowest AC
(ties: lowest current hp, then initiative list order) - instead of its
nearest one.
The party side (any side other than the designated monster side) delegates
to ``default_policy`` unchanged, so one policy object drives the whole
battle and the danger-map comparison isolates exactly the monster-side
change. Spellcasting, class abilities and maneuvers stay on the default
flow: the spike measures what target selection alone is worth.
"""
from __future__ import annotations
from pf1e_simulator.combat import (
Action,
CombatantState,
CombatEngine,
Policy,
default_policy,
)
def focus_target(engine: CombatEngine, state: CombatantState) -> CombatantState | None:
"""The enemy the whole monster side concentrates on: lowest AC, then hp, then order.
AC is the combatant's current total AC as exposed on the frozen profile
(the spike does not re-derive buff/condition modifiers); hp is current
hit points (a dying target ranks below a healthy one). Deterministic:
ties break on the engine's initiative list order.
"""
enemies = [s for s in engine.states if s.side != state.side and s.active]
if not enemies:
return None
return min(enemies, key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)))
def _strike(
engine: CombatEngine, state: CombatantState, target: CombatantState
) -> tuple[Action, ...]:
"""Attack the focus target with the default melee/ranged action ladder."""
if engine.weapon_for(state, target) is not None:
return (Action(kind="full_attack", target_id=target.combatant.id),)
charge = engine.can_charge(state, target)
if charge is not None and charge.weapon is not None:
return (Action(kind="charge", target_id=target.combatant.id),)
if engine.can_step5_to_attack(state, target):
return (
Action(kind="5foot_step", target_id=target.combatant.id),
Action(kind="full_attack", target_id=target.combatant.id),
)
return (
Action(kind="move", target_id=target.combatant.id),
Action(kind="attack", target_id=target.combatant.id),
)
def focus_fire_policy(monster_side: str = "monsters") -> Policy:
"""A ``Policy`` where the monster side focus-fires; every other side plays default.
Pass to ``run_encounter(policy=focus_fire_policy("monsters"))``. The
party side delegates to ``default_policy``, so the only behavioural
change is monster target concentration.
"""
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
if state.side != monster_side:
return default_policy(engine, state)
if engine.has_condition_flag(state, "must_flee"):
target = engine.nearest_enemy(state)
if target is None:
return (Action(kind="wait"),)
return (Action(kind="withdraw", target_id=target.combatant.id),)
target = focus_target(engine, state)
if target is None:
return (Action(kind="wait"),)
return _strike(engine, state, target)
return policy
+234
View File
@@ -0,0 +1,234 @@
"""Externally-loaded, versioned prompt pack for the phase-2 LLM layer (P1).
Prompts do NOT live in this repository (architecture doc section 14 -- the
repo is code only). They live in a separate, versioned pack directory,
resolved through configuration (env var ``PF1E_PROMPT_PACK_DIR``, never
hard-coded -- section 15). This module loads a pack once per resolved path,
exposes each prompt by name and computes the ``prompt_version`` that must
enter the response-cache composite key (decision D6), the ``ReplayRecord``
and the sweep reports, so a campaign stays reproducible after a prompt edit
(traceability rule, section 14).
Pack layout
-----------
<pack_dir>/
version.txt # single line: semver or commit hash (mandatory)
<name>.md # one file per prompt name, e.g. ``nl_default.md``
<name>.examples.json # optional few-shot examples for that prompt: a
# JSON array of ``{"observation": {...},
# "actions": [{...}, ...]}`` teaching pairs
Loading is eager and strict. A missing pack, a missing ``version.txt`` or a
missing ``<name>.md`` raises a typed ``PromptPackError`` -- there is no
silent fallback (validation rule, section 14). An optional
``<name>.examples.json`` file carries few-shot examples (P5.2): it comes
from the external pack, never from this repo (section 14), and is validated
at load time -- an unreadable, malformed or orphaned examples file is a
pack error, not a silent skip. ``prompt_version`` covers the examples
content alongside the prompt text, so the response-cache key (D6) stays in
sync with any pack edit. Unit tests use the minimal local fixtures under
``tests/fixtures/prompt_pack/`` so CI never depends on the external prompt
repository (pinning rule, section 14).
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, field
from functools import cache
from typing import TYPE_CHECKING, NoReturn, TypeGuard
from pf1e_simulator.config import ENV_PROMPT_PACK_DIR, env_path
if TYPE_CHECKING:
from pathlib import Path
_PROMPT_SUFFIX = ".md"
_EXAMPLES_SUFFIX = ".examples.json"
_VERSION_FILE = "version.txt"
_HASH_PREFIX_LEN = 12
_HASH_SEPARATOR = "\0"
_EXAMPLE_KEYS = frozenset({"observation", "actions"})
class PromptPackError(RuntimeError):
"""A prompt pack is missing, invalid, or lacks a required prompt."""
def _pack_error(kind: str, message: str) -> NoReturn:
msg = f"{kind}: {message}"
raise PromptPackError(msg)
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
return isinstance(value, dict)
def _is_object_list(value: object) -> TypeGuard[list[object]]:
return isinstance(value, list)
@dataclass(frozen=True)
class FewShotExample:
"""One structured few-shot pair: an example observation and a valid reply.
``observation`` is a plain JSON object (the pack author composes it, the
builder renders it verbatim); ``actions`` is a JSON array of Action-shaped
objects teaching the exact reply format the parser accepts.
"""
observation: dict[str, object]
actions: tuple[dict[str, object], ...]
def _parse_examples(path: Path, text: str) -> tuple[FewShotExample, ...]:
"""Parse and strictly validate one few-shot examples file."""
try:
raw = json.loads(text)
except json.JSONDecodeError as exc:
msg = f"{path.name} is not valid JSON: {exc}"
_pack_error("invalid pack", msg)
if not _is_object_list(raw):
_pack_error("invalid pack", f"{path.name} must be a JSON array of examples")
examples: list[FewShotExample] = []
for index, item in enumerate(raw):
if not _is_object_dict(item):
msg = f"{path.name}[{index}] must be an object with 'observation' and 'actions'"
_pack_error("invalid pack", msg)
unknown = sorted(set(item) - _EXAMPLE_KEYS)
if unknown:
msg = f"{path.name}[{index}] has unknown key(s): {', '.join(unknown)}"
_pack_error("invalid pack", msg)
observation = item.get("observation")
actions = item.get("actions")
if not _is_object_dict(observation):
msg = f"{path.name}[{index}] 'observation' must be a JSON object"
_pack_error("invalid pack", msg)
if not _is_object_list(actions):
msg = f"{path.name}[{index}] 'actions' must be a JSON array of objects"
_pack_error("invalid pack", msg)
action_dicts: list[dict[str, object]] = []
for action in actions:
if not _is_object_dict(action):
msg = f"{path.name}[{index}] 'actions' must contain only JSON objects"
_pack_error("invalid pack", msg)
action_dicts.append(action)
examples.append(FewShotExample(observation=observation, actions=tuple(action_dicts)))
return tuple(examples)
@dataclass(frozen=True)
class PromptPack:
"""A versioned set of prompts loaded from one external directory.
Instances are immutable (frozen), so a loaded pack can be shared by
parallel sweep workers without drift. ``_prompts`` maps prompt names
(file stems) to their full text, read at load time; ``_examples`` maps
prompt names to their parsed few-shot pairs (empty when the pack ships
none) and ``_examples_text`` keeps the raw file bytes for hashing.
"""
root: Path
version: str
_prompts: dict[str, str] = field(repr=False)
_examples: dict[str, tuple[FewShotExample, ...]] = field(repr=False)
_examples_text: dict[str, str] = field(repr=False)
def get(self, name: str) -> str:
"""Return the prompt ``name``'s text, or raise ``PromptPackError``."""
try:
return self._prompts[name]
except KeyError as exc:
msg = (
f"prompt '{name}' missing from pack at {self.root} "
f"(expected file '{name}{_PROMPT_SUFFIX}')"
)
raise PromptPackError(msg) from exc
def examples(self, name: str) -> tuple[FewShotExample, ...]:
"""Return the few-shot pairs for ``name`` (empty when none)."""
return self._examples.get(name, ())
def prompt_version(self, name: str) -> str:
"""Pack version plus per-prompt content hash (cache-key component, D6).
The digest covers the prompt text and, when the pack ships a
``<name>.examples.json`` file, that file's bytes too -- any pack edit
invalidates the cache key. Without an examples file the digest is
byte-identical to the pre-few-shot scheme.
"""
content = self.get(name)
examples_text = self._examples_text.get(name)
digest_input = content
if examples_text is not None:
digest_input = f"{content}{_HASH_SEPARATOR}{examples_text}"
digest = hashlib.sha256(digest_input.encode("utf-8")).hexdigest()[:_HASH_PREFIX_LEN]
return f"{self.version}:{name}:{digest}"
@classmethod
def load(cls, root: Path) -> PromptPack:
"""Load and validate the pack rooted at ``root`` (strict, no fallback)."""
if not root.is_dir():
_pack_error(
"pack directory not found",
f"{root} does not exist or is not a directory",
)
version_file = root / _VERSION_FILE
if not version_file.is_file():
_pack_error("invalid pack", f"{root} has no {_VERSION_FILE} file")
version = version_file.read_text(encoding="utf-8").strip()
if not version:
_pack_error("invalid pack", f"{root / _VERSION_FILE} is empty")
prompts: dict[str, str] = {}
for path in sorted(root.glob(f"*{_PROMPT_SUFFIX}")):
prompts[path.stem] = path.read_text(encoding="utf-8")
examples: dict[str, tuple[FewShotExample, ...]] = {}
examples_text: dict[str, str] = {}
for path in sorted(root.glob(f"*{_EXAMPLES_SUFFIX}")):
stem = path.name.removesuffix(_EXAMPLES_SUFFIX)
if stem not in prompts:
msg = f"{path.name} has no matching '{stem}{_PROMPT_SUFFIX}' prompt"
_pack_error("invalid pack", msg)
text = path.read_text(encoding="utf-8")
examples[stem] = _parse_examples(path, text)
examples_text[stem] = text
return cls(
root=root,
version=version,
_prompts=prompts,
_examples=examples,
_examples_text=examples_text,
)
@cache
def _load_cached(root: Path) -> PromptPack:
"""Load once per resolved path (thread-safe; parallel sweeps share it)."""
return PromptPack.load(root)
def load_prompt_pack(pack_dir: Path | None = None) -> PromptPack:
"""Load the prompt pack from ``pack_dir`` or the configured directory.
With ``pack_dir`` given, load that directory directly (tests use this
with their local fixture). Otherwise resolve
``PF1E_PROMPT_PACK_DIR``; when unset, raise a clear error instead of
falling back to a bundled (non-existent) pack.
"""
if pack_dir is not None:
return _load_cached(pack_dir)
configured = env_path(ENV_PROMPT_PACK_DIR)
if configured is None:
_pack_error(
"prompt pack not configured",
f"set {ENV_PROMPT_PACK_DIR} to the external prompt pack directory "
"(architecture doc section 14)",
)
return _load_cached(configured)
def clear_pack_cache() -> None:
"""Drop the per-path pack cache (used by tests that relocate fixtures)."""
_load_cached.cache_clear()
+204
View File
@@ -0,0 +1,204 @@
"""Prompt builder for the phase-2 LLM layer (P1.4).
Builds a deterministic prompt from an ``Observation`` plus the external
prompt pack: template + optional few-shot examples + compact observation +
plausible actions + JSON action schema. The examples (P5.2) come from the
pack's ``<name>.examples.json`` file when present and are mounted between
the template and the live observation, so the query stays the last section
before the reply (few-shot recency) and packs without examples keep the
original section order bit-for-bit. The schema is derived from the
``Action`` dataclass via ``typing.get_type_hints`` so it stays in sync with
the engine.
"""
from __future__ import annotations
import json
from dataclasses import MISSING, asdict, dataclass, fields
from types import UnionType
from typing import TYPE_CHECKING, Literal, Union, get_args, get_origin, get_type_hints
from pf1e_simulator.candidates import candidates
from pf1e_simulator.combat import UNSUPPORTED_ACTION_KINDS, Action, Observation
from pf1e_simulator.llm_view import compact_observation
if TYPE_CHECKING:
from pf1e_simulator.prompt import FewShotExample, PromptPack
def _supported_kinds() -> tuple[str, ...]:
"""Return the 27 action kinds supported by the LLM layer (declaration order)."""
kind_literal = get_type_hints(Action)["kind"]
all_kinds = get_args(kind_literal)
return tuple(k for k in all_kinds if k not in UNSUPPORTED_ACTION_KINDS)
_EMPTY_LITERAL_MSG = "empty Literal"
_UNSUPPORTED_LITERAL_MSG = "unsupported Literal values: {}"
_MULTI_UNION_MSG = "multi-type union not supported in Action: {}"
_UNSUPPORTED_TYPE_MSG = "unsupported type in Action schema: {}"
_TUPLE_ELLIPSIS_LEN = 2
def _json_schema_base(tp: object) -> dict[str, object] | None:
"""Return the JSON Schema fragment for a base type, else None."""
if tp is type(None):
return {"type": "null"}
if tp is str:
return {"type": "string"}
if tp is int:
return {"type": "integer"}
if tp is bool:
return {"type": "boolean"}
return None
def _json_schema_literal(args: tuple[object, ...]) -> dict[str, object]:
"""Return the JSON Schema fragment for a Literal type."""
if not args:
raise ValueError(_EMPTY_LITERAL_MSG)
first = args[0]
if isinstance(first, str):
base_type = "string"
elif isinstance(first, int):
base_type = "integer"
else:
msg = _UNSUPPORTED_LITERAL_MSG.format(args)
raise TypeError(msg)
return {"type": base_type, "enum": list(args)}
def _json_schema_tuple(args: tuple[object, ...]) -> dict[str, object]:
"""Return the JSON Schema fragment for a tuple type."""
if not args:
return {"type": "array"}
if len(args) == _TUPLE_ELLIPSIS_LEN and args[1] is Ellipsis:
# tuple[T, ...] → homogeneous array
item_schema = _json_schema_for(args[0])
return {"type": "array", "items": item_schema}
# Fixed-length tuple → prefixItems + minItems/maxItems
prefix = [_json_schema_for(arg) for arg in args]
return {
"type": "array",
"prefixItems": prefix,
"minItems": len(args),
"maxItems": len(args),
}
def _json_schema_union(args: tuple[object, ...]) -> dict[str, object]:
"""Return the JSON Schema fragment for a nullable single-member union."""
non_none = [a for a in args if a is not type(None)]
if len(non_none) == 1:
member_schema = _json_schema_for(non_none[0])
# Merge "null" into the type array
member_type = member_schema.get("type")
if isinstance(member_type, str):
member_schema["type"] = [member_type, "null"]
# If member already has an enum (e.g. nullable Literal), keep it.
return member_schema
msg = _MULTI_UNION_MSG.format(args)
raise ValueError(msg)
def _json_schema_for(tp: object) -> dict[str, object]:
"""Convert a Python type annotation to a JSON Schema fragment."""
base_schema = _json_schema_base(tp)
if base_schema is not None:
return base_schema
origin = get_origin(tp)
args = get_args(tp)
if origin is Literal:
return _json_schema_literal(args)
if origin is tuple:
return _json_schema_tuple(args)
if origin is Union or origin is UnionType:
return _json_schema_union(args)
msg = _UNSUPPORTED_TYPE_MSG.format(tp)
raise ValueError(msg)
def action_schema() -> dict[str, object]:
"""JSON Schema for a single Action object."""
hints = get_type_hints(Action)
properties = {name: _json_schema_for(tp) for name, tp in hints.items()}
# Override kind with the filtered 27-kind enum (source Literal order)
properties["kind"] = {"type": "string", "enum": list(_supported_kinds())}
# Only 'kind' lacks a default
required = [
f.name
for f in fields(Action)
if f.default is MISSING and f.default_factory is MISSING
]
return {
"type": "object",
"properties": properties,
"required": required,
"additionalProperties": False,
}
def render_candidates(actions: tuple[Action, ...]) -> tuple[str, ...]:
"""Render the plausible-action menu as numbered lines with compact JSON."""
lines: list[str] = []
for index, action in enumerate(actions):
payload = {
key: value
for key, value in asdict(action).items()
if key != "kind" and value is not None and value != () and value is not False
}
suffix = f" {json.dumps(payload, sort_keys=True, separators=(',', ':'))}" if payload else ""
lines.append(f"{index}. {action.kind}{suffix}")
return tuple(lines)
def _render_examples(examples: tuple[FewShotExample, ...]) -> str:
"""Render few-shot pairs as observation-to-response teaching blocks."""
blocks: list[str] = []
for index, example in enumerate(examples):
observation_json = json.dumps(example.observation, sort_keys=True)
actions_json = json.dumps(example.actions, sort_keys=True)
blocks.append(
f"### example {index + 1}\n"
f"observation:\n{observation_json}\n\n"
f"response:\n{actions_json}"
)
return "## few-shot examples\n" + "\n\n".join(blocks)
@dataclass(frozen=True)
class PromptBuilder:
"""Builds the complete LLM prompt from a prompt pack and an observation."""
pack: PromptPack
def prompt_version(self, name: str) -> str:
"""Pack version plus per-prompt content hash (cache-key component, D6)."""
return self.pack.prompt_version(name)
def build(self, observation: object, *, prompt_name: str = "nl_default") -> str:
"""Assemble the full prompt for the given observation."""
if not isinstance(observation, Observation):
msg = "observation must be an Observation instance"
raise TypeError(msg)
template = self.pack.get(prompt_name)
obs_text = compact_observation(observation).rstrip("\n")
menu = candidates(observation)
actions_text = "\n".join(render_candidates(menu))
schema_text = json.dumps(action_schema(), sort_keys=True, indent=2)
sections: list[str] = [template]
examples = self.pack.examples(prompt_name)
if examples:
sections.append(_render_examples(examples))
sections.extend([
"## observation\n" + obs_text,
"## plausible actions\n" + actions_text,
"## action schema\n" + schema_text,
])
return "\n\n".join(sections)
+227
View File
@@ -0,0 +1,227 @@
"""Real OpenAI-compatible chat client for the phase-2 LLM layer (P5.1).
``OpenAILLMClient`` speaks the OpenAI chat-completions wire format over HTTPS
using only the standard library (``urllib.request``), so the repository ships
no HTTP dependency. The base URL, model and API key are resolved from the
environment (:mod:`pf1e_simulator.config`, section 15); the key is never
embedded in code.
Error taxonomy (shared with :mod:`pf1e_simulator.client`):
- ``ProviderError`` for transient failures (transport errors, HTTP 429, any
5xx) -- retried by ``LLMClient``'s backoff loop and counted by the breaker;
- ``ProviderRejectedError`` for permanent 4xx rejections (bad key, unknown
model, quota) -- raised immediately, never retried;
- ``ProviderResponseError`` for a 2xx body that is not a valid
chat-completions payload -- raised immediately, never retried.
"""
from __future__ import annotations
import json
import time
import urllib.request
from typing import TYPE_CHECKING, Protocol, Self, TypeGuard
from urllib.error import HTTPError
from urllib.request import Request
from pf1e_simulator.client import (
CircuitBreaker,
ClientError,
ClientParams,
Completion,
CostBudget,
LLMClient,
Pricing,
ProviderError,
ResponseCache,
TokenUsage,
estimate_tokens,
)
from pf1e_simulator.combat import OBSERVATION_SCHEMA_VERSION
from pf1e_simulator.config import (
ENV_LLM_API_KEY,
llm_api_key,
llm_base_url,
llm_model,
)
if TYPE_CHECKING:
from collections.abc import Callable
# Status-code buckets of the chat-completions wire (RFC 9110): 2xx is a
# success, 429 and every 5xx mean "try again later" (the only statuses safe
# to retry), and any other 4xx is a permanent rejection.
_OK_MIN = 200
_OK_EXCLUSIVE = 300
_RATE_LIMITED = 429
_SERVER_ERROR_MIN = 500
class ProviderRejectedError(ClientError):
"""A permanent 4xx provider rejection - never retried."""
class ProviderResponseError(ClientError):
"""A 2xx body that is not a valid chat-completions payload."""
class HttpResponse(Protocol):
"""The minimal ``urllib.response`` surface the wire call uses."""
status: int
def read(self) -> bytes: ...
def __enter__(self) -> Self: ...
def __exit__(self, _exc_type: object, _exc_value: object, _traceback: object) -> None: ...
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
return isinstance(value, dict)
def _is_object_list(value: object) -> TypeGuard[list[object]]:
return isinstance(value, list)
def _parse_completion(payload: dict[str, object]) -> str:
"""Extract the completion text from a chat-completions payload."""
choices_raw = payload.get("choices")
if not _is_object_list(choices_raw) or not choices_raw:
msg = "provider response has no choices"
raise ProviderResponseError(msg)
choice = choices_raw[0]
if not _is_object_dict(choice):
msg = "provider response choice is not a JSON object"
raise ProviderResponseError(msg)
message_raw = choice.get("message")
if not _is_object_dict(message_raw):
msg = "provider response message is not a JSON object"
raise ProviderResponseError(msg)
content = message_raw.get("content")
if not isinstance(content, str):
msg = "provider response content is not a string"
raise ProviderResponseError(msg)
return content
def _parse_usage(payload: dict[str, object], prompt: str, text: str) -> TokenUsage:
"""Read the usage block, or estimate tokens when the provider omits it."""
usage_raw = payload.get("usage")
if not _is_object_dict(usage_raw):
return TokenUsage(
prompt_tokens=estimate_tokens(prompt),
completion_tokens=estimate_tokens(text),
)
prompt_tokens = usage_raw.get("prompt_tokens")
completion_tokens = usage_raw.get("completion_tokens")
if not isinstance(prompt_tokens, int) or not isinstance(completion_tokens, int):
msg = "provider usage block is malformed"
raise ProviderResponseError(msg)
return TokenUsage(prompt_tokens=prompt_tokens, completion_tokens=completion_tokens)
class OpenAILLMClient(LLMClient):
"""Real chat-completions client (OpenAI or any compatible gateway).
Endpoint, model and key default to the environment (section 15); all may
be overridden explicitly. ``urlopen`` and ``sleep`` are injectable so
tests exercise the full retry/breaker machinery offline.
"""
def __init__(
self,
*,
prompt_version: str,
model: str | None = None,
api_key: str | None = None,
base_url: str | None = None,
timeout_s: float = 30.0,
urlopen: Callable[..., HttpResponse] = urllib.request.urlopen,
schema_version: int = OBSERVATION_SCHEMA_VERSION,
cache: ResponseCache | None = None,
budget: CostBudget | None = None,
pricing: Pricing | None = None,
retries: int = 2,
backoff_base_s: float = 0.25,
breaker: CircuitBreaker | None = None,
sleep: Callable[[float], None] | None = None,
) -> None:
key = api_key if api_key is not None else llm_api_key()
if key is None or key == "":
msg = (
f"no LLM API key: set the {ENV_LLM_API_KEY} environment "
"variable or pass an explicit api_key"
)
raise ValueError(msg)
if timeout_s <= 0:
msg = f"timeout_s must be > 0, got {timeout_s}"
raise ValueError(msg)
super().__init__(
model=model if model is not None else llm_model(),
prompt_version=prompt_version,
schema_version=schema_version,
cache=cache,
budget=budget,
pricing=pricing,
retries=retries,
backoff_base_s=backoff_base_s,
sleep=sleep if sleep is not None else time.sleep,
breaker=breaker,
)
resolved_base_url = base_url if base_url is not None else llm_base_url()
self._api_key = key
self._base_url = resolved_base_url.rstrip("/")
self._timeout_s = timeout_s
self._urlopen = urlopen
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
"""One chat-completions attempt; raise ``ProviderError`` when retryable."""
body = {
"model": self.model,
"messages": [{"role": "user", "content": prompt}],
"temperature": params.temperature,
"max_tokens": params.max_tokens,
"top_p": params.top_p,
"response_format": {"type": "json_object"},
}
request = Request(
f"{self._base_url}/chat/completions",
data=json.dumps(body).encode("utf-8"),
headers={
"Authorization": f"Bearer {self._api_key}",
"Content-Type": "application/json",
},
method="POST",
)
try:
with self._urlopen(request, timeout=self._timeout_s) as response:
status = response.status
data = response.read()
except HTTPError as exc:
status = exc.code
data = exc.read()
except OSError as exc:
msg = f"provider unreachable: {exc}"
raise ProviderError(msg) from exc
if status == _RATE_LIMITED or status >= _SERVER_ERROR_MIN:
msg = f"provider transient failure: HTTP {status}"
raise ProviderError(msg)
if not _OK_MIN <= status < _OK_EXCLUSIVE:
msg = f"provider rejected request: HTTP {status}"
raise ProviderRejectedError(msg)
try:
payload = json.loads(data.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
msg = f"provider response is not valid JSON: {exc}"
raise ProviderResponseError(msg) from exc
if not _is_object_dict(payload):
msg = "provider payload is not a JSON object"
raise ProviderResponseError(msg)
text = _parse_completion(payload)
usage = _parse_usage(payload, prompt, text)
return Completion(text=text, usage=usage)
+250
View File
@@ -0,0 +1,250 @@
"""Replay log for observation-only policies (docs/phase-2 P0.5).
A ``ReplayLog`` records every ``(observation, action)`` pair an
observation-only policy produced during one battle, keyed by a monotonic
``call_seq`` assigned in execution order. It also carries the ``EncounterSpec``
plus the battle ``seed`` and ``round_cap`` so a recorded battle can be re-run
standalone. ``replay(log)`` turns a log back into an ``LLMPolicy`` that returns
the recorded action for each observation, raising a typed ``ReplayError`` (a
``ValueError``) on an out-of-sequence observation or once the log is exhausted.
Why ``call_seq`` and not the observation as the key: the same observation can
legitimately recur (a stalled battle), and delayed turns / summons can revisit
an earlier position — the sequence position is the only unambiguous key.
The JSONL serialization is self-contained (a header line with the spec, seed
and round cap, then one record per line) so ``ReplayLog.from_jsonl(text)``
round-trips without any out-of-band state. ``Observation`` uses the engine's
canonical ``to_dict()`` (``schema_version`` included); ``Action`` and the spec
are dumped through Pydantic type adapters so their tuple fields survive as
JSON arrays and are coerced back to tuples on load. Observations are matched
during playback by :func:`observation_key` (their canonical JSON), not raw
dataclass equality: JSON collapses tuples and lists, and the snapshot's inner
``dict[str, object]`` holds both.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import TYPE_CHECKING, Self, TypeGuard
from pydantic import TypeAdapter
from pf1e_simulator.combat import Action, Observation
from pf1e_simulator.map import MapSpec
from pf1e_simulator.models import Combatant
from pf1e_simulator.runner import EncounterSpec
if TYPE_CHECKING:
from pf1e_simulator.combat import LLMPolicy
_HEADER_KIND = "header"
_RECORD_KIND = "record"
# ``EncounterSpec`` is a stdlib dataclass whose annotations reference
# ``MapSpec``/``Combatant``; ``runner`` imports those only under TYPE_CHECKING,
# so the adapter's default namespace cannot resolve them. Rebuild it with the
# two runtime classes in scope.
_SPEC_ADAPTER = TypeAdapter(EncounterSpec)
_SPEC_ADAPTER.rebuild(_types_namespace={"MapSpec": MapSpec, "Combatant": Combatant})
_OBSERVATION_ADAPTER = TypeAdapter(Observation)
_ACTION_ADAPTER = TypeAdapter(Action)
class ReplayError(ValueError):
"""Base error for replay-log construction and playback."""
class ReplayMismatchError(ReplayError):
"""A replay policy received an observation that differs from the recorded one."""
class ReplayExhaustedError(ReplayError):
"""A replay policy was called past the last recorded ``call_seq``."""
@dataclass(frozen=True)
class ReplayRecord:
"""One recorded policy call: the snapshot shown and the action chosen.
``call_seq`` is the position of this call in the execution order of the
battle (0-based, contiguous), assigned by the recorder.
"""
call_seq: int
observation: Observation
action: Action
class ReplayLog:
"""A recorded battle: spec + seed + round cap + ordered policy calls."""
def __init__(self, *, spec: EncounterSpec, seed: int, round_cap: int) -> None:
self.spec = spec
self.seed = seed
self.round_cap = round_cap
self._records: list[ReplayRecord] = []
def add(self, record: ReplayRecord) -> None:
"""Append one call; require a strictly increasing ``call_seq``.
The log is a sequence: a duplicate or out-of-order ``call_seq`` would
make the recorded execution order ambiguous, so both are rejected.
"""
if self._records:
last = self._records[-1].call_seq
if record.call_seq == last:
msg = f"duplicate call_seq {record.call_seq}"
raise ReplayError(msg)
if record.call_seq < last:
msg = f"call_seq {record.call_seq} is not after {last}"
raise ReplayError(msg)
self._records.append(record)
@property
def records(self) -> tuple[ReplayRecord, ...]:
"""The recorded calls ordered by ``call_seq`` (replay order)."""
return tuple(sorted(self._records, key=lambda record: record.call_seq))
def __len__(self) -> int:
return len(self._records)
def to_jsonl(self) -> str:
"""Serialize to newline-delimited JSON (header line, then one per call)."""
lines = [self._header_json()]
lines.extend(self._record_json(record) for record in self.records)
return "".join(line + "\n" for line in lines)
@classmethod
def from_jsonl(cls, text: str) -> Self:
"""Rebuild a log from ``to_jsonl`` output; raise ``ReplayError`` if invalid."""
lines = [line for line in text.splitlines() if line.strip()]
if not lines:
msg = "empty replay log"
raise ReplayError(msg)
header = _as_object(_loads(lines[0]), "header")
if _as_str(header.get("kind"), "kind") != _HEADER_KIND:
msg = "first line is not a replay header"
raise ReplayError(msg)
log = cls(
spec=_SPEC_ADAPTER.validate_python(header["spec"]),
seed=_as_int(header.get("seed"), "seed"),
round_cap=_as_int(header.get("round_cap"), "round_cap"),
)
for line in lines[1:]:
data = _as_object(_loads(line), "record")
if _as_str(data.get("kind"), "kind") != _RECORD_KIND:
msg = "record line is not a replay record"
raise ReplayError(msg)
log.add(
ReplayRecord(
call_seq=_as_int(data.get("call_seq"), "call_seq"),
observation=_OBSERVATION_ADAPTER.validate_python(
data["observation"]
),
action=_ACTION_ADAPTER.validate_python(data["action"]),
)
)
return log
def _header_json(self) -> str:
payload: dict[str, object] = {
"kind": _HEADER_KIND,
"seed": self.seed,
"round_cap": self.round_cap,
"spec": _SPEC_ADAPTER.dump_python(self.spec, mode="json"),
}
return json.dumps(
payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
def _record_json(self, record: ReplayRecord) -> str:
payload: dict[str, object] = {
"kind": _RECORD_KIND,
"call_seq": record.call_seq,
"observation": record.observation.to_dict(),
"action": _ACTION_ADAPTER.dump_python(
record.action, mode="json", exclude_defaults=True
),
}
return json.dumps(
payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
def observation_key(observation: Observation) -> str:
"""Return the canonical identity of an observation for replay matching.
``Observation.to_dict()`` is already JSON-shaped, so dumping it with sorted
keys yields a value that is insensitive to dict insertion order and treats
tuples and lists alike (JSON has no tuple type).
"""
return json.dumps(
observation.to_dict(), sort_keys=True, separators=(",", ":"), ensure_ascii=True
)
def replay(log: ReplayLog) -> LLMPolicy:
"""Build an observation-only policy that replays ``log`` in ``call_seq`` order.
The returned policy walks the log once, comparing the observation it is
handed to the recorded one before returning that call's action (wrapped in
a one-action tuple). An out-of-sequence observation raises
``ReplayMismatchError``; a call past the end raises ``ReplayExhaustedError``.
"""
ordered = log.records
keys = [observation_key(record.observation) for record in ordered]
cursor = 0
def policy(observation: Observation) -> tuple[Action, ...]:
nonlocal cursor
if cursor >= len(ordered):
msg = f"replay log exhausted after {len(ordered)} calls"
raise ReplayExhaustedError(msg)
record = ordered[cursor]
if keys[cursor] != observation_key(observation):
msg = (
f"replay mismatch at call_seq {record.call_seq}: "
"the observation differs from the recorded one"
)
raise ReplayMismatchError(msg)
cursor += 1
return (record.action,)
return policy
def _loads(text: str) -> object:
"""Parse one JSONL line, mapping a JSON error to a ``ReplayError``."""
try:
return json.loads(text)
except json.JSONDecodeError as exc:
msg = f"invalid JSON in replay log: {exc}"
raise ReplayError(msg) from exc
def _as_object(value: object, what: str) -> dict[str, object]:
if _is_str_object_dict(value):
return value
msg = f"expected a JSON object for {what}"
raise ReplayError(msg)
def _as_str(value: object, what: str) -> str:
if isinstance(value, str):
return value
msg = f"expected a string for {what}"
raise ReplayError(msg)
def _as_int(value: object, what: str) -> int:
# bool is an int subclass but never a valid call_seq/seed/round_cap.
if isinstance(value, int) and not isinstance(value, bool):
return value
msg = f"expected an integer for {what}"
raise ReplayError(msg)
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
return isinstance(value, dict)
+682
View File
@@ -0,0 +1,682 @@
"""Robustness layer: alternate min-max and danger cartography
(docs/phase-2-llm-architecture.md sections 3.2 and 10; P4.2 and P4.3).
A party's danger is only meaningful against an adversary that plays to
win (D12). This module implements the two robustness layers on top of
the compiled Etage-A pipeline:
- P4.2 - alternate min-max (section 3.2: max_PJ . min_monstres):
``adversarial_analysis`` alternately fixes one side and best-responds
the other (the party maximizes its own ``DangerMetrics``, then the
adversary minimizes the party's), every evaluation being a
``matchup_benchmark`` of one party ``StrategySpec`` vs one adversary
``AdversarySpec`` over a ``ScenarioSpec``. The same measure gives both
senses: danger for the party AND danger for the monsters.
- P4.3 - danger cartography (non exhaustive, D14): ``danger_map`` sweeps
a grid of per-side initial-HP fractions (``SideSpec.hp_fraction``) and
reports both sides' danger per cell, ``frontier`` bisects the party HP
fraction where a danger criterion crosses a threshold, and
``sensitivity`` ranks which single adversary lever moves the party
danger the most.
Deterministic throughout: no RNG, no LLM, no network. The engine's
``SeededRng`` under ``run_encounter`` is the only randomness and it is
seeded, so identical inputs produce bit-identical frozen results.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from pf1e_simulator.objectives import DangerMetrics
from pf1e_simulator.runner import EncounterReport
from pf1e_simulator.strategy import StrategySpec
from pf1e_simulator.adversary import AdversarySpec, adversary_name, compile_matchup
from pf1e_simulator.objectives import danger_metrics
from pf1e_simulator.runner import run_encounter
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec, build_encounter
_CRITERIA = ("p_survives", "p_tpk", "p_any_death", "expected_hp_lost")
_MATCHUP_SIDES = 2 # a matchup is exactly one arena pair (two sides)
def _check_criterion(criterion: str) -> None:
if criterion not in _CRITERIA:
msg = (
"criterion must be one of ('p_survives', 'p_tpk', 'p_any_death', "
f"'expected_hp_lost'), got {criterion!r}"
)
raise ValueError(msg)
def _danger_value(danger: DangerMetrics, criterion: str) -> float:
"""The `criterion` value of a danger profile (explicit dispatch, no getattr)."""
if criterion == "p_survives":
return danger.p_survives
if criterion == "p_tpk":
return danger.p_tpk
if criterion == "p_any_death":
return danger.p_any_death
return danger.expected_hp_lost
@dataclass(frozen=True)
class MatchupBenchmark:
"""One (party strategy x adversary spec) matchup over a scenario.
``report`` aggregates all runs; ``party_danger`` is the danger of the
``party_side`` side and ``adversary_danger`` the danger of the other
side (same measure, both senses: PJ and monsters).
"""
scenario: str
party: str
adversary: str
runs: int
seed: int
report: EncounterReport
party_danger: DangerMetrics
adversary_danger: DangerMetrics
party_side: str
def _other_side(scenario: ScenarioSpec, party_side: str) -> str:
"""The unique side different from ``party_side`` (exactly-2-sides contract)."""
names = tuple(side.name for side in scenario.sides)
if len(names) != _MATCHUP_SIDES:
msg = f"matchup requires exactly 2 sides, got {len(names)}"
raise ValueError(msg)
if party_side not in names:
msg = f"party_side {party_side!r} is not a side of the scenario"
raise ValueError(msg)
(other,) = (name for name in names if name != party_side)
return other
def matchup_benchmark(
scenario: ScenarioSpec,
party: StrategySpec,
adversary: AdversarySpec,
*,
party_side: str = "players",
runs: int = 100,
seed: int = 1,
) -> MatchupBenchmark:
"""Compile one matchup and measure both sides' danger over `runs` battles.
Deterministic: same inputs -> bit-identical ``MatchupBenchmark``
(frozen). Exactly two sides are required (one arena pair), the party
side name must be non-empty and be one of them, and ``runs`` >= 1.
"""
if not party_side:
msg = "party_side must be a non-empty string"
raise ValueError(msg)
adversary_side = _other_side(scenario, party_side)
if runs < 1:
msg = "runs must be >= 1"
raise ValueError(msg)
encounter = build_encounter(scenario)
policy = compile_matchup(party, adversary, party_side=party_side)
report, outcomes = run_encounter(
encounter, runs, seed, policy=policy, collect_runs=True
)
return MatchupBenchmark(
scenario=scenario.name,
party=party.name,
adversary=adversary.name,
runs=runs,
seed=seed,
report=report,
party_danger=danger_metrics(outcomes, party_side),
adversary_danger=danger_metrics(outcomes, adversary_side),
party_side=party_side,
)
@dataclass(frozen=True)
class RobustnessEntry:
"""One turn of the alternating min-max: the pair kept after that turn."""
iteration: int
party: StrategySpec
adversary: AdversarySpec
benchmark: MatchupBenchmark
def _party_rank(benchmark: MatchupBenchmark) -> tuple[float, float, str]:
"""Key: the party best-response prefers high survival, low HP lost, then name."""
danger = benchmark.party_danger
return (danger.p_survives, -danger.expected_hp_lost, benchmark.party)
def _adversary_rank(benchmark: MatchupBenchmark) -> tuple[float, float, str]:
"""Key: the adversary best-response prefers low party survival, high HP lost."""
danger = benchmark.party_danger
return (danger.p_survives, -danger.expected_hp_lost, benchmark.adversary)
def adversarial_analysis(
scenario: ScenarioSpec,
*,
party_candidates: Sequence[StrategySpec],
adversary_candidates: Sequence[AdversarySpec],
party_side: str = "players",
runs: int = 100,
seed: int = 1,
iterations: int = 3,
max_total_runs: int | None = None,
) -> tuple[RobustnessEntry, ...]:
"""Alternate (party, adversary) best responses over `iterations` turns.
Each turn re-evaluates every party candidate against the current
adversary and keeps the best for the party (max own survival), then
every adversary candidate against the kept party and keeps the worst
for the party (min own survival, max party HP lost) -- the min-max
alternation of section 3.2. The kept pair is reported with its
benchmark.
Budget: at most ``iterations * (party + adversary candidates) * runs``
battles. When ``max_total_runs`` is set and the budget exceeds it, no
battle is fought and a ValueError is raised.
"""
if not party_candidates:
msg = "party_candidates must be non-empty"
raise ValueError(msg)
if not adversary_candidates:
msg = "adversary_candidates must be non-empty"
raise ValueError(msg)
if iterations < 1:
msg = f"iterations must be >= 1, got {iterations}"
raise ValueError(msg)
if runs < 1:
msg = f"runs must be >= 1, got {runs}"
raise ValueError(msg)
cost = iterations * (len(party_candidates) + len(adversary_candidates)) * runs
if max_total_runs is not None and cost > max_total_runs:
msg = (
f"robustness budget exceeded: {cost} runs > "
f"{max_total_runs} max_total_runs"
)
raise ValueError(msg)
party = party_candidates[0]
adversary = adversary_candidates[0]
entries: list[RobustnessEntry] = []
for iteration in range(1, iterations + 1):
party_pairs: list[tuple[StrategySpec, MatchupBenchmark]] = [
(
candidate,
matchup_benchmark(
scenario,
candidate,
adversary,
party_side=party_side,
runs=runs,
seed=seed,
),
)
for candidate in party_candidates
]
party = max(party_pairs, key=lambda pair: _party_rank(pair[1]))[0]
adversary_pairs: list[tuple[AdversarySpec, MatchupBenchmark]] = [
(
candidate,
matchup_benchmark(
scenario,
party,
candidate,
party_side=party_side,
runs=runs,
seed=seed,
),
)
for candidate in adversary_candidates
]
adversary, benchmark = min(
adversary_pairs, key=lambda pair: _adversary_rank(pair[1])
)
entries.append(
RobustnessEntry(
iteration=iteration,
party=party,
adversary=adversary,
benchmark=benchmark,
)
)
return tuple(entries)
@dataclass(frozen=True)
class DangerCell:
"""One (party HP x adversary HP) cell of the danger map."""
party_hp_fraction: float
adversary_hp_fraction: float
benchmark: MatchupBenchmark
def _side_variant(
scenario: ScenarioSpec,
*,
party_side: str,
party_hp_fraction: float,
adversary_hp_fraction: float,
) -> ScenarioSpec:
"""A deterministic clone of `scenario` with per-side initial-HP fractions.
The `party_side` side carries `party_hp_fraction`; every other side
carries `adversary_hp_fraction` (the matchup contract is exactly 2
sides). Name: "<base> party_hp=<f> adv_hp=<g>" so cells stay distinct.
"""
sides = tuple(
SideSpec(
name=side.name,
zone=side.zone,
combatant_files=side.combatant_files,
hp_fraction=(
party_hp_fraction if side.name == party_side else adversary_hp_fraction
),
aware=side.aware,
)
for side in scenario.sides
)
return ScenarioSpec(
name=(
f"{scenario.name} party_hp={party_hp_fraction} "
f"adv_hp={adversary_hp_fraction}"
),
map_file=scenario.map_file,
sides=sides,
round_cap=scenario.round_cap,
)
def danger_map(
scenario: ScenarioSpec,
party: StrategySpec,
adversary: AdversarySpec,
*,
party_side: str = "players",
fractions: Sequence[float] = (0.5, 0.7, 1.0),
runs: int = 100,
seed: int = 1,
) -> tuple[DangerCell, ...]:
"""The grid of party HP x adversary HP danger cells (non exhaustive, D14).
One ``DangerCell`` per combination, party-major (party fraction
outermost), each with a distinct scenario name
"<base> party_hp=<f> adv_hp=<g>". Every fraction must lie in (0, 1].
"""
if not fractions:
msg = "fractions must be non-empty"
raise ValueError(msg)
for fraction in fractions:
if not 0.0 < fraction <= 1.0:
msg = f"fraction must be in (0, 1], got {fraction}"
raise ValueError(msg)
return tuple(
DangerCell(
party_hp_fraction=party_frac,
adversary_hp_fraction=adv_frac,
benchmark=matchup_benchmark(
_side_variant(
scenario,
party_side=party_side,
party_hp_fraction=party_frac,
adversary_hp_fraction=adv_frac,
),
party,
adversary,
party_side=party_side,
runs=runs,
seed=seed,
),
)
for party_frac in fractions
for adv_frac in fractions
)
def _validate_frontier(
criterion: str,
target: float,
min_fraction: float,
max_fraction: float,
precision: float,
) -> None:
"""Validate frontier parameters (keeps ``frontier`` itself flat)."""
_check_criterion(criterion)
if criterion == "expected_hp_lost":
if target < 0:
msg = f"target must be >= 0, got {target}"
raise ValueError(msg)
elif not 0.0 < target < 1.0:
msg = f"target must be in (0, 1), got {target}"
raise ValueError(msg)
if min_fraction <= 0:
msg = f"min_fraction must be > 0, got {min_fraction}"
raise ValueError(msg)
if max_fraction > 1:
msg = f"max_fraction must be <= 1, got {max_fraction}"
raise ValueError(msg)
if min_fraction > max_fraction:
msg = (
f"min_fraction must be <= max_fraction, "
f"got min={min_fraction} max={max_fraction}"
)
raise ValueError(msg)
if precision <= 0:
msg = f"precision must be > 0, got {precision}"
raise ValueError(msg)
def frontier(
scenario: ScenarioSpec,
party: StrategySpec,
adversary: AdversarySpec,
*,
party_side: str = "players",
criterion: str = "p_tpk",
target: float = 0.2,
min_fraction: float = 0.25,
max_fraction: float = 1.0,
precision: float = 0.05,
runs: int = 100,
seed: int = 1,
) -> float | None:
"""The smallest party HP fraction where `criterion` crosses `target`.
The adversary HP fraction stays at 1.0. Crossing convention (section
3.2): ``value >= target`` for p_tpk, p_any_death and expected_hp_lost;
``value <= target`` for p_survives. Monotone danger (less party HP is
strictly worse): a crossing at ``min_fraction`` already yields the
smallest probed fraction, ceiled to the ``precision`` grid; otherwise
bisection converges to the smallest crossing fraction and returns
``ceil(hi / precision) * precision`` (ceiling keeps the fraction on
the crossing side of the threshold). None when not even
``max_fraction`` crosses. Probabilities require ``0 < target < 1``;
expected_hp_lost requires ``target >= 0``.
"""
_validate_frontier(criterion, target, min_fraction, max_fraction, precision)
def _crossed(fraction: float) -> bool:
cell = matchup_benchmark(
_side_variant(
scenario,
party_side=party_side,
party_hp_fraction=fraction,
adversary_hp_fraction=1.0,
),
party,
adversary,
party_side=party_side,
runs=runs,
seed=seed,
)
value = _danger_value(cell.party_danger, criterion)
if criterion == "p_survives":
return value <= target
return value >= target
if _crossed(min_fraction):
return math.ceil(min_fraction / precision) * precision
if not _crossed(max_fraction):
return None
lo = min_fraction
hi = max_fraction
while hi - lo > precision:
mid = (lo + hi) / 2.0
if _crossed(mid):
hi = mid
else:
lo = mid
return math.ceil(hi / precision) * precision
@dataclass(frozen=True)
class SensitivityEntry:
"""One adversary lever raised from the baseline, with its danger delta."""
levers: str
spec: AdversarySpec
benchmark: MatchupBenchmark
delta: float
def sensitivity(
scenario: ScenarioSpec,
party: StrategySpec,
*,
party_side: str = "players",
criterion: str = "expected_hp_lost",
runs: int = 100,
seed: int = 1,
max_total_runs: int | None = None,
) -> tuple[SensitivityEntry, ...]:
"""Rank which single adversary lever moves the party danger the most.
Baseline is the all-default ``AdversarySpec()``; each of the 7
candidate levers varies exactly one knob (target_priority x3,
spell_policy=none, retreat x2, use_charges=false). ``delta`` is the
signed change of the party ``criterion`` value vs the baseline
(positive = more dangerous). Sorted by |delta| descending, ties by
lever label ascending. Budget is exactly 8 x runs (baseline + 7
levers), enforced before any battle.
"""
_check_criterion(criterion)
cost = 8 * runs
if max_total_runs is not None and cost > max_total_runs:
msg = (
f"robustness budget exceeded: {cost} runs > "
f"{max_total_runs} max_total_runs"
)
raise ValueError(msg)
levers = (
AdversarySpec(target_priority="weakest_ac"),
AdversarySpec(target_priority="lowest_hp"),
AdversarySpec(target_priority="healer"),
AdversarySpec(spell_policy="none"),
AdversarySpec(retreat_hp_fraction=0.25),
AdversarySpec(retreat_hp_fraction=0.5),
AdversarySpec(use_charges=False),
)
baseline = matchup_benchmark(
scenario,
party,
AdversarySpec(),
party_side=party_side,
runs=runs,
seed=seed,
)
base = _danger_value(baseline.party_danger, criterion)
entries: list[SensitivityEntry] = []
for spec in levers:
benchmark = matchup_benchmark(
scenario, party, spec, party_side=party_side, runs=runs, seed=seed
)
entries.append(
SensitivityEntry(
levers=adversary_name(spec),
spec=spec,
benchmark=benchmark,
delta=_danger_value(benchmark.party_danger, criterion) - base,
)
)
entries.sort(key=lambda entry: (-abs(entry.delta), entry.levers))
return tuple(entries)
def _render_line(values: Sequence[str], widths: Sequence[int]) -> str:
"""One aligned row: every cell ljust-padded to its column width."""
return " ".join(value.ljust(width) for value, width in zip(values, widths, strict=True))
def _render_table(headers: Sequence[str], rows: Sequence[Sequence[str]]) -> str:
"""A plain-text grid: header, a full-width dash separator, data rows.
Column widths are the max width over header and rows (same algorithm
as ``sweep.format_sweep``), so every line has identical length and the
separator spans the whole header line.
"""
columns: tuple[Sequence[str], ...] = (headers, *rows)
widths = tuple(
max(len(row[column]) for row in columns) for column in range(len(headers))
)
header = _render_line(headers, widths)
lines = [header, "-" * len(header)]
lines.extend(_render_line(row, widths) for row in rows)
return "\n".join(lines)
def format_matchup(benchmark: MatchupBenchmark) -> str:
"""Render one matchup: per-side win rate, danger profile and rounds."""
headers = (
"Scenario",
"Runs",
"Side",
"Win%",
"Survive",
"TPK",
">=1-death",
"HP-lost",
"Rounds",
)
adversary_side = next(
side for side in benchmark.report.wins if side != benchmark.party_side
)
party_row = (
benchmark.scenario,
str(benchmark.runs),
"party",
f"{benchmark.report.win_rate(benchmark.party_side):.0%}",
f"{benchmark.party_danger.p_survives:.0%}",
f"{benchmark.party_danger.p_tpk:.0%}",
f"{benchmark.party_danger.p_any_death:.0%}",
f"{benchmark.party_danger.expected_hp_lost:.1f}",
f"{benchmark.report.avg_rounds:.1f}",
)
adversary_row = (
"",
"",
"adversary",
f"{benchmark.report.win_rate(adversary_side):.0%}",
f"{benchmark.adversary_danger.p_survives:.0%}",
f"{benchmark.adversary_danger.p_tpk:.0%}",
f"{benchmark.adversary_danger.p_any_death:.0%}",
f"{benchmark.adversary_danger.expected_hp_lost:.1f}",
f"{benchmark.report.avg_rounds:.1f}",
)
return _render_table(headers, (party_row, adversary_row))
def format_robustness(entries: Sequence[RobustnessEntry]) -> str:
"""Render the alternating min-max log, one aligned line per turn."""
headers = (
"Iter",
"Party",
"Adversary",
"P(party-survive)",
"P(party-TPK)",
"Party-HP",
"P(adv-survive)",
"P(adv-TPK)",
"Adv-HP",
)
rows = tuple(
(
str(entry.iteration),
entry.party.name,
entry.adversary.name,
f"{entry.benchmark.party_danger.p_survives:.0%}",
f"{entry.benchmark.party_danger.p_tpk:.0%}",
f"{entry.benchmark.party_danger.expected_hp_lost:.1f}",
f"{entry.benchmark.adversary_danger.p_survives:.0%}",
f"{entry.benchmark.adversary_danger.p_tpk:.0%}",
f"{entry.benchmark.adversary_danger.expected_hp_lost:.1f}",
)
for entry in entries
)
return _render_table(headers, rows)
def format_danger_map(cells: Sequence[DangerCell]) -> str:
"""Render the danger grid: rows = party HP, columns = adversary HP.
Each cell shows party survive% / adversary survive%, both as 0%
percentages (order preserved from ``danger_map``, hence the
dict.fromkeys dedupe instead of a set).
"""
party_fractions: tuple[float, ...] = tuple(
dict.fromkeys(cell.party_hp_fraction for cell in cells)
)
adversary_fractions: tuple[float, ...] = tuple(
dict.fromkeys(cell.adversary_hp_fraction for cell in cells)
)
by_cell: dict[tuple[float, float], DangerCell] = {
(cell.party_hp_fraction, cell.adversary_hp_fraction): cell for cell in cells
}
headers = (
"Party\\Adv",
*(f"adv={fraction}" for fraction in adversary_fractions),
)
rows: list[tuple[str, ...]] = []
for fraction in party_fractions:
row: list[str] = [f"party={fraction}"]
for adv in adversary_fractions:
cell = by_cell[(fraction, adv)]
row.append(
f"{cell.benchmark.party_danger.p_survives:.0%}/"
f"{cell.benchmark.adversary_danger.p_survives:.0%}"
)
rows.append(tuple(row))
return _render_table(headers, rows)
def format_frontier(
criterion: str,
target: float,
fraction: float | None,
*,
min_fraction: float,
max_fraction: float,
) -> str:
"""Render a frontier result (None -> "no frontier" message)."""
if fraction is None:
return (
f"no frontier for {criterion} {target} "
f"in [{min_fraction:.2f}, {max_fraction:.2f}]"
)
return (
f"frontier {criterion} {target}: "
f"party hp_fraction = {fraction:.2f}"
)
def _format_criterion_value(danger: DangerMetrics, criterion: str) -> str:
"""One danger criterion as text: 0% for probabilities, 1 decimal for HP."""
value = _danger_value(danger, criterion)
if criterion == "expected_hp_lost":
return f"{value:.1f}"
return f"{value:.0%}"
def format_sensitivity(entries: Sequence[SensitivityEntry], criterion: str) -> str:
"""Render the lever ranking: rank, levers, party criterion value, delta."""
headers = ("Rank", "Levers", "Value", "Delta")
probability = criterion != "expected_hp_lost"
rows = tuple(
(
str(index),
entry.levers,
_format_criterion_value(entry.benchmark.party_danger, criterion),
f"{entry.delta:+.0%}" if probability else f"{entry.delta:+.1f}",
)
for index, entry in enumerate(entries, start=1)
)
return _render_table(headers, rows)
+170 -26
View File
@@ -10,10 +10,12 @@ attrition averages. All statistical helpers live in metrics.py.
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Literal, overload
if TYPE_CHECKING:
from pf1e_simulator.combat import Policy
from collections.abc import Callable
from pf1e_simulator.combat import LegacyLLMPolicy, LLMPolicy, Policy
from pf1e_simulator.grid import Grid
from pf1e_simulator.map import MapSpec
from pf1e_simulator.models import Combatant
@@ -24,6 +26,7 @@ from pf1e_simulator.combat import (
CombatantState,
CombatantStats,
CombatEngine,
CombatResult,
)
from pf1e_simulator.grid import Grid
from pf1e_simulator.map import Pos, zone_cells
@@ -61,6 +64,39 @@ class CombatantAttrition:
healing_done: float
@dataclass(frozen=True)
class CombatantOutcome:
"""Final state of one combatant at the end of one battle (per-run record).
``dead`` is the engine's own death condition (``CombatantState.dead``:
hp at or below the combatant's death threshold - CON for living PCs, 0
for undead/constructs and under the simple monster-death rule). A dying
PC (hp < 0 but above -CON) is NOT dead here. ``hp`` may be negative;
``hp_max`` is carried so objectives can compute hit points lost.
"""
id: str
side: str
hp: int
hp_max: int
dead: bool
@dataclass(frozen=True)
class RunOutcome:
"""Minimal per-battle outcome for the danger/optimization objectives.
Collected only when ``run_encounter`` is called with
``collect_runs=True``; the aggregate ``EncounterReport`` stays the
default (cheap) output. ``combatants`` covers every creature that
existed at battle end, including summons.
"""
winner: str | None
rounds: int
combatants: tuple[CombatantOutcome, ...]
@dataclass(frozen=True)
class EncounterReport:
"""Aggregated results of a Monte Carlo encounter.
@@ -169,16 +205,113 @@ def build_states(spec: EncounterSpec) -> list[CombatantState]:
return states
def _accumulate_stats(totals: dict[str, CombatantStats], result: CombatResult) -> None:
"""Add one battle's per-combatant stats into the running totals, in place.
Accumulate whatever ids the engine reports: summoned creatures act in the
battle and appear in result.stats, so seeding from the initial states
only would KeyError on them. Ids seen for the first time get a fresh
accumulator. Duplicate summons of the same creature are disambiguated by
the engine the same way build_states does (celestial_wolf,
celestial_wolf-2, ...), so each summoned creature owns its own slot and
their attrition is split, not merged.
"""
for cid, stats in result.stats.items():
current = totals.get(cid)
if current is None:
totals[cid] = stats
continue
totals[cid] = CombatantStats(
hits=current.hits + stats.hits,
crits=current.crits + stats.crits,
damage_dealt=current.damage_dealt + stats.damage_dealt,
damage_taken=current.damage_taken + stats.damage_taken,
healing_done=current.healing_done + stats.healing_done,
)
def _run_outcome(engine: CombatEngine, result: CombatResult) -> RunOutcome:
"""One battle's per-run summary, read from the engine's final live states."""
return RunOutcome(
winner=result.winner,
rounds=result.rounds,
combatants=tuple(
CombatantOutcome(
id=state.combatant.id,
side=state.side,
hp=state.hp,
hp_max=state.combatant.hp_max,
dead=state.dead,
)
for state in engine.states
),
)
def _check_policy_args(
policy: Policy | None,
llm_policy: LLMPolicy | LegacyLLMPolicy | None,
policy_factory: Callable[[int], Policy | None] | None,
) -> None:
"""Reject conflicting policy entry points (P0.4).
``policy_factory`` owns the per-run policy slot, so it cannot be combined
with ``policy`` or ``llm_policy``. ``policy`` + ``llm_policy`` is left to
``CombatEngine`` (single source of truth for the mutual-exclusion error).
"""
if policy_factory is not None and policy is not None:
msg = "policy_factory and policy are mutually exclusive"
raise ValueError(msg)
if policy_factory is not None and llm_policy is not None:
msg = "policy_factory and llm_policy are mutually exclusive"
raise ValueError(msg)
@overload
def run_encounter(
spec: EncounterSpec,
runs: int,
seed: int,
*,
policy: Policy | None = None,
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
policy_factory: Callable[[int], Policy | None] | None = None,
spell_registry: dict[str, SpellSpec] | None = None,
capture_first_transcript: bool = False,
monsters_die_at_zero: bool = False,
) -> EncounterReport:
collect_runs: Literal[False] = False,
) -> EncounterReport: ...
@overload
def run_encounter(
spec: EncounterSpec,
runs: int,
seed: int,
*,
policy: Policy | None = None,
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
policy_factory: Callable[[int], Policy | None] | None = None,
spell_registry: dict[str, SpellSpec] | None = None,
capture_first_transcript: bool = False,
monsters_die_at_zero: bool = False,
collect_runs: Literal[True],
) -> tuple[EncounterReport, tuple[RunOutcome, ...]]: ...
def run_encounter(
spec: EncounterSpec,
runs: int,
seed: int,
*,
policy: Policy | None = None,
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
policy_factory: Callable[[int], Policy | None] | None = None,
spell_registry: dict[str, SpellSpec] | None = None,
capture_first_transcript: bool = False,
monsters_die_at_zero: bool = False,
collect_runs: bool = False,
) -> EncounterReport | tuple[EncounterReport, tuple[RunOutcome, ...]]:
"""Run `runs` battles (one SeededRng per run) and aggregate balance metrics.
The spell registry is loaded once (from ``data/spells/`` unless overridden)
@@ -190,10 +323,30 @@ def run_encounter(
``monsters_die_at_zero`` enables the optional simple monster-death rule
(CLI ``--simple-monster-death``): monsters die at hp <= 0; PCs keep the
PF1e hp <= -CON rule. Default False keeps behavior bit-identical.
``collect_runs`` (Phase 2, danger cartography) additionally returns a
tuple ``(report, run_outcomes)`` where ``run_outcomes`` holds one
``RunOutcome`` per battle (winner, rounds, final per-combatant
id/side/hp/dead). Default False keeps the signature and behavior of
every existing caller unchanged (the report alone).
``llm_policy`` (Phase 2, P0.4) forwards an observation-only (or legacy
two-argument) policy straight to every ``CombatEngine``. It is mutually
exclusive with ``policy``; the engine raises ``ValueError`` when both are
given, so that conflict surfaces here.
``policy_factory`` (Phase 2, P0.4) builds the engine-style ``Policy`` per
run: it is called once per battle with the run index (``0..runs-1``) and
its result is passed as that run's ``policy`` (``None`` falls back to
``default_policy``). It is mutually exclusive with both ``policy`` and
``llm_policy``. Note the per-run callable is a ``Policy`` (it receives the
live engine), not an ``LLMPolicy``; use ``llm_policy`` for the
observation-only entry point.
"""
if runs <= 0:
msg = "runs must be positive"
raise ValueError(msg)
_check_policy_args(policy, llm_policy, policy_factory)
grid = Grid.from_spec(spec.map)
states = build_states(spec)
if spell_registry is None:
@@ -201,28 +354,27 @@ def run_encounter(
wins = {side.name: 0 for side in spec.sides}
draws = 0
total_rounds = 0
# Accumulate whatever ids the engine reports: summoned creatures act in the
# battle and appear in result.stats, so seeding from the initial states
# only would KeyError on them. Ids seen for the first time get a fresh
# accumulator. Duplicate summons of the same creature are disambiguated by
# the engine the same way build_states does (celestial_wolf,
# celestial_wolf-2, ...), so each summoned creature owns its own slot and
# their attrition is split, not merged.
totals: dict[str, CombatantStats] = {}
transcript: tuple[str, ...] | None = None
outcomes: list[RunOutcome] = []
for index in range(runs):
if index > 0:
states = build_states(spec)
rng = SeededRng(seed + index)
result = CombatEngine(
run_policy = policy if policy_factory is None else policy_factory(index)
engine = CombatEngine(
rng,
grid,
states,
round_cap=spec.round_cap,
policy=policy,
policy=run_policy,
llm_policy=llm_policy,
spell_registry=spell_registry,
monsters_die_at_zero=monsters_die_at_zero,
).run()
)
result = engine.run()
if collect_runs:
outcomes.append(_run_outcome(engine, result))
if capture_first_transcript and index == 0:
transcript = result.transcript
if result.winner is None:
@@ -230,18 +382,7 @@ def run_encounter(
else:
wins[result.winner] = wins.get(result.winner, 0) + 1
total_rounds += result.rounds
for cid, stats in result.stats.items():
current = totals.get(cid)
if current is None:
totals[cid] = stats
continue
totals[cid] = CombatantStats(
hits=current.hits + stats.hits,
crits=current.crits + stats.crits,
damage_dealt=current.damage_dealt + stats.damage_dealt,
damage_taken=current.damage_taken + stats.damage_taken,
healing_done=current.healing_done + stats.healing_done,
)
_accumulate_stats(totals, result)
attrition = {
cid: CombatantAttrition(
hits=stats.hits / runs,
@@ -252,7 +393,7 @@ def run_encounter(
)
for cid, stats in totals.items()
}
return EncounterReport(
report = EncounterReport(
runs=runs,
seed=seed,
wins=wins,
@@ -261,3 +402,6 @@ def run_encounter(
attrition=attrition,
transcript=transcript,
)
if collect_runs:
return report, tuple(outcomes)
return report
+203
View File
@@ -0,0 +1,203 @@
"""Scenario search space and deterministic sampling (docs/phase-2-llm-architecture.md
sections 3.1, 6, 10; P3.1).
``ScenarioSpec`` is the reusable scenario search space of section 6: a map
file, the sides that fill its deployment zones, and a battle round cap. It is
compiled deterministically into a runner ``EncounterSpec`` by
``build_encounter``, which loads the map and every combatant file and applies
each side's initial-HP fraction. ``scenario_variants`` implements the coarse
sampling of section 3.1 ("echantillonnage latin"): deterministic per-side HP
fraction variants of a base scenario, with no RNG.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path
from pf1e_simulator.models import Combatant
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.runner import EncounterSpec, Side
@dataclass(frozen=True)
class SideSpec:
"""One side of a scenario: a deployment zone letter and the combatant
files that fill it, with optional initial-hp scaling."""
name: str # side name, e.g. "players"
zone: str # deployment zone letter, e.g. "A"
combatant_files: tuple[Path, ...] # monster/PJ JSON files, loaded via load_combatant
hp_fraction: float = 1.0 # initial HP as a fraction of hp_max, in (0, 1]
aware: bool | None = None # SRD "Surprise" override; None keeps Combatant.aware (True)
@dataclass(frozen=True)
class ScenarioSpec:
"""A reusable scenario: map file, sides, and a battle round cap (docs
section 6 -- the ScenarioSpec search space). Deterministically compiled
to an EncounterSpec by build_encounter."""
name: str
map_file: Path
sides: tuple[SideSpec, ...]
round_cap: int = 100
def __post_init__(self) -> None:
if not self.name:
msg = f"scenario name must be non-empty, got {self.name!r}"
raise ValueError(msg)
if not self.sides:
msg = "scenario sides must be non-empty"
raise ValueError(msg)
for side in self.sides:
if not side.zone:
msg = (
f"side {side.name!r}: zone must be a single non-empty "
f"string, got {side.zone!r}"
)
raise ValueError(msg)
if not 0.0 < side.hp_fraction <= 1.0:
msg = (
f"side {side.name!r}: hp_fraction must be in (0, 1], "
f"got {side.hp_fraction}"
)
raise ValueError(msg)
if self.round_cap < 1:
msg = f"round_cap must be >= 1, got {self.round_cap}"
raise ValueError(msg)
def _apply_hp_fraction(combatant: Combatant, fraction: float) -> Combatant:
"""Scale initial HP via model_copy, the same pattern as runner.build_states."""
if fraction == 1.0:
return combatant
return combatant.model_copy(
update={"hp_max": max(1, round(combatant.hp_max * fraction))}
)
def _apply_aware(combatant: Combatant, *, aware: bool | None) -> Combatant:
"""Apply a side's SRD "Surprise" awareness override via model_copy.
None keeps the loaded combatant unchanged (Combatant.aware defaults to
True, models.py:416-420). SRD "Surprise": only some combatants aware
means a surprise round precedes round 1; unaware combatants do not act
in it.
"""
if aware is None:
return combatant
return combatant.model_copy(update={"aware": aware})
def build_encounter(scenario: ScenarioSpec) -> EncounterSpec:
"""Compile a ScenarioSpec into a runner EncounterSpec (deterministic).
- load_map(scenario.map_file); for each SideSpec, load each combatant
file with load_combatant, then apply hp_fraction to hp_max via
pydantic model_copy(update={"hp_max": max(1, round(hp_max * fraction))})
(same model_copy pattern as runner.build_states).
- When a side's aware is not None, it is applied to that side's
combatants via model_copy(update={"aware": ...}) at the same point
where hp_fraction is applied (SRD "Surprise" round semantics: only
some aware -> surprise round; Combatant.aware defaults to True, so
aware=None keeps the loaded default and is bit-identical to the
behaviour without the field).
- Duplicate combatant ids are NOT disambiguated here: build_states
(called later by run_encounter) handles that. Just list each loaded
combatant once per occurrence in combatant_files.
- Returns EncounterSpec(map=..., sides=tuple(Side[...]), round_cap=...).
"""
map_spec = load_map(scenario.map_file)
if map_spec.zones is not None and not map_spec.deployment:
msg = (
"map has a zones layer but no deployment section (each zone "
"letter needs a side)"
)
raise ValueError(msg)
for side in scenario.sides:
if not side.combatant_files:
msg = f"side {side.name!r}: at least one combatant file is required"
raise ValueError(msg)
letters = [side.zone for side in scenario.sides]
shared = sorted({letter for letter in letters if letters.count(letter) > 1})
if shared:
msg = (
"sides share zone letters (each side needs its own distinct "
f"zone, or they deploy on top of each other): {shared}"
)
raise ValueError(msg)
if map_spec.deployment:
declared = set(map_spec.deployment.values())
for side in scenario.sides:
if side.zone not in declared:
msg = (
f"side {side.name!r} zone {side.zone!r} not declared in "
"map deployment"
)
raise ValueError(msg)
sides = tuple(
Side(
name=side.name,
combatants=tuple(
_apply_aware(
_apply_hp_fraction(load_combatant(path), side.hp_fraction),
aware=side.aware,
)
for path in side.combatant_files
),
zone=side.zone,
)
for side in scenario.sides
)
return EncounterSpec(map=map_spec, sides=sides, round_cap=scenario.round_cap)
def scenario_variants(
base: ScenarioSpec,
*,
hp_fractions: Sequence[float] = (1.0,),
) -> tuple[ScenarioSpec, ...]:
"""Deterministic variants of `base` over per-side HP fractions.
Returns one ScenarioSpec per hp_fraction in `hp_fractions`, each with
every side's hp_fraction set to that value (scenario names suffixed
" <name> hp=<fraction>", e.g. "goblins-vs-orcs hp=0.5"; the identity
fraction 1.0 keeps the base name, so the default call returns base
itself). Same deps as __post_init__ validation (each fraction in
(0, 1]); empty tuple of fractions -> ValueError. Deterministic given
the same inputs: no RNG.
"""
if not hp_fractions:
msg = "hp_fractions must be non-empty"
raise ValueError(msg)
for fraction in hp_fractions:
if not 0.0 < fraction <= 1.0:
msg = f"hp_fraction must be in (0, 1], got {fraction}"
raise ValueError(msg)
variants: list[ScenarioSpec] = []
for fraction in hp_fractions:
name = base.name if fraction == 1.0 else f"{base.name} hp={fraction}"
variants.append(
ScenarioSpec(
name=name,
map_file=base.map_file,
sides=tuple(
SideSpec(
name=side.name,
zone=side.zone,
combatant_files=side.combatant_files,
hp_fraction=fraction,
)
for side in base.sides
),
round_cap=base.round_cap,
)
)
return tuple(variants)
+318
View File
@@ -0,0 +1,318 @@
"""Strategy search and playbook optimization (docs/phase-2-llm-architecture.md
sections 3.1, 7, 10; P3.2/P3.3).
``evaluate`` measures one strategy against one scenario and returns its
``StrategyBenchmark`` (danger metrics, win counts, and the party's 3-sigma
win-rate band). ``optimize`` searches the strategy space in three stages - a
coarse 24-cell grid over the target/spell/charge/class-ability levers, a
retreat-threshold refinement of the top grid cells, then any LLM
``candidates`` - and returns a ranked ``PlaybookEntry`` tuple.
``strategy_name`` renders a deterministic label for a spec and
``format_playbook`` renders the ranked playbook as aligned ASCII columns.
Deterministic: the only randomness comes from the fixed ``seed`` passed to
the sweep runner.
"""
from __future__ import annotations
from dataclasses import dataclass, replace
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from pf1e_simulator.objectives import DangerMetrics
from pf1e_simulator.runner import EncounterSpec
from pf1e_simulator.scenarios import ScenarioSpec
from pf1e_simulator.sweep import SweepCellResult
from pf1e_simulator.scenarios import build_encounter
from pf1e_simulator.strategy import StrategySpec, compile_strategy
from pf1e_simulator.sweep import SweepCell, SweepRunner
@dataclass(frozen=True)
class StrategyBenchmark:
"""The measured outcome of one strategy on one scenario (P3.2)."""
strategy: StrategySpec
scenario_name: str
danger: DangerMetrics
wins: dict[str, int]
draws: int
avg_rounds: float
win_rate_band: tuple[float, float]
@dataclass(frozen=True)
class PlaybookEntry:
"""One ranked row of an optimized playbook (P3.2/P3.3)."""
rank: int
strategy: StrategySpec
benchmark: StrategyBenchmark
def strategy_name(strategy: StrategySpec) -> str:
"""A deterministic label for `strategy` from its non-default levers.
Levers render in field order (target_priority, spell_policy, use_charges,
use_class_abilities, retreat_hp_fraction), joined with ", "; the `name`
field is never part of the label. An all-default spec is "default".
"""
levers: list[str] = []
if strategy.target_priority != "nearest":
levers.append(strategy.target_priority)
if strategy.spell_policy != "default":
levers.append(f"spell_policy={strategy.spell_policy}")
if not strategy.use_charges:
levers.append("use_charges=false")
if not strategy.use_class_abilities:
levers.append("use_class_abilities=false")
if strategy.retreat_hp_fraction != 0.0:
levers.append(f"retreat={strategy.retreat_hp_fraction}")
if not levers:
return "default"
return ", ".join(levers)
def _grid_specs() -> tuple[StrategySpec, ...]:
"""The coarse 24-cell grid: 3 targets x 2 spell policies x 2 x 2, retreat 0."""
specs: list[StrategySpec] = []
for target_priority in ("nearest", "weakest_ac", "lowest_hp"):
for spell_policy in ("default", "none"):
for use_charges in (True, False):
for use_class_abilities in (True, False):
raw = StrategySpec(
target_priority=target_priority,
spell_policy=spell_policy,
use_charges=use_charges,
use_class_abilities=use_class_abilities,
)
specs.append(replace(raw, name=strategy_name(raw)))
return tuple(specs)
def _refine_specs(top: Sequence[StrategySpec]) -> tuple[StrategySpec, ...]:
"""Retreat-threshold refinements of the top grid specs (3 per spec)."""
refined: list[StrategySpec] = []
for spec in top:
for fraction in (0.25, 0.5, 0.75):
raw = replace(spec, retreat_hp_fraction=fraction)
refined.append(replace(raw, name=strategy_name(raw)))
return tuple(refined)
def _rank_key(
strategy: StrategySpec, danger: DangerMetrics
) -> tuple[float, float, float, float, str]:
"""Sort key: best first (survive high, hp lost low, tail high, rounds low)."""
return (
-danger.p_survives,
danger.expected_hp_lost,
-danger.worst_decile_party_hp,
danger.expected_rounds,
strategy_name(strategy),
)
def _cell(
scenario: ScenarioSpec,
encounter: EncounterSpec,
spec: StrategySpec,
runs: int,
seed: int,
) -> SweepCell:
"""One sweep cell for `spec` under its compiled strategy."""
return SweepCell(
scenario=scenario.name,
strategy=strategy_name(spec),
spec=encounter,
policy=compile_strategy(spec),
runs=runs,
seed=seed,
)
def _cells(
scenario: ScenarioSpec,
encounter: EncounterSpec,
specs: Sequence[StrategySpec],
runs: int,
seed: int,
) -> tuple[SweepCell, ...]:
"""One cell per spec, in spec order."""
return tuple(_cell(scenario, encounter, spec, runs, seed) for spec in specs)
def _benchmark(
strategy: StrategySpec,
scenario_name: str,
result: SweepCellResult,
party: str,
) -> StrategyBenchmark:
"""Build the benchmark for one strategy from its sweep cell result."""
return StrategyBenchmark(
strategy=strategy,
scenario_name=scenario_name,
danger=result.danger,
wins=result.report.wins,
draws=result.report.draws,
avg_rounds=result.report.avg_rounds,
win_rate_band=result.report.win_rate_band(party),
)
def evaluate( # noqa: PLR0913 - pinned Phase-2 API (6 knobs)
strategy: StrategySpec,
scenario: ScenarioSpec,
*,
party: str,
runs: int,
seed: int,
workers: int = 4,
) -> StrategyBenchmark:
"""Measure one strategy on one scenario (P3.2).
Compiles `strategy` and runs a single sweep cell of `runs` seeded battles;
the benchmark carries the party's danger metrics, win counts, and 3-sigma
win-rate band.
"""
encounter = build_encounter(scenario)
runner = SweepRunner(party=party, workers=workers)
(result,) = runner.run((_cell(scenario, encounter, strategy, runs, seed),))
return _benchmark(strategy, scenario.name, result, party)
def optimize( # noqa: PLR0913 - pinned Phase-2 API (7 knobs)
scenario: ScenarioSpec,
*,
party: str,
runs: int,
seed: int,
workers: int = 4,
max_total_runs: int | None = None,
candidates: Sequence[StrategySpec] = (),
) -> tuple[PlaybookEntry, ...]:
"""Search the strategy space and return a ranked playbook (P3.2/P3.3).
Stage 1 sweeps the coarse 24-cell grid (target priority x spell policy x
charges x class abilities, retreat 0); stage 2 refines the top 3 grid
cells over retreat thresholds 0.25/0.5/0.75; stage 3 adds the LLM
`candidates`, deduplicated by field equality against every spec already
run. Entries are ranked by (-p_survives, +expected_hp_lost,
-worst_decile_party_hp, +avg_rounds, strategy_name). `max_total_runs`
caps the total Monte Carlo cost before anything executes (mirrors
SweepRunner.run). The signature is pinned by the Phase-2 API, hence the
PLR0913 noqa.
"""
encounter = build_encounter(scenario)
runner = SweepRunner(party=party, workers=workers)
stage1_specs = _grid_specs()
seen: set[StrategySpec] = set(stage1_specs)
stage3_specs: list[StrategySpec] = []
for spec in candidates:
if spec not in seen:
seen.add(spec)
stage3_specs.append(spec)
total_runs = (len(stage1_specs) + 9 + len(stage3_specs)) * runs
if max_total_runs is not None and total_runs > max_total_runs:
msg = (
f"optimize budget exceeded: {total_runs} runs > "
f"{max_total_runs} max_total_runs"
)
raise ValueError(msg)
stage1_results = runner.run(_cells(scenario, encounter, stage1_specs, runs, seed))
stage1_ranked = sorted(
zip(stage1_specs, stage1_results, strict=True),
key=lambda pair: _rank_key(pair[0], pair[1].danger),
)
top3 = [spec for spec, _ in stage1_ranked[:3]]
stage2_specs = _refine_specs(top3)
stage2_results = runner.run(_cells(scenario, encounter, stage2_specs, runs, seed))
seen = set(stage1_specs) | set(stage2_specs)
stage3_specs = []
for spec in candidates:
if spec not in seen:
seen.add(spec)
stage3_specs.append(spec)
stage3_results = (
runner.run(_cells(scenario, encounter, stage3_specs, runs, seed))
if stage3_specs
else ()
)
pairs = [
*zip(stage1_specs, stage1_results, strict=True),
*zip(stage2_specs, stage2_results, strict=True),
*zip(stage3_specs, stage3_results, strict=True),
]
benchmarks = [
_benchmark(spec, scenario.name, result, party) for spec, result in pairs
]
ordered = sorted(benchmarks, key=lambda b: _rank_key(b.strategy, b.danger))
return tuple(
PlaybookEntry(rank=index, strategy=benchmark.strategy, benchmark=benchmark)
for index, benchmark in enumerate(ordered, start=1)
)
def _win_rate_text(benchmark: StrategyBenchmark, party: str) -> str:
"""The party's win rate with its 3-sigma band, e.g. "93.0 [89.5, 96.5]"."""
rate = benchmark.wins.get(party, 0) / benchmark.danger.runs
low, high = benchmark.win_rate_band
return f"{rate * 100:.1f} [{low * 100:.1f}, {high * 100:.1f}]"
def format_playbook(entries: Sequence[PlaybookEntry], party: str) -> str:
"""Render a ranked playbook as aligned plain-text columns (ASCII only).
One row per entry in order, under a header line and a separator row of
dashes. Percentages and HP values use one decimal; the Win% column shows
the party's win rate with its 3-sigma band. Column widths are computed
from the content; columns are separated by a single space.
"""
headers = (
"Rank",
"Strategy",
"Runs",
"P(survive)",
"P(TPK)",
"P(>=1 death)",
"HP lost",
"Worst-decile HP",
"Rounds",
"Win% (3σ)",
)
rows = [
(
str(entry.rank),
entry.benchmark.strategy.name,
str(entry.benchmark.danger.runs),
f"{entry.benchmark.danger.p_survives:.1%}",
f"{entry.benchmark.danger.p_tpk:.1%}",
f"{entry.benchmark.danger.p_any_death:.1%}",
f"{entry.benchmark.danger.expected_hp_lost:.1f}",
f"{entry.benchmark.danger.worst_decile_party_hp:.1f}",
f"{entry.benchmark.danger.expected_rounds:.1f}",
_win_rate_text(entry.benchmark, party),
)
for entry in entries
]
widths = [
max(len(header), *(len(row[index]) for row in rows))
for index, header in enumerate(headers)
]
def _line(values: Sequence[str]) -> str:
return " ".join(
value.ljust(width) for value, width in zip(values, widths, strict=True)
)
lines = [_line(headers), _line(["-" * width for width in widths])]
lines.extend(_line(row) for row in rows)
return "\n".join(lines)
+193
View File
@@ -0,0 +1,193 @@
"""Etage A strategy compiler (docs/phase-2-llm-architecture.md sections 2 and 10, P2.1).
Deterministic, additive: the engine (combat.py) is never modified. A
``StrategySpec`` is a small set of frozen knobs (target priority, retreat
threshold, spell policy, charge/class-ability toggles) compiled by
``compile_strategy`` into a ``Policy`` callable. The all-default spec compiles
to the module-level ``default_policy`` object itself (identity), so the common
case is bit-identical and zero-cost. Non-default specs mirror
``default_policy``'s decision flow with the spec's values substituted.
``StrategySpec`` feeds the SweepRunner (sweep.py, later batch) and the LLM
NL-to-spec translation (later batch): the optimizer searches over specs, the
LLM produces specs from natural language, and this module is the single
deterministic executor both rely on.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
from pf1e_simulator.combat import (
Action,
CombatantState,
CombatEngine,
Policy,
# The decision helpers below are module-private by convention but stable:
# the engine stays unchanged, and duplicating their spell/hex/channel
# logic here would drift.
_choose_spell_action, # pyright: ignore[reportPrivateUsage]
_emit_channel_energy, # pyright: ignore[reportPrivateUsage]
_emit_hex, # pyright: ignore[reportPrivateUsage]
_spell_combat_actions, # pyright: ignore[reportPrivateUsage]
_swift_bonus_actions, # pyright: ignore[reportPrivateUsage]
default_policy,
)
TargetPriority = Literal["nearest", "weakest_ac", "lowest_hp", "healer"]
SpellPolicy = Literal["default", "none"]
@dataclass(frozen=True)
class StrategySpec:
"""Deterministic strategy knobs compiled to a ``Policy`` (Etage A).
All fields default to the behaviour of ``default_policy``; a spec built
with every default compiles to ``default_policy`` itself (identity).
"""
name: str = "default"
target_priority: TargetPriority = "nearest"
retreat_hp_fraction: float = 0.0 # 0.0 = never retreat
spell_policy: SpellPolicy = "default"
use_charges: bool = True
use_class_abilities: bool = True
def __post_init__(self) -> None:
if not self.name:
msg = "name must be a non-empty string"
raise ValueError(msg)
if not 0.0 <= self.retreat_hp_fraction <= 1.0:
msg = "retreat_hp_fraction must be within [0, 1]"
raise ValueError(msg)
def select_target(
engine: CombatEngine, state: CombatantState, priority: TargetPriority
) -> CombatantState | None:
"""The active enemy selected by ``priority``; None when no enemy is active.
Deterministic: ties break on ``engine.states`` index order.
"""
if priority == "nearest":
return engine.nearest_enemy(state)
enemies = [s for s in engine.states if s.side != state.side and s.active]
if not enemies:
return None
match priority:
case "weakest_ac":
return min(
enemies,
key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)),
)
case "lowest_hp":
return min(enemies, key=lambda s: (s.hp, engine.states.index(s)))
case "healer":
# Prefer the caster/healer (non-empty combatant.spells); among
# several, min by (ac, hp, list order). No active caster on the
# enemy side -> identical to weakest_ac.
casters = [s for s in enemies if s.combatant.spells]
pool = casters or enemies
return min(
pool,
key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)),
)
def _melee_actions(
engine: CombatEngine,
state: CombatantState,
target: CombatantState,
spec: StrategySpec,
) -> tuple[Action, ...]:
"""The strike ladder: full-attack, charge (if enabled), step, or approach."""
if engine.weapon_for(state, target) is not None:
return (Action(kind="full_attack", target_id=target.combatant.id),)
if spec.use_charges:
charge = engine.can_charge(state, target)
if charge is not None and charge.weapon is not None:
return (Action(kind="charge", target_id=target.combatant.id),)
if engine.can_step5_to_attack(state, target):
return (
Action(kind="5foot_step", target_id=target.combatant.id),
Action(kind="full_attack", target_id=target.combatant.id),
)
return (
Action(kind="move", target_id=target.combatant.id),
Action(kind="attack", target_id=target.combatant.id),
)
def _normal_actions(
engine: CombatEngine,
state: CombatantState,
target: CombatantState,
spec: StrategySpec,
) -> tuple[Action, ...]:
"""The spell/melee action sequence (without swift/free class-ability add-ons)."""
if spec.spell_policy != "none" and state.combatant.spells:
spell_action = _choose_spell_action(engine, state, target)
if spell_action is not None:
return spell_action
return _melee_actions(engine, state, target, spec)
def _class_ability_action(
engine: CombatEngine, state: CombatantState, target: CombatantState
) -> tuple[Action, ...] | None:
"""The channel/hex replacement action when a class ability fires."""
channel_mode = _emit_channel_energy(engine, state)
if channel_mode is not None:
return (
Action(
kind="channel_energy",
target_id=target.combatant.id,
channel_mode=channel_mode,
),
)
if _emit_hex(engine, state, target):
return (Action(kind="hex", target_id=target.combatant.id, hex_name="evil_eye"),)
return None
def _compiled_policy(spec: StrategySpec) -> Policy:
"""Build the ``Policy`` that executes ``spec`` (non-default specs only)."""
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
target = select_target(engine, state, spec.target_priority)
if target is None:
return (Action(kind="wait"),)
if engine.has_condition_flag(state, "must_flee") or (
spec.retreat_hp_fraction > 0.0
and state.hp < spec.retreat_hp_fraction * state.combatant.hp_max
):
return (Action(kind="withdraw", target_id=target.combatant.id),)
if spec.use_class_abilities:
class_action = _class_ability_action(engine, state, target)
if class_action is not None:
return class_action
normal = _normal_actions(engine, state, target, spec)
if not spec.use_class_abilities:
return normal
spell_combat = _spell_combat_actions(engine, state, target)
if spell_combat is not None:
return spell_combat
swift_bonus = _swift_bonus_actions(engine, state, target, normal)
return (*swift_bonus, *normal) if swift_bonus else normal
return policy
_DEFAULT_SPEC = StrategySpec()
def compile_strategy(spec: StrategySpec) -> Policy:
"""Compile a ``StrategySpec`` into a ``Policy`` callable.
The all-default spec returns the module-level ``default_policy`` object
itself (identity), so the common case is bit-identical and zero-cost.
"""
if spec == _DEFAULT_SPEC:
return default_policy
return _compiled_policy(spec)
+201
View File
@@ -0,0 +1,201 @@
"""Parallel, cost-bounded sweep runner for Phase-2 danger cartography.
Executes a grid of (scenario x strategy) cells (docs/phase-2-llm-architecture.md
section 10, P2.3; decisions D9/D10): each cell runs ``run_encounter(...,
collect_runs=True)`` for a fixed number of seeded runs under one policy and
reports the aggregate ``EncounterReport`` plus the party's ``DangerMetrics``
(objectives.danger_metrics). Cells are independent, so they run on a thread
pool (D10); the spell registry is loaded once and shared read-only across
threads. The runner is strategy-agnostic: policies are passed in as callables
(``None`` = ``run_encounter``'s own ``default_policy``) and no strategy module
is imported here.
"""
from __future__ import annotations
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
from pf1e_simulator.combat import Policy
from pf1e_simulator.runner import EncounterReport, EncounterSpec
from pf1e_simulator.spells import SpellSpec
from pf1e_simulator.objectives import DangerMetrics, danger_metrics
from pf1e_simulator.runner import run_encounter
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
@dataclass(frozen=True)
class SweepCell:
"""One (scenario, strategy) cell of a sweep: N seeded runs with one policy."""
scenario: str
strategy: str
spec: EncounterSpec
policy: Policy | None # None = default_policy (run_encounter's own default)
runs: int
seed: int
@dataclass(frozen=True)
class SweepCellResult:
"""Aggregate + danger metrics for one cell."""
cell: SweepCell
report: EncounterReport
danger: DangerMetrics
class SweepRunner:
"""Run a grid of (scenario, strategy) cells in parallel, bounded by cost.
``party`` is the side name whose danger is measured (passed to
``danger_metrics``); ``workers`` bounds the thread pool. The spell
registry is loaded once in ``__init__`` and shared read-only across all
cells (reloading it per cell would be wasteful; a read-only dict is safe
to share between threads).
"""
def __init__(self, *, party: str, workers: int = 4) -> None:
if not party:
msg = "party must be a non-empty side name"
raise ValueError(msg)
if workers < 1:
msg = "workers must be >= 1"
raise ValueError(msg)
self._party: str = party
self._workers: int = workers
self._registry: dict[str, SpellSpec] = load_spell_registry(SPELLS_DIR)
def run(
self,
cells: Sequence[SweepCell],
*,
max_total_runs: int | None = None,
) -> tuple[SweepCellResult, ...]:
"""Run every cell on the thread pool and return results in input order.
``max_total_runs`` caps the total Monte Carlo cost (docs section 9,
D9/D14 "Budget - plafonds petits"): when the cells' runs sum exceeds
it, a ``ValueError`` is raised before anything executes. Each cell
must run at least once. Exceptions from a cell propagate loudly - a
failing cell fails the whole sweep.
"""
total_runs = sum(cell.runs for cell in cells)
if max_total_runs is not None and total_runs > max_total_runs:
msg = (
f"sweep budget exceeded: {total_runs} runs > "
f"{max_total_runs} max_total_runs"
)
raise ValueError(msg)
for cell in cells:
if cell.runs < 1:
msg = (
f"cell runs must be >= 1 "
f"(scenario={cell.scenario!r}, strategy={cell.strategy!r})"
)
raise ValueError(msg)
with ThreadPoolExecutor(max_workers=self._workers) as pool:
futures = [pool.submit(self._run_cell, cell) for cell in cells]
results = [future.result() for future in futures]
return tuple(results)
def _run_cell(self, cell: SweepCell) -> SweepCellResult:
"""One cell's work: run the encounter and measure the party's danger."""
report, outcomes = run_encounter(
cell.spec,
cell.runs,
cell.seed,
policy=cell.policy,
spell_registry=self._registry,
collect_runs=True,
)
danger = danger_metrics(outcomes, self._party)
return SweepCellResult(cell=cell, report=report, danger=danger)
def run_grid(
self,
scenarios: Sequence[tuple[str, EncounterSpec]],
strategies: Sequence[tuple[str, Policy | None]],
*,
runs: int,
seed: int,
max_total_runs: int | None = None,
) -> tuple[SweepCellResult, ...]:
"""Build the scenario-major (scenario x strategy) grid and run it.
Cells are built in SCENARIO-major order (for each scenario, for each
strategy - mirrors the danger-map table of docs section 16), every
cell sharing the same ``runs`` and ``seed``. Names come from the
tuples; a ``None`` policy stays ``None`` (default_policy).
"""
cells = [
SweepCell(
scenario=scenario_name,
strategy=strategy_name,
spec=spec,
policy=policy,
runs=runs,
seed=seed,
)
for scenario_name, spec in scenarios
for strategy_name, policy in strategies
]
return self.run(cells, max_total_runs=max_total_runs)
def format_sweep(results: Sequence[SweepCellResult], party: str) -> str:
"""Render a sweep as aligned plain-text columns (ASCII only).
One row per result, in input order, under a header line and a separator
row of dashes. ``party`` is the side whose danger was measured: the Win%
column shows that side's win rate (``report.wins.get(party, 0) /
report.runs``) as a percentage with one decimal. Column widths are
computed from the content; columns are separated by a single space.
"""
headers = (
"Scenario",
"Strategy",
"Runs",
"Win%",
"Draws",
"Rounds",
"P(survive)",
"P(TPK)",
"P(>=1 death)",
"HP lost",
"Worst-decile HP",
)
rows = [
(
result.cell.scenario,
result.cell.strategy,
str(result.cell.runs),
f"{result.report.wins.get(party, 0) / result.report.runs * 100:.1f}",
str(result.report.draws),
f"{result.report.avg_rounds:.1f}",
f"{result.danger.p_survives:.1%}",
f"{result.danger.p_tpk:.1%}",
f"{result.danger.p_any_death:.1%}",
f"{result.danger.expected_hp_lost:.1f}",
f"{result.danger.worst_decile_party_hp:.1f}",
)
for result in results
]
widths = [
max(len(header), *(len(row[index]) for row in rows))
for index, header in enumerate(headers)
]
def _line(values: Sequence[str]) -> str:
return " ".join(
value.ljust(width) for value, width in zip(values, widths, strict=True)
)
lines = [_line(headers), _line(["-" * width for width in widths])]
lines.extend(_line(row) for row in rows)
return "\n".join(lines)
+5
View File
@@ -0,0 +1,5 @@
You are the tactical director for a Pathfinder 1e battle.
You see the current battle state as a compact observation view, followed by
the list of plausible actions the acting combatant could take.
Respond with the JSON action sequence described in the schema.
+1
View File
@@ -0,0 +1 @@
0.1.0-test
@@ -0,0 +1,6 @@
[
{
"observation": {"round": 1, "combatant": "goblin", "enemy": "orc", "distance_ft": 10},
"actions": [{"kind": "attack", "target_id": "orc"}]
}
]
+5
View File
@@ -0,0 +1,5 @@
You are the tactical director for a Pathfinder 1e battle.
You see the current battle state as a compact observation view, followed by
the list of plausible actions the acting combatant could take.
Respond with the JSON action sequence described in the schema.
+1
View File
@@ -0,0 +1 @@
0.1.0-test
+1 -1
View File
@@ -229,7 +229,7 @@ def test_results_visible_in_next_turn_observation() -> None:
# the second invocation must already see the first turn's rejection.
seen: list[Observation] = []
def llm_policy(obs: Observation, state: CombatantState) -> tuple[Action, ...]:
def llm_policy(obs: Observation) -> tuple[Action, ...]:
seen.append(obs)
if len(seen) == 1:
return (Action(kind="move"),)
+2 -2
View File
@@ -1,7 +1,7 @@
"""Package U8 — final validation: an observation-only LLM policy drives the
Phase-2 capabilities (U1-U7b) end-to-end through the real engine.
Each test wires a ``llm_policy`` (``LLMPolicy(observation, state)``) that
Each test wires a ``llm_policy`` (``LLMPolicy(observation)``) that
reads ONLY the frozen Observation snapshot — never ``engine.*`` — and emits
one newly-unblocked tactical decision per capability: ``ready`` (U4),
``delay`` (U3), ``total_defense`` (U5a), ``move`` with an explicit
@@ -121,7 +121,7 @@ def _counting(decide: Decide) -> LLMPolicy:
"""LLMPolicy calling ``decide(id, call_no, obs)`` per turn (obs only)."""
calls: dict[str, int] = {}
def policy(obs: Observation, _state: CombatantState) -> tuple[Action, ...]:
def policy(obs: Observation) -> tuple[Action, ...]:
cid = str(_me(obs)["id"])
n = calls.get(cid, 0)
calls[cid] = n + 1
+105
View File
@@ -0,0 +1,105 @@
"""Unit tests for the Phase-2 objective primitives (objectives.py).
Synthetic RunOutcome tuples with hand-computed expectations: the metrics
are pure frequencies/means over per-run summaries, so every number below is
verifiable by eye. No engine, no RNG.
"""
from __future__ import annotations
from pf1e_simulator.objectives import (
danger_metrics,
expected_hp_lost,
expected_rounds,
p_any_death,
p_survives,
p_tpk,
worst_decile_party_hp,
)
from pf1e_simulator.runner import CombatantOutcome, RunOutcome
def _pc(cid: str, hp: int, hp_max: int = 10) -> CombatantOutcome:
dead = hp <= -12
return CombatantOutcome(id=cid, side="players", hp=hp, hp_max=hp_max, dead=dead)
def _mob(cid: str, hp: int, hp_max: int = 10) -> CombatantOutcome:
dead = hp <= -12
return CombatantOutcome(id=cid, side="monsters", hp=hp, hp_max=hp_max, dead=dead)
def _run(rounds: int, *combatants: CombatantOutcome) -> RunOutcome:
return RunOutcome(winner=None, rounds=rounds, combatants=combatants)
def test_empty_batch_returns_zero() -> None:
assert p_survives((), "players") == 0.0
assert p_tpk((), "players") == 0.0
assert p_any_death((), "players") == 0.0
assert expected_hp_lost((), "players") == 0.0
assert expected_rounds(()) == 0.0
assert worst_decile_party_hp((), "players") == 0.0
def test_survival_tpk_and_death_rates() -> None:
# run 1: both players clean (hp 10/10, 8/10) -> survives, no death
# run 2: one player dying (hp -1, NOT dead at -CON 12), one at 3 hp
# -> survives (nobody dead), no death counted
# run 3: one player dead (hp -12), one at 0 hp -> any death, not TPK
# run 4: both dead (hp -13, -12) -> TPK
runs = (
_run(2, _pc("a", 10), _pc("b", 8), _mob("m", 5)),
_run(3, _pc("a", -1), _pc("b", 3)),
_run(4, _pc("a", -12), _pc("b", 0)),
_run(5, _pc("a", -13), _pc("b", -12)),
)
assert p_survives(runs, "players") == 2 / 4
assert p_any_death(runs, "players") == 2 / 4
assert p_tpk(runs, "players") == 1 / 4
# monsters absent from a run: vacuously survived, never TPK
assert p_survives(runs, "monsters") == 1.0
assert p_tpk(runs, "monsters") == 0.0
def test_expected_hp_lost_floors_per_member() -> None:
# run 1: a lost 2, b lost 7 (total 9). Run 2: a at -12 counts as the
# full 10 lost (clamped), b at 10 lost nothing. Mean of the two totals
# is 9.5.
runs = (
_run(1, _pc("a", 8), _pc("b", 3)),
_run(2, _pc("a", -12), _pc("b", 10)),
)
assert expected_hp_lost(runs, "players") == 9.5
assert expected_rounds(runs) == 1.5
def test_worst_decile_takes_the_tail_not_the_mean() -> None:
# 20 runs: 19 clean (hp lost 0) + 1 disaster (both dead, 20 lost).
# Mean hp lost = 1.0 but the worst decile (2 runs: disaster + one clean)
# averages 10 hp remaining - the tail is visible.
clean = _run(1, _pc("a", 10), _pc("b", 10))
disaster = _run(9, _pc("a", -12), _pc("b", -13))
runs = (disaster, *([clean] * 19))
assert expected_hp_lost(runs, "players") == 1.0
assert worst_decile_party_hp(runs, "players") == (0 + 20) / 2
def test_worst_decile_minimum_one_run() -> None:
runs = (_run(1, _pc("a", 4)),)
assert worst_decile_party_hp(runs, "players") == 4
def test_danger_metrics_aggregates_every_primitive() -> None:
runs = (
_run(2, _pc("a", 10), _mob("m", 0)),
_run(4, _pc("a", -12), _mob("m", -13)),
)
m = danger_metrics(runs, "players")
assert m.runs == 2
assert m.p_survives == 0.5
assert m.p_tpk == 0.5
assert m.p_any_death == 0.5
assert m.expected_hp_lost == 5.0
assert m.expected_rounds == 3.0
assert m.worst_decile_party_hp == 0.0
+73 -5
View File
@@ -9,9 +9,12 @@ Covers the pinned schema:
- Terrain/elevation layers.
- Frozen copies: mutating live state after construction does not change the
already-built Observation.
- ``to_dict()`` is JSON-serializable and exposes no ``engine.*`` callables.
- ``to_dict()`` is JSON-serializable, carries the pinned ``schema_version``
key, and exposes no ``engine.*`` callables.
- The ``llm_policy`` adapter builds the Observation at the point of policy
invocation and delegates to ``llm_policy(observation, state)``.
invocation and delegates to ``llm_policy(observation)`` (D1: observation
only); a legacy ``llm_policy(observation, state)`` callable keeps working
through the compatibility adapter.
"""
from __future__ import annotations
@@ -20,7 +23,13 @@ import json
import pytest
from pf1e_simulator.combat import Action, CombatantState, CombatEngine, Observation
from pf1e_simulator.combat import (
OBSERVATION_SCHEMA_VERSION,
Action,
CombatantState,
CombatEngine,
Observation,
)
from pf1e_simulator.conditions import SHAKEN
from pf1e_simulator.dice import parse_dice
from pf1e_simulator.effects import ActiveCondition
@@ -441,11 +450,11 @@ def test_observation_frozen_copies() -> None:
def test_llm_policy_wrapper_e2e() -> None:
"""Given: a CombatEngine with an llm_policy that reads ONLY the observation
"""Given: a CombatEngine with an observation-only llm_policy (D1 signature)
When: the battle runs
Then: the policy's decisions are honored and the transcript is coherent."""
def llm_policy(obs: Observation, state: CombatantState) -> tuple:
def llm_policy(obs: Observation) -> tuple:
# Read only the observation: full_attack if the target is adjacent,
# otherwise move toward it.
target_id = obs.combatants[obs.pairs[0]["target_index"]]["id"]
@@ -470,6 +479,65 @@ def test_llm_policy_wrapper_e2e() -> None:
assert any("move" in line for line in result.transcript)
def test_legacy_two_arg_llm_policy_still_works() -> None:
"""Given: a pre-D1 llm_policy taking (observation, state)
When: the battle runs through the compatibility adapter
Then: the policy receives BOTH arguments and its decisions are honored."""
seen_states: list[CombatantState] = []
def legacy_policy(obs: Observation, state: CombatantState) -> tuple:
seen_states.append(state)
assert state.combatant.id == obs.combatants[obs.acting_combatant]["id"]
target_id = obs.combatants[obs.pairs[0]["target_index"]]["id"]
return (Action(kind="move", target_id=target_id),)
hero = make_combatant("hero")
foe = make_combatant("foe")
hero_state = make_state(hero, "players", (0, 0))
foe_state = make_state(foe, "monsters", (0, 3))
engine = CombatEngine(
ScriptedRng([10, 9, 8, 15, 3, 5]),
make_grid(),
[hero_state, foe_state],
round_cap=1,
llm_policy=legacy_policy,
)
result = engine.run()
assert seen_states, "legacy policy was never invoked"
assert any("move" in line for line in result.transcript)
def test_observation_schema_version_pinned() -> None:
"""Given: a built Observation
When: to_dict() is called
Then: the top-level key set is exactly the pinned keys and
schema_version == OBSERVATION_SCHEMA_VERSION, JSON-serializable."""
hero = make_combatant("hero")
foe = make_combatant("foe")
hero_state = make_state(hero, "players", (0, 0))
foe_state = make_state(foe, "monsters", (0, 3))
engine = make_engine([hero_state, foe_state])
data = engine._build_observation(hero_state).to_dict()
assert data["schema_version"] == OBSERVATION_SCHEMA_VERSION
assert set(data) == {
"schema_version",
"round",
"acting_combatant",
"combatants",
"pairs",
"terrain",
"elevation",
"threatened",
"ranged_threatened",
"terrain_semantics",
"condition_definitions",
"initiative_order",
}
json.dumps(data) # must not raise
# ---------------------------------------------------------------------------
# 12. policy + llm_policy → ValueError
# ---------------------------------------------------------------------------
+4 -2
View File
@@ -1051,7 +1051,8 @@ def test_observation_enriched_terrain_semantics_no_callable_no_share() -> None:
def test_observation_enriched_e5_existing_keys_preserved() -> None:
"""Given: an Observation built for the L5 fields
When: the top-level dict is inspected
Then: all pre-existing top-level keys remain (round, acting_combatant,
Then: all pre-existing top-level keys remain (schema_version, round,
acting_combatant,
combatants, pairs, terrain, elevation) and the new keys
(threatened, ranged_threatened, terrain_semantics,
condition_definitions, initiative_order) are added — exact key set."""
@@ -1063,6 +1064,7 @@ def test_observation_enriched_e5_existing_keys_preserved() -> None:
obs = _snapshot(engine, hero_state)
assert set(obs) == {
"schema_version",
"round",
"acting_combatant",
"combatants",
@@ -1138,7 +1140,7 @@ def _best_usable_attack(obs: Observation) -> dict[str, object] | None:
return best
def _enriched_llm_policy(obs: Observation, state: CombatantState) -> tuple[Action, ...]:
def _enriched_llm_policy(obs: Observation) -> tuple[Action, ...]:
"""Read ONLY the observation — never state.combatant.* nor engine.*."""
cast = _castable_spell_action(obs)
if cast is not None:
+249
View File
@@ -0,0 +1,249 @@
"""Phase-2 adversary compilation tests (docs/phase-2-llm-architecture.md 10 P4.1).
Deterministic, offline: no network, no LLM. Pins the ``AdversarySpec`` /
``compile_adversary`` / ``compile_matchup`` contract (default-policy identity,
per-side routing on a real ``CombatEngine``, empty ``party_side`` rejection),
the additive ``"healer"`` target priority (prefer the enemy caster, fall back
to ``weakest_ac``), and ``SideSpec.aware`` landing on the right side's
combatants plus its SRD "Surprise" round effect. Conventions follow
tests/test_phase2_search.py: small n (runs <= 5), fixed seed, ROOT-relative
path constants.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.adversary import (
AdversarySpec,
adversary_name,
compile_adversary,
compile_matchup,
)
from pf1e_simulator.combat import CombatantState, CombatEngine, default_policy
from pf1e_simulator.grid import Grid
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import build_states, run_encounter
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec, build_encounter
from pf1e_simulator.strategy import StrategySpec, compile_strategy, select_target
if TYPE_CHECKING:
from pf1e_simulator.models import Combatant
from pf1e_simulator.runner import EncounterSpec
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
ORC = ROOT / "data" / "monsters" / "orc.json"
NAIRDA = ROOT / "data" / "pcs" / "nairda.json"
RUNS = 5
SEED = 7
ROUND_CAP = 20
PARTY = "players"
_MAP_SPEC = load_map(MAP)
def _c(path: Path) -> Combatant:
return load_combatant(path)
def _scenario(*, monsters_aware: bool | None = None) -> ScenarioSpec:
"""3 goblins (zone A) vs 2 orcs (zone B), optionally with side awareness."""
return ScenarioSpec(
name="goblins-vs-orcs",
map_file=MAP,
sides=(
SideSpec(
name=PARTY,
zone="A",
combatant_files=(GOBLIN, GOBLIN, GOBLIN),
),
SideSpec(
name="monsters",
zone="B",
combatant_files=(ORC, ORC),
aware=monsters_aware,
),
),
round_cap=ROUND_CAP,
)
def _encounter() -> EncounterSpec:
return build_encounter(_scenario())
def _state(
combatant: Combatant, side: str, pos: tuple[int, int], *, hp: int | None = None
) -> CombatantState:
"""A bare runtime state over a loaded combatant (no battle run)."""
return CombatantState(
combatant=combatant,
side=side,
pos=pos,
hp=combatant.hp_max if hp is None else hp,
resources_left={},
)
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
"""A live engine over the sample arena with the given states (no battle run)."""
return CombatEngine(SeededRng(SEED), Grid.from_spec(_MAP_SPEC), states)
# --- identity ---------------------------------------------------------------
def test_compile_adversary_default_is_default_policy() -> None:
"""The all-default AdversarySpec compiles to default_policy by identity."""
assert compile_adversary(AdversarySpec()) is default_policy
def test_compile_matchup_defaults_is_default_policy() -> None:
"""Two all-default specs return default_policy itself, not a wrapper."""
assert compile_matchup(StrategySpec(), AdversarySpec()) is default_policy
def test_adversary_name_mirrors_strategy_name() -> None:
"""Labels render non-default levers in field order; the name field is excluded."""
assert adversary_name(AdversarySpec()) == "default"
assert adversary_name(AdversarySpec(target_priority="healer")) == "healer"
assert (
adversary_name(AdversarySpec(name="custom", spell_policy="none"))
== "spell_policy=none"
)
assert (
adversary_name(
AdversarySpec(target_priority="weakest_ac", retreat_hp_fraction=0.5)
)
== "weakest_ac, retreat=0.5"
)
# --- validation ------------------------------------------------------------
def test_compile_matchup_empty_party_side_raises() -> None:
"""An empty party_side is a ValueError (pre-assigned msg, EM101/EM102)."""
with pytest.raises(ValueError, match="party_side"):
compile_matchup(StrategySpec(), AdversarySpec(), party_side="")
# --- routing ----------------------------------------------------------------
def test_compile_matchup_routes_by_side() -> None:
"""On a real engine, each side of the matchup delegates to its own policy."""
party_spec = StrategySpec(name="party-focus", target_priority="weakest_ac")
adversary_spec = AdversarySpec(name="adv-focus", target_priority="lowest_hp")
matchup = compile_matchup(party_spec, adversary_spec)
assert matchup is not default_policy
encounter = _encounter()
states = build_states(encounter)
engine = CombatEngine(SeededRng(SEED), Grid.from_spec(encounter.map), states)
party_state = next(s for s in states if s.side == PARTY)
monster_state = next(s for s in states if s.side == "monsters")
assert matchup(engine, party_state) == compile_strategy(party_spec)(
engine, party_state
)
assert matchup(engine, monster_state) == compile_adversary(adversary_spec)(
engine, monster_state
)
def test_matchup_and_healer_strategy_run_deterministically() -> None:
"""A healer spec compiles and runs evaluate-style cells bit-identically."""
healer = StrategySpec(name="healer-party", target_priority="healer")
policy = compile_strategy(healer)
assert policy is not default_policy
matchup = compile_matchup(healer, AdversarySpec(target_priority="weakest_ac"))
spec = _encounter()
_, first = run_encounter(spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True)
_, again = run_encounter(spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True)
assert first == again
_, matchup_a = run_encounter(
spec, runs=RUNS, seed=SEED, policy=matchup, collect_runs=True
)
_, matchup_b = run_encounter(
spec, runs=RUNS, seed=SEED, policy=matchup, collect_runs=True
)
assert matchup_a == matchup_b
# --- healer targeting -------------------------------------------------------
def test_healer_prefers_the_enemy_caster() -> None:
"""With a monster-side caster (nairda) among non-casters, healer picks her."""
actor = _state(_c(GOBLIN), PARTY, (1, 1))
nairda = _state(_c(NAIRDA), "monsters", (1, 8))
orc = _state(_c(ORC), "monsters", (1, 16))
assert nairda.combatant.spells # caster precondition (nairda.json)
engine = _mini_engine([actor, nairda, orc])
assert select_target(engine, actor, "healer") is nairda
# Discriminator: weakest_ac sees equal AC 13 and prefers the wounded orc.
assert select_target(engine, actor, "weakest_ac") is orc
def test_healer_falls_back_to_weakest_ac_without_casters() -> None:
"""No active caster on the enemy side -> identical to weakest_ac."""
actor = _state(_c(GOBLIN), PARTY, (1, 1))
orc = _state(_c(ORC), "monsters", (1, 8))
goblin_enemy = _state(_c(GOBLIN), "monsters", (1, 16))
assert not orc.combatant.spells
assert not goblin_enemy.combatant.spells
engine = _mini_engine([actor, orc, goblin_enemy])
healer = select_target(engine, actor, "healer")
assert healer is select_target(engine, actor, "weakest_ac")
assert healer is orc # AC 13 < goblin AC 16
# --- SideSpec.aware (SRD "Surprise") ---------------------------------------
def test_side_spec_aware_applies_per_side() -> None:
"""aware=False lands on that side's combatants only; the other side keeps True."""
encounter = build_encounter(_scenario(monsters_aware=False))
players, monsters = encounter.sides
assert all(c.aware is True for c in players.combatants)
assert all(c.aware is False for c in monsters.combatants)
states = build_states(encounter)
assert all(s.combatant.aware is True for s in states if s.side == PARTY)
assert all(s.combatant.aware is False for s in states if s.side == "monsters")
untouched = build_encounter(_scenario())
assert all(c.aware is True for side in untouched.sides for c in side.combatants)
def test_side_spec_aware_triggers_surprise_round() -> None:
"""Only some aware -> a surprise round runs; the default (aware=None) has none."""
surprise = _encounter_states_and_run(_scenario(monsters_aware=False))
assert "surprise round" in surprise
default = _encounter_states_and_run(_scenario())
assert "surprise round" not in default
def _encounter_states_and_run(scenario: ScenarioSpec) -> tuple[str, ...]:
"""Run one seeded battle of `scenario` and return its transcript."""
encounter = build_encounter(scenario)
engine = CombatEngine(
SeededRng(SEED),
Grid.from_spec(encounter.map),
build_states(encounter),
round_cap=ROUND_CAP,
)
return engine.run().transcript
+198
View File
@@ -0,0 +1,198 @@
"""Phase-2 candidate actions (docs/phase-2-llm-architecture.md P0.3).
``candidates.candidates`` derives a deterministic menu of plausible
``Action`` proposals from the ``Observation`` snapshot's pairs. This module
pins the contract: a normal turn yields a non-empty, capped menu whose kinds
stay within the declared set; the menu is deterministic; no candidate ever
targets an ally (the pairs cover every other combatant, so the side filter is
what keeps the menu hostile-only); a weaponless combatant still gets
movement/defensive options; an enemy in weapon reach gets attack/full_attack;
and harmless spells (cure, buffs) target the caster rather than an enemy.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
from pf1e_simulator.candidates import _CANDIDATE_KINDS, MAX_CANDIDATES, candidates
from pf1e_simulator.combat import (
Action,
CombatantState,
CombatEngine,
Observation,
default_policy,
)
from pf1e_simulator.grid import Grid
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import EncounterSpec, Side, build_states
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
if TYPE_CHECKING:
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
class _CaptureAll:
"""Policy wrapper that snapshots every observation, then delegates."""
def __init__(self) -> None:
self.observations: dict[str, list[Observation]] = {}
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
obs = engine._build_observation(state)
self.observations.setdefault(state.combatant.id, []).append(obs)
return default_policy(engine, state)
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _representative_spec() -> EncounterSpec:
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars)."""
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(
name="players",
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
zone="A",
),
Side(
name="monsters",
combatants=(
_c("monsters/lich.json"),
_c("monsters/thrune_agent.json"),
_c("monsters/orc.json"),
_c("monsters/goblin.json"),
),
zone="B",
),
),
)
def _weaponless_spec() -> EncounterSpec:
"""1v1: a goblin with no usable weapons against an orc."""
goblin = _c("monsters/goblin.json").model_copy(update={"attacks": ()})
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(name="players", combatants=(goblin,), zone="A"),
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
),
)
def _run_observations(
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
) -> dict[str, list[Observation]]:
grid = Grid.from_spec(spec.map)
states = build_states(spec)
cap = _CaptureAll()
CombatEngine(
SeededRng(seed),
grid,
states,
round_cap=spec.round_cap,
policy=cap.policy,
spell_registry=registry,
).run()
return cap.observations
def _pair_target_side(pair: dict[str, object], obs: Observation) -> object:
"""Side of the pair's target combatant, or None when out of range."""
index = pair.get("target_index")
if not isinstance(index, int) or not (0 <= index < len(obs.combatants)):
return None
return obs.combatants[index].get("side")
def test_candidates_normal_turn_non_empty_and_supported_kinds() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_representative_spec(), 1, registry)
obs = next(obs_list[0] for obs_list in observations.values())
menu = candidates(obs)
assert menu
assert len(menu) <= MAX_CANDIDATES
kinds = {a.kind for a in menu}
assert kinds <= _CANDIDATE_KINDS
assert {"move", "total_defense", "delay"} <= kinds
def test_candidates_deterministic() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_representative_spec(), 1, registry)
obs = next(obs_list[0] for obs_list in observations.values())
assert candidates(obs) == candidates(obs)
def test_candidates_no_allies_targeted() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_representative_spec(), 1, registry)
for obs_list in observations.values():
for obs in obs_list:
acting = obs.combatants[obs.acting_combatant]
acting_side = acting["side"]
acting_id = acting["id"]
for action in candidates(obs):
if action.target_id is None or action.target_id == acting_id:
continue
target = next(
(c for c in obs.combatants if c["id"] == action.target_id), None
)
assert target is not None
assert target["side"] != acting_side
def test_candidates_weaponless_gets_movement_and_defense() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_weaponless_spec(), 1, registry)
obs = observations["goblin"][0]
kinds = {a.kind for a in candidates(obs)}
assert {"move", "total_defense", "ready", "delay"} <= kinds
assert not ({"attack", "full_attack", "charge"} & kinds)
def test_candidates_in_range_gets_attack() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_representative_spec(), 1, registry)
for obs_list in observations.values():
for obs in obs_list:
acting = obs.combatants[obs.acting_combatant]
acting_side = acting["side"]
enemy_pairs = [
p for p in obs.pairs if _pair_target_side(p, obs) != acting_side
]
if not any(p.get("weapon_for") is not None for p in enemy_pairs):
continue
kinds = {a.kind for a in candidates(obs)}
assert "attack" in kinds
assert "full_attack" in kinds
return
msg = "no observation with an enemy in weapon reach found"
raise AssertionError(msg)
def test_candidates_harmless_spells_target_self() -> None:
registry = load_spell_registry(SPELLS_DIR)
observations = _run_observations(_representative_spec(), 1, registry)
obs = observations["oni"][0]
acting = obs.combatants[obs.acting_combatant]
acting_id = acting["id"]
harmless_casts = [
a
for a in candidates(obs)
if a.kind == "cast_spell"
and a.spell_name in {"Guidance", "Shield of Faith", "Stabilize"}
]
assert harmless_casts
assert all(a.target_id == acting_id for a in harmless_casts)
+378
View File
@@ -0,0 +1,378 @@
"""Phase-2 LLM client layer (docs/phase-2 P1.1/P1.2, decisions D6/D10).
``client.py`` wraps every LLM call with the shared safeguards: composite
cache (D6: model, params, prompt_version, schema_version, prompt hash),
cost/token budgets + circuit breaker (P1.2, section 13), retry/backoff on
transient provider failures, and ``FakeLLMClient`` (P1.1) to drive battles
fully offline. These tests pin the ledger, the budgets and the breaker
behaviour without any network.
"""
from __future__ import annotations
import pytest
from pf1e_simulator.client import (
BudgetExceededError,
CircuitBreaker,
CircuitOpenError,
ClientParams,
ClientStats,
Completion,
CostBudget,
FakeLLMClient,
LLMClient,
Pricing,
ResponseCache,
RetryExhaustedError,
ScriptExhaustedError,
TokenUsage,
estimate_tokens,
response_cache_key,
)
_PARAMS = ClientParams()
_PROMPT = "observation rendered prompt v1"
class _UnusedClient(LLMClient):
"""Concrete client whose provider is never invoked (validation tests)."""
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
del prompt, params # validation-only: the provider never runs
msg = "provider must never be called"
raise AssertionError(msg)
# ── composite cache key (D6) ───────────────────────────────────────────────
def test_cache_key_is_composite_and_deterministic() -> None:
"""Each D6 component changes the key; identical inputs are stable."""
base = response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt=_PROMPT,
)
assert (
response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt=_PROMPT,
)
== base
)
assert (
response_cache_key(
model="other",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt=_PROMPT,
)
!= base
)
assert (
response_cache_key(
model="m",
params=ClientParams(temperature=0.7),
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt=_PROMPT,
)
!= base
)
assert (
response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:9.9.9:nl:xyz",
schema_version=1,
prompt=_PROMPT,
)
!= base
)
assert (
response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=2,
prompt=_PROMPT,
)
!= base
)
assert (
response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt="a different observation",
)
!= base
)
def test_response_cache_round_trip_and_counters() -> None:
"""get/put round-trip; hits and misses are counted separately."""
cache = ResponseCache(shard_count=4)
key = response_cache_key(
model="m",
params=_PARAMS,
prompt_version="pack:1.0.0:nl:abc",
schema_version=1,
prompt=_PROMPT,
)
assert cache.get(key) is None
assert cache.hits == 0
assert cache.misses == 1
cache.put(key, "action json 1")
assert cache.get(key) == "action json 1"
assert cache.hits == 1
assert cache.misses == 1
def test_response_cache_clear_drops_entries_and_counters() -> None:
"""clear() empties every shard and resets the counters (tests only)."""
cache = ResponseCache(shard_count=2)
key = response_cache_key(
model="m",
params=_PARAMS,
prompt_version="p:1:nl:a",
schema_version=1,
prompt=_PROMPT,
)
cache.put(key, "action json 1")
assert cache.get(key) == "action json 1"
cache.clear()
assert cache.get(key) is None
assert cache.hits == 0
assert cache.misses == 1
def test_response_cache_requires_positive_shard_count() -> None:
"""A zero shard count is a data error, not a silent default."""
with pytest.raises(ValueError, match="shard_count"):
ResponseCache(shard_count=0)
# ── FakeLLMClient: scripted offline behaviour (P1.1) ───────────────────────
def test_fake_serves_script_in_order() -> None:
"""The fake returns scripted responses one per successful call."""
client = FakeLLMClient(script=("action a", "action b"))
assert client.complete("observation one") == "action a"
assert client.complete("observation two") == "action b"
assert client.stats.calls == 2
def test_fake_script_exhaustion_raises_typed_error() -> None:
"""A call past the script is a test-authoring bug, loudly reported."""
client = FakeLLMClient(script=("action a",))
assert client.complete("observation one") == "action a"
with pytest.raises(ScriptExhaustedError, match="script exhausted"):
client.complete("observation two")
def test_fake_reports_estimated_token_usage() -> None:
"""Token usage is estimated from character counts (no provider metric)."""
client = FakeLLMClient(script=("0123456789",))
client.complete(_PROMPT)
stats = client.stats
assert stats.prompt_tokens == estimate_tokens(_PROMPT)
assert stats.completion_tokens == estimate_tokens("0123456789")
assert isinstance(stats, ClientStats)
# ── cache integration on the client (D6) ───────────────────────────────────
def test_client_cache_hit_consumes_no_script() -> None:
"""A cached response is returned without a provider call."""
client = FakeLLMClient(script=("action a", "action b"))
first = client.complete(_PROMPT)
second = client.complete(_PROMPT)
assert first == second == "action a"
assert client.calls == 1 # the second call came from the cache
assert client.cache_misses == 1
assert client.cache_hits == 1
def test_client_cache_miss_on_different_prompt() -> None:
"""A different prompt is a different cache key: script advances."""
client = FakeLLMClient(script=("action a", "action b"))
assert client.complete(_PROMPT) == "action a"
assert client.complete("another observation") == "action b"
assert client.calls == 2
def test_client_cache_respects_prompt_version() -> None:
"""The composite key includes prompt_version: packs do not collide."""
cache = ResponseCache()
a = FakeLLMClient(
script=("action a",),
prompt_version="pack:1.0.0:nl:aaa",
cache=cache,
)
b = FakeLLMClient(
script=("action b",),
prompt_version="pack:1.0.0:nl:bbb",
cache=cache,
)
assert a.complete(_PROMPT) == "action a"
assert b.complete(_PROMPT) == "action b"
assert a.calls == 1
assert b.calls == 1
def test_client_cache_respects_model() -> None:
"""The composite key includes the model name."""
cache = ResponseCache()
a = FakeLLMClient(script=("action a",), model="model-a", cache=cache)
b = FakeLLMClient(script=("action b",), model="model-b", cache=cache)
assert a.complete(_PROMPT) == "action a"
assert b.complete(_PROMPT) == "action b"
assert a.calls == 1
assert b.calls == 1
# ── retry/backoff (P1.2) ───────────────────────────────────────────────────
def test_retry_recovers_after_transient_failures() -> None:
"""ProviderError is retried; the script is only consumed by successes."""
client = FakeLLMClient(
script=("action a",),
failures=2,
retries=3,
)
assert client.complete(_PROMPT) == "action a"
assert client.calls == 3 # two failures + one success
def test_retry_exhausted_after_too_many_failures() -> None:
"""Past the retry budget, a typed RetryExhaustedError is raised."""
client = FakeLLMClient(script=("action a",), failures=5, retries=2)
with pytest.raises(RetryExhaustedError, match="after 3 attempt"):
client.complete(_PROMPT)
def test_retry_counter_requires_positive_value() -> None:
"""A negative retry count is a data error."""
with pytest.raises(ValueError, match="retries"):
FakeLLMClient(script=("a",), retries=-1)
# ── circuit breaker (P1.2, section 13) ─────────────────────────────────────
def test_breaker_opens_refuses_probes_and_recovers() -> None:
"""Open after N failures: refusals, then a probe that closes on success."""
breaker = CircuitBreaker(failure_threshold=2, open_refusals=1)
client = FakeLLMClient(
script=("action a",),
failures=2, # needs two attempts before real service
retries=0,
breaker=breaker,
)
with pytest.raises(RetryExhaustedError):
client.complete(_PROMPT) # failure 1: not open yet
with pytest.raises(RetryExhaustedError):
client.complete(_PROMPT) # failure 2: threshold reached, opens
assert breaker.is_open
with pytest.raises(CircuitOpenError, match="circuit breaker open"):
client.complete(_PROMPT) # refused
assert client.complete(_PROMPT) == "action a" # probe succeeds now
assert not breaker.is_open
def test_breaker_probe_failure_reopens() -> None:
"""A failed probe reopens the breaker for another refusal window."""
breaker = CircuitBreaker(failure_threshold=1, open_refusals=1)
client = FakeLLMClient(
script=("action a",),
failures=10, # every attempt fails, probe included
retries=0,
breaker=breaker,
)
with pytest.raises(RetryExhaustedError):
client.complete(_PROMPT)
assert breaker.is_open
with pytest.raises(CircuitOpenError):
client.complete(_PROMPT)
with pytest.raises(RetryExhaustedError):
client.complete(_PROMPT) # the probe fails
assert breaker.is_open
with pytest.raises(CircuitOpenError):
client.complete(_PROMPT) # reopened: refuses again
def test_breaker_validation() -> None:
"""Breaker thresholds must be sensible numbers."""
with pytest.raises(ValueError, match="failure_threshold"):
CircuitBreaker(failure_threshold=0)
with pytest.raises(ValueError, match="open_refusals"):
CircuitBreaker(open_refusals=-1)
# ── cost/token budgets (P1.2) ──────────────────────────────────────────────
def test_call_budget_ceiling() -> None:
"""max_calls refuses further calls with a typed error."""
client = FakeLLMClient(
script=("a", "b", "c"),
budget=CostBudget(max_calls=1),
)
assert client.complete(_PROMPT) == "a"
with pytest.raises(BudgetExceededError, match="call budget exceeded"):
client.complete("a different observation")
def test_token_budget_ceiling() -> None:
"""A completion over max_total_tokens is rejected and not cached."""
client = FakeLLMClient(
script=("x" * 400,), # ~100 estimated completion tokens
budget=CostBudget(max_total_tokens=50),
)
with pytest.raises(BudgetExceededError, match="token budget exceeded"):
client.complete(_PROMPT)
def test_cost_budget_ceiling_and_ledger() -> None:
"""Estimated USD cost is accumulated and capped by max_cost_usd."""
pricing = Pricing(prompt_per_1k_usd=1.0, completion_per_1k_usd=1.0)
long_prompt = "p" * 2000 # ~500 prompt tokens ≈ 0.5 USD
client = FakeLLMClient(
script=("c" * 2000, "c" * 2000), # ~500 completion tokens each ≈ 0.5 USD
pricing=pricing,
budget=CostBudget(max_cost_usd=1.0),
)
assert client.complete(long_prompt) == "c" * 2000
assert client.cost_usd == pytest.approx(1.0, abs=1e-6)
with pytest.raises(BudgetExceededError, match="cost budget exceeded"):
client.complete("q" * 2000)
assert client.cost_usd == pytest.approx(2.0, abs=1e-6)
def test_backoff_must_be_positive() -> None:
"""A negative backoff is a data error (validation-only client)."""
with pytest.raises(ValueError, match="backoff_base_s"):
_UnusedClient(
model="m",
prompt_version="p:1:nl:a",
backoff_base_s=-1.0,
)
def test_token_usage_total() -> None:
"""TokenUsage.total is the sum of both counters."""
usage = TokenUsage(prompt_tokens=3, completion_tokens=4)
assert usage.total == 7
assert estimate_tokens("abcd") >= 1
+193
View File
@@ -0,0 +1,193 @@
"""Phase-2 danger-cartography spike (docs/phase-2-llm-architecture.md 3.1/3.2, 10 P3/P4).
Proof of value WITHOUT any LLM: the same party measured against the default
monster policy and against the strengthened focus-fire policy (spike module
pf1e_simulator.phase2_spike). A danger only exists against an adversary that
plays to win, so the DELTA between the two columns is the danger the default
Monte Carlo report hides. Prints a compact per-scenario block (party win
rate +/- 3 sigma, P(TPK), P(>=1 death), expected hp lost, rounds, worst
decile hp) and pins loose assertions: everything is deterministic under a
fixed seed, so the pinned inequalities are exact reproductions, not noise.
Scenarios (sample arena, 100 runs, fixed seed):
warband - 4 level-4 PCs vs a thrune agent + 2 orcs + a goblin: the party
"wins" either way, but focus fire nearly doubles P(>=1 death).
lich - 2 level-2/4 PCs vs the lich + an orc: the party loses both
ways, but against the default policy nobody ever actually
DIES (downed at hp<0, above -CON); focus fire converts defeat
into real deaths and a nonzero TPK rate.
"""
from __future__ import annotations
import sys
from pathlib import Path
from typing import TYPE_CHECKING
from pf1e_simulator.combat import CombatantState, CombatEngine, default_policy
from pf1e_simulator.grid import Grid
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.objectives import DangerMetrics, danger_metrics
from pf1e_simulator.phase2_spike import focus_fire_policy
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import EncounterSpec, RunOutcome, Side, run_encounter
if TYPE_CHECKING:
import pytest
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
RUNS = 100
SEED = 11
ROUND_CAP = 20
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _spec(pcs: list[str], monsters: list[str]) -> EncounterSpec:
return EncounterSpec(
map=load_map(MAP),
round_cap=ROUND_CAP,
sides=(
Side(name="players", combatants=tuple(_c(p) for p in pcs), zone="A"),
Side(name="monsters", combatants=tuple(_c(p) for p in monsters), zone="B"),
),
)
WARBAND = _spec(
["pcs/jeanne.json", "pcs/tammara.json", "pcs/harvie.json", "pcs/ierlieth.json"],
["monsters/thrune_agent.json", "monsters/orc.json", "monsters/orc.json",
"monsters/goblin.json"],
)
LICH = _spec(
["pcs/harvie.json", "pcs/ierlieth.json"],
["monsters/lich.json", "monsters/orc.json"],
)
def _emit(line: str) -> None:
sys.stdout.write(line + "\n")
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
return CombatEngine(SeededRng(1), Grid.from_spec(WARBAND.map), states)
def _state(cid: str, side: str, pos: tuple[int, int], hp: int = 10) -> CombatantState:
c = _c("monsters/orc.json").model_copy(update={"id": cid})
return CombatantState(combatant=c, side=side, pos=pos, hp=hp, resources_left={})
def test_focus_fire_concentrates_on_lowest_ac_not_nearest() -> None:
"""The monster ignores the adjacent tough target and converges on the squishy one."""
policy = focus_fire_policy("monsters")
orc = _state("orc1", "monsters", (1, 1))
tough = _c("pcs/jeanne.json") # AC 23, adjacent to the orc
squishy = _c("pcs/nairda.json") # AC 13, far away
tough_state = CombatantState(
combatant=tough, side="players", pos=(1, 2), hp=tough.hp_max, resources_left={}
)
squishy_state = CombatantState(
combatant=squishy, side="players", pos=(1, 16), hp=squishy.hp_max, resources_left={}
)
engine = _mini_engine([orc, tough_state, squishy_state])
actions = policy(engine, orc)
assert all(a.target_id == "nairda" for a in actions), actions
def test_focus_fire_tie_break_prefers_wounded() -> None:
"""Equal AC: the monster focuses the member with the lower current hp."""
policy = focus_fire_policy("monsters")
orc = _state("orc1", "monsters", (1, 1))
fresh = _c("monsters/goblin.json").model_copy(update={"id": "fresh"})
hurt = _c("monsters/goblin.json").model_copy(update={"id": "hurt"})
fresh_state = CombatantState(
combatant=fresh, side="players", pos=(5, 5), hp=fresh.hp_max, resources_left={}
)
hurt_state = CombatantState(
combatant=hurt, side="players", pos=(5, 10), hp=1, resources_left={}
)
engine = _mini_engine([orc, fresh_state, hurt_state])
actions = policy(engine, orc)
assert all(a.target_id == "hurt" for a in actions), actions
def test_focus_fire_leaves_the_party_side_on_default_policy() -> None:
"""Only the monster side changes: a party-side turn gets default_policy verbatim."""
policy = focus_fire_policy("monsters")
pc = _state("p", "players", (1, 1))
mob = _state("m", "monsters", (1, 8))
engine = _mini_engine([pc, mob])
assert policy(engine, pc) == default_policy(engine, pc)
def _report(name: str, m: DangerMetrics, win: float, sigma: float) -> None:
_emit(
f" {name:<9} win={win:6.1%} +/-{3 * sigma:4.1%} surv={m.p_survives:5.1%} "
f"tpk={m.p_tpk:5.1%} >=1-death={m.p_any_death:5.1%} "
f"hp-lost={m.expected_hp_lost:6.1f} rounds={m.expected_rounds:4.1f} "
f"tail-hp={m.worst_decile_party_hp:5.1f}"
)
def _measure(
spec: EncounterSpec,
*,
strengthened: bool
) -> tuple[float, float, DangerMetrics, tuple[RunOutcome, ...]]:
policy = focus_fire_policy("monsters") if strengthened else None
report, outcomes = run_encounter(
spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True
)
win = report.win_rate("players")
return win, report.win_rate_sigma("players"), danger_metrics(outcomes, "players"), outcomes
def test_danger_map_default_vs_strengthened(capsys: pytest.CaptureFixture[str]) -> None:
"""Print the danger map sketch and pin the delta (deterministic, fixed seed)."""
_emit("=== PHASE2 SUMMARY (danger cartography spike) ===")
deltas: dict[str, tuple[float, float]] = {}
metrics: dict[str, tuple[DangerMetrics, DangerMetrics]] = {}
for name, spec in (("warband", WARBAND), ("lich", LICH)):
_emit(f"scenario {name}: {len(spec.sides[0].combatants)} PCs vs "
f"{len(spec.sides[1].combatants)} monsters, runs={RUNS}, seed={SEED}")
w0, s0, m0, _ = _measure(spec, strengthened=False)
w1, s1, m1, _ = _measure(spec, strengthened=True)
_report("default", m0, w0, s0)
_report("focusfire", m1, w1, s1)
_emit(
f" delta death-rate {m1.p_any_death - m0.p_any_death:+.1%} "
f"hp-lost {m1.expected_hp_lost - m0.expected_hp_lost:+.1f}"
)
deltas[name] = (m0.p_any_death, m1.p_any_death)
metrics[name] = (m0, m1)
out = capsys.readouterr().out
assert "PHASE2 SUMMARY" in out
# Loose, deterministic pins: the strengthened adversary never hides danger.
for m0, m1 in metrics.values():
assert 0.0 <= m1.p_any_death <= 1.0
assert m1.p_tpk >= m0.p_tpk - 0.05
assert m1.p_survives <= m0.p_survives + 0.05
# warband: focus fire must strictly raise the death rate
d0, d1 = deltas["warband"]
assert d1 > d0
# lich: default policy shows ZERO real deaths; focus fire shows some
assert metrics["lich"][0].p_any_death == 0.0
assert metrics["lich"][1].p_any_death > 0.0
assert metrics["lich"][1].p_tpk > 0.0
def test_danger_map_is_reproducible() -> None:
"""Same seed, same policy -> byte-identical per-run outcome tuples."""
a = _measure(LICH, strengthened=True)[3]
b = _measure(LICH, strengthened=True)[3]
assert a == b
+183
View File
@@ -0,0 +1,183 @@
"""Phase-2 measurement spike (docs/phase-2-llm-architecture.md section 12.b).
Two numbers decide whether a per-turn LLM is viable for the Monte Carlo
sweep: the OBSERVATION SIZE (bytes / rough tokens) and the POLICY CALLS PER
BATTLE (hence per sweep cell). This module measures both through the real
engine path on a representative caster-heavy encounter, prints a summary
block for the orchestrator, and pins LOOSE upper bounds so a regression
(bigger snapshot schema, policy called twice per turn, ...) fails loudly.
No engine behaviour is changed here: the recorder only reads
``engine._build_observation`` (pure read, consumes no RNG) at each policy
invocation and delegates to ``default_policy``.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
from typing import TYPE_CHECKING
from pf1e_simulator.combat import Action, CombatantState, CombatEngine, default_policy
from pf1e_simulator.grid import Grid
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import EncounterSpec, Side, build_states
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
if TYPE_CHECKING:
import pytest
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
# Loose ceilings: a regression past these fails loudly; the real values are
# far below (printed by the tests for the orchestrator to quote).
MAX_OBS_BYTES = 200_000
MAX_CALLS_PER_BATTLE = 200
# Round cap for the measurement battles: generous enough to exercise buffs,
# conditions and summons, small enough to bound the calls/battle figure.
MEASURE_ROUND_CAP = 20
class _Recorder:
"""Counting policy wrapper: counts invocations, snapshots observation size.
Delegates every decision to ``default_policy`` so the battle trajectory
(and therefore the call count) is the real engine path. The observation
is rebuilt at each call via ``engine._build_observation(state)`` - the
same snapshot the engine hands to an ``LLMPolicy`` - and its JSON size
is tracked (first call = start of battle, max = fullest mid-battle
snapshot with buffs/conditions active).
"""
def __init__(self) -> None:
self.calls = 0
self.first_bytes: int | None = None
self.max_bytes = 0
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
self.calls += 1
obs = engine._build_observation(state).to_dict()
size = len(json.dumps(obs))
if self.first_bytes is None:
self.first_bytes = size
self.max_bytes = max(self.max_bytes, size)
return default_policy(engine, state)
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _representative_spec() -> EncounterSpec:
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars).
Oni (channel + heal + summon), misty (spell combat), nairda (hexes) vs
the lich (high-level caster, fear aura, paralysis touch), a thrune
agent, an orc and a goblin: buffs, conditions, spells and pairs all
load the snapshot.
"""
return EncounterSpec(
map=load_map(MAP),
round_cap=MEASURE_ROUND_CAP,
sides=(
Side(
name="players",
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
zone="A",
),
Side(
name="monsters",
combatants=(
_c("monsters/lich.json"),
_c("monsters/thrune_agent.json"),
_c("monsters/orc.json"),
_c("monsters/goblin.json"),
),
zone="B",
),
),
)
def _duel_spec() -> EncounterSpec:
"""Small 2v2 baseline to show how calls/battle scales with combatants."""
return EncounterSpec(
map=load_map(MAP),
round_cap=MEASURE_ROUND_CAP,
sides=(
Side(name="players", combatants=(_c("monsters/goblin.json"),) * 2, zone="A"),
Side(name="monsters", combatants=(_c("monsters/orc.json"),) * 2, zone="B"),
),
)
def _run_battle(
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
) -> _Recorder:
grid = Grid.from_spec(spec.map)
states = build_states(spec)
rec = _Recorder()
CombatEngine(
SeededRng(seed),
grid,
states,
round_cap=spec.round_cap,
policy=rec.policy,
spell_registry=registry,
).run()
return rec
def _emit(line: str) -> None:
sys.stdout.write(line + "\n")
def test_observation_size_and_policy_calls(capsys: pytest.CaptureFixture[str]) -> None:
"""Measure both numbers on the representative battle; pin loose bounds."""
registry = load_spell_registry(SPELLS_DIR)
spec = _representative_spec()
rec = _run_battle(spec, seed=1, registry=registry)
assert rec.first_bytes is not None
# Determinism: same seed -> same trajectory -> same numbers.
rec2 = _run_battle(spec, seed=1, registry=registry)
assert rec2.calls == rec.calls
assert rec2.first_bytes == rec.first_bytes
assert rec2.max_bytes == rec.max_bytes
# Loose bounds (regression guards, not exact pins).
assert rec.first_bytes < MAX_OBS_BYTES
assert rec.max_bytes < MAX_OBS_BYTES
assert rec.calls < MAX_CALLS_PER_BATTLE
duel = _run_battle(_duel_spec(), seed=1, registry=registry)
assert duel.calls < MAX_CALLS_PER_BATTLE
assert duel.first_bytes is not None
first_tokens = rec.first_bytes // 4
max_tokens = rec.max_bytes // 4
_emit("=== PHASE2 SUMMARY (measurement spike) ===")
_emit(f"encounter: 7 combatants, round_cap={MEASURE_ROUND_CAP}, seed=1")
_emit(f"observation bytes (first call): {rec.first_bytes} (~{first_tokens} tokens)")
_emit(f"observation bytes (max in battle): {rec.max_bytes} (~{max_tokens} tokens)")
_emit(f"policy calls/battle (7 combatants): {rec.calls}")
_emit(f"policy calls/battle (2v2 duel): {duel.calls}")
_emit(f"observation 2v2 duel bytes: {duel.first_bytes} (~{duel.first_bytes // 4} tokens)")
for runs in (1, 100, 1000):
_emit(f"projected calls per sweep cell at runs={runs}: {rec.calls * runs}")
out = capsys.readouterr().out
assert "PHASE2 SUMMARY" in out
def test_projection_is_linear_in_runs() -> None:
"""Calls per sweep cell = calls/battle x runs (fixed per-battle count)."""
registry = load_spell_registry(SPELLS_DIR)
rec = _run_battle(_representative_spec(), seed=7, registry=registry)
assert rec.calls > 0
assert rec.calls * 1000 == (rec.calls * 100) * 10
+414
View File
@@ -0,0 +1,414 @@
"""Tests for the natural-language policy (P1.3)."""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.client import (
ClientParams,
Completion,
FakeLLMClient,
ResponseCache,
ScriptExhaustedError,
)
from pf1e_simulator.combat import Action, Observation
from pf1e_simulator.nl_policy import (
NLPolicy,
NLPolicyError,
NLPolicyStats,
fallback_default,
parse_actions,
)
if TYPE_CHECKING:
from collections.abc import Sequence
from pf1e_simulator.prompt import PromptPack
from pf1e_simulator.prompt_builder import PromptBuilder
_FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
def _obs() -> Observation:
"""Minimal valid observation for deterministic tests."""
return Observation(
round=1,
acting_combatant=0,
combatants=(
{"id": "goblin", "side": "players", "active": True, "pos": (1, 1)},
),
pairs=(),
terrain=((None,),),
elevation=((0,),),
)
def _pack() -> PromptPack:
return PromptPack.load(_FIXTURE_PACK)
def _builder() -> PromptBuilder:
return PromptBuilder(pack=_pack())
def _make_client(script: list[str], *, builder: PromptBuilder, **kwargs) -> FakeLLMClient:
"""Create a FakeLLMClient with the correct prompt_version from the builder."""
return FakeLLMClient(
script=script,
prompt_version=builder.prompt_version("nl_default"),
**kwargs,
)
def _make_policy(
client: FakeLLMClient, builder: PromptBuilder, *, max_repairs: int = 1
) -> NLPolicy:
return NLPolicy(client=client, builder=builder, max_repairs=max_repairs)
class _RecordingClient(FakeLLMClient):
"""Scripted client that records every prompt sent to the model.
The repair-loop tests assert on the accumulated error trail, so the
prompts themselves must be observable.
"""
def __init__(self, script: Sequence[str], *, prompt_version: str) -> None:
super().__init__(script=script, prompt_version=prompt_version)
self.prompts: list[str] = []
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
self.prompts.append(prompt)
return super()._call_once(prompt, params=params)
@pytest.fixture
def builder() -> PromptBuilder:
"""A builder over the fixture pack, shared by the NLPolicy integration tests."""
return _builder()
# ---- parse_actions tests -----------------------------------------------------
def test_parse_actions_ok() -> None:
reply = '[{"kind": "move", "destination": [2, 3]}]'
actions = parse_actions(reply)
assert actions == (Action(kind="move", destination=(2, 3)),)
def test_parse_actions_empty_array() -> None:
assert parse_actions("[]") == ()
def test_parse_actions_invalid_json() -> None:
with pytest.raises(NLPolicyError, match=r"reply is not valid JSON"):
parse_actions("not json")
def test_parse_actions_not_a_list() -> None:
with pytest.raises(NLPolicyError, match=r"reply must be a JSON array of action objects"):
parse_actions('{"kind": "attack"}')
def test_parse_actions_unknown_field() -> None:
with pytest.raises(NLPolicyError, match=r"unknown action field\(s\): bogus"):
parse_actions('[{"kind": "attack", "bogus": 1}]')
def test_parse_actions_missing_kind() -> None:
with pytest.raises(NLPolicyError, match=r"missing or unknown 'kind'"):
parse_actions('[{"target_id": "orc"}]')
def test_parse_actions_unknown_kind() -> None:
with pytest.raises(NLPolicyError, match=r"missing or unknown 'kind'"):
parse_actions('[{"kind": "dance"}]')
def test_parse_actions_channel_mode_invalid() -> None:
with pytest.raises(NLPolicyError, match=r"'channel_mode' must be one of: heal, harm"):
parse_actions('[{"kind": "channel_energy", "channel_mode": "loud"}]')
def test_parse_actions_destination_invalid_not_pair() -> None:
with pytest.raises(NLPolicyError, match=r"'destination' must be a \[row, col\] pair"):
parse_actions('[{"kind": "move", "destination": [2]}]')
def test_parse_actions_destination_invalid_strings() -> None:
with pytest.raises(NLPolicyError, match=r"'destination' must be a \[row, col\] pair"):
parse_actions('[{"kind": "move", "destination": ["2", "3"]}]')
def test_parse_actions_aid_mode_invalid() -> None:
with pytest.raises(NLPolicyError, match=r"'aid_mode' must be one of: attack, ac"):
parse_actions('[{"kind": "aid_another", "aid_mode": "damage"}]')
def test_parse_actions_trick_condition_invalid() -> None:
with pytest.raises(
NLPolicyError,
match=(
r"'trick_condition' must be one of: dazzled, blinded, "
r"deafened, entangled, shaken, sickened"
),
):
parse_actions('[{"kind": "cast_spell", "trick_condition": "nauseated"}]')
def test_parse_actions_ready_trigger_invalid() -> None:
with pytest.raises(
NLPolicyError,
match=(
r"'ready_trigger' must be one of: enemy_enters_reach, "
r"enemy_casts, enemy_attacks, enemy_hits_ally"
),
):
parse_actions('[{"kind": "ready", "ready_trigger": "enemy_sneezed"}]')
def test_parse_actions_channel_exclude_not_list() -> None:
with pytest.raises(NLPolicyError, match=r"'channel_exclude' must be a list of strings"):
parse_actions('[{"kind": "channel_energy", "channel_exclude": "orc"}]')
def test_parse_actions_channel_exclude_ok() -> None:
actions = parse_actions('[{"kind": "channel_energy", "channel_exclude": ["orc"]}]')
assert actions[0].channel_exclude == ("orc",)
def test_parse_actions_tumble_not_bool() -> None:
with pytest.raises(NLPolicyError, match=r"'tumble' must be a boolean"):
parse_actions('[{"kind": "attack", "tumble": "yes"}]')
def test_parse_actions_cast_defensively_not_bool() -> None:
with pytest.raises(NLPolicyError, match=r"'cast_defensively' must be a boolean"):
parse_actions('[{"kind": "cast_spell", "cast_defensively": 1}]')
def test_parse_actions_metamagic_ok() -> None:
actions = parse_actions('[{"kind": "cast_spell", "metamagic": ["Empower", "Maximize"]}]')
assert actions[0].metamagic == ("Empower", "Maximize")
# ---- fallback_default tests --------------------------------------------------
def test_fallback_default_prefers_full_attack() -> None:
menu = (
Action(kind="total_defense"),
Action(kind="full_attack", target_id="orc"),
Action(kind="move"),
)
assert fallback_default(menu) == (Action(kind="full_attack", target_id="orc"),)
def test_fallback_default_prefers_attack_over_charge() -> None:
menu = (
Action(kind="charge", target_id="orc"),
Action(kind="attack", target_id="orc"),
)
assert fallback_default(menu) == (Action(kind="attack", target_id="orc"),)
def test_fallback_default_prefers_charge_over_move() -> None:
menu = (
Action(kind="move", target_id="orc"),
Action(kind="charge", target_id="orc"),
)
assert fallback_default(menu) == (Action(kind="charge", target_id="orc"),)
def test_fallback_default_falls_back_to_first() -> None:
menu = (Action(kind="total_defense"), Action(kind="delay"))
assert fallback_default(menu) == (Action(kind="total_defense"),)
def test_fallback_default_empty_menu_returns_wait() -> None:
assert fallback_default(()) == (Action(kind="wait"),)
# ---- NLPolicy integration tests ---------------------------------------------
def test_nl_policy_parse_ok(builder: PromptBuilder) -> None:
"""Parse OK path with valid JSON from FakeLLMClient."""
script = ['[{"kind": "move", "destination": [2, 3]}]']
client = _make_client(script, builder=builder)
policy = _make_policy(client, builder)
actions = policy(_obs())
assert actions == (Action(kind="move", destination=(2, 3)),)
assert client.stats.calls == 1
def test_nl_policy_retry_then_succeeds() -> None:
"""One bad reply, then a good one → parsed successfully."""
builder = _builder()
script = ["not json", '[{"kind": "move", "destination": [2, 3]}]']
client = _make_client(script, builder=builder)
policy = _make_policy(client, builder)
actions = policy(_obs())
assert actions == (Action(kind="move", destination=(2, 3)),)
assert client.stats.calls == 2
def test_nl_policy_retry_then_fallback() -> None:
"""Two bad replies → fallback used."""
builder = _builder()
script = ["not json", "not json"]
client = _make_client(script, builder=builder)
policy = _make_policy(client, builder)
# Minimal obs menu is (total_defense, delay)
actions = policy(_obs())
# fallback_default picks first decisive action, but menu has only defensive
assert actions == (Action(kind="total_defense"),)
assert client.stats.calls == 2
def test_nl_policy_prompt_version_mismatch() -> None:
"""Client default prompt_version vs builder pack version → ValueError."""
builder = _builder()
# Use default prompt_version ("fixture:fake:0") which differs from pack
client = FakeLLMClient(script=['[{"kind": "wait"}]'])
with pytest.raises(ValueError, match=r"prompt_version"):
_make_policy(client, builder)
def test_nl_policy_script_exhausted_propagates() -> None:
"""ScriptExhaustedError from FakeLLMClient propagates (not caught)."""
builder = _builder()
client = _make_client([], builder=builder)
policy = _make_policy(client, builder)
with pytest.raises(ScriptExhaustedError):
policy(_obs())
def test_nl_policy_shared_cache_hit() -> None:
"""Two clients sharing a ResponseCache: second call hits cache."""
builder = _builder()
cache = ResponseCache()
# First client has one scripted response
client1 = _make_client(['[{"kind": "wait"}]'], builder=builder, cache=cache)
# Second client has NO script; relies on cache hit
client2 = _make_client([], builder=builder, cache=cache)
policy1 = _make_policy(client1, builder)
policy2 = _make_policy(client2, builder)
obs = _obs()
actions1 = policy1(obs)
actions2 = policy2(obs)
assert actions2 == actions1 == (Action(kind="wait"),)
# Second call was a cache hit (no ScriptExhaustedError)
assert client2.stats.cache_hits == 1
assert client1.stats.cache_misses == 1
# ---- P5.2 repair loop --------------------------------------------------------
def test_nl_policy_stats_defaults_to_zero() -> None:
"""A fresh NLPolicy reports zero calls until the first decision."""
builder = _builder()
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
policy = _make_policy(client, builder)
assert policy.stats == NLPolicyStats()
assert policy.stats.calls == 0
def test_nl_policy_max_repairs_zero_falls_back_immediately() -> None:
"""max_repairs=0 disables repair: first bad reply triggers fallback."""
builder = _builder()
client = _make_client(["bad one"], builder=builder)
policy = _make_policy(client, builder, max_repairs=0)
actions = policy(_obs())
assert actions == (Action(kind="total_defense"),)
assert policy.stats.calls == 1
assert policy.stats.repairs_attempted == 0
assert policy.stats.repairs_succeeded == 0
assert policy.stats.fallbacks == 1
def test_nl_policy_max_repairs_two_repairs_then_succeeds() -> None:
"""Two invalid replies are repaired, then the valid reply is parsed."""
builder = _builder()
script = ["bad one", "bad two", '[{"kind": "move", "destination": [2, 3]}]']
client = _make_client(script, builder=builder)
policy = _make_policy(client, builder, max_repairs=2)
actions = policy(_obs())
assert actions == (Action(kind="move", destination=(2, 3)),)
assert policy.stats.calls == 3
assert policy.stats.repairs_attempted == 2
assert policy.stats.repairs_succeeded == 1
assert policy.stats.fallbacks == 0
def test_nl_policy_max_repairs_two_then_falls_back() -> None:
"""Past max_repairs the policy stops repairing and falls back."""
builder = _builder()
script = ["bad one", "bad two", "bad three"]
client = _make_client(script, builder=builder)
policy = _make_policy(client, builder, max_repairs=2)
actions = policy(_obs())
assert actions == (Action(kind="total_defense"),)
assert policy.stats.calls == 3
assert policy.stats.repairs_attempted == 2
assert policy.stats.repairs_succeeded == 0
assert policy.stats.fallbacks == 1
def test_nl_policy_repair_trail_accumulates() -> None:
"""Each retry prompt carries the full accumulated error trail."""
builder = _builder()
script = ["bad one", "bad two", '[{"kind": "wait"}]']
client = _RecordingClient(
script=script,
prompt_version=builder.prompt_version("nl_default"),
)
policy = _make_policy(client, builder, max_repairs=2)
actions = policy(_obs())
assert actions == (Action(kind="wait"),)
# First call: bare prompt with no trail.
assert len(client.prompts) == 3
assert "-- previous reply invalid" not in client.prompts[0]
# First retry: trail carries attempt 1 only.
assert "attempt 1" in client.prompts[1]
assert "attempt 2" not in client.prompts[1]
assert "Return a valid JSON action sequence." in client.prompts[1]
# Second retry: trail carries attempts 1 AND 2, in order.
attempt1 = client.prompts[2].index("-- previous reply invalid (attempt 1) --")
attempt2 = client.prompts[2].index("-- previous reply invalid (attempt 2) --")
assert attempt1 < attempt2
# The base prompt is preserved at the head of every retry.
assert client.prompts[1].startswith(client.prompts[0])
def test_nl_policy_max_repairs_negative_rejected() -> None:
"""A negative max_repairs is a configuration error, not a fallback."""
builder = _builder()
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
with pytest.raises(ValueError, match=r"max_repairs must be >= 0, got -1"):
_make_policy(client, builder, max_repairs=-1)
def test_nl_policy_stats_after_direct_success() -> None:
"""A clean first-try success leaves the repair counters untouched."""
builder = _builder()
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
policy = _make_policy(client, builder)
actions = policy(_obs())
assert actions == (Action(kind="wait"),)
assert policy.stats.calls == 1
assert policy.stats.repairs_attempted == 0
assert policy.stats.repairs_succeeded == 0
assert policy.stats.fallbacks == 0
+87
View File
@@ -0,0 +1,87 @@
"""Offline battles driven by FakeLLMClient through the NL policy (P1.3/P1.4)."""
from __future__ import annotations
from pathlib import Path
from pf1e_simulator.client import FakeLLMClient
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.nl_policy import NLPolicy
from pf1e_simulator.prompt import PromptPack
from pf1e_simulator.prompt_builder import PromptBuilder
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
def _c(path: str):
return load_combatant(ROOT / "data" / path)
def _duel_spec() -> EncounterSpec:
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
),
)
def _pack() -> PromptPack:
return PromptPack.load(ROOT / "tests" / "fixtures" / "prompt_pack")
def _builder() -> PromptBuilder:
return PromptBuilder(pack=_pack())
def _make_client(script: list[str], *, builder: PromptBuilder) -> FakeLLMClient:
return FakeLLMClient(
script=script,
prompt_version=builder.prompt_version("nl_default"),
)
def test_offline_battle_total_defense_draw() -> None:
"""Both sides play total_defense → draw at round_cap (20 rounds, 40 calls)."""
spec = _duel_spec()
builder = _builder()
# 60 responses is plenty for 40 turns (20 rounds x 2 combatants)
script = ['[{"kind": "total_defense"}]'] * 60
client = _make_client(script, builder=builder)
policy = NLPolicy(client=client, builder=builder)
report, _outcomes = run_encounter(spec, runs=1, seed=1, llm_policy=policy, collect_runs=True)
# Draw at round_cap
assert report.draws == 1
assert report.wins.get("players", 0) == 0
assert report.wins.get("monsters", 0) == 0
assert report.avg_rounds == 20
# 20 rounds x 2 combatants = 40 policy invocations
assert client.stats.calls == 40
def test_offline_battle_fallback() -> None:
"""Fallback-driven battle: script of invalid JSON forces fallback on every turn."""
spec = _duel_spec()
builder = _builder()
# Each turn: 2 completes (first parse fail, retry parse fail → fallback).
# Max 20 rounds x 2 combatants x 2 = 80 completes. Use 100 for margin.
script = ["not json"] * 100
client = _make_client(script, builder=builder)
policy = NLPolicy(client=client, builder=builder)
report, _outcomes = run_encounter(spec, runs=1, seed=1, llm_policy=policy, collect_runs=True)
# The battle should complete (not exhaust script)
assert report.runs == 1
assert report.draws + sum(report.wins.values()) == 1
# At least some calls happened; script not exhausted
assert client.stats.calls > 0
assert client.stats.calls <= 100
+321
View File
@@ -0,0 +1,321 @@
"""Phase-2 config and external prompt pack (docs/phase-2 P1a, sections 14-15).
``config.py`` centralizes every phase-2 path/env read (no hard-coded data
paths - section 15). ``prompt.py`` loads the EXTERNAL, versioned prompt pack
(section 14): prompts never live in this repo; the pack carries its own
version and each prompt has an individual content hash, so ``prompt_version``
can enter the response-cache composite key (D6). The minimal pack under
``tests/fixtures/prompt_pack/`` is the local fixture so CI never depends on
the external prompt repository, and missing/invalid packs raise a clear
``PromptPackError`` instead of falling back silently.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from pf1e_simulator.config import (
ENV_LLM_API_KEY,
ENV_PROMPT_PACK_DIR,
env_path,
env_str,
llm_api_key,
)
from pf1e_simulator.prompt import (
FewShotExample,
PromptPack,
PromptPackError,
clear_pack_cache,
load_prompt_pack,
)
FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
_FIXTURE_PACK_FS = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
_FIXTURE_VERSION = "0.1.0-test"
_FIXTURE_PROMPT_NAME = "nl_default"
def _write_pack(root: Path, *, version: str, prompts: dict[str, str]) -> None:
"""Write a minimal prompt pack directory (version.txt + <name>.md)."""
root.mkdir(parents=True, exist_ok=True)
(root / "version.txt").write_text(version, encoding="utf-8")
for name, text in prompts.items():
(root / f"{name}.md").write_text(text, encoding="utf-8")
def _write_examples(root: Path, examples: dict[str, str]) -> None:
"""Write <name>.examples.json files into an existing pack directory."""
for name, text in examples.items():
(root / f"{name}.examples.json").write_text(text, encoding="utf-8")
# ── config.py ──────────────────────────────────────────────────────────────
def test_env_str_and_path_round_trip(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
"""env_str/env_path read the environment and respect deletes."""
monkeypatch.setenv("PF1E_SIM_TEST_STR", "abc")
monkeypatch.setenv("PF1E_SIM_TEST_PATH", str(tmp_path / "x"))
assert env_str("PF1E_SIM_TEST_STR") == "abc"
assert env_str("PF1E_SIM_TEST_UNSET") is None
assert env_str("PF1E_SIM_TEST_UNSET", default="d") == "d"
assert env_path("PF1E_SIM_TEST_PATH") == tmp_path / "x"
assert env_path("PF1E_SIM_TEST_UNSET") is None
monkeypatch.delenv("PF1E_SIM_TEST_STR")
assert env_str("PF1E_SIM_TEST_STR") is None
def test_llm_api_key_reads_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""llm_api_key surfaces the dedicated env var (None when unset)."""
assert llm_api_key() is None
monkeypatch.setenv(ENV_LLM_API_KEY, "sk-test")
assert llm_api_key() == "sk-test"
# ── prompt.py — loading and validation ─────────────────────────────────────
def test_load_fixture_pack() -> None:
"""The local fixture loads: version pinned, prompts readable by name."""
pack = load_prompt_pack(FIXTURE_PACK)
assert pack.version == _FIXTURE_VERSION
assert pack.root == FIXTURE_PACK
text = pack.get(_FIXTURE_PROMPT_NAME)
assert "tactical director" in text
# The pack has exactly the fixture prompts (only <name>.md files count).
assert pack.get("nl_default") == (FIXTURE_PACK / "nl_default.md").read_text(
encoding="utf-8"
)
def test_prompt_version_stable_and_content_sensitive(tmp_path: Path) -> None:
"""prompt_version carries pack version + per-prompt content hash (D6)."""
pack = load_prompt_pack(FIXTURE_PACK)
a = pack.prompt_version(_FIXTURE_PROMPT_NAME)
b = pack.prompt_version(_FIXTURE_PROMPT_NAME)
assert a == b # deterministic within a loaded pack
assert a.startswith(f"{_FIXTURE_VERSION}:{_FIXTURE_PROMPT_NAME}:")
other = tmp_path / "other"
_write_pack(other, version=_FIXTURE_VERSION, prompts={_FIXTURE_PROMPT_NAME: "x"})
assert load_prompt_pack(other).prompt_version(_FIXTURE_PROMPT_NAME) != a
bumped = tmp_path / "bumped"
_write_pack(bumped, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
assert load_prompt_pack(bumped).prompt_version(_FIXTURE_PROMPT_NAME).startswith(
"1.0.0:"
)
def test_missing_prompt_raises_clear_error(tmp_path: Path) -> None:
"""A missing prompt name fails loudly, naming the expected file."""
other = tmp_path / "other"
_write_pack(other, version="1.0.0", prompts={"other": "x"})
with pytest.raises(PromptPackError, match=r"missing.*nl_default\.md"):
load_prompt_pack(other).get("nl_default")
def test_missing_pack_dir_raises(tmp_path: Path) -> None:
"""A nonexistent pack path is a clear error, not a silent fallback."""
with pytest.raises(PromptPackError, match="not found"):
load_prompt_pack(tmp_path / "nope")
def test_missing_version_file_raises(tmp_path: Path) -> None:
"""A pack without version.txt is invalid (traceability, section 14)."""
root = tmp_path / "noversion"
_write_pack(root, version="", prompts={})
(root / "version.txt").unlink()
with pytest.raises(PromptPackError, match="version\\.txt"):
load_prompt_pack(root)
def test_empty_version_file_raises(tmp_path: Path) -> None:
"""An empty version.txt is invalid (no untraceable pack)."""
root = tmp_path / "emptyversion"
_write_pack(root, version="", prompts={_FIXTURE_PROMPT_NAME: "x"})
with pytest.raises(PromptPackError, match="empty"):
load_prompt_pack(root)
def test_unset_pack_dir_raises(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Without PF1E_PROMPT_PACK_DIR, loading raises instead of falling back."""
monkeypatch.delenv(ENV_PROMPT_PACK_DIR, raising=False)
with pytest.raises(PromptPackError, match=ENV_PROMPT_PACK_DIR):
load_prompt_pack()
def test_pack_dir_from_env(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""load_prompt_pack() resolves the pack through the env var (section 15)."""
monkeypatch.setenv(ENV_PROMPT_PACK_DIR, str(FIXTURE_PACK))
assert load_prompt_pack().version == _FIXTURE_VERSION
monkeypatch.delenv(ENV_PROMPT_PACK_DIR)
def test_cache_and_clear(tmp_path: Path) -> None:
"""Repeated loads share the cached instance; clear_pack_cache drops it."""
root = tmp_path / "cached"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
first = load_prompt_pack(root)
assert load_prompt_pack(root) is first
clear_pack_cache()
assert load_prompt_pack(root) is not first
def test_load_returns_prompt_pack_instance() -> None:
"""The loaded object is a PromptPack (typed API for the client layer)."""
pack = load_prompt_pack(FIXTURE_PACK)
assert isinstance(pack, PromptPack)
assert pack.version == _FIXTURE_VERSION
# ── prompt.py — few-shot examples (P5.2) ────────────────────────────────────
def test_examples_absent_returns_empty() -> None:
"""A prompt without an examples file exposes an empty example tuple."""
pack = load_prompt_pack(FIXTURE_PACK)
assert pack.examples(_FIXTURE_PROMPT_NAME) == ()
def test_few_shot_fixture_pack_loads_examples() -> None:
"""prompt_pack_fs ships one teaching pair, parsed into FewShotExample."""
pack = load_prompt_pack(_FIXTURE_PACK_FS)
examples = pack.examples(_FIXTURE_PROMPT_NAME)
assert len(examples) == 1
example = examples[0]
assert isinstance(example, FewShotExample)
assert example.observation == {
"round": 1,
"combatant": "goblin",
"enemy": "orc",
"distance_ft": 10,
}
assert example.actions == ({"kind": "attack", "target_id": "orc"},)
def test_prompt_version_includes_examples_content() -> None:
"""With an examples file, the content hash covers it (cache key, D6)."""
plain = load_prompt_pack(FIXTURE_PACK)
with_examples = load_prompt_pack(_FIXTURE_PACK_FS)
# Same prompt text, same pack version: the digest must differ because the
# few-shot file is hashed alongside the prompt.
assert plain.get(_FIXTURE_PROMPT_NAME) == with_examples.get(_FIXTURE_PROMPT_NAME)
assert plain.prompt_version(_FIXTURE_PROMPT_NAME) != with_examples.prompt_version(
_FIXTURE_PROMPT_NAME
)
def test_prompt_version_byte_identical_without_examples(tmp_path: Path) -> None:
"""No examples file → digest is byte-identical to the pre-few-shot scheme."""
root = tmp_path / "plain"
_write_pack(root, version=_FIXTURE_VERSION, prompts={_FIXTURE_PROMPT_NAME: "x"})
pack = load_prompt_pack(root)
digest = pack.prompt_version(_FIXTURE_PROMPT_NAME).split(":")[-1]
assert len(digest) == 12
def test_examples_invalid_json_raises(tmp_path: Path) -> None:
"""A malformed examples file is a pack error, not a silent skip."""
root = tmp_path / "badjson"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(root, {_FIXTURE_PROMPT_NAME: "not json"})
with pytest.raises(PromptPackError, match="is not valid JSON"):
load_prompt_pack(root)
def test_examples_not_an_array_raises(tmp_path: Path) -> None:
"""The examples file must be a JSON array at the top level."""
root = tmp_path / "notarray"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(root, {_FIXTURE_PROMPT_NAME: '{"observation": {}}'})
with pytest.raises(PromptPackError, match="must be a JSON array of examples"):
load_prompt_pack(root)
def test_examples_item_not_object_raises(tmp_path: Path) -> None:
"""Each array entry must be an object with observation and actions."""
root = tmp_path / "itemnotobject"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(root, {_FIXTURE_PROMPT_NAME: "[42]"})
with pytest.raises(
PromptPackError,
match="must be an object with 'observation' and 'actions'",
):
load_prompt_pack(root)
def test_examples_unknown_key_raises(tmp_path: Path) -> None:
"""Unexpected keys inside an example are rejected (strict validation)."""
root = tmp_path / "unknownkey"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(
root,
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": [], "hint": 1}]'},
)
with pytest.raises(PromptPackError, match="unknown key\\(s\\): hint"):
load_prompt_pack(root)
def test_examples_observation_not_object_raises(tmp_path: Path) -> None:
"""'observation' must be a JSON object."""
root = tmp_path / "badobs"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(
root,
{_FIXTURE_PROMPT_NAME: '[{"observation": "goblin", "actions": []}]'},
)
with pytest.raises(PromptPackError, match="'observation' must be a JSON object"):
load_prompt_pack(root)
def test_examples_actions_not_array_raises(tmp_path: Path) -> None:
"""'actions' must be a JSON array."""
root = tmp_path / "badactions"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(
root,
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": {"kind": "wait"}}]'},
)
with pytest.raises(PromptPackError, match="'actions' must be a JSON array"):
load_prompt_pack(root)
def test_examples_action_entry_not_object_raises(tmp_path: Path) -> None:
"""Each action inside 'actions' must be an object."""
root = tmp_path / "badactionentry"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(
root,
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": ["jump"]}]'},
)
with pytest.raises(PromptPackError, match="must contain only JSON objects"):
load_prompt_pack(root)
def test_examples_empty_array_accepted(tmp_path: Path) -> None:
"""An empty examples array is valid and yields no teaching pairs."""
root = tmp_path / "emptyexamples"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(root, {_FIXTURE_PROMPT_NAME: "[]"})
pack = load_prompt_pack(root)
assert pack.examples(_FIXTURE_PROMPT_NAME) == ()
def test_orphan_examples_file_raises(tmp_path: Path) -> None:
"""An .examples.json without a matching <name>.md is a pack error."""
root = tmp_path / "orphan"
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
_write_examples(root, {"stray": "[]"})
with pytest.raises(PromptPackError, match="has no matching 'stray\\.md' prompt"):
load_prompt_pack(root)
+297
View File
@@ -0,0 +1,297 @@
"""Tests for the phase-2 prompt builder (P1.4)."""
from __future__ import annotations
import hashlib
import json
from dataclasses import fields
from pathlib import Path
from pf1e_simulator.combat import Action, Observation
from pf1e_simulator.nl_policy import KindValue
from pf1e_simulator.nl_policy import get_args as _get_args
from pf1e_simulator.prompt import PromptPack
from pf1e_simulator.prompt_builder import (
PromptBuilder,
_render_examples,
_supported_kinds,
action_schema,
render_candidates,
)
_FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
_FIXTURE_PACK_FS = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
_FIXTURE_VERSION = "0.1.0-test"
_FIXTURE_PROMPT_NAME = "nl_default"
def _obs() -> Observation:
"""Minimal valid observation for deterministic tests."""
return Observation(
round=1,
acting_combatant=0,
combatants=(
{"id": "goblin", "side": "players", "active": True, "pos": (1, 1)},
),
pairs=(),
terrain=((None,),),
elevation=((0,),),
)
def _pack() -> PromptPack:
return PromptPack.load(_FIXTURE_PACK)
def _builder() -> PromptBuilder:
return PromptBuilder(pack=_pack())
def test_supported_kinds_filters_unsupported() -> None:
kinds = _supported_kinds()
assert "swift" not in kinds
assert "free" not in kinds
assert "full_round" not in kinds
# 27 kinds remain
assert len(kinds) == 27
# Order preserved from source Literal
assert kinds == (
"attack", "move", "wait", "immediate", "full_attack", "charge",
"withdraw", "5foot_step", "cast_spell", "maneuver", "channel_energy",
"hex", "deed", "spell_combat", "inspire_courage", "retrieve_weapon",
"repair_weapon", "activate_power_attack", "delay", "ready",
"total_defense", "fight_defensively", "stand_up", "drop_prone",
"aid_another", "run", "hide",
)
def test_action_schema_structure() -> None:
schema = action_schema()
assert schema["type"] == "object"
assert schema["required"] == ["kind"]
assert schema["additionalProperties"] is False
props = schema["properties"]
# Properties match Action fields exactly
action_field_names = {f.name for f in fields(Action)}
assert set(props.keys()) == action_field_names
# Kind enum is the filtered 27
kind_prop = props["kind"]
assert kind_prop["type"] == "string"
assert kind_prop["enum"] == list(_supported_kinds())
def test_action_schema_spot_check_fragments() -> None:
schema = action_schema()
props = schema["properties"]
# target_id accepts either a string or null
assert props["target_id"] == {"type": ["string", "null"]}
# destination is a nullable two-int pair (prefixItems + min/maxItems)
dest = props["destination"]
assert dest["type"] == ["array", "null"]
assert dest["prefixItems"] == [{"type": "integer"}, {"type": "integer"}]
assert dest["minItems"] == 2
assert dest["maxItems"] == 2
# channel_exclude: string[] (array of strings)
ch_ex = props["channel_exclude"]
assert ch_ex["type"] == "array"
assert ch_ex["items"] == {"type": "string"}
# cast_defensively is a boolean field
assert props["cast_defensively"] == {"type": "boolean"}
# tumble is a boolean field
assert props["tumble"] == {"type": "boolean"}
# channel_mode: nullable enum
ch_mode = props["channel_mode"]
assert ch_mode["type"] == ["string", "null"]
assert ch_mode["enum"] == ["heal", "harm"]
# aid_mode: nullable enum
aid = props["aid_mode"]
assert aid["type"] == ["string", "null"]
assert aid["enum"] == ["attack", "ac"]
# trick_condition: nullable enum (6 values)
tc = props["trick_condition"]
assert tc["type"] == ["string", "null"]
assert tc["enum"] == ["dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"]
# ready_trigger: nullable enum (4 values)
rt = props["ready_trigger"]
assert rt["type"] == ["string", "null"]
assert rt["enum"] == ["enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"]
# metamagic: string[]
meta = props["metamagic"]
assert meta["type"] == "array"
assert meta["items"] == {"type": "string"}
# class_ability is a nullable string field
assert props["class_ability"] == {"type": ["string", "null"]}
def test_render_candidates() -> None:
# Single action with payload
rendered = render_candidates((Action(kind="move", destination=(2, 3)),))
assert rendered == ('0. move {"destination":[2,3]}',)
# Action with no payload (all defaults/None)
rendered = render_candidates((Action(kind="total_defense"),))
assert rendered == ("0. total_defense",)
# Action with target_id
rendered = render_candidates((Action(kind="attack", target_id="orc"),))
assert rendered == ('0. attack {"target_id":"orc"}',)
# Empty menu
assert render_candidates(()) == ()
# Multiple actions
rendered = render_candidates((
Action(kind="attack", target_id="a"),
Action(kind="move", destination=(1, 1)),
))
assert rendered == (
'0. attack {"target_id":"a"}',
'1. move {"destination":[1,1]}',
)
def test_prompt_builder_build_deterministic() -> None:
builder = _builder()
obs = _obs()
prompt1 = builder.build(obs)
prompt2 = builder.build(obs)
assert prompt1 == prompt2
def test_prompt_builder_sections_present() -> None:
builder = _builder()
obs = _obs()
prompt = builder.build(obs)
# Template content
assert "tactical director" in prompt
assert "battle state" in prompt
# Sections
assert "## observation\n" in prompt
assert "## plausible actions\n" in prompt
assert "## action schema\n" in prompt
# Schema is valid JSON and contains kind enum
schema_part = prompt.split("## action schema\n")[1]
parsed = json.loads(schema_part)
assert parsed["properties"]["kind"]["enum"] == list(_supported_kinds())
def test_prompt_version_format() -> None:
builder = _builder()
pv = builder.prompt_version(_FIXTURE_PROMPT_NAME)
assert pv.startswith(f"{_FIXTURE_VERSION}:{_FIXTURE_PROMPT_NAME}:")
digest = pv.split(":")[-1]
assert len(digest) == 12
assert all(c in "0123456789abcdef" for c in digest)
# Cross-check against actual pack content
pack = _pack()
content = pack.get(_FIXTURE_PROMPT_NAME)
expected_digest = hashlib.sha256(content.encode("utf-8")).hexdigest()[:12]
assert digest == expected_digest
def test_prompt_builder_custom_prompt_name() -> None:
builder = _builder()
obs = _obs()
# nl_default is the only fixture, but the API accepts any name
prompt = builder.build(obs, prompt_name=_FIXTURE_PROMPT_NAME)
assert "tactical director" in prompt
def test_prompt_version_delegates_to_pack() -> None:
builder = _builder()
pack = _pack()
assert builder.prompt_version(_FIXTURE_PROMPT_NAME) == pack.prompt_version(_FIXTURE_PROMPT_NAME)
def test_action_schema_cross_pin_with_nl_policy_kindvalue() -> None:
"""The kind enum in the schema must match nl_policy.KindValue exactly."""
schema = action_schema()
assert schema["properties"]["kind"]["enum"] == list(_get_args(KindValue))
# ---- few-shot examples (P5.2) ----------------------------------------------
_FS_PACK = PromptPack.load(_FIXTURE_PACK_FS)
def _fs_builder() -> PromptBuilder:
return PromptBuilder(pack=_FS_PACK)
def test_render_examples_block_format() -> None:
"""Few-shot pairs render as numbered observation-to-response blocks."""
rendered = _render_examples((_FS_PACK.examples(_FIXTURE_PROMPT_NAME)[0],))
assert rendered == (
"## few-shot examples\n"
"### example 1\n"
"observation:\n"
'{"combatant": "goblin", "distance_ft": 10, "enemy": "orc", "round": 1}\n'
"\n"
"response:\n"
'[{"kind": "attack", "target_id": "orc"}]'
)
def test_build_without_examples_unchanged() -> None:
"""Packs without examples keep the pre-few-shot bit-for-bit section order."""
plain = _builder().build(_obs())
assert "## few-shot examples" not in plain
# Template, then observation, then candidates, then schema — in order.
sections = [
"## observation\n",
"## plausible actions\n",
"## action schema\n",
]
positions = [plain.index(section) for section in sections]
assert positions == sorted(positions)
# The template comes first, unmodified.
assert plain.startswith("You are the tactical director")
def test_build_with_examples_inserts_between_template_and_observation() -> None:
"""The few-shot section sits between the template and the observation."""
prompt = _fs_builder().build(_obs())
assert "## few-shot examples\n" in prompt
assert prompt.index("## few-shot examples\n") < prompt.index("## observation\n")
# Template still leads the prompt.
assert prompt.startswith("You are the tactical director")
# Observation/candidates/schema keep their relative order.
assert prompt.index("## observation\n") < prompt.index("## plausible actions\n")
assert prompt.index("## plausible actions\n") < prompt.index("## action schema\n")
def test_build_with_examples_includes_example_content() -> None:
"""The rendered example content is embedded verbatim (sorted keys)."""
prompt = _fs_builder().build(_obs())
assert '{"combatant": "goblin", "distance_ft": 10, "enemy": "orc", "round": 1}' in prompt
assert '[{"kind": "attack", "target_id": "orc"}]' in prompt
assert "### example 1" in prompt
def test_empty_examples_array_no_section(tmp_path: Path) -> None:
"""An empty examples array yields no few-shot section (not an error)."""
root = tmp_path / "emptyfs"
root.mkdir(parents=True, exist_ok=True)
(root / "version.txt").write_text("1.0.0", encoding="utf-8")
(root / "nl_default.md").write_text("T", encoding="utf-8")
(root / "nl_default.examples.json").write_text("[]", encoding="utf-8")
pack = PromptPack.load(root)
assert pack.examples("nl_default") == ()
prompt = PromptBuilder(pack=pack).build(_obs())
assert "## few-shot examples" not in prompt
+336
View File
@@ -0,0 +1,336 @@
"""Phase-2 real provider client (docs/phase-2 P5.1, sections 15/16).
``OpenAILLMClient`` (``provider.py``) speaks the OpenAI chat-completions
wire format over ``urllib.request`` with zero HTTP dependency. These tests
pin the wire format, the error taxonomy (transient 429/5xx retried, 4xx
rejected, invalid payload rejected), the env-key resolution, the token
ledger and the cache behaviour - all through an injectable fake transport,
never the network. Every key used below is a fake test credential.
"""
from __future__ import annotations
import io
import json
from typing import TYPE_CHECKING, Self
from urllib.error import HTTPError
import pytest
from pf1e_simulator.client import (
RetryExhaustedError,
estimate_tokens,
)
from pf1e_simulator.config import (
ENV_LLM_API_KEY,
ENV_LLM_BASE_URL,
ENV_LLM_MODEL,
)
from pf1e_simulator.provider import (
OpenAILLMClient,
ProviderRejectedError,
ProviderResponseError,
)
if TYPE_CHECKING:
from collections.abc import Sequence
from urllib.request import Request
_KEY = "sk-test"
_ENV_KEY = "sk-env-key"
_PROMPT_VERSION = "fixture:provider:0"
_BASE_URL = "https://api.openai.com/v1"
_MODEL = "gpt-4o-mini"
class _FakeResponse:
"""Minimal ``urllib.response`` double: status + body, context manager."""
def __init__(self, status: int, body: bytes) -> None:
self.status = status
self._body = body
def read(self) -> bytes:
return self._body
def __enter__(self) -> Self:
return self
def __exit__(self, _exc_type: object, _exc_value: object, _traceback: object) -> None:
return None
class _FakeTransport:
"""Injectable ``urlopen``: replays a script, keeping the last entry.
An ``Exception`` entry is raised; provider code sees it exactly as it
would see a real transport failure.
"""
def __init__(self, script: Sequence[_FakeResponse | Exception]) -> None:
self._script = tuple(script)
self._calls: list[tuple[Request, float]] = []
def __call__(self, request: Request, *, timeout: float) -> _FakeResponse:
self._calls.append((request, timeout))
index = min(len(self._calls) - 1, len(self._script) - 1)
entry = self._script[index]
if isinstance(entry, Exception):
raise entry
return entry
@property
def calls(self) -> tuple[tuple[Request, float], ...]:
return tuple(self._calls)
def _http_error(status: int) -> HTTPError:
return HTTPError(
f"{_BASE_URL}/chat/completions",
status,
"simulated HTTP error",
hdrs={},
fp=io.BytesIO(b"{}"),
)
def _ok_response(*, content: str = "action json", include_usage: bool = True) -> _FakeResponse:
body: dict[str, object] = {
"choices": [{"message": {"content": content}}],
}
if include_usage:
body["usage"] = {"prompt_tokens": 12, "completion_tokens": 7}
return _FakeResponse(200, json.dumps(body).encode("utf-8"))
def _fake_transport(script: Sequence[_FakeResponse | Exception]) -> _FakeTransport:
return _FakeTransport(script)
def _client(
transport: _FakeTransport,
*,
api_key: str = _KEY,
retries: int = 2,
timeout_s: float = 30.0,
base_url: str = _BASE_URL,
model: str = _MODEL,
) -> OpenAILLMClient:
return OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
model=model,
api_key=api_key,
base_url=base_url,
timeout_s=timeout_s,
urlopen=transport,
retries=retries,
sleep=lambda _: None,
)
# ── wire format ─────────────────────────────────────────────────────────────
def test_wire_format_matches_openai_chat_completions() -> None:
"""The request is a POST to /v1/chat/completions with the raw body."""
transport = _fake_transport((_ok_response(),))
client = _client(transport)
assert client.complete("hello") == "action json"
assert len(transport.calls) == 1
request, timeout = transport.calls[0]
assert timeout == 30.0
assert request.full_url == f"{_BASE_URL}/chat/completions"
assert request.method == "POST"
assert request.get_header("Authorization") == f"Bearer {_KEY}"
# urllib capitalizes header keys via str.capitalize(): "Content-Type" is
# stored as "Content-type".
assert request.get_header("Content-type") == "application/json"
assert request.data is not None
body = json.loads(request.data)
assert body["model"] == _MODEL
assert body["messages"] == [{"role": "user", "content": "hello"}]
assert body["temperature"] == 0.0
assert body["max_tokens"] == 1024
assert body["top_p"] == 1.0
assert body["response_format"] == {"type": "json_object"}
def test_ledger_records_provider_usage() -> None:
"""Provider-reported token counts land in the client ledger."""
transport = _fake_transport((_ok_response(),))
client = _client(transport)
client.complete("hello")
assert client.stats.calls == 1
assert client.stats.prompt_tokens == 12
assert client.stats.completion_tokens == 7
assert client.stats.cost_usd == 0.0
assert client.stats.cache_hits == 0
assert client.stats.cache_misses == 1
def test_usage_estimated_when_provider_omits_it() -> None:
"""Without a usage block, tokens are estimated from character counts."""
transport = _fake_transport((_ok_response(include_usage=False),))
client = _client(transport)
client.complete("hello")
assert client.stats.prompt_tokens == estimate_tokens("hello")
assert client.stats.completion_tokens == estimate_tokens("action json")
# ── key resolution ──────────────────────────────────────────────────────────
def test_missing_or_empty_api_key_raises(monkeypatch: pytest.MonkeyPatch) -> None:
"""A missing or empty key is rejected with the env-var name in the error."""
monkeypatch.delenv(ENV_LLM_API_KEY, raising=False)
with pytest.raises(ValueError, match="PF1E_LLM_API_KEY"):
OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
api_key=None,
base_url=_BASE_URL,
urlopen=_fake_transport((_ok_response(),)),
)
with pytest.raises(ValueError, match="PF1E_LLM_API_KEY"):
OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
api_key="",
base_url=_BASE_URL,
urlopen=_fake_transport((_ok_response(),)),
)
def test_api_key_read_from_environment(monkeypatch: pytest.MonkeyPatch) -> None:
"""With no explicit key, PF1E_LLM_API_KEY is used for the bearer token."""
monkeypatch.setenv(ENV_LLM_API_KEY, _ENV_KEY)
transport = _fake_transport((_ok_response(),))
client = OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
api_key=None,
base_url=_BASE_URL,
urlopen=transport,
sleep=lambda _: None,
)
client.complete("hello")
request, _timeout = transport.calls[0]
assert request.get_header("Authorization") == f"Bearer {_ENV_KEY}"
# ── error taxonomy and retries ──────────────────────────────────────────────
def test_rate_limit_retries_then_succeeds() -> None:
"""HTTP 429 is transient: with retries=2 the third attempt succeeds."""
transport = _fake_transport(
(_http_error(429), _http_error(429), _ok_response()),
)
client = _client(transport, retries=2)
assert client.complete("hello") == "action json"
assert client.calls == 3
def test_transient_oserror_retries_then_exhausts() -> None:
"""A transport error is retried, then RetryExhaustedError names it."""
transport = _fake_transport((TimeoutError("simulated timeout"),))
client = _client(transport, retries=1)
with pytest.raises(RetryExhaustedError, match="unreachable"):
client.complete("hello")
assert client.calls == 2
def test_permanent_4xx_is_not_retried() -> None:
"""HTTP 401 is a permanent rejection: one attempt, then raise."""
transport = _fake_transport((_http_error(401),))
client = _client(transport, retries=3)
with pytest.raises(ProviderRejectedError, match="HTTP 401"):
client.complete("hello")
assert client.calls == 1
def test_server_error_retries_then_exhausts() -> None:
"""HTTP 500 is transient: retries exhausted raises with the status."""
transport = _fake_transport((_http_error(500),))
client = _client(transport, retries=1)
with pytest.raises(RetryExhaustedError, match="HTTP 500"):
client.complete("hello")
assert client.calls == 2
# ── payload validation ──────────────────────────────────────────────────────
def test_invalid_json_body_raises_provider_response_error() -> None:
"""A 2xx body that is not JSON is a permanent response error."""
transport = _fake_transport((_FakeResponse(200, b"<html>oops</html>"),))
client = _client(transport)
with pytest.raises(ProviderResponseError, match="not valid JSON"):
client.complete("hello")
@pytest.mark.parametrize(
("payload", "message"),
[
({"choices": []}, "no choices"),
({"choices": [{}]}, "message"),
({"choices": [{"message": {}}]}, "content"),
([], "not a JSON object"),
],
)
def test_malformed_payload_raises_provider_response_error(
payload: object,
message: str,
) -> None:
"""Non-chat-completions payloads are rejected with a stable message."""
transport = _fake_transport(
(_FakeResponse(200, json.dumps(payload).encode("utf-8")),),
)
client = _client(transport)
with pytest.raises(ProviderResponseError, match=message):
client.complete("hello")
# ── caching and environment defaults ────────────────────────────────────────
def test_cache_serves_second_call_without_provider() -> None:
"""An identical second call is served from the cache, no wire traffic."""
transport = _fake_transport((_ok_response(),))
client = _client(transport)
assert client.complete("hello") == "action json"
assert client.complete("hello") == "action json"
assert len(transport.calls) == 1
assert client.calls == 1
assert client.cache_hits == 1
assert client.cache_misses == 1
def test_base_url_and_model_read_from_environment(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""PF1E_LLM_BASE_URL/PF1E_LLM_MODEL drive the endpoint and the body."""
monkeypatch.setenv(ENV_LLM_BASE_URL, "http://localhost:8080/v1/")
monkeypatch.setenv(ENV_LLM_MODEL, "local-model")
transport = _fake_transport((_ok_response(),))
client = OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
api_key=_KEY,
base_url=None,
model=None,
urlopen=transport,
sleep=lambda _: None,
)
client.complete("hello")
request, _timeout = transport.calls[0]
assert request.full_url == "http://localhost:8080/v1/chat/completions"
assert request.data is not None
assert json.loads(request.data)["model"] == "local-model"
def test_invalid_timeout_rejected() -> None:
"""A non-positive timeout_s is a construction error, not a wire call."""
with pytest.raises(ValueError, match="timeout_s"):
OpenAILLMClient(
prompt_version=_PROMPT_VERSION,
api_key=_KEY,
base_url=_BASE_URL,
timeout_s=0,
)
+234
View File
@@ -0,0 +1,234 @@
"""Phase-2 replay log (docs/phase-2-llm-architecture.md P0.5).
``replay.py`` records every ``(observation, action)`` pair a policy produced,
keyed by the execution-order ``call_seq``, and serializes the whole battle
(spec + seed + round cap + calls) to JSONL. ``replay()`` turns the log back
into an observation-only policy returning the recorded actions.
These tests pin the contract: JSONL round-trips to canonically-equal records
(``observation_key`` ignores JSON's tuple/list collapse), a replayed battle is
bit-identical to the recorded one (including from a serialized log), a
duplicate ``call_seq`` is rejected, and playback fails loudly on a mismatched
observation and once the log is exhausted.
"""
from __future__ import annotations
from dataclasses import replace
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.candidates import candidates
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.replay import (
ReplayError,
ReplayExhaustedError,
ReplayLog,
ReplayMismatchError,
ReplayRecord,
observation_key,
replay,
)
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
if TYPE_CHECKING:
from pf1e_simulator.combat import Action, CombatantState, Observation
from pf1e_simulator.models import Combatant
from pf1e_simulator.runner import EncounterReport
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
SEED = 7
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _duel_spec() -> EncounterSpec:
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
),
)
class _Recorder:
"""Legacy two-argument policy: log every call, then pick ``candidates[0]``."""
def __init__(self, spec: EncounterSpec, seed: int, round_cap: int) -> None:
self.log = ReplayLog(spec=spec, seed=seed, round_cap=round_cap)
def policy(
self, observation: Observation, _state: CombatantState
) -> tuple[Action, ...]:
action = candidates(observation)[0]
self.log.add(
ReplayRecord(
call_seq=len(self.log),
observation=observation,
action=action,
)
)
return (action,)
def _record(spec: EncounterSpec, *, seed: int = SEED) -> tuple[_Recorder, EncounterReport]:
"""Record one battle and return the recorder plus its captured report."""
registry = load_spell_registry(SPELLS_DIR)
recorder = _Recorder(spec, seed=seed, round_cap=spec.round_cap)
report = run_encounter(
spec,
runs=1,
seed=seed,
llm_policy=recorder.policy,
spell_registry=registry,
capture_first_transcript=True,
)
return recorder, report
def test_jsonl_round_trip_preserves_records() -> None:
"""A log serializes to JSONL and loads back to structurally-equal records."""
spec = _duel_spec()
recorder, _report = _record(spec)
log = recorder.log
assert len(log) > 0
assert [record.call_seq for record in log.records] == list(range(len(log)))
text = log.to_jsonl()
restored = ReplayLog.from_jsonl(text)
assert len(restored) == len(log)
assert [record.call_seq for record in restored.records] == [
record.call_seq for record in log.records
]
assert [
observation_key(record.observation) for record in restored.records
] == [observation_key(record.observation) for record in log.records]
assert [record.action for record in restored.records] == [
record.action for record in log.records
]
assert restored.spec == log.spec
assert restored.seed == log.seed
assert restored.round_cap == log.round_cap
assert restored.to_jsonl() == text
def test_replay_reproduces_the_recorded_battle() -> None:
"""Replaying the in-memory log matches the recorded battle exactly."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
recorder, recorded = _record(spec)
replayed = run_encounter(
spec,
runs=1,
seed=SEED,
llm_policy=replay(recorder.log),
spell_registry=registry,
capture_first_transcript=True,
)
assert replayed.wins == recorded.wins
assert replayed.draws == recorded.draws
assert replayed.avg_rounds == recorded.avg_rounds
assert replayed.transcript == recorded.transcript
def test_replay_from_serialized_log_reproduces_the_battle() -> None:
"""A log thawed from JSONL replays the battle bit-for-bit."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
recorder, recorded = _record(spec)
restored = ReplayLog.from_jsonl(recorder.log.to_jsonl())
replayed = run_encounter(
spec,
runs=1,
seed=SEED,
llm_policy=replay(restored),
spell_registry=registry,
capture_first_transcript=True,
)
assert replayed.wins == recorded.wins
assert replayed.draws == recorded.draws
assert replayed.avg_rounds == recorded.avg_rounds
assert replayed.transcript == recorded.transcript
def test_replay_rejects_a_mismatched_observation() -> None:
"""An observation that differs from the recorded one raises a mismatch error."""
spec = _duel_spec()
recorder, _report = _record(spec)
log = recorder.log
policy = replay(log)
first = log.records[0].observation
tampered = replace(first, round=first.round + 1)
with pytest.raises(ReplayMismatchError):
policy(tampered)
def test_replay_returns_each_recorded_action_then_exhausts() -> None:
"""Playback walks every record in order, then raises once past the end."""
spec = _duel_spec()
recorder, _report = _record(spec)
log = recorder.log
policy = replay(log)
for record in log.records:
assert policy(record.observation) == (record.action,)
with pytest.raises(ReplayExhaustedError):
policy(log.records[-1].observation)
def test_replay_log_rejects_a_duplicate_call_seq() -> None:
"""Adding two records with the same ``call_seq`` is an ambiguity error."""
spec = _duel_spec()
recorder, _report = _record(spec)
record = recorder.log.records[-1]
with pytest.raises(ReplayError, match="duplicate call_seq"):
recorder.log.add(
ReplayRecord(
call_seq=record.call_seq,
observation=record.observation,
action=record.action,
)
)
def test_replay_log_rejects_an_out_of_order_call_seq() -> None:
"""A ``call_seq`` below the last one is an ambiguity error (not a duplicate)."""
spec = _duel_spec()
recorder, _report = _record(spec)
record = recorder.log.records[0]
with pytest.raises(ReplayError, match="is not after"):
recorder.log.add(
ReplayRecord(
call_seq=record.call_seq,
observation=record.observation,
action=record.action,
)
)
def test_from_jsonl_rejects_garbage() -> None:
"""An empty or unparseable payload raises ``ReplayError``, not a raw error."""
with pytest.raises(ReplayError):
ReplayLog.from_jsonl("")
with pytest.raises(ReplayError):
ReplayLog.from_jsonl("{not json}")
+599
View File
@@ -0,0 +1,599 @@
"""Tests for the robustness layer (P4.2/P4.3, docs section 10)."""
from __future__ import annotations
import itertools
from pathlib import Path
import pytest
from pf1e_simulator import robustness
from pf1e_simulator.adversary import AdversarySpec
from pf1e_simulator.robustness import (
DangerCell,
MatchupBenchmark,
RobustnessEntry,
SensitivityEntry,
adversarial_analysis,
danger_map,
format_danger_map,
format_frontier,
format_matchup,
format_robustness,
format_sensitivity,
frontier,
matchup_benchmark,
sensitivity,
)
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec
from pf1e_simulator.strategy import StrategySpec
ROOT = Path(__file__).resolve().parent.parent
RUNS = 5
SEED = 7
ROUND_CAP = 20
PARTY = "players"
ARENA = ROOT / "data" / "maps" / "sample_arena.yaml"
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
ORC = ROOT / "data" / "monsters" / "orc.json"
def _scenario(
*, name: str = "goblins-vs-orcs", sides: tuple[SideSpec, ...] | None = None
) -> ScenarioSpec:
return ScenarioSpec(
name=name,
map_file=ARENA,
sides=sides
or (
SideSpec(PARTY, "A", (GOBLIN, GOBLIN, GOBLIN)),
SideSpec("monsters", "B", (ORC, ORC)),
),
round_cap=ROUND_CAP,
)
def _party(*, name: str = "default") -> StrategySpec:
return StrategySpec(name=name)
def _three_side_scenario() -> ScenarioSpec:
return ScenarioSpec(
name="three-sides",
map_file=ARENA,
sides=(
SideSpec(PARTY, "A", (GOBLIN,)),
SideSpec("monsters", "B", (ORC,)),
SideSpec("spectators", "C", (GOBLIN,)),
),
round_cap=ROUND_CAP,
)
def test_matchup_requires_exactly_two_sides() -> None:
with pytest.raises(ValueError, match="exactly 2 sides"):
matchup_benchmark(
_three_side_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED
)
def test_matchup_requires_valid_party_side() -> None:
scenario = _scenario()
with pytest.raises(ValueError, match="party_side"):
matchup_benchmark(
scenario, _party(), AdversarySpec(), party_side="", runs=RUNS, seed=SEED
)
with pytest.raises(ValueError, match="is not a side"):
matchup_benchmark(
scenario,
_party(),
AdversarySpec(),
party_side="spectators",
runs=RUNS,
seed=SEED,
)
def test_matchup_reports_both_dangers_and_is_reproducible() -> None:
scenario = _scenario()
party = _party()
adversary = AdversarySpec()
first = matchup_benchmark(scenario, party, adversary, runs=RUNS, seed=SEED)
assert isinstance(first, MatchupBenchmark)
assert first.scenario == "goblins-vs-orcs"
assert first.party == "default"
assert first.adversary == "default"
assert first.runs == RUNS
assert first.seed == SEED
assert first.party_side == PARTY
assert first.report.runs == RUNS
for danger in (first.party_danger, first.adversary_danger):
assert 0.0 <= danger.p_survives <= 1.0
assert 0.0 <= danger.p_tpk <= 1.0
assert 0.0 <= danger.p_any_death <= 1.0
assert danger.expected_hp_lost >= 0.0
second = matchup_benchmark(scenario, party, adversary, runs=RUNS, seed=SEED)
assert first == second
assert second.party_danger == first.party_danger
def test_adversarial_analysis_alternates_and_converges() -> None:
scenario = _scenario()
party = _party()
specs = (
AdversarySpec(name="retreat", retreat_hp_fraction=0.5),
AdversarySpec(name="plain"),
)
entries = adversarial_analysis(
scenario,
party_candidates=[party],
adversary_candidates=specs,
runs=RUNS,
seed=SEED,
iterations=2,
)
assert len(entries) == 2
assert all(isinstance(entry, RobustnessEntry) for entry in entries)
assert [entry.iteration for entry in entries] == [1, 2]
# the adversary best response is the candidate with the worst party outlook
benchmarks = {
spec.name: matchup_benchmark(
scenario, party, spec, runs=RUNS, seed=SEED
)
for spec in specs
}
expected = min(
benchmarks.values(),
key=lambda b: (
b.party_danger.p_survives,
-b.party_danger.expected_hp_lost,
b.adversary,
),
)
assert all(entry.benchmark == expected for entry in entries)
assert entries[0].adversary.name == expected.adversary
def test_adversarial_party_best_response() -> None:
scenario = _scenario()
parties = (
StrategySpec(),
StrategySpec(name="bully", target_priority="lowest_hp"),
)
adversary = AdversarySpec()
entries = adversarial_analysis(
scenario,
party_candidates=parties,
adversary_candidates=[adversary],
runs=RUNS,
seed=SEED,
iterations=1,
)
benchmarks = {
spec.name: matchup_benchmark(
scenario, spec, adversary, runs=RUNS, seed=SEED
)
for spec in parties
}
expected = max(
benchmarks.values(),
key=lambda b: (
b.party_danger.p_survives,
-b.party_danger.expected_hp_lost,
b.party,
),
)
assert entries[0].benchmark == expected
def test_adversarial_candidates_required() -> None:
scenario = _scenario()
party = _party()
with pytest.raises(ValueError, match="party_candidates must be non-empty"):
adversarial_analysis(
scenario,
party_candidates=[],
adversary_candidates=[AdversarySpec()],
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="adversary_candidates must be non-empty"):
adversarial_analysis(
scenario,
party_candidates=[party],
adversary_candidates=[],
runs=RUNS,
seed=SEED,
)
def test_adversarial_budget_is_enforced_before_any_battle() -> None:
scenario = _scenario()
party = _party()
specs = (
AdversarySpec(name="retreat", retreat_hp_fraction=0.5),
AdversarySpec(name="plain"),
)
cost = 2 * (1 + len(specs)) * RUNS
with pytest.raises(ValueError, match="robustness budget exceeded"):
adversarial_analysis(
scenario,
party_candidates=[party],
adversary_candidates=specs,
runs=RUNS,
seed=SEED,
iterations=2,
max_total_runs=cost - 1,
)
entries = adversarial_analysis(
scenario,
party_candidates=[party],
adversary_candidates=specs,
runs=RUNS,
seed=SEED,
iterations=2,
max_total_runs=cost,
)
assert len(entries) == 2
def test_danger_map_is_3x3_party_major_with_variant_names() -> None:
scenario = _scenario()
cells = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
assert len(cells) == 9
assert all(isinstance(cell, DangerCell) for cell in cells)
order = [(cell.party_hp_fraction, cell.adversary_hp_fraction) for cell in cells]
assert order == [(p, a) for p in (0.5, 0.7, 1.0) for a in (0.5, 0.7, 1.0)]
assert cells[0].benchmark.scenario == "goblins-vs-orcs party_hp=0.5 adv_hp=0.5"
assert cells[-1].benchmark.scenario == "goblins-vs-orcs party_hp=1.0 adv_hp=1.0"
for cell in cells:
assert 0.0 <= cell.benchmark.party_danger.p_survives <= 1.0
def test_danger_map_rejects_out_of_range_fractions() -> None:
scenario = _scenario()
with pytest.raises(ValueError, match="fraction must be in"):
danger_map(
scenario, _party(), AdversarySpec(), fractions=(0.0,), runs=RUNS, seed=SEED
)
with pytest.raises(ValueError, match="fraction must be in"):
danger_map(
scenario, _party(), AdversarySpec(), fractions=(1.5,), runs=RUNS, seed=SEED
)
with pytest.raises(ValueError, match="fractions must be non-empty"):
danger_map(
scenario, _party(), AdversarySpec(), fractions=(), runs=RUNS, seed=SEED
)
def test_danger_map_is_reproducible() -> None:
scenario = _scenario()
first = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
second = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
assert first == second
def test_frontier_finds_reached_threshold() -> None:
scenario = _scenario()
result = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
min_fraction=0.25,
max_fraction=1.0,
precision=0.05,
runs=RUNS,
seed=SEED,
)
# expected hp lost is always >= 0, so the threshold is crossed everywhere:
# bisection converges to the smallest probed fraction, rounded to precision
assert result == pytest.approx(0.25)
def test_frontier_returns_none_when_threshold_unreachable() -> None:
scenario = _scenario()
result = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=1e9,
min_fraction=0.25,
max_fraction=1.0,
precision=0.05,
runs=RUNS,
seed=SEED,
)
assert result is None
def test_frontier_is_reproducible() -> None:
scenario = _scenario()
first = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
runs=RUNS,
seed=SEED,
)
second = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
runs=RUNS,
seed=SEED,
)
assert first == second
def test_frontier_returns_precision_multiple() -> None:
scenario = _scenario()
result = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
min_fraction=0.3,
max_fraction=1.0,
precision=0.1,
runs=RUNS,
seed=SEED,
)
assert result is not None
# bisection stops within precision of min_fraction; the result is the
# nearest precision multiple in [min_fraction, min_fraction + precision]
assert 0.3 <= result <= 0.4
assert result == pytest.approx(round(result / 0.1) * 0.1)
def test_frontier_p_survives_uses_inverse_threshold() -> None:
scenario = _scenario()
result = frontier(
scenario,
_party(),
AdversarySpec(),
criterion="p_survives",
target=0.9999,
min_fraction=0.5,
max_fraction=1.0,
precision=0.25,
runs=RUNS,
seed=SEED,
)
# convention: p_survives crosses when value <= target; a present result
# is a precision multiple within the probed range
if result is not None:
assert 0.5 <= result <= 0.75
assert result == pytest.approx(round(result / 0.25) * 0.25)
def test_frontier_validates_arguments() -> None:
scenario = _scenario()
with pytest.raises(ValueError, match="criterion must be one of"):
frontier(
scenario, _party(), AdversarySpec(), criterion="bogus", runs=RUNS, seed=SEED
)
with pytest.raises(ValueError, match="target must be in"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="p_tpk",
target=0.0,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="target must be in"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="p_survives",
target=1.0,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="target must be >= 0"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=-1.0,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="min_fraction must be > 0"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
min_fraction=0.0,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="max_fraction must be <= 1"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
max_fraction=1.5,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="min_fraction must be <= max_fraction"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
min_fraction=0.9,
max_fraction=0.5,
runs=RUNS,
seed=SEED,
)
with pytest.raises(ValueError, match="precision must be > 0"):
frontier(
scenario,
_party(),
AdversarySpec(),
criterion="expected_hp_lost",
target=0.0,
precision=0.0,
runs=RUNS,
seed=SEED,
)
def test_sensitivity_ranks_levers_and_reports_signed_deltas() -> None:
scenario = _scenario()
party = _party()
entries = sensitivity(scenario, party, runs=RUNS, seed=SEED)
assert len(entries) == 7
assert all(isinstance(entry, SensitivityEntry) for entry in entries)
assert {entry.levers for entry in entries} == {
"weakest_ac",
"lowest_hp",
"healer",
"spell_policy=none",
"retreat=0.25",
"retreat=0.5",
"use_charges=false",
}
for previous, current in itertools.pairwise(entries):
assert abs(previous.delta) >= abs(current.delta)
if abs(previous.delta) == abs(current.delta):
assert previous.levers <= current.levers
# delta is the signed change vs the all-default baseline
baseline = matchup_benchmark(
scenario, party, AdversarySpec(), runs=RUNS, seed=SEED
)
base = baseline.party_danger.expected_hp_lost
for entry in entries:
assert entry.delta == pytest.approx(
entry.benchmark.party_danger.expected_hp_lost - base
)
def test_sensitivity_rejects_unknown_criterion() -> None:
scenario = _scenario()
with pytest.raises(ValueError, match="criterion must be one of"):
sensitivity(scenario, _party(), criterion="bogus", runs=RUNS, seed=SEED)
def test_sensitivity_budget_is_enforced() -> None:
scenario = _scenario()
party = _party()
with pytest.raises(ValueError, match="robustness budget exceeded"):
sensitivity(
scenario, party, runs=RUNS, seed=SEED, max_total_runs=8 * RUNS - 1
)
entries = sensitivity(
scenario, party, runs=RUNS, seed=SEED, max_total_runs=8 * RUNS
)
assert len(entries) == 7
def test_format_matchup_ascii_aligned() -> None:
benchmark_ = matchup_benchmark(
_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED
)
text = format_matchup(benchmark_)
lines = text.splitlines()
assert lines[0].split() == [
"Scenario",
"Runs",
"Side",
"Win%",
"Survive",
"TPK",
">=1-death",
"HP-lost",
"Rounds",
]
assert lines[1] == "-" * len(lines[0])
assert "party" in lines[2]
assert "adversary" in lines[3]
assert all(len(line) == len(lines[0]) for line in lines)
assert text.isascii()
def test_format_robustness_ascii_aligned() -> None:
scenario = _scenario()
entries = adversarial_analysis(
scenario,
party_candidates=[_party()],
adversary_candidates=[AdversarySpec()],
runs=RUNS,
seed=SEED,
iterations=2,
)
text = format_robustness(entries)
lines = text.splitlines()
assert lines[0].split() == [
"Iter",
"Party",
"Adversary",
"P(party-survive)",
"P(party-TPK)",
"Party-HP",
"P(adv-survive)",
"P(adv-TPK)",
"Adv-HP",
]
assert len(lines) == 2 + 2
assert all(len(line) == len(lines[0]) for line in lines)
assert text.isascii()
def test_format_danger_map_ascii_aligned() -> None:
cells = danger_map(_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED)
text = format_danger_map(cells)
lines = text.splitlines()
assert lines[0].startswith("Party\\Adv")
assert "adv=0.5" in lines[0]
assert "party=0.5" in lines[2]
assert "party=1.0" in lines[4]
assert "/" in lines[2]
assert all(len(line) == len(lines[0]) for line in lines)
assert text.isascii()
def test_format_frontier_messages() -> None:
assert (
format_frontier("p_tpk", 0.2, None, min_fraction=0.25, max_fraction=1.0)
== "no frontier for p_tpk 0.2 in [0.25, 1.00]"
)
assert (
format_frontier("p_tpk", 0.2, 0.35, min_fraction=0.25, max_fraction=1.0)
== "frontier p_tpk 0.2: party hp_fraction = 0.35"
)
def test_format_sensitivity_ascii_aligned() -> None:
entries = sensitivity(_scenario(), _party(), runs=RUNS, seed=SEED)
text = format_sensitivity(entries, "expected_hp_lost")
lines = text.splitlines()
assert lines[0].split() == ["Rank", "Levers", "Value", "Delta"]
assert len(lines) == 2 + 7
assert lines[1].split()[-1].startswith(("+", "-"))
assert all(len(line) == len(lines[0]) for line in lines)
assert text.isascii()
def test_robustness_module_does_not_import_search() -> None:
source = Path(robustness.__file__).resolve().read_text(encoding="utf-8")
assert "search" not in source
+153
View File
@@ -0,0 +1,153 @@
"""Phase-2 runner policy entry points (docs/phase-2-llm-architecture.md P0.4).
``run_encounter`` grows two Phase-2 keyword arguments: ``llm_policy`` (the
observation-only/legacy policy every ``CombatEngine`` gets) and
``policy_factory`` (a per-run ``Policy`` builder called once per battle with
the run index). These tests pin the contract: the factory is invoked exactly
once per run in index order, its result is the run's policy (``None`` falls
back to ``default_policy``), the three entry points are mutually exclusive,
and an ``llm_policy`` really drives a full battle through the runner.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.combat import Action, default_policy
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
if TYPE_CHECKING:
from pf1e_simulator.combat import Observation, Policy
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _duel_spec() -> EncounterSpec:
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
),
)
def _wait_policy(_observation: Observation) -> tuple[Action, ...]:
"""An observation-only policy that never attacks (forces a draw)."""
return (Action(kind="wait"),)
def test_policy_factory_called_once_per_run_in_order() -> None:
"""The factory sees each run index once, and its policy drives the run."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
calls: list[int] = []
def factory(index: int) -> Policy | None:
calls.append(index)
return default_policy
report = run_encounter(
spec, runs=4, seed=3, policy_factory=factory, spell_registry=registry
)
direct = run_encounter(
spec, runs=4, seed=3, policy=default_policy, spell_registry=registry
)
assert calls == [0, 1, 2, 3]
# A per-run factory returning default_policy is bit-identical to passing
# default_policy directly (same seed stream, same policy each run).
assert report.wins == direct.wins
assert report.draws == direct.draws
assert report.avg_rounds == direct.avg_rounds
assert report.attrition == direct.attrition
def test_policy_factory_none_falls_back_to_default_policy() -> None:
"""A factory returning None leaves the engine on default_policy."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
report = run_encounter(
spec,
runs=2,
seed=5,
policy_factory=lambda _index: None,
spell_registry=registry,
)
direct = run_encounter(spec, runs=2, seed=5, spell_registry=registry)
assert report.wins == direct.wins
assert report.avg_rounds == direct.avg_rounds
def test_policy_entry_points_mutually_exclusive() -> None:
"""policy / llm_policy / policy_factory reject every conflicting pairing."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
def factory(_index: int) -> Policy | None:
return default_policy
with pytest.raises(ValueError, match="policy_factory and policy"):
run_encounter(
spec,
runs=1,
seed=1,
policy=default_policy,
policy_factory=factory,
spell_registry=registry,
)
with pytest.raises(ValueError, match="policy_factory and llm_policy"):
run_encounter(
spec,
runs=1,
seed=1,
llm_policy=_wait_policy,
policy_factory=factory,
spell_registry=registry,
)
with pytest.raises(ValueError, match="policy and llm_policy"):
run_encounter(
spec,
runs=1,
seed=1,
policy=default_policy,
llm_policy=_wait_policy,
spell_registry=registry,
)
def test_llm_policy_drives_a_full_battle_through_the_runner() -> None:
"""The observation-only policy is invoked each turn and settles the battle."""
spec = _duel_spec()
registry = load_spell_registry(SPELLS_DIR)
seen: list[int] = []
def wait(observation: Observation) -> tuple[Action, ...]:
seen.append(observation.round)
return (Action(kind="wait"),)
report = run_encounter(
spec, runs=1, seed=1, llm_policy=wait, spell_registry=registry
)
# Both combatants wait every round: the policy is called for every turn
# and nobody can win, so the battle draws at the round cap.
assert seen
assert report.draws == 1
assert report.wins == {"players": 0, "monsters": 0}
+206
View File
@@ -0,0 +1,206 @@
"""Phase-2 scenario spec tests (docs/phase-2-llm-architecture.md 3.1, 6, 10 P3.1).
``ScenarioSpec``/``SideSpec`` are the reusable scenario search space of
section 6; ``build_encounter`` compiles one deterministically into a runner
``EncounterSpec`` (map + sides + round cap), and ``scenario_variants``
implements the deterministic per-side HP-fraction sampling of section 3.1.
These tests pin the round-trip (side order, ids, hp scaling, round cap), the
validation errors of both layers, the variant naming/scaling, determinism,
and one cheap integration run proving the compiled encounter is playable.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from pf1e_simulator.runner import run_encounter
from pf1e_simulator.scenarios import (
ScenarioSpec,
SideSpec,
build_encounter,
scenario_variants,
)
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
ORC = ROOT / "data" / "monsters" / "orc.json"
def _scenario(goblins: int = 3, orcs: int = 2) -> ScenarioSpec:
return ScenarioSpec(
name="goblins-vs-orcs",
map_file=MAP,
sides=(
SideSpec(
name="players",
zone="A",
combatant_files=tuple(GOBLIN for _ in range(goblins)),
),
SideSpec(
name="monsters",
zone="B",
combatant_files=tuple(ORC for _ in range(orcs)),
),
),
)
def test_build_encounter_round_trip() -> None:
"""Side order/names/zones, combatant counts and ids, round cap pass through."""
spec = build_encounter(_scenario(goblins=3, orcs=2))
assert [side.name for side in spec.sides] == ["players", "monsters"]
assert [side.zone for side in spec.sides] == ["A", "B"]
assert [len(side.combatants) for side in spec.sides] == [3, 2]
assert [c.id for c in spec.sides[0].combatants] == ["goblin", "goblin", "goblin"]
assert [c.id for c in spec.sides[1].combatants] == ["orc", "orc"]
assert spec.round_cap == 100
assert spec.map.deployment == {"players": "A", "monsters": "B"}
def test_build_encounter_hp_fraction() -> None:
"""hp_fraction 0.5 halves hp_max (rounded); 1.0 leaves it identical."""
half = build_encounter(
ScenarioSpec(
name="half-hp",
map_file=MAP,
sides=(
SideSpec(
name="players",
zone="A",
combatant_files=(GOBLIN,),
hp_fraction=0.5,
),
SideSpec(
name="monsters",
zone="B",
combatant_files=(ORC,),
hp_fraction=0.5,
),
),
)
)
assert half.sides[0].combatants[0].hp_max == 3
assert half.sides[1].combatants[0].hp_max == 3
full = build_encounter(_scenario(goblins=1, orcs=1))
assert full.sides[0].combatants[0].hp_max == 6
assert full.sides[1].combatants[0].hp_max == 6
def test_scenario_spec_validation() -> None:
"""Empty name, empty sides, bad hp_fraction, and round_cap 0 all raise."""
with pytest.raises(ValueError, match="name"):
ScenarioSpec(name="", map_file=MAP, sides=(_scenario().sides[0],))
with pytest.raises(ValueError, match="sides"):
ScenarioSpec(name="x", map_file=MAP, sides=())
for fraction in (0.0, 1.1, -0.5):
with pytest.raises(ValueError, match="hp_fraction"):
ScenarioSpec(
name="x",
map_file=MAP,
sides=(
SideSpec(
name="players",
zone="A",
combatant_files=(GOBLIN,),
hp_fraction=fraction,
),
),
)
with pytest.raises(ValueError, match="round_cap"):
ScenarioSpec(
name="x",
map_file=MAP,
sides=(_scenario().sides[0],),
round_cap=0,
)
def test_build_encounter_validation() -> None:
"""Shared zone letters, undeclared zones, and empty combatant lists raise."""
with pytest.raises(ValueError, match="share zone letters"):
build_encounter(
ScenarioSpec(
name="x",
map_file=MAP,
sides=(
SideSpec(name="a", zone="A", combatant_files=(GOBLIN,)),
SideSpec(name="b", zone="A", combatant_files=(ORC,)),
),
)
)
with pytest.raises(ValueError, match="not declared in map deployment"):
build_encounter(
ScenarioSpec(
name="x",
map_file=MAP,
sides=(
SideSpec(name="a", zone="A", combatant_files=(GOBLIN,)),
SideSpec(name="b", zone="C", combatant_files=(ORC,)),
),
)
)
with pytest.raises(ValueError, match="combatant file"):
build_encounter(
ScenarioSpec(
name="x",
map_file=MAP,
sides=(
SideSpec(name="a", zone="A", combatant_files=()),
SideSpec(name="b", zone="B", combatant_files=(ORC,)),
),
)
)
def test_variants_default_returns_base() -> None:
"""The default hp_fractions=(1.0,) yields exactly one variant equal to base."""
base = _scenario()
assert scenario_variants(base) == (base,)
def test_variants_fractions_names_and_scaling() -> None:
"""(0.5, 1.0) yields two variants with suffixed names and scaled HP."""
base = _scenario()
variants = scenario_variants(base, hp_fractions=(0.5, 1.0))
assert len(variants) == 2
assert variants[0].name == "goblins-vs-orcs hp=0.5"
assert all(side.hp_fraction == 0.5 for side in variants[0].sides)
assert variants[1] == base
half = build_encounter(variants[0])
assert half.sides[0].combatants[0].hp_max == 3
assert half.sides[1].combatants[0].hp_max == 3
def test_variants_empty_fractions_raise() -> None:
"""An empty hp_fractions tuple is a ValueError."""
with pytest.raises(ValueError, match="hp_fractions"):
scenario_variants(_scenario(), hp_fractions=())
def test_variants_invalid_fraction_raises() -> None:
"""A fraction outside (0, 1] is a ValueError."""
with pytest.raises(ValueError, match="hp_fraction"):
scenario_variants(_scenario(), hp_fractions=(1.5,))
def test_variants_deterministic() -> None:
"""Two calls with the same inputs produce equal variant tuples."""
base = _scenario()
fractions = (0.25, 0.5, 0.75, 1.0)
assert scenario_variants(base, hp_fractions=fractions) == scenario_variants(
base, hp_fractions=fractions
)
def test_build_encounter_playable() -> None:
"""A compiled encounter runs end-to-end (20 seeded battles)."""
spec = build_encounter(_scenario(goblins=3, orcs=2))
report = run_encounter(spec, runs=20, seed=1)
assert report.runs == 20
+210
View File
@@ -0,0 +1,210 @@
"""Phase-2 strategy search tests (docs/phase-2-llm-architecture.md 10 P3.2/P3.3).
``evaluate`` measures one strategy on one scenario; ``optimize`` searches the
strategy space (coarse grid, retreat refinement, LLM candidates) and returns a
ranked playbook; ``strategy_name`` renders deterministic labels; and
``format_playbook`` renders the playbook as aligned ASCII columns. These tests
pin the contract: benchmark fields, label rendering, rank ordering, candidate
deduplication, determinism, the ``max_total_runs`` cost bound, and the text
renderer. Small n (runs <= 5) keeps the file fast and deterministic.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec
from pf1e_simulator.search import evaluate, format_playbook, optimize, strategy_name
from pf1e_simulator.strategy import StrategySpec
if TYPE_CHECKING:
from pf1e_simulator.search import PlaybookEntry
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
ORC = ROOT / "data" / "monsters" / "orc.json"
RUNS = 5
SEED = 7
ROUND_CAP = 20
PARTY = "players"
def _scenario() -> ScenarioSpec:
return ScenarioSpec(
name="goblins-vs-orcs",
map_file=MAP,
sides=(
SideSpec(
name="players",
zone="A",
combatant_files=(GOBLIN, GOBLIN, GOBLIN),
),
SideSpec(
name="monsters",
zone="B",
combatant_files=(ORC, ORC),
),
),
round_cap=ROUND_CAP,
)
@pytest.fixture(scope="module")
def playbook() -> tuple[PlaybookEntry, ...]:
return optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
def test_evaluate_default_strategy() -> None:
"""evaluate returns a benchmark with sane danger metrics and win band."""
benchmark = evaluate(
StrategySpec(),
_scenario(),
party=PARTY,
runs=100,
seed=SEED,
)
assert benchmark.scenario_name == "goblins-vs-orcs"
assert benchmark.strategy == StrategySpec()
assert benchmark.danger.runs == 100
assert 0.0 <= benchmark.danger.p_survives <= 1.0
assert 0.0 <= benchmark.danger.p_tpk <= 1.0
assert benchmark.danger.expected_rounds > 0.0
assert set(benchmark.wins) == {"players", "monsters"}
assert benchmark.draws >= 0
low, high = benchmark.win_rate_band
assert 0.0 <= low <= high <= 1.0
def test_strategy_name() -> None:
"""Labels render non-default levers in field order; the name field is excluded."""
assert strategy_name(StrategySpec()) == "default"
assert strategy_name(StrategySpec(target_priority="weakest_ac")) == "weakest_ac"
assert (
strategy_name(StrategySpec(target_priority="weakest_ac", spell_policy="none"))
== "weakest_ac, spell_policy=none"
)
assert (
strategy_name(
StrategySpec(target_priority="lowest_hp", retreat_hp_fraction=0.5)
)
== "lowest_hp, retreat=0.5"
)
assert (
strategy_name(StrategySpec(name="custom", use_charges=False))
== "use_charges=false"
)
def test_optimize_small_ranks_and_default(playbook: tuple[PlaybookEntry, ...]) -> None:
"""A small optimize returns contiguous ranks ordered by the rank key."""
assert len(playbook) >= 24
assert [entry.rank for entry in playbook] == list(range(1, len(playbook) + 1))
assert any(entry.strategy == StrategySpec() for entry in playbook)
keys = [
(
-entry.benchmark.danger.p_survives,
entry.benchmark.danger.expected_hp_lost,
-entry.benchmark.danger.worst_decile_party_hp,
entry.benchmark.danger.expected_rounds,
strategy_name(entry.strategy),
)
for entry in playbook
]
assert keys == sorted(keys)
def test_optimize_deterministic() -> None:
"""Two optimize calls with the same inputs return identical playbooks."""
first = optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
second = optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
assert first == second
def test_optimize_candidates_deduped() -> None:
"""LLM candidates run once each; duplicates and grid duplicates collapse."""
in_grid = StrategySpec(target_priority="weakest_ac", spell_policy="none")
custom = StrategySpec(
target_priority="lowest_hp",
spell_policy="none",
use_charges=False,
use_class_abilities=False,
retreat_hp_fraction=0.75,
)
playbook = optimize(
_scenario(),
party=PARTY,
runs=RUNS,
seed=SEED,
candidates=(custom, custom, in_grid),
)
custom_matches = [entry for entry in playbook if entry.strategy == custom]
assert len(custom_matches) == 1
in_grid_matches = [entry for entry in playbook if entry.strategy == in_grid]
assert len(in_grid_matches) == 1
direct = evaluate(custom, _scenario(), party=PARTY, runs=RUNS, seed=SEED)
assert custom_matches[0].benchmark.danger == direct.danger
def test_budget_raises_below_required_total() -> None:
"""A max_total_runs below the required total raises before running."""
with pytest.raises(ValueError, match="budget") as excinfo:
optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED, max_total_runs=1)
assert "max_total_runs" in str(excinfo.value)
def test_budget_generous_cap_raises_nothing() -> None:
"""A generous max_total_runs cap does not raise."""
playbook = optimize(
_scenario(),
party=PARTY,
runs=RUNS,
seed=SEED,
max_total_runs=10_000,
)
assert len(playbook) >= 24
def test_format_playbook_columns_and_ascii(
playbook: tuple[PlaybookEntry, ...],
) -> None:
"""The renderer shows header tokens, a dash separator, aligned rows, ASCII."""
text = format_playbook(playbook, PARTY)
for token in (
"Rank",
"Strategy",
"Runs",
"P(survive)",
"P(TPK)",
"P(>=1 death)",
"HP lost",
"Worst-decile HP",
"Rounds",
"Win% (3σ)",
):
assert token in text
lines = text.splitlines()
assert len(lines) == 2 + len(playbook)
separator = lines[1]
assert separator
assert set(separator) <= {"-", " "}
assert len({len(line) for line in lines}) == 1
non_ascii = {char for char in text if ord(char) > 127}
assert non_ascii <= {"σ"}
top = playbook[0]
rate = top.benchmark.wins.get(PARTY, 0) / top.benchmark.danger.runs
low, high = top.benchmark.win_rate_band
expected = f"{rate * 100:.1f} [{low * 100:.1f}, {high * 100:.1f}]"
assert expected in text
+231
View File
@@ -0,0 +1,231 @@
"""Phase-2 Etage A strategy compiler tests (docs/phase-2-llm-architecture.md 10 P2.1).
Deterministic, offline: no network, no LLM. Pins the ``StrategySpec`` /
``compile_strategy`` contract: the all-default spec compiles to
``default_policy`` by identity, ``select_target`` honours each priority with
deterministic tie-breaks, non-default specs drive real battles through
``run_encounter`` bit-identically under a fixed seed, and invalid specs are
rejected at construction.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.combat import (
Action,
CombatantState,
CombatEngine,
default_policy,
)
from pf1e_simulator.grid import Grid
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
from pf1e_simulator.strategy import StrategySpec, compile_strategy, select_target
if TYPE_CHECKING:
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
_MAP_SPEC = load_map(MAP)
_SPELLS = load_spell_registry(SPELLS_DIR)
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _spec() -> EncounterSpec:
"""A fast 2v2 (goblin archer + orc vs orc + goblin) on the sample arena."""
return EncounterSpec(
map=_MAP_SPEC,
round_cap=20,
sides=(
Side(
name="players",
combatants=(_c("monsters/goblin.json"), _c("monsters/orc.json")),
zone="A",
),
Side(
name="monsters",
combatants=(_c("monsters/orc.json"), _c("monsters/goblin.json")),
zone="B",
),
),
)
def _state(
cid: str, side: str, pos: tuple[int, int], hp: int = 10, path: str = "monsters/orc.json"
) -> CombatantState:
"""A bare runtime state over a loaded combatant (no battle run)."""
combatant = _c(path).model_copy(update={"id": cid})
return CombatantState(combatant=combatant, side=side, pos=pos, hp=hp, resources_left={})
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
"""A live engine over the 2v2 map with the given states (no battle run)."""
return CombatEngine(SeededRng(1), Grid.from_spec(_MAP_SPEC), states, spell_registry=_SPELLS)
def test_compile_strategy_default_is_default_policy() -> None:
"""The all-default spec compiles to ``default_policy`` by identity."""
assert compile_strategy(StrategySpec()) is default_policy
def test_select_target_nearest() -> None:
"""nearest picks the closest active enemy (ties keep list order)."""
orc = _state("orc1", "monsters", (1, 1))
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
far_orc = _state("orc2", "players", (1, 16), hp=6)
engine = _mini_engine([orc, goblin, far_orc])
assert select_target(engine, orc, "nearest") is goblin
def test_select_target_weakest_ac() -> None:
"""weakest_ac picks the lowest total AC (goblin 16 > orc 13)."""
orc = _state("orc1", "monsters", (1, 1))
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
far_orc = _state("orc2", "players", (1, 16), hp=6)
engine = _mini_engine([orc, goblin, far_orc])
assert select_target(engine, orc, "weakest_ac") is far_orc
def test_select_target_lowest_hp() -> None:
"""lowest_hp picks the most wounded active enemy."""
orc = _state("orc1", "monsters", (1, 1))
healthy = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
wounded = _state("goblin2", "players", (1, 16), hp=1, path="monsters/goblin.json")
engine = _mini_engine([orc, healthy, wounded])
assert select_target(engine, orc, "lowest_hp") is wounded
def test_select_target_no_active_enemy() -> None:
"""No active enemy -> None for every priority."""
orc = _state("orc1", "monsters", (1, 1))
dying = _state("goblin1", "players", (1, 2), hp=-5, path="monsters/goblin.json")
engine = _mini_engine([orc, dying])
assert select_target(engine, orc, "nearest") is None
assert select_target(engine, orc, "weakest_ac") is None
assert select_target(engine, orc, "lowest_hp") is None
def test_compiled_policy_melee_ladder() -> None:
"""A non-default spec still resolves the melee ladder on a live engine."""
spec = StrategySpec(name="no-charge", use_charges=False)
policy = compile_strategy(spec)
assert policy is not default_policy
orc = _state("orc1", "monsters", (1, 1))
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
engine = _mini_engine([orc, goblin])
assert policy(engine, orc) == (Action(kind="full_attack", target_id="goblin1"),)
def test_compiled_policy_charge_disabled() -> None:
"""use_charges=False skips the charge and falls to step + full-attack."""
spec = StrategySpec(name="no-charge", use_charges=False)
policy = compile_strategy(spec)
# The celestial wolf is melee-only (Bite, reach 5 ft): at 2 cells no
# weapon_for short-circuit, so the ladder reaches the charge branch. The
# orc sits DIAGONALLY at (2, 3): straight ahead at (1, 3) the only
# attackable charge end is 1 cell away (below the 2-cell minimum charge
# move), while from (1, 1) the diagonal end (1, 3) is a legal 2-cell
# straight line AND a single 5-foot step to (1, 2) still enters reach.
wolf = _state("wolf1", "players", (1, 1), hp=13, path="monsters/celestial_wolf.json")
orc = _state("orc1", "monsters", (2, 3))
engine = _mini_engine([wolf, orc])
assert default_policy(engine, wolf) == (Action(kind="charge", target_id="orc1"),)
assert policy(engine, wolf) == (
Action(kind="5foot_step", target_id="orc1"),
Action(kind="full_attack", target_id="orc1"),
)
def test_compiled_policy_retreats_below_threshold() -> None:
"""retreat_hp_fraction withdraws below the threshold, not at/above it."""
spec = StrategySpec(name="coward", retreat_hp_fraction=0.5)
policy = compile_strategy(spec)
orc = _state("orc1", "monsters", (1, 1), hp=2) # 2 < 0.5 * 6 = 3
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
engine = _mini_engine([orc, goblin])
assert policy(engine, orc) == (Action(kind="withdraw", target_id="goblin1"),)
orc_ok = _state("orc2", "monsters", (1, 1), hp=3) # 3 is not below 3
engine_ok = _mini_engine([orc_ok, goblin])
assert all(a.kind != "withdraw" for a in policy(engine_ok, orc_ok))
def test_compiled_policy_spell_policy_none_skips_casting() -> None:
"""spell_policy="none" never proposes a cast even for a caster with slots."""
spec = StrategySpec(name="no-spells", spell_policy="none")
policy = compile_strategy(spec)
oni = _c("pcs/oni.json")
oni_state = CombatantState(
combatant=oni,
side="players",
pos=(1, 1),
hp=oni.hp_max,
resources_left={},
spell_slots_left={1: 4, 2: 3},
)
goblin = _state("goblin1", "monsters", (1, 2), hp=6, path="monsters/goblin.json")
engine = _mini_engine([oni_state, goblin])
# With the default spell policy the same setup casts (Shield of Faith,
# slot 1): no ally is below 50% HP and Sound Burst is an area spell, so
# _find_offensive_spell (top-level DamageEffect only) skips it.
assert default_policy(engine, oni_state) == (
Action(kind="cast_spell", target_id="oni", spell_name="Shield of Faith"),
)
# With spell_policy="none" the melee ladder runs instead.
actions = policy(engine, oni_state)
assert all(a.kind != "cast_spell" for a in actions)
def test_compiled_policy_class_abilities_disabled() -> None:
"""use_class_abilities=False skips channel/hex even when they would fire."""
spec = StrategySpec(name="no-class", use_class_abilities=False)
policy = compile_strategy(spec)
oni = _c("pcs/oni.json")
oni_state = CombatantState(
combatant=oni,
side="players",
pos=(1, 1),
hp=oni.hp_max,
resources_left={"channel_energy": 6},
spell_slots_left={1: 4, 2: 3},
)
hurt = _state("hurt", "players", (1, 3), hp=1, path="monsters/goblin.json")
goblin = _state("goblin1", "monsters", (1, 2), hp=6, path="monsters/goblin.json")
engine = _mini_engine([oni_state, hurt, goblin])
# With class abilities enabled the channel fires (ally below 50% hp).
assert default_policy(engine, oni_state) == (
Action(kind="channel_energy", target_id="goblin1", channel_mode="heal"),
)
# Disabled: the channel is skipped entirely.
actions = policy(engine, oni_state)
assert all(a.kind != "channel_energy" for a in actions)
def test_compiled_policy_deterministic_runs() -> None:
"""A compiled policy drives bit-identical battles under a fixed seed."""
spec = StrategySpec(name="focus", target_priority="weakest_ac")
policy = compile_strategy(spec)
_, runs_a = run_encounter(_spec(), runs=3, seed=7, policy=policy, collect_runs=True)
_, runs_b = run_encounter(_spec(), runs=3, seed=7, policy=policy, collect_runs=True)
assert runs_a == runs_b
def test_strategy_spec_validation() -> None:
"""Invalid specs are rejected at construction."""
with pytest.raises(ValueError, match="non-empty"):
StrategySpec(name="")
with pytest.raises(ValueError, match="within"):
StrategySpec(retreat_hp_fraction=1.5)
with pytest.raises(ValueError, match="within"):
StrategySpec(retreat_hp_fraction=-0.1)
+203
View File
@@ -0,0 +1,203 @@
"""Phase-2 sweep runner tests (docs/phase-2-llm-architecture.md 10 P2.3, D9/D10).
The ``SweepRunner`` executes a grid of (scenario x strategy) cells in
parallel, each cell running ``run_encounter(..., collect_runs=True)`` under
one policy and reporting the aggregate report plus the party's danger
metrics. These tests pin the contract: scenario-major grid order, bit-for-bit
determinism, thread-pool equivalence (workers=1 vs workers=4), the
``max_total_runs`` cost bound, constructor/cell validation, the equivalence
of a ``None`` policy cell with a direct ``run_encounter`` call, and the
aligned text renderer. All cells use small n (runs <= 5) so the file stays
fast and deterministic.
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.objectives import danger_metrics
from pf1e_simulator.phase2_spike import focus_fire_policy
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
from pf1e_simulator.sweep import SweepCell, SweepRunner, format_sweep
if TYPE_CHECKING:
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
RUNS = 5
SEED = 7
ROUND_CAP = 20
PARTY = "players"
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _spec(goblins: int, orcs: int) -> EncounterSpec:
return EncounterSpec(
map=load_map(MAP),
round_cap=ROUND_CAP,
sides=(
Side(
name="players",
combatants=tuple(_c("monsters/goblin.json") for _ in range(goblins)),
zone="A",
),
Side(
name="monsters",
combatants=tuple(_c("monsters/orc.json") for _ in range(orcs)),
zone="B",
),
),
)
VARIANT_A = _spec(goblins=3, orcs=2)
VARIANT_B = _spec(goblins=2, orcs=3)
SCENARIOS = (("variant_a", VARIANT_A), ("variant_b", VARIANT_B))
STRATEGIES = (("default", None), ("focus_fire_monsters", focus_fire_policy("monsters")))
def _cells() -> tuple[SweepCell, ...]:
return (
SweepCell(
scenario="variant_a",
strategy="default",
spec=VARIANT_A,
policy=None,
runs=RUNS,
seed=SEED,
),
SweepCell(
scenario="variant_b",
strategy="focus_fire_monsters",
spec=VARIANT_B,
policy=focus_fire_policy("monsters"),
runs=RUNS,
seed=SEED,
),
)
def test_run_grid_2x2_scenario_major() -> None:
"""A 2x2 grid returns 4 results in scenario-major order with sane metrics."""
runner = SweepRunner(party=PARTY, workers=2)
results = runner.run_grid(SCENARIOS, STRATEGIES, runs=RUNS, seed=SEED)
assert len(results) == 4
assert [r.cell.scenario for r in results] == [
"variant_a",
"variant_a",
"variant_b",
"variant_b",
]
assert [r.cell.strategy for r in results] == [
"default",
"focus_fire_monsters",
"default",
"focus_fire_monsters",
]
for result in results:
assert result.report.runs == RUNS
assert result.danger.runs == RUNS
assert 0.0 <= result.danger.p_survives <= 1.0
assert 0.0 <= result.danger.p_tpk <= 1.0
def test_run_is_deterministic() -> None:
"""Two runs over identical cells return identical frozen results."""
runner = SweepRunner(party=PARTY, workers=2)
cells = _cells()
first = runner.run(cells)
second = runner.run(cells)
assert first == second
def test_parallel_equivalence_workers_1_vs_4() -> None:
"""workers=1 and workers=4 produce identical results (thread-safe registry)."""
cells = _cells()
serial = SweepRunner(party=PARTY, workers=1).run(cells)
parallel = SweepRunner(party=PARTY, workers=4).run(cells)
assert serial == parallel
def test_cost_bound_raises_on_budget_exceeded() -> None:
"""Cells summing past max_total_runs raise a ValueError naming runs/budget."""
runner = SweepRunner(party=PARTY, workers=2)
cells = (
SweepCell(
scenario="a", strategy="s", spec=VARIANT_A, policy=None, runs=3, seed=1
),
SweepCell(
scenario="b", strategy="s", spec=VARIANT_B, policy=None, runs=3, seed=1
),
)
with pytest.raises(ValueError, match="runs") as excinfo:
runner.run(cells, max_total_runs=5)
assert "budget" in str(excinfo.value)
def test_validation_workers_party_runs() -> None:
"""workers=0, an empty party, and a runs=0 cell all raise ValueError."""
with pytest.raises(ValueError, match="workers"):
SweepRunner(party=PARTY, workers=0)
with pytest.raises(ValueError, match="party"):
SweepRunner(party="", workers=2)
runner = SweepRunner(party=PARTY, workers=2)
bad = SweepCell(
scenario="a", strategy="s", spec=VARIANT_A, policy=None, runs=0, seed=1
)
with pytest.raises(ValueError, match="runs"):
runner.run((bad,))
def test_policy_none_matches_direct_run_encounter() -> None:
"""A None-policy cell equals a direct run_encounter on the same scenario."""
runner = SweepRunner(party=PARTY, workers=2)
cell = SweepCell(
scenario="variant_a",
strategy="default",
spec=VARIANT_A,
policy=None,
runs=RUNS,
seed=SEED,
)
(result,) = runner.run((cell,))
report, outcomes = run_encounter(VARIANT_A, RUNS, SEED, collect_runs=True)
assert result.report == report
assert result.danger == danger_metrics(outcomes, PARTY)
def test_format_sweep_columns_and_win_pct() -> None:
"""The renderer shows header tokens, one row per result, and the party win%."""
runner = SweepRunner(party=PARTY, workers=2)
results = runner.run_grid(SCENARIOS, STRATEGIES, runs=RUNS, seed=SEED)
text = format_sweep(results, PARTY)
for token in ("Scenario", "Win%", "P(TPK)"):
assert token in text
lines = text.splitlines()
assert len(lines) == 2 + len(results)
first = results[0]
expected_win = f"{first.report.wins.get(PARTY, 0) / first.report.runs * 100:.1f}"
assert lines[2].split()[3] == expected_win
+279
View File
@@ -0,0 +1,279 @@
"""Phase-2 compact observation view (docs/phase-2-llm-architecture.md P0.2).
``llm_view.build_view`` + ``render`` turn the raw ``Observation`` snapshot
into a compact, deterministic text view for the LLM prompt. This module
measures the compaction on a representative caster-heavy encounter (same
shape as the measurement spike, but with its OWN spec builder - the spike
file is out of scope), prints a summary block for the orchestrator, and
pins LOOSE upper bounds so a regression (a dropped key re-added, a sparse
grid turned dense, ...) fails loudly. It also pins the structural contract:
sparse grids only carry non-default cells, dropped keys stay dropped, the
nested attack/spell expansions stay compact, and the render is
deterministic.
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
from typing import TYPE_CHECKING
from pf1e_simulator.combat import (
OBSERVATION_SCHEMA_VERSION,
Action,
CombatantState,
CombatEngine,
Observation,
default_policy,
)
from pf1e_simulator.grid import Grid
from pf1e_simulator.llm_view import build_view, render
from pf1e_simulator.loaders import load_combatant
from pf1e_simulator.map import load_map
from pf1e_simulator.rng import SeededRng
from pf1e_simulator.runner import EncounterSpec, Side, build_states
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
if TYPE_CHECKING:
import pytest
from pf1e_simulator.models import Combatant
ROOT = Path(__file__).resolve().parents[1]
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
# Loose ceilings (regression guards, not exact pins): the docs target a
# compact view of ~2-4k tokens against a raw snapshot of ~16k tokens, i.e.
# 12.5-25% of the raw size. 25% of raw bytes and a 4k-token byte ceiling
# (4 bytes/token) are the outer bounds.
MAX_COMPACT_RATIO = 0.25
MAX_COMPACT_BYTES = 16_000
# Keys the compact view deliberately drops from each combatant snapshot.
_DROPPED_COMBATANT_KEYS = frozenset(
{
"occupied_cells",
"abilities",
"skills",
"initiative_mod",
"initiative_total",
"temp_hp_rounds_left",
"features",
"feature_details",
"carried_items",
"weapon_damage",
"aware",
"alignment",
"armor_spell_failure",
"arcane_caster",
"run_rounds",
}
)
# Every pair key must survive (the pairs are the tactical core); the raw
# ``target_index`` is replaced by the stable combatant ``target`` id.
_PAIR_KEYS = frozenset(
{
"target",
"distance_ft",
"weapon_for",
"weapon_reason",
"range_penalty",
"can_charge",
"charge_reason",
"can_step5_to_attack",
"has_line_of_effect",
"flanking",
"movement_cost_ft",
"target_threatens_me",
"enemies_threatening_me",
"cover_bonus",
}
)
# Keys dropped from the nested attack/spell expansions.
_ATTACK_DROPPED_KEYS = frozenset({"splash", "hardness", "hit_points"})
_SPELL_DROPPED_KEYS = frozenset(
{"summary", "school", "half_on_save", "save_negates", "harmless", "components"}
)
class _Capture:
"""Policy wrapper that snapshots the first observation, then delegates."""
def __init__(self) -> None:
self.observation: Observation | None = None
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
if self.observation is None:
self.observation = engine._build_observation(state)
return default_policy(engine, state)
def _c(path: str) -> Combatant:
return load_combatant(ROOT / "data" / path)
def _representative_spec() -> EncounterSpec:
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars)."""
return EncounterSpec(
map=load_map(MAP),
round_cap=20,
sides=(
Side(
name="players",
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
zone="A",
),
Side(
name="monsters",
combatants=(
_c("monsters/lich.json"),
_c("monsters/thrune_agent.json"),
_c("monsters/orc.json"),
_c("monsters/goblin.json"),
),
zone="B",
),
),
)
def _first_observation(
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
) -> Observation:
grid = Grid.from_spec(spec.map)
states = build_states(spec)
cap = _Capture()
CombatEngine(
SeededRng(seed),
grid,
states,
round_cap=spec.round_cap,
policy=cap.policy,
spell_registry=registry,
).run()
assert cap.observation is not None
return cap.observation
def _emit(line: str) -> None:
sys.stdout.write(line + "\n")
def _is_default_terrain(semantics: dict[str, object]) -> bool:
"""Mirror of ``llm_view._is_default_terrain`` for the structural test."""
return (
semantics.get("move_cost") == 1
and not semantics.get("blocks_los")
and not semantics.get("cover")
and semantics.get("cover_bonus") == 0
)
def test_compact_view_size(capsys: pytest.CaptureFixture[str]) -> None:
"""Measure raw-vs-compact on the representative battle; pin loose bounds."""
registry = load_spell_registry(SPELLS_DIR)
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
raw = json.dumps(obs.to_dict())
view = build_view(obs)
compact = render(view)
raw_bytes = len(raw)
compact_bytes = len(compact)
# Determinism: rebuilding the view from the same observation renders
# byte-identically.
assert render(build_view(obs)) == compact
# Schema version is pinned to the engine's contract.
assert view.schema_version == OBSERVATION_SCHEMA_VERSION
# Loose bounds (regression guards, not exact pins).
assert compact_bytes < raw_bytes * MAX_COMPACT_RATIO
assert compact_bytes < MAX_COMPACT_BYTES
_emit("=== PHASE2 SUMMARY (compact observation view) ===")
_emit("encounter: 7 combatants, round_cap=20, seed=1")
_emit(f"raw observation bytes: {raw_bytes} (~{raw_bytes // 4} tokens)")
_emit(f"compact view bytes: {compact_bytes} (~{compact_bytes // 4} tokens)")
_emit(f"compaction ratio: {compact_bytes / raw_bytes:.1%} of raw")
_emit(f"combatants in view: {len(view.combatants)}")
_emit(f"pairs in view: {len(view.pairs)}")
_emit(f"sparse terrain cells: {len(view.terrain)}")
_emit(f"sparse elevation cells: {len(view.elevation)}")
_emit(f"threatened cells: {len(view.threatened)}")
_emit(f"ranged_threatened cells: {len(view.ranged_threatened)}")
out = capsys.readouterr().out
assert "PHASE2 SUMMARY" in out
def test_view_sparse_grids() -> None:
"""Pin the sparse-grid contract: only non-default/non-zero/True cells."""
registry = load_spell_registry(SPELLS_DIR)
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
raw = obs.to_dict()
view = build_view(obs)
semantics_by_type = {str(sem["type"]): sem for sem in raw["terrain_semantics"]}
# Dimensions match the raw terrain grid.
assert view.dimensions == (len(raw["terrain"]), len(raw["terrain"][0]))
# Sparse terrain only carries non-default cells, with the raw char.
for r, c, terrain_char in view.terrain:
assert not _is_default_terrain(semantics_by_type[terrain_char])
assert raw["terrain"][r][c] == terrain_char
# Sparse elevation only carries non-zero cells.
for r, c, height in view.elevation:
assert height != 0
assert raw["elevation"][r][c] == height
# Sparse threatened grids only carry True cells.
for r, c in view.threatened:
assert raw["threatened"][r][c] is True
for r, c in view.ranged_threatened:
assert raw["ranged_threatened"][r][c] is True
def test_view_dropped_keys() -> None:
"""Pin the key contract: dropped keys stay dropped, pair keys survive."""
registry = load_spell_registry(SPELLS_DIR)
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
view = build_view(obs)
for combatant in view.combatants:
assert _DROPPED_COMBATANT_KEYS.isdisjoint(combatant.keys())
for pair in view.pairs:
assert _PAIR_KEYS.issubset(pair.keys())
# Nested expansions stay compact: no dropped attack/spell keys, and the
# attack damage is rendered as formula+type strings.
for combatant in view.combatants:
for attack in combatant["attacks"]:
assert _ATTACK_DROPPED_KEYS.isdisjoint(attack.keys())
assert all(isinstance(d, str) for d in attack["damage"])
for spell in combatant["spells"]:
assert _SPELL_DROPPED_KEYS.isdisjoint(spell.keys())
def test_view_ids_and_render() -> None:
"""Pin the id contract and the deterministic line-based render."""
registry = load_spell_registry(SPELLS_DIR)
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
raw = obs.to_dict()
view = build_view(obs)
# The acting combatant and pair targets are rendered as ids.
assert view.acting_combatant in raw["initiative_order"]
ids = {combatant["id"] for combatant in view.combatants}
for pair in view.pairs:
assert pair["target"] in ids
# The render is line-based and self-describing.
text = render(view)
assert text.startswith("schema_version=1\n")
assert "combatants:" in text
assert "pairs:" in text
assert "condition_definitions" not in text
+78 -1
View File
@@ -22,7 +22,13 @@ from pf1e_simulator.models import (
DamageComponent,
Saves,
)
from pf1e_simulator.runner import EncounterSpec, Side, build_states, run_encounter
from pf1e_simulator.runner import (
EncounterReport,
EncounterSpec,
Side,
build_states,
run_encounter,
)
def make_combatant(
@@ -391,3 +397,74 @@ def test_run_encounter_reports_healing_done() -> None:
report = run_encounter(spec, runs=5, seed=11, policy=_self_healing_policy)
assert report.attrition["healer"].healing_done > 0.0
assert report.attrition["healer"].damage_taken > 0.0
def _duel_spec() -> EncounterSpec:
return EncounterSpec(
map=make_map(),
sides=(
Side(name="players", combatants=(make_combatant("p1", hp=20),), zone="A"),
Side(name="monsters", combatants=(make_combatant("m1", hp=20),), zone="B"),
),
round_cap=20,
)
def test_run_encounter_default_unchanged_without_collect_runs() -> None:
"""Without collect_runs the call still returns the bare EncounterReport."""
result = run_encounter(_duel_spec(), runs=5, seed=3)
assert isinstance(result, EncounterReport)
def test_run_encounter_collect_runs_returns_report_and_summaries() -> None:
"""collect_runs=True returns (report, per-run summaries) with consistent aggregates."""
spec = _duel_spec()
report, outcomes = run_encounter(spec, runs=10, seed=3, collect_runs=True)
assert isinstance(report, EncounterReport)
assert len(outcomes) == 10
wins = sum(1 for o in outcomes if o.winner == "players")
draws = sum(1 for o in outcomes if o.winner is None)
assert wins == report.wins["players"]
assert draws == report.draws
assert sum(o.rounds for o in outcomes) / 10 == report.avg_rounds
for o in outcomes:
assert {c.id for c in o.combatants} == {"p1", "m1"}
by_id = {c.id: c for c in o.combatants}
assert by_id["p1"].side == "players"
assert by_id["m1"].side == "monsters"
assert by_id["p1"].hp_max == 20
# dead is the engine death condition: hp <= -CON (con 12 here)
for c in o.combatants:
assert c.dead == (c.hp <= -12)
def test_run_encounter_collect_runs_deterministic() -> None:
"""Same seed -> identical per-run summaries."""
spec = _duel_spec()
_, a = run_encounter(spec, runs=8, seed=9, collect_runs=True)
_, b = run_encounter(spec, runs=8, seed=9, collect_runs=True)
assert a == b
def test_run_encounter_collect_runs_dead_matches_winner() -> None:
"""A side that never wins still ends runs with its member dead in the summaries."""
spec = EncounterSpec(
map=make_map(),
sides=(
Side(name="players", combatants=(make_combatant("weak", hp=4),), zone="A"),
Side(
name="monsters",
combatants=(make_combatant("strong", hp=40, attack_bonus=10, damage="2d6"),) * 2,
zone="B",
),
),
round_cap=20,
)
report, outcomes = run_encounter(spec, runs=5, seed=5, collect_runs=True)
assert report.wins["monsters"] == 5
for o in outcomes:
weak = next(c for c in o.combatants if c.id == "weak")
# PF1e: the battle ends when weak is INACTIVE (hp < 0), not necessarily
# dead - dead must mirror the engine condition hp <= -CON, not the loss.
assert weak.hp < 0
assert weak.dead == (weak.hp <= -12)