Compare commits
30 Commits
7e4a9f5edf
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
| f61fc5c61f | |||
| b0bd0b95f4 | |||
| 98985ee6fc | |||
| c454a67b84 | |||
| dc890fc8fd | |||
| d4db4f6d33 | |||
| 42802f7bf5 | |||
| 790df17623 | |||
| cd69312ff4 | |||
| 016d6e1459 | |||
| dd64c2ebd6 | |||
| 115f54a7e5 | |||
| dbd1a5822e | |||
| 15d24be638 | |||
| de4645838b | |||
| f0617a0de6 | |||
| 26c5b51b98 | |||
| 6c6de16e60 | |||
| 334f8f9d05 | |||
| ac7fe20054 | |||
| 8b5043072a | |||
| 9da9537995 | |||
| 25418f164f | |||
| de88543c5c | |||
| e5c4892a57 | |||
| 3c3e30634e | |||
| 3fd9860b5c | |||
| d794828c7e | |||
| 630f5321e0 | |||
| be64451ba4 |
@@ -0,0 +1,67 @@
|
||||
# Architecture & Plan d'implémentation
|
||||
|
||||
## Vue d'ensemble
|
||||
|
||||
- [Architecture technique](technical-architecture.md) — modules, flux de données, interfaces
|
||||
- [Plan Phase 2 (LLM)](phase-2-llm-architecture.md) — étapes P5.1–P5.3, décisions, risques
|
||||
- [Conformité SRD](conformance-srd.md) — écarts documentés, corrigés, hors périmètre
|
||||
- [Feuille de route](roadmap.md) — Phases 0, 1, 2, 3 et au-delà
|
||||
|
||||
## Structure du code
|
||||
|
||||
```
|
||||
src/pf1e_simulator/
|
||||
├── combat.py # CombatEngine, résolution attaques, actions, states
|
||||
├── dice.py # parseur notation dés (2d4+1)
|
||||
├── rng.py # SeededRng, ScriptedRng
|
||||
├── models.py # schémas Pydantic v2 (Combatant, AttackSpec, etc.)
|
||||
├── map.py # MapSpec, chargement YAML, validation
|
||||
├── grid.py # grille 5-10-5, Dijkstra, airborne_view
|
||||
├── los.py # ligne d'effet, couvert (règle des coins)
|
||||
├── areas.py # cône, ligne, burst (géométrie zones)
|
||||
├── effects.py # StatModifier, resolve_modifiers (empilement bonus)
|
||||
├── conditions.py # 29 conditions PF1e + flags comportementaux
|
||||
├── abilities.py # dons (Weapon Focus, Power Attack, etc.)
|
||||
├── spells.py # SpellSpec, effets, scalage, métamagie
|
||||
├── metrics.py # win_rate, win_rate_sigma, win_rate_band
|
||||
├── runner.py # EncounterSpec, run_encounter, EncounterReport
|
||||
├── objectives.py # primitives Phase 2 (P(survie), P(TPK), etc.)
|
||||
├── phase2_spike.py # politique "focus fire", danger cartography
|
||||
├── cli.py # argparse, rapport FR, flags LLM
|
||||
├── loaders/
|
||||
│ ├── combatant.py # chargeur PJ + monstres + dons/sorts
|
||||
│ └── foundry.py # migration Foundry (one-shot, incomplet)
|
||||
└── report.py # LLM report writer (Phase 3)
|
||||
```
|
||||
|
||||
## Décisions clés (D-table)
|
||||
|
||||
| ID | Domaine | Décision |
|
||||
|----|---------|----------|
|
||||
| D1 | Économie d'action | Résolue par deux étages (budget général + filtres spécialisés) |
|
||||
| D2 | Cartographie danger | Non exhaustive : région d'échec + frontière + sensibilité |
|
||||
| D3 | Priorité v1 | Cartographie d'abord, puis optimiseur (mêmes briques) |
|
||||
| D4 | Fournisseur/modèle | Client agnostique ; implémentation distante d'abord, locale ensuite ; mode JSON/schéma requis |
|
||||
| D5 | Budget | Plafonds petits (≤ 50 appels/campagne, ≤ 20 combats vitrine) + coupe-circuit |
|
||||
| D6 | Reproductibilité | Cache obligatoire ; clé composite incluant le modèle ; replay `call_seq` |
|
||||
| D7 | Prompts | Externes / dépôt séparé (voir §14) |
|
||||
|
||||
## Gate de validation
|
||||
|
||||
```bash
|
||||
uv run pytest -q
|
||||
uv run ruff check src tests
|
||||
uv run basedpyright src
|
||||
```
|
||||
|
||||
- Ruff : `select = ["ALL"]`, longueur ligne 100
|
||||
- Basedpyright : `typeCheckingMode = "strict"` sur `src` et `tests`
|
||||
- Transcripts épinglés + assertions statistiques (matchup symétrique dans bande 3σ)
|
||||
|
||||
## Feuille de route
|
||||
|
||||
- **Phase 0** ✅ — socle combat (initiative, attaques, critiques, RD, états, déplacement, mêlée/distance)
|
||||
- **Phase 1** ✅ — magie/états (108 sorts, DD/sauvegardes, concentration, invocation, conditions, manœuvres, dons, capacités de classe)
|
||||
- **Phase 2** ✅ — couche tactique LLM (few-shot, réparation bornée, contrat Observation/LLMPolicy)
|
||||
- **Phase 3** ✅ — rapporteur LLM local (agrégation stats + rédaction FR, `report.py`, `--llm-report`)
|
||||
- **Phase 4** (futur) — optimiseur de scénarios, campagnes multi-rencontres
|
||||
@@ -0,0 +1,183 @@
|
||||
# Architecture Phase 2 — Couche tactique LLM
|
||||
|
||||
> Document consolidé depuis `phase-2-llm-architecture.md` (historique complet conservé dans l'original à la racine de `docs/`)
|
||||
|
||||
---
|
||||
|
||||
## 1. Vision générale
|
||||
|
||||
La Phase 2 ajoute une **couche tactique LLM** au simulateur déterministe Phase 0/1 :
|
||||
|
||||
- **P5.1** ✅ Fournisseur réel (`ProviderClient`) — client OpenAI-compatible HTTP, cache, budget, observabilité
|
||||
- **P5.2** ✅ Few-shot + réparation bornée — prompts externes versionnés, boucle `max_repairs` tracée par `NLPolicyStats`
|
||||
- **P5.3** ✅ Rapporteur LLM local — agrégation stats + rédaction FR (`report.py`, `--llm-report`)
|
||||
|
||||
Le contrat d'observation (`Observation` + wrapper `LLMPolicy`) est en place ; la couche LLM elle-même est désormais implémentée.
|
||||
|
||||
---
|
||||
|
||||
## 2. Contraintes & décisions arbitrées (D-table)
|
||||
|
||||
| ID | Domaine | Décision |
|
||||
|----|---------|----------|
|
||||
| D1 | Économie d'action | Résolue par deux étages (budget général + filtres spécialisés) |
|
||||
| D2 | Cartographie danger | Non exhaustive : région d'échec + frontière + sensibilité |
|
||||
| D3 | Priorité v1 | Cartographie d'abord, puis optimiseur (mêmes briques) |
|
||||
| D4 | Fournisseur/modèle | Client agnostique ; implémentation distante d'abord, locale ensuite ; **mode JSON/schéma requis** |
|
||||
| D5 | Budget | **Plafonds petits** (≤ 50 appels de compilation/campagne, ≤ 20 combats en vitrine) + coupe-circuit dans le client |
|
||||
| D6 | Reproductibilité | **Cache obligatoire** ; clé composite incluant le modèle ; replay `call_seq` |
|
||||
| D7 | Prompts | **Externes / dépôt séparé** (voir §14) |
|
||||
|
||||
---
|
||||
|
||||
## 3. P5.1 — ProviderClient (fournisseur réel)
|
||||
|
||||
**Fichier** : `src/pf1e_simulator/provider.py` — `OpenAILLMClient`
|
||||
|
||||
- Speaks OpenAI chat-completions wire format over HTTPS (stdlib `urllib.request`)
|
||||
- Base URL, model, API key résolus depuis l'env (`config.py` : `PF1E_LLM_API_KEY`, `PF1E_LLM_BASE_URL`, `PF1E_LLM_MODEL`)
|
||||
- Aucune dépendance HTTP externe
|
||||
- Error taxonomy : `ProviderError` (transient 429/5xx), `ProviderRejectedError` (4xx permanent), `ProviderResponseError` (2xx body invalide)
|
||||
|
||||
### Safeguards intégrés (dans `LLMClient` base class)
|
||||
|
||||
- **Cache** : `ResponseCache` (clé composite = modèle|params|prompt_version|schema|prompt_hash), sharded, thread-safe
|
||||
- **Budget** : `CostBudget` (max_calls, max_total_tokens, max_cost_usd) — appliqué avant chaque appel
|
||||
- **Retry/Backoff** : `retries` (défaut 2), `backoff_base_s` (0.25s), exponentiel
|
||||
- **Circuit Breaker** : `CircuitBreaker` (seuil échecs consécutifs, probes après refus)
|
||||
|
||||
### Interface
|
||||
|
||||
```python
|
||||
class LLMClient(ABC):
|
||||
def __init__(self, *, model: str, prompt_version: str, ...)
|
||||
def complete(self, prompt: str, params: ClientParams | None = None) -> str
|
||||
```
|
||||
|
||||
`prompt_version` = `version:name:hash12` (pack version + nom prompt + hash contenu) — requis pour validation cache D6.
|
||||
|
||||
---
|
||||
|
||||
## 4. P5.2 — Few-shot + réparation bornée
|
||||
|
||||
### PromptPack externe (§14)
|
||||
|
||||
- Répertoire pointé par `PF1E_PROMPT_PACK_DIR` (hors repo)
|
||||
- Layout : `version.txt` + `<name>.md` + `<name>.examples.json` (optionnel)
|
||||
- `PromptPack.load(root)` : strict, pas de fallback, lève `PromptPackError` si invalide
|
||||
- `prompt_version = version:name:hash12` (hash couvre prompt + exemples)
|
||||
|
||||
### NLPolicy
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class NLPolicy:
|
||||
client: LLMClient
|
||||
builder: PromptBuilder
|
||||
prompt_name: str = "nl_default"
|
||||
fallback: Fallback = fallback_default
|
||||
max_repairs: int = 1
|
||||
stats: NLPolicyStats = field(default_factory=NLPolicyStats)
|
||||
|
||||
def __call__(self, observation: Observation) -> tuple[Action, ...]:
|
||||
# 1. build prompt via PromptBuilder
|
||||
# 2. client.complete(prompt)
|
||||
# 3. parse_actions() → tuple[Action]
|
||||
# 4. si échec → retry jusqu'à max_repairs (injecte erreur + instruction)
|
||||
# 5. fallback scripté si épuisé
|
||||
```
|
||||
|
||||
- `NLPolicyStats` : `calls`, `repairs_attempted`, `repairs_succeeded`, `fallbacks` — « matière des rapports de P5.3 »
|
||||
- `FakeLLMClient` pour tests 100% offline (script de réponses, zéro réseau)
|
||||
|
||||
---
|
||||
|
||||
## 5. P5.3 — Rapporteur LLM local (Phase 3)
|
||||
|
||||
**Module** : `src/pf1e_simulator/report.py`
|
||||
|
||||
### Structures
|
||||
|
||||
- `ReportContext` : agrège `EncounterReport`, `DangerMetrics`, `NLPolicyStats`, `ClientStats`, `MatchupBenchmark`, `DangerCell`, `SensitivityEntry`
|
||||
- `build_report_context()` : fonction pure, calcule `DangerMetrics` depuis `RunOutcome` si fourni
|
||||
- `build_report_prompt(ctx, template)` : rend rapport factuel + danger + robustesse + observabilité LLM
|
||||
- `LLMReportPolicy` : distinct de `NLPolicy` (sortie texte libre, retry simple, `ReportStats`)
|
||||
- `ReportBudget` : plafonds vitrine (`max_calls=1`, ajusté pour retries via `for_retries()`)
|
||||
- `format_llm_report(ctx, llm_text)` : préfixe factuel + `=== Récit d'équilibrage (LLM) ===` + narratif
|
||||
|
||||
### CLI
|
||||
|
||||
- `--llm-report` : active mode LLM (exclusif avec `--transcript`)
|
||||
- `--output FILE.md` : écrit rapport complet (factuel + LLM) dans fichier
|
||||
- `--llm-model`, `--llm-base-url`, `--llm-api-key` : overrides CLI
|
||||
- Exige `PF1E_LLM_API_KEY` + `PF1E_PROMPT_PACK_DIR` env
|
||||
|
||||
### Pack FR
|
||||
|
||||
- `tests/fixtures/prompt_pack_fs/report.md` : narrateur FR, structure 4 sections (Bilan, Danger, Robustesse, Observabilité)
|
||||
- `report.examples.json` : 2 exemples few-shot
|
||||
- Découvert via `PromptPack` comme `nl_default`
|
||||
|
||||
---
|
||||
|
||||
## 6. Risques & mitigations
|
||||
|
||||
| Risque | Impact | Mitigation |
|
||||
|--------|--------|------------|
|
||||
| Bandes 3σ trompeuses à petit n | Mauvaises conclusions quantitatives | Étage A pour le quantitatif (formule fermée) |
|
||||
| Événements rares (TPK) | Monte Carlo naïf inefficace | Ciblage par scénarios, cartographie danger |
|
||||
| Explosion combinatoire | Énumération impossible | Coarse-to-fine (danger_map, frontier, sensitivity) |
|
||||
| Prompt trop long | Coût/tokens élevés | Vue compacte ~4k tokens, budget ≤ 50 appels |
|
||||
| Hallucination chiffres | Rapport non fiable | Injection stats verbatim, pas d'invention |
|
||||
| Tests flaky (texte libre) | CI instable | `FakeLLMClient` scripté + assertions structure |
|
||||
|
||||
---
|
||||
|
||||
## 7. Intégration CLI (P5.3)
|
||||
|
||||
```bash
|
||||
# Variables d'env requises
|
||||
export PF1E_LLM_API_KEY=sk-...
|
||||
export PF1E_PROMPT_PACK_DIR=/chemin/vers/prompt-pack
|
||||
|
||||
# Rapport LLM + sortie fichier
|
||||
uv run pf1e-sim \
|
||||
--map data/maps/sample_arena.yaml \
|
||||
--side players data/monsters/goblin.json \
|
||||
--side monsters data/monsters/orc.json \
|
||||
--runs 100 --seed 1 \
|
||||
--llm-report --output rapport.md
|
||||
```
|
||||
|
||||
Options ajoutées :
|
||||
- `--llm-report` : active LLM (exclusif `--transcript`)
|
||||
- `--output FILE.md` : écrit rapport complet (factuel + LLM)
|
||||
- `--llm-model`, `--llm-base-url`, `--llm-api-key` : overrides
|
||||
|
||||
---
|
||||
|
||||
## 8. Tests & Gate
|
||||
|
||||
### Nouveaux tests (27)
|
||||
- `test_report_context.py` (4) — `ReportContext` + `build_report_context`
|
||||
- `test_report_prompt_pack.py` (3) — prompt pack version/render
|
||||
- `test_llm_report_policy.py` (4) — `LLMReportPolicy` + retry/fallback
|
||||
- `test_format_llm_report.py` (4) — `format_llm_report` empty/with narrative
|
||||
- `test_cli_llm_report.py` (4) — flags mutuellement exclusifs, env req
|
||||
- `test_report_budget.py` (8) — `ReportBudget`, `CostBudget`, retries
|
||||
|
||||
### Gate complète
|
||||
```bash
|
||||
uv run pytest -q
|
||||
uv run ruff check src tests
|
||||
uv run basedpyright src
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Liens transverses
|
||||
|
||||
- [Architecture technique](../architecture/technical-architecture.md)
|
||||
- [Conformité SRD](../conformance-srd.md)
|
||||
- [Formats de données](../data-formats/index.md)
|
||||
- [Règles Phase 0](../rules/phase-0-implemented.md)
|
||||
@@ -0,0 +1,64 @@
|
||||
# Architecture technique
|
||||
|
||||
## Modules principaux
|
||||
|
||||
| Module | Responsabilité |
|
||||
|--------|----------------|
|
||||
| `combat.py` | `CombatEngine` : initiative, actions, résolution attaques, états de vie, incantation, invocation, manœuvres, transcript, politique par défaut, `Observation`/`LLMPolicy` |
|
||||
| `dice.py` | Parseur notation dés (`2d4+1`) + lancer |
|
||||
| `rng.py` | `SeededRng` (flux reproductible par graine), `ScriptedRng` (tests) |
|
||||
| `models.py` | Schémas Pydantic v2 : `Combatant`, `AbilityScores`, `ACProfile`, `AttackSpec`, `DamageComponent`, `Saves`, `DamageReduction` |
|
||||
| `map.py` | `MapSpec`/`TerrainType`, chargement YAML, validation (dimensions, chars, zones, déploiement, markers) |
|
||||
| `grid.py` | Grille 5-10-5 : `distance`, `step_cost`, `diagonal_allowed`, Dijkstra `reachable`, `airborne_view` |
|
||||
| `los.py` | Ligne d'effet, couvert par règle des coins : gate dans `weapon_for`, bonus dans `resolve_attack` |
|
||||
| `areas.py` | Géométrie gabarits zone : `cone_cells` (quart cercle 90°), `line_cells` (Bresenham 5ft), bursts via `_area_creatures` |
|
||||
| `combat.py` | `CombatEngine` déterministe : initiative, actions, résolution attaques, tailles/allonges, high ground, actions immédiates opt-in, états de vie, incantation, invocation, manœuvres, capacités de classe, transcripts, politique par défaut, `Observation`/`LLMPolicy` |
|
||||
| `effects.py` | `StatModifier` + `resolve_modifiers` (empilement PF1e : dodge/racial/trait/sans-type s'additionnent ; moral/sacré/profane/enhancement gardent le max) |
|
||||
| `conditions.py` | 29 conditions PF1e : `Condition` (modificateurs + flags comportementaux + `miss_chance`) |
|
||||
| `abilities.py` | Dons : `AbilitySpec` + fabriques (Weapon Focus, Toughness, Power Attack actif −X/+2X, X=BAB//4+1) |
|
||||
| `spells.py` | `SpellSpec` (niveau, école, portée, save, DC, `requires_ranged_touch`, `harmless`, `cast_time`, `components`) + effets (`DamageEffect`, `ConditionEffect`, `HealEffect`, `BuffEffect`, `AreaEffect`, `OneShotEffect`) + scalage (`scaling` dés, `scaling_flat` bonus plat) |
|
||||
| `metrics.py` | `win_rate`, `win_rate_sigma`, `win_rate_band` (bande 3σ bornée [0,1]) |
|
||||
| `runner.py` | `EncounterSpec`/`Side`, `build_states` (placement + désambiguïsation ids), `run_encounter` (1 `SeededRng`/run), `EncounterReport` + attrition |
|
||||
| `objectives.py` | Primitives Phase 2 : P(survie), P(TPK), P(≥1 mort), PV perdus, rounds, dcile PV restants |
|
||||
| `phase2_spike.py` | Politique "focus fire" + cartographie danger |
|
||||
| `cli.py` | `argparse` `pf1e-sim`, rapport FR, codes sortie, flags `--llm-report`/`--output` |
|
||||
| `loaders/combatant.py` | Chargeur unifié PJ + monstres : résolution dons → `AbilitySpec`, sorts → `Combatant.spells` |
|
||||
| `loaders/foundry.py` | Migration Foundry one-shot (incomplet — ne pas régénérer `data/pcs/*.json`) |
|
||||
| `report.py` | LLM report writer P5.3 : `ReportContext`, `LLMReportPolicy`, `ReportBudget`, `format_llm_report` |
|
||||
|
||||
## Flux de données principal
|
||||
|
||||
```
|
||||
CLI (args) → _build_spec() → EncounterSpec
|
||||
↓
|
||||
load_spell_registry() → spell_registry
|
||||
↓
|
||||
run_encounter(spec, runs, seed, spell_registry, ...)
|
||||
↓
|
||||
Pour chaque run: SeededRng(seed+i) → CombatEngine → CombatResult
|
||||
↓
|
||||
Agrégation → EncounterReport (wins, draws, avg_rounds, attrition)
|
||||
↓
|
||||
format_report() → sortie FR
|
||||
```
|
||||
|
||||
## Frontière LLM (Phase 2/3)
|
||||
|
||||
```
|
||||
CombatEngine(llm_policy=LLMPolicy) → avant chaque appel politique:
|
||||
Observation snapshot (to_dict() → JSON isolé)
|
||||
LLMPolicy(observation) → tuple[Action, ...]
|
||||
```
|
||||
|
||||
- LLM ne reçoit **que** le snapshot, jamais `engine.*`
|
||||
- `PromptPack` externe (`PF1E_PROMPT_PACK_DIR`) : `version.txt` + `<name>.md` + `<name>.examples.json`
|
||||
- `prompt_version = version:name:hash12` (clé cache composite D6)
|
||||
- Cache `ResponseCache` (clé = modèle|params|prompt_version|schema|prompt_hash)
|
||||
- Budget `CostBudget` + `CircuitBreaker` + retry/backoff
|
||||
|
||||
## Tests & qualité
|
||||
|
||||
- Transcripts épinglés (combats 1v1/2v2 scriptés via `ScriptedRng`)
|
||||
- Assertions statistiques forme fermée (matchup symétrique dans bande 3σ)
|
||||
- Gate : `uv run pytest -q && uv run ruff check src tests && uv run basedpyright src`
|
||||
- Ruff `select = ["ALL"]`, ligne 100 ; basedpyright strict sur `src` + `tests`
|
||||
@@ -0,0 +1,149 @@
|
||||
# Guidelines de contribution
|
||||
|
||||
Merci de votre intérêt pour contribuer à pf1e-simulator !
|
||||
|
||||
## Standards de code
|
||||
|
||||
### Langage & outils
|
||||
- **Python ≥ 3.12** — typage strict, pas de `Any`, pas de `# type: ignore`
|
||||
- **uv** pour la gestion de l'environnement (`uv sync --dev`)
|
||||
- **Ruff** : `select = ["ALL"]`, longueur de ligne 100
|
||||
- **Basedpyright** : mode strict sur `src` et `tests`
|
||||
- **Pytest** : tests déterministes, transcripts épinglés
|
||||
|
||||
### Conventions Python
|
||||
- **Dataclasses frozen** par défaut (`@dataclass(frozen=True)`)
|
||||
- **Types explicites** — pas d'inférence ambiguë
|
||||
- **Docstrings** sur les API publiques (modules, classes, fonctions publiques)
|
||||
- **EM101/EM102** : messages d'erreur explicites (`msg = ...; raise X(msg)`)
|
||||
- **Pas de `type: ignore` / `cast()` / `Any`** — le typage strict est la norme
|
||||
|
||||
### Architecture
|
||||
- **Modules petits** (< 250 lignes) — séparation des responsabilités
|
||||
- **Pas de cycles d'import** — dépendances unidirectionnelles
|
||||
- **Interfaces explicites** — `Protocol` ou classes abstraites pour l'injection
|
||||
- **Tests purs** — pas de réseau, pas d'état global, `FakeLLMClient` pour LLM
|
||||
|
||||
---
|
||||
|
||||
## Processus de développement
|
||||
|
||||
### 1. Branche & commits
|
||||
```bash
|
||||
git checkout -b feat/ma-fonctionnalité
|
||||
# Travail...
|
||||
git add -A
|
||||
git commit -m "feat(scope): description courte
|
||||
|
||||
Description plus longue si nécessaire, wrapping à 72 caractères.
|
||||
|
||||
Refs: #issue-number"
|
||||
```
|
||||
|
||||
**Style de commit** : Conventional Commits
|
||||
- `feat(scope): ...` — nouvelle fonctionnalité
|
||||
- `fix(scope): ...` — correction de bug
|
||||
- `docs(scope): ...` — documentation
|
||||
- `refactor(scope): ...` — refactoring sans changement fonctionnel
|
||||
- `test(scope): ...` — ajout/modification de tests
|
||||
- `chore: ...` — maintenance (deps, config, etc.)
|
||||
|
||||
### 2. Gate de validation (obligatoire avant PR)
|
||||
```bash
|
||||
uv run pytest -q
|
||||
uv run ruff check src tests
|
||||
uv run basedpyright src
|
||||
```
|
||||
**Tous doivent passer** — pas d'exceptions.
|
||||
|
||||
### 3. Tests
|
||||
- **Couverture** : nouvelles fonctionnalités = nouveaux tests
|
||||
- **Determinisme** : `ScriptedRng` / `FakeLLMClient` pour tests LLM
|
||||
- **Transcripts épinglés** : combats 1v1/2v2 scriptés
|
||||
- **Assertions statistiques** : forme fermée (bande 3σ sur matchup symétrique)
|
||||
- **Noms de tests** : `test_<fonctionnalité>_<scénario>_<résultat_attendu>`
|
||||
|
||||
### 4. Documentation
|
||||
- **API publiques** : docstrings complètes
|
||||
- **Architecture** : mises à jour dans `docs/architecture/`
|
||||
- **Règles** : conformité SRD dans `docs/rules/conformance-srd.md`
|
||||
- **Formats** : exemples dans `docs/data-formats/`
|
||||
|
||||
---
|
||||
|
||||
## Types de contributions
|
||||
|
||||
### Bug fixes
|
||||
1. Ouvrir une issue (template bug) avec reproduction minimale
|
||||
2. Écrire un test qui échoue (reproduction)
|
||||
3. Corriger le code
|
||||
3. Vérifier gate complète
|
||||
|
||||
### Nouvelles fonctionnalités
|
||||
1. Ouvrir une issue (template feature) avec design/discussion
|
||||
2. Découper en tranches atomiques testables (cf. `docs/architecture/phase-2-llm-architecture.md`)
|
||||
3. Implémenter tranche par tranche, gate à chaque étape
|
||||
4. Doc + tests + architecture doc mise à jour
|
||||
|
||||
### Refactoring
|
||||
- Gate complète avant et après
|
||||
- Pas de changement fonctionnel
|
||||
- Commits `refactor(scope): ...`
|
||||
|
||||
### Documentation
|
||||
- Corrections typo/clarification : commits `docs: ...`
|
||||
- Nouvelles sections : suivi structure `docs/`
|
||||
|
||||
---
|
||||
|
||||
## Configuration locale recommandée
|
||||
|
||||
```bash
|
||||
# Pre-commit (optionnel mais recommandé)
|
||||
uv pip install pre-commit
|
||||
pre-commit install
|
||||
|
||||
# Éditeur : config basedpyright + ruff
|
||||
# VS Code : extensions Python, Ruff, Basedpyright
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code review
|
||||
|
||||
### Ce qu'on regarde
|
||||
- ✅ Gate complète passe
|
||||
- ✅ Tests pertinents + passants
|
||||
- ✅ Typage strict (pas de `Any`, `ignore`, `cast`)
|
||||
- ✅ Architecture respectée (modules, dépendances)
|
||||
- ✅ Docstrings sur API publiques
|
||||
- ✅ Messages d'erreur explicites (EM101/EM102)
|
||||
- ✅ Pas de code mort / code commenté
|
||||
- ✅ Commits atomiques, messages clairs
|
||||
|
||||
### Red flags (rejettent la PR)
|
||||
- ❌ Gate qui ne passe pas
|
||||
- ❌ `type: ignore` / `# type: ignore` / `cast()`
|
||||
- ❌ Tests supprimés sans remplacement
|
||||
- ❌ Code non testé (nouvelle logique sans test)
|
||||
- ❌ Dépendances circulaires
|
||||
- ❌ Code mort / `pass` / `TODO` sans issue
|
||||
|
||||
---
|
||||
|
||||
## Signalement de sécurité
|
||||
|
||||
Pour les vulnérabilités potentielles : **ne pas ouvrir d'issue publique**.
|
||||
Envoyer un email à l'équipe de maintenance (voir `SECURITY.md` ou `AUTHORS`).
|
||||
|
||||
---
|
||||
|
||||
## Licence
|
||||
|
||||
En contribuant, vous acceptez que votre contribution soit sous la même licence que le projet (voir `LICENSE`).
|
||||
|
||||
---
|
||||
|
||||
## Questions ?
|
||||
|
||||
Ouvrez une discussion GitHub ou consultez `docs/architecture/` pour l'architecture technique.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Formats de données
|
||||
|
||||
- [Carte (YAML)](map.md)
|
||||
- [Monstre (JSON)](monster.md)
|
||||
- [Fiches de personnages PJ](pc-sheets.md)
|
||||
|
||||
Chaque format est documenté en détail dans son fichier respectif.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Format de carte (YAML)
|
||||
|
||||
Chaque couche est un bloc ASCII de mêmes dimensions ; un caractère = une case
|
||||
de `square_size_ft` pieds. Seule `terrain` est obligatoire.
|
||||
|
||||
```yaml
|
||||
name: "Arène d'essai"
|
||||
square_size_ft: 5
|
||||
|
||||
terrain: | # couche obligatoire
|
||||
####################
|
||||
#......######......#
|
||||
#..C...######...C..#
|
||||
#......~..~........#
|
||||
#..TT..~..~...TT...#
|
||||
#......~..~........#
|
||||
#..C............C..#
|
||||
####################
|
||||
|
||||
legend: # un caractère = un type de terrain
|
||||
"#": { type: wall, move_cost: null, blocks_los: true } # bloque déplacement + LoS
|
||||
".": { type: floor, move_cost: 1 }
|
||||
"T": { type: rubble, move_cost: 2 } # terrain difficile
|
||||
"~": { type: water, move_cost: 2 }
|
||||
"C": { type: pillar, move_cost: null, cover: true } # bloque le pas, couvert (+4)
|
||||
"W": { type: ruin, move_cost: null, cover_bonus: 2 } # couvert partiel (+2) ; 4 = standard, 8 = amélioré
|
||||
|
||||
elevation: | # optionnel : chiffres = hauteur (appliqué : coût de pas + high ground)
|
||||
....................
|
||||
..1111..............
|
||||
..1111..............
|
||||
....................
|
||||
|
||||
zones: | # optionnel : lettres de zone de déploiement
|
||||
....................
|
||||
.AAAA..........BBBB.
|
||||
.AAAA..........BBBB.
|
||||
....................
|
||||
|
||||
markers: # optionnel : points nommés (validés, pas encore utilisés)
|
||||
autel: [1, 16]
|
||||
|
||||
deployment: # obligatoire si zones : nom de camp -> lettre de zone
|
||||
players: A
|
||||
monsters: B
|
||||
```
|
||||
|
||||
## Détail des champs
|
||||
|
||||
### `square_size_ft`
|
||||
Taille d'une case en pieds (entier > 0). Par défaut 5 ft (1,5 m).
|
||||
|
||||
### `terrain` (obligatoire)
|
||||
Bloc ASCII représentant le terrain. Chaque ligne = une rangée de cases.
|
||||
|
||||
### `legend`
|
||||
Dictionnaire associant chaque caractère du terrain à ses propriétés :
|
||||
- `type` : identificateur du type de terrain
|
||||
- `move_cost` : entier > 0 pour terrain praticable, `null` pour infranchissable (mur, pilier)
|
||||
- `blocks_los` : booléen, bloque la ligne d'effet
|
||||
- `cover` : booléen, couvert standard (+4)
|
||||
- `cover_bonus` : optionnel, niveau de couvert (2 = partiel, 4 = standard, 8 = amélioré)
|
||||
|
||||
**Important** : omettre `move_cost` équivaut à `null` (infranchissable). Un terrain praticable doit **toujours** déclarer explicitement son coût.
|
||||
|
||||
### `elevation` (optionnel)
|
||||
Chiffres = hauteur de sol (marche). Monter d'un niveau coûte +1 case de mouvement par niveau ; la descente est gratuite. Une créature plus haute que sa cible gagne +1 aux jets d'attaque de mêlée et à la confirmation de critique (high ground). La ligne d'effet reste 2D.
|
||||
|
||||
### `zones` (optionnel)
|
||||
Lettres de zone de déploiement. Chaque lettre = une zone distincte.
|
||||
|
||||
### `markers` (optionnel)
|
||||
Points nommés (validés à la lecture, pas encore utilisés en jeu).
|
||||
|
||||
### `deployment` (obligatoire si `zones` présent)
|
||||
Mapping nom de camp -> lettre de zone. Chaque camp du déploiement doit être fourni via `--side`, et toutes les zones doivent être couvertes. Deux camps ne peuvent pas partager la même lettre de zone.
|
||||
|
||||
## Sémantique de `blocks_los`
|
||||
|
||||
**Ligne d'effet, non ligne de vision** (SRD « Aiming a Spell — Line of Effect ») :
|
||||
> « A line of effect is a straight, unblocked path that indicates what a spell can affect. A line of effect is canceled by a solid barrier. It's like line of sight for ranged weapons, except that it's not blocked by fog, darkness, and other factors that limit normal sight. »
|
||||
|
||||
Le drapeau encode donc une **barrière solide** ; un terrain qui ne limite que la vision normale (brouillard, obscurité, dissimulation) n'est pas modélisé — le moteur n'a qu'un seul drapeau, et la dissimulation passe par les conditions (`miss_chance`), pas par la carte.
|
||||
|
||||
## Vol et terrain
|
||||
|
||||
Une créature **aérienne** (vol ≥ sol) ignore les coûts de `move_cost` > 1 et **enjambe** les cases `move_cost: null` qui ne portent **pas** `blocks_los` (fosses, eau, gravats, piliers non bloquants). Une case `blocks_los` (barrière solide, hauteur supposée pleine) l'arrête aussi. La règle des coins suit la même vue : un aérien peut passer en diagonale au ras d'une fosse mais pas au ras d'un coin de mur.
|
||||
@@ -0,0 +1,117 @@
|
||||
# Format de monstre (JSON)
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "Goblin",
|
||||
"level": 1,
|
||||
"size": "Small",
|
||||
"cr": "1/3",
|
||||
"xp": 135,
|
||||
"source": "Bestiary > Goblin",
|
||||
"abilities": {
|
||||
"str_score": 11, "dex_score": 15, "con_score": 12,
|
||||
"int_score": 10, "wis_score": 9, "cha_score": 6
|
||||
},
|
||||
"hp_max": 6,
|
||||
"ac": { "total": 16, "touch": 13, "flat_footed": 14 },
|
||||
"bab": 1,
|
||||
"initiative_mod": 6,
|
||||
"speed_land_ft": 30,
|
||||
"saves": { "fort": 3, "ref": 2, "will": -1 },
|
||||
"attacks": [
|
||||
{
|
||||
"name": "short sword",
|
||||
"kind": "melee",
|
||||
"attack_bonus": 2,
|
||||
"damage": [{ "formula": "1d4", "types": ["slashing"] }],
|
||||
"crit_range": 19,
|
||||
"crit_mult": 2
|
||||
},
|
||||
{
|
||||
"name": "short bow",
|
||||
"kind": "ranged",
|
||||
"attack_bonus": 4,
|
||||
"damage": [{ "formula": "1d4", "types": ["piercing"] }],
|
||||
"crit_mult": 3,
|
||||
"range_increment_ft": 60
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Champs principaux
|
||||
|
||||
| Champ | Type | Requis | Description |
|
||||
|-------|------|--------|-------------|
|
||||
| `name` | string | oui | Nom du monstre |
|
||||
| `level` | int | oui | Niveau / HD |
|
||||
| `size` | string | oui | `Tiny`, `Diminutive`, `Fine`, `Small`, `Medium`, `Large`, `Huge`, `Gargantuan`, `Colossal` |
|
||||
| `cr` | string | oui | Challenge Rating (ex: `"1/3"`, `"5"`) |
|
||||
| `xp` | int | oui | Points d'expérience |
|
||||
| `source` | string | oui | Source (ex: `"Bestiary > Goblin"`) |
|
||||
| `abilities` | object | oui | 6 scores : `str_score`, `dex_score`, `con_score`, `int_score`, `wis_score`, `cha_score` |
|
||||
| `hp_max` | int | oui | PV maximum |
|
||||
| `ac` | object | oui | `{ total, touch, flat_footed }` — valeurs **finales** (bonus de taille déjà inclus) |
|
||||
| `bab` | int | oui | Base Attack Bonus |
|
||||
| `initiative_mod` | int | oui | Modificateur d'initiative |
|
||||
| `speed_land_ft` | int | oui | Vitesse au sol en ft |
|
||||
| `speed_fly_ft` | int | non | Vitesse de vol en ft (si absent = pas de vol) |
|
||||
| `saves` | object | oui | `{ fort, ref, will }` — jets de sauvegarde finaux |
|
||||
| `attacks` | array | oui | Liste des attaques (voir ci-dessous) |
|
||||
|
||||
## Attaques (`attacks[]`)
|
||||
|
||||
Chaque attaque est un objet avec les champs suivants :
|
||||
|
||||
| Champ | Type | Requis | Description |
|
||||
|-------|------|--------|-------------|
|
||||
| `name` | string | oui | Nom de l'arme |
|
||||
| `kind` | string | oui | `"melee"`, `"ranged"`, ou `"touch"` |
|
||||
| `attack_bonus` | int | oui | Bonus d'attaque (final, taille incluse) |
|
||||
| `damage` | array | oui | Liste de composantes : `{ "formula": "1d4", "types": ["slashing"] }` |
|
||||
| `crit_range` | int | non | Menace de critique (défaut 20) |
|
||||
| `crit_mult` | int | oui | Multiplicateur de critique (2, 3, etc.) |
|
||||
| `range_increment_ft` | int | non* | Incrément de portée en ft (> 0), requis pour `ranged` et `touch` |
|
||||
| `damage_bonus` | int | non | Bonus de dégâts plat |
|
||||
| `dr` | string | non | Type de contournement de RD (ex: `"magic"`, `"cold_iron"`) |
|
||||
| `two_handed` | bool | non | Arme à deux mains (Power Attack +3X au lieu de +2X) |
|
||||
| `reach_weapon` | bool | non | Arme à allonge (anneau, dead zone) |
|
||||
| `reach_ft` | int | non | Allonge naturelle (pour attaques naturelles) |
|
||||
| `natural` | bool | non | Attaque naturelle (pas d'itératifs, peut être `secondary`) |
|
||||
| `secondary` | bool | non | Attaque naturelle secondaire (BAB −5, Multiattack → −2) |
|
||||
| `firearm` | bool | non | Arme à feu (touch dans 1er incrément, misfire sur 1 naturel) |
|
||||
| `splash` | object | non | Arme à éclaboussures : `{ "damage": 1, "type": "fire" }` |
|
||||
| `spell_resistance` | int | non | Résistance aux sorts |
|
||||
| `resistances` | object | non | Résistances énergétiques : `{ "fire": 5 }` |
|
||||
| `immunities` | array | non | Immunités : `["fire", "poison"]` |
|
||||
|
||||
* `range_increment_ft` : entier **strictement positif** (zéro ou négatif = erreur). Pénalité −2 par incrément complet au-delà du premier. Arme `ranged` utilisable jusqu'à 10 incréments, `touch` jusqu'à 5.
|
||||
|
||||
## Champs optionnels avancés
|
||||
|
||||
- `carried_items` : array d'objets transportés non maniés/portés serré (volables via manœuvre *Steal*). Armes maniées = dans `attacks` ; équipement porté serré (armure, sac à dos, anneaux) exclus.
|
||||
- `aware` : bool (défaut `true`) — conscience au début du combat (règle *Surprise*).
|
||||
- `alignment` : code à deux axes (`LG`, `NG`, `CG`, `LN`, `N`, `CN`, `LE`, `NE`, `CE`), défaut `"N"`.
|
||||
- `spell_slots` : dict niveau → nombre (ex: `{"1": 4, "2": 3}`) — budget d'encounter, **pas** calculé depuis la caractéristique.
|
||||
- `spells` : array de noms de sorts connus (doivent exister dans `data/spells/`).
|
||||
- `metamagic_feats` : array de noms de dons de métamagie connus.
|
||||
- `features` : noms de capacités de classe / dons (résolus par le chargeur).
|
||||
|
||||
## Taille et allonge
|
||||
|
||||
| Taille | Allonge naturelle | Footprint |
|
||||
|--------|-------------------|-----------|
|
||||
| Fine / Diminutive / Tiny | 0 ft | 1×1 |
|
||||
| Small / Medium | 5 ft | 1×1 |
|
||||
| Large | 10 ft | 2×2 |
|
||||
| Huge | 15 ft | 3×3 |
|
||||
| Gargantuan | 20 ft | 4×4 |
|
||||
| Colossal | 30 ft | 6×6 |
|
||||
|
||||
Une **arme à allonge** (`reach_weapon: true`) double l'allonge naturelle mais crée une **dead zone** (ne frappe pas à l'allonge naturelle ou moins). Une **arme naturelle** avec `reach_ft` explicite conserve cette allonge (le `reach_ft` d'une attaque naturelle EST son allonge naturelle).
|
||||
|
||||
## Notes importantes
|
||||
|
||||
- Les valeurs `ac` et `attack_bonus` des JSON sont des valeurs **FINALES**, déjà majorées du modificateur de taille PF1e. Le moteur n'applique **jamais** de bonus de taille aux jets d'attaque ni à la CA.
|
||||
- La taille n'intervient que pour : manœuvres (CMB/CMD), allonge naturelle, footprint, occupation.
|
||||
- Les JSON de `data/monsters/` et `data/pcs/` sont la référence. Le chargeur `loaders/foundry.py` est un outil de migration **one-shot et incomplet** — ne pas régénérer les JSON de PJ sans réappliquer les corrections SRD (voir `docs/rules/conformance-srd.md`).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Fiches de personnages (PJ)
|
||||
|
||||
Les PJ sont définis au format JSON (schéma `Combatant`) dans `data/pcs/`. Huit fiches
|
||||
sont fournies : `esha.json`, `harvie.json`, `ierlieth.json`, `jeanne.json`,
|
||||
`misty.json`, `nairda.json`, `oni.json`, `tammara.json`.
|
||||
|
||||
Elles se passent directement à la CLI comme les monstres :
|
||||
|
||||
```bash
|
||||
uv run pf1e-sim --map data/maps/sample_arena.yaml \
|
||||
--side players data/pcs/oni.json \
|
||||
--side monsters data/pcs/tammara.json --runs 200
|
||||
```
|
||||
|
||||
## Format JSON
|
||||
|
||||
Le schéma est identique aux monstres (voir [`monster.md`](monster.md)), avec
|
||||
ces particularités PJ :
|
||||
|
||||
- `source` : pointe vers la fiche Foundry VTT d'origine sous `fiches_personnages/`
|
||||
(ex: `"fiches_personnages/oni.foundry.json"`).
|
||||
- `spell_slots` : budget d'encounter (pas progression RAW — voir note ci-dessous).
|
||||
- `spells` : noms de sorts connus (doivent exister dans `data/spells/`).
|
||||
- `features` : noms de capacités de classe / dons (résolus par le chargeur).
|
||||
- `carried_items` : objets volables (filtrés depuis l'inventaire Foundry per APG *Steal*).
|
||||
|
||||
## Note importante : `spell_slots` = budget d'encounter
|
||||
|
||||
Le champ `spell_slots` est une valeur **finale** par encounter. Le moteur ne calcule
|
||||
**jamais** les sorts bonus de caractéristique d'incantation élevée (SRD Table 1-3).
|
||||
Les éventuels sorts bonus doivent être **encodés dans le JSON par l'auteur**.
|
||||
|
||||
Preuve : les fiches PJ actuelles encodent la table de classe de BASE sans bonus.
|
||||
Exemple : Oni (clerc 4, SAG 18) → `{"1":4, "2":3}` = table de base 4/3,
|
||||
alors que le RAW avec bonus serait 5/4. Le liche (INT 20, niveau 11) porte
|
||||
`{"1":4, "2":4, "3":4, "4":4, "5":4, "6":3}` qui ne correspond à AUCUNE table
|
||||
RAW — les slots sont donc un budget d'encounter authored, pas une progression RAW.
|
||||
|
||||
Conséquence assumée : les PJ incanteurs tiennent moins de sorts par encounter
|
||||
que le RAW ne le permettrait.
|
||||
|
||||
## Fiches Foundry (source)
|
||||
|
||||
`fiches_personnages/` contient les 8 fiches au format Foundry VTT
|
||||
(`pf1-sheet/v1`) ; le chargeur `loaders/foundry.py` sait les lire. C'est un
|
||||
outil de migration **one-shot et incomplet** : il ne reproduit pas les corrections
|
||||
manuelles/SRD appliquées aux JSON (drapeaux `natural`/`secondary`/`firearm`,
|
||||
allonge sabots Jeanne 10 ft, vol Ierlieth 40 ft, morsure Misty 1d6, dons au-delà
|
||||
de 8 entrées, `spell_slots`/`resources`/`casting_ability`).
|
||||
|
||||
> **`data/pcs/*.json` est la référence canonique des PJ.** Ne pas régénérer
|
||||
> depuis `foundry.py` sans réappliquer les corrections SRD consignées dans
|
||||
> [`docs/rules/conformance-srd.md`](conformance-srd.md).
|
||||
@@ -0,0 +1,705 @@
|
||||
# Phase 2 — Architecture et plan d'implémentation de la couche de décision LLM
|
||||
|
||||
> Document de conception (révisé). **Aucun code n'est encore écrit.**
|
||||
> Révision intégrant deux demandes : **optimiser la meilleure stratégie de
|
||||
> survie** et **cartographier tous les cas où un camp est en difficulté**.
|
||||
|
||||
## 1. Objectif et périmètre
|
||||
|
||||
Le simulateur doit répondre à **trois questions distinctes**, souvent confondues :
|
||||
|
||||
| Question | Nature | Réponse apportée par |
|
||||
|---|---|---|
|
||||
| « Comment *exécuter* une tactique donnée ? » | Exécution | Étage A (compilé) / Étage B (LLM par tour) |
|
||||
| « Quelle est la *meilleure* stratégie de survie ? » | **Optimisation** | Couche **Optimiseur** (recherche sur `StrategySpec`) |
|
||||
| « *Où* les PJ (ou les monstres) sont-ils en difficulté ? » | **Robustesse adversariale** | Couche **Robustesse** (min-max + cartographie de scénarios) |
|
||||
|
||||
**Hors périmètre** : la rédaction finale du rapport d'équilibrage (phase 3).
|
||||
|
||||
**Principe directeur** : les **règles** du moteur restent inchangées ; la couche
|
||||
LLM et les couches d'analyse sont **additives**. La LLM n'est ni l'optimiseur ni
|
||||
l'exécutant des sweeps.
|
||||
|
||||
## 2. Exécution — les deux étages (rappel)
|
||||
|
||||
Mettre la LLM à chaque tour dans une boucle Monte Carlo est prohibitif
|
||||
(`runs × rounds × combattants` appels, caches non mutualisables). D'où :
|
||||
|
||||
- **Étage A — stratégie compilée (quantitatif).** La LLM traduit une stratégie NL
|
||||
en un **objet de paramètres** (`StrategySpec`) ; un **compilateur déterministe**
|
||||
en fait une `Policy` Python exécutée **sans appel LLM**. → bandes valides, coût
|
||||
maîtrisé, **déterminisme bit-à-bit**.
|
||||
- **Étage B — LLM par tour (qualitatif, vitrine).** `NLPolicy(observation)` à
|
||||
chaque tour, sur **≤ 20 runs**. → démonstration du raisonnement, transcripts
|
||||
exploitables ; **pas** de conclusion statistique.
|
||||
|
||||
`default_policy` devient un cas particulier : `compile_strategy(SPEC_PAR_DEFAUT)`.
|
||||
|
||||
## 3. Décider — les deux nouvelles couches
|
||||
|
||||
L'étage A fournit une **fonction d'évaluation déterministe** : c'est le bon
|
||||
substrat pour *chercher* et pour *éprouver*.
|
||||
|
||||
### 3.1 Optimiseur — « meilleure stratégie de survie »
|
||||
|
||||
- **Espace** : les leviers de `StrategySpec` (par PJ ou par rôle).
|
||||
- **Oracle** : `evaluate(spec, scenario, n, seeds) → métriques` (runs parallèles).
|
||||
- **Objectifs** : `P(survivre tous)`, `P(TPK)`, `P(≥1 mort)`, espérance de PV
|
||||
perdus, ressources consommées, rounds, **risque de queue (CVaR / pire décile)**.
|
||||
- **Algorithmes** : grille grossière / échantillonnage latin → affinage
|
||||
(descente coordonnée, **CMA-ES**, **optimisation bayésienne**) ; la LLM peut
|
||||
**amorcer** avec des stratégies candidates (NL → `StrategySpec`).
|
||||
- **Sortie** : un **playbook** classé (« si groupés : Buff → Contrôle → Focus
|
||||
lanceur ; reculer sous 30 % PV ») avec incertitude.
|
||||
|
||||
### 3.2 Robustesse et cartographie — « où est le danger »
|
||||
|
||||
Un danger n'existe que **contre un adversaire qui cherche à gagner** : évaluer
|
||||
les PJ contre la politique par défaut **sous-estime** le risque. Il faut :
|
||||
|
||||
- **`AdversarySpec`** (symétrique de `StrategySpec`) : priorité de cible, focus
|
||||
fire, usage des capacités, embuscade, positionnement.
|
||||
- **Min-max** : pour chaque scénario, `max_politique_PJ · min_politique_monstres`
|
||||
(optimisation alternée = meilleures réponses) → pire cas réaliste. La même
|
||||
mesure donne **les deux sens** : difficulté pour les PJ *et* pour les monstres
|
||||
(rencontre trop facile).
|
||||
- **Cartographie** (au lieu d'une énumération impossible) :
|
||||
1. grille grossière sur `ScenarioSpec` → cellules d'échec ;
|
||||
2. **affinage près de la frontière** (bissection / apprentissage actif) ;
|
||||
3. **frontière de danger** : pour chaque mix, le seuil (ex. « < 45 % PV
|
||||
cumulés et sans soigneur à portée ⇒ `P(TPK) > 20 %` ») ;
|
||||
4. **analyse de sensibilité** : facteurs dominants (surprise, terrain,
|
||||
épuisement des ressources, initiative).
|
||||
|
||||
## 4. État des lieux (socle vérifié)
|
||||
|
||||
| Élément | Emplacement | Vérifié |
|
||||
|---|---|---|
|
||||
| `Observation` figée, `to_dict()` JSON, sans `engine.*` | `combat.py:1262` / `1310` | oui |
|
||||
| `Policy` / `LLMPolicy` / `ReactionPolicy` | `combat.py:10573/10575/10577` | oui |
|
||||
| Câblage `policy`/`llm_policy` exclusifs | `combat.py:1615-1623` | oui |
|
||||
| `ActionResult` + `ACTION_RESULT_REASONS` + `last_turn_results` | `combat.py:1483`, `1528` | oui |
|
||||
| Démo « observation seule » | `tests/test_llm_engine_capabilities.py` | oui |
|
||||
| Déterminisme `SeededRng(seed+i)` + métriques 3σ + attrition | `runner.py`, `metrics.py` | oui |
|
||||
| Base de règles conforme | `docs/conformance-srd.md` (gate 1981) | oui |
|
||||
|
||||
**Manques confirmés** : `run_encounter` sans `llm_policy` (`runner.py:177`) ;
|
||||
CLI sans politique ; pas de `schema_version` ; pas de vue compacte ; pas d'API
|
||||
d'actions légales ; pas de mémoire inter-tours ; pas de journal de replay ;
|
||||
`LLMPolicy(obs, state)` transmet l'état vivant ; **aucune couche d'objectifs,
|
||||
de recherche, d'adversaire ou de scénarios**.
|
||||
|
||||
## 5. Architecture cible
|
||||
|
||||
```
|
||||
Moteur (règles inchangées ; frontière additive)
|
||||
|
|
||||
|-- EXÉCUTION
|
||||
| |- Étage A : compile_strategy(StrategySpec) -> Policy (sweeps)
|
||||
| \-- Étage B : NLPolicy(observation) (vitrine <=20 runs)
|
||||
|
|
||||
|-- DÉCISION / ANALYSE
|
||||
| |- objectives.py : métriques de survie/danger (dont queue)
|
||||
| |- scenarios.py : ScenarioSpec + échantillonnage
|
||||
| |- search.py : optimiseur (objectif = objectives), oracle = simulateur
|
||||
| |- adversary.py : AdversarySpec + politiques de monstres fortes (red team)
|
||||
| \-- robustness.py : min-max, cartographie de danger, frontière, sensibilité
|
||||
|
|
||||
\-- SUPPORT
|
||||
|- view.py : ObservationView (objet structuré, versionné) + render()
|
||||
|- candidates.py : actions plausibles dérivées des pairs
|
||||
|- prompt.py : chargement EXTERNE (pack séparé) + versionnage
|
||||
|- client.py : LLMClient + cache composite + FakeLLMClient scripté + coût
|
||||
|- replay.py : JSONL clé call_seq + replay()
|
||||
\-- sweep.py : exécution parallèle des runs (substrat commun)
|
||||
```
|
||||
|
||||
**Interfaces clés** : `ObservationView` (objet + `render()`), `candidates()`
|
||||
(dérivées des `pairs`, pas de 2ᵉ moteur de légalité), `ResponseCache` (clé
|
||||
composite), `ReplayRecord` (clé `call_seq`), `PolicyFactory` (par run),
|
||||
`evaluate(...)` (oracle d'optimisation), `danger_map(...)` (cartographie).
|
||||
|
||||
## 6. Espaces de recherche
|
||||
|
||||
| Espace | Contenu | Utilisé par |
|
||||
|---|---|---|
|
||||
| `StrategySpec` | priorité de cible, style d'engagement, distance, esquive d'AoO, seuil de repli, buffs d'ouverture, politique de sorts, ressources, focus fire | Optimiseur / Étage A |
|
||||
| `AdversarySpec` | mêmes leviers côté monstres (focus, ciblage du soigneur/lanceur, contrôle, embuscade) | Robustesse |
|
||||
| `ScenarioSpec` | composition/niveaux, **PV et ressources initiaux**, déploiement/positionnement, carte/terrain, **surprise**, distance, mix et nombre de monstres, CR | Cartographie |
|
||||
|
||||
## 7. Objectifs et métriques
|
||||
|
||||
| Métrique | Pourquoi |
|
||||
|---|---|
|
||||
| `P(survivre tous)` / `P(TPK)` / `P(≥ 1 mort)` | lecture directe de la survie |
|
||||
| Espérance de PV perdus (groupe / PJ) | gravité |
|
||||
| Ressources consommées (sorts, potions, canal) | soutenabilité |
|
||||
| Rounds | tempo |
|
||||
| **CVaR / pire décile** | **risque de queue** — la moyenne cache les TPK |
|
||||
| Bandes 3σ (n suffisant) | incertitude — **étage A uniquement** |
|
||||
|
||||
## 8. Rôle de la LLM (recadré)
|
||||
|
||||
| Usage | Pourquoi la LLM |
|
||||
|---|---|
|
||||
| NL → `StrategySpec` / `AdversarySpec` | traduction structurée (petit, robuste) |
|
||||
| **Générateur d'hypothèses** (candidats variés à amorcer) | créativité, pas de gradient |
|
||||
| **Red Team** (tactiques de monstres fortes) | révéler les faiblesses |
|
||||
| **Explicateur** (lire les transcripts dangereux) | causalité en langage naturel |
|
||||
| **Rapporteur** (playbook + carte de risque) | rédaction (phase 3) |
|
||||
|
||||
**Non-usage** : optimisation numérique (faite par `search.py`) et exécution par
|
||||
tour en sweep (faite par l'étage A).
|
||||
|
||||
## 9. Décisions
|
||||
|
||||
> Toutes les décisions ci-dessous sont **arbitrées** (voir §13) ; le tableau
|
||||
> conserve la recommandation validée. Les prompts sont traités au §14.
|
||||
| # | Décision | Recommandation |
|
||||
|---|---|---|
|
||||
| D1 | Entrée : `(observation, state)` ou `(observation)` ? | **`(observation)`** + adaptateur. **À trancher en P0.0.** |
|
||||
| D2 | Vue dédiée vs `to_dict()` brut | Vue compacte, structurée, versionnée |
|
||||
| D3 | Mémoire inter-tours | **Retirée de la v1** |
|
||||
| D4 | Sortie d'actions | JSON strict, **schéma généré depuis `Action`**, `swift/free/full_round` exclus |
|
||||
| D5 | Légalité | **Pas de 2ᵉ moteur** : candidats dérivés des `pairs` |
|
||||
| D6 | Reproductibilité | Cache composite `(model, params, prompt_version, schema_version, hash)` |
|
||||
| D7 | Runner/CLI | `llm_policy` + `policy_factory` sur `run_encounter` ; entrée de sweep dédiée |
|
||||
| D8 | Périmètre v1 | Sous-ensemble curé ; `ReactionPolicy` par défaut conservée |
|
||||
| D9 | Économie | **Deux étages** (exécution) |
|
||||
| D10 | Parallélisme | Runs indépendants → pool de threads, cache shardé |
|
||||
| D11 | Réactions | v1 par défaut ; LLM via `ready` |
|
||||
| D12 | Danger = **adversarial** | Mesurer contre des monstres **optimisés**, sinon sous-estimation |
|
||||
| D13 | Objectifs | Inclure le **risque de queue**, pas seulement la moyenne |
|
||||
| D14 | Cartographie | **Non exhaustive** : coarse-to-fine + frontière + sensibilité |
|
||||
|
||||
## 10. Plan d'implémentation
|
||||
|
||||
### P0 — Frontière durcie (sans LLM ; débloque tout)
|
||||
P0.0 trancher D1 (obs seule + adaptateur) · P0.1 `schema_version` + clés pinnées ·
|
||||
P0.2 `ObservationView` + mesure de taille · P0.3 `candidates()` (depuis `pairs`) ·
|
||||
P0.4 `run_encounter(..., llm_policy=, policy_factory=)` · P0.5 `ReplayLog`
|
||||
(`call_seq`) + `replay()` identique.
|
||||
**Sortie** : gate verte ; un run enregistré se rejoue à l'identique.
|
||||
|
||||
### P1 — Client et politiques
|
||||
P1.1 `LLMClient` + `FakeLLMClient` scripté + cache composite · P1.2 **garde-fous
|
||||
coût/tokens + retry/backoff** · P1.3 `NLPolicy` (étage B : parse, valide, 1
|
||||
re-essai, repli) · P1.4 `PromptBuilder` versionné + schéma d'action généré.
|
||||
**Sortie** : bataille complète pilotée par le Fake, hors ligne.
|
||||
|
||||
### P2 — Étage A : compilation, objectifs, sweep
|
||||
P2.1 compilateur NL → `Policy` paramétrée · P2.2 `objectives.py` · P2.3
|
||||
`SweepRunner` **parallèle**.
|
||||
**Sortie** : sweep ≥ 2×2 avec métriques, borné en coût.
|
||||
|
||||
### P3 — Optimiseur (« meilleure stratégie »)
|
||||
P3.1 `scenarios.py` (`ScenarioSpec` + échantillonnage) · P3.2 `search.py`
|
||||
(grille → affinage) · P3.3 amorçage par candidats LLM.
|
||||
**Sortie** : playbook classé avec incertitude pour ≥ 1 rencontre type.
|
||||
|
||||
### P4 — Adversaire et cartographie (« où est le danger »)
|
||||
P4.1 `AdversarySpec` + politiques de monstres fortes · P4.2 `robustness.py`
|
||||
(min-max alterné) · P4.3 `danger_map` (coarse-to-fine, frontière, sensibilité).
|
||||
**Sortie** : carte de danger (PJ **et** monstres) + frontière seuils.
|
||||
|
||||
### P5 — Fournisseur réel et rapport (frontière phase 3)
|
||||
P5.1 `ProviderClient` (clé env, coût/observabilité) · P5.2 few-shot + réparation
|
||||
complète · **P5.3** ✅ rapport FR (phase 3) — module `report.py`, prompt FR `report.md`,
|
||||
CLI `--llm-report`/`--output`, budget `ReportBudget`.
|
||||
|
||||
## 11. Risques
|
||||
|
||||
1. **Économie per-tour** → résolue par les deux étages.
|
||||
2. **Adversaire trop faible** : danger sous-estimé → monstres **optimisés** (D12).
|
||||
3. **Replay sous-spécifié** (`delay`, invocations) → clé `call_seq`.
|
||||
4. **Cache sans identité de modèle** → clé composite.
|
||||
5. **Divergence de légalité** → candidats dérivés des `pairs`.
|
||||
6. **Bascule tardive de D1** → à trancher en P0.0.
|
||||
7. **Bandes 3σ trompeuses** à petit n → étage A pour le quantitatif.
|
||||
8. **Explosion combinatoire** → coarse-to-fine, pas d'énumération (D14).
|
||||
9. **Événements rares (TPK)** : n énorme en Monte Carlo naïf → ciblage par
|
||||
scénarios, ou annoncer la limite.
|
||||
|
||||
## 12. Première action recommandée
|
||||
|
||||
Un **spike** (sans LLM) qui :
|
||||
(a) tranche **D1** + `schema_version` + test de clés pinné ;
|
||||
(b) **mesure** taille de prompt et appels par cellule de sweep (décide per-tour
|
||||
vs compilé) ;
|
||||
(c) esquisse `ScenarioSpec` + `objectives.py` et **valide la chaîne de
|
||||
cartographie** sur une rencontre type, en comparant `default_policy` contre une
|
||||
politique de monstres renforcée — ce dernier point prouve immédiatement la valeur
|
||||
« prédire où sont les difficultés », avant toute LLM.
|
||||
|
||||
## 13. Décisions arbitrées
|
||||
|
||||
| # | Décision | Choix retenu |
|
||||
|---|---|---|
|
||||
| D1 | Entrée de la politique | **`(observation)` seule** + adaptateur de compatibilité |
|
||||
| D9 | Exécution | **Deux étages** (compilé pour les sweeps, per-tour pour la vitrine) |
|
||||
| D12 | Mesure du danger | **Contre un adversaire optimisé** (`AdversarySpec` + meilleure réponse) |
|
||||
| D14 | Cartographie | **Non exhaustive** : région d'échec + frontière + sensibilité |
|
||||
| — | Priorité v1 | **Cartographie d'abord**, puis optimiseur (mêmes briques) |
|
||||
| — | Fournisseur/modèle | **Client agnostique** ; implémentation distante d'abord, locale ensuite ; **mode JSON/schéma requis** |
|
||||
| — | Budget | **Plafonds petits** (≤ 50 appels de compilation/campagne, ≤ 20 combats en vitrine) + coupe-circuit dans le client |
|
||||
| — | Reproductibilité | **Cache obligatoire** ; clé composite incluant le modèle ; replay `call_seq` |
|
||||
| — | Prompts | **Externes / dépôt séparé** (voir §14) |
|
||||
|
||||
Reste à préciser au fil des lots (non bloquant) : forme exacte des leviers de
|
||||
`StrategySpec`/`AdversarySpec`, choix de l'algorithme d'optimisation et seuils
|
||||
statistiques par défaut (le fournisseur distant est **arbitré en P5.1** : un
|
||||
gateway OpenAI-compatible, §16).
|
||||
|
||||
## 14. Prompts externes (décision utilisateur)
|
||||
|
||||
Les prompts **ne vivent pas dans ce dépôt** : ce dépôt ne doit contenir que le
|
||||
**code du simulateur** (calcul + couche LLM + rapport). Conséquences :
|
||||
|
||||
- **Chargement configurable** : le code lit les prompts depuis un **chemin
|
||||
externe** (variable d'environnement / fichier de configuration), jamais codés
|
||||
en dur.
|
||||
- **Versionnage côté pack** : le dépôt de prompts porte sa propre version
|
||||
(semver ou hash de commit) ; chaque prompt a un hash individuel.
|
||||
- **Traçabilité** : le `prompt_version` (version du pack + hash du prompt)
|
||||
entre dans la **clé du cache**, dans le `ReplayRecord` et dans les rapports de
|
||||
sweep — sinon une campagne n'est pas reproductible après modification d'un
|
||||
prompt.
|
||||
- **Tests** : le pack externe est **épinglé** pour l'intégration (version fixée) ;
|
||||
les tests unitaires utilisent un **fixture de prompt minimal** local pour que
|
||||
la CI ne dépende pas du dépôt externe.
|
||||
- **Validation** : un prompt manquant/invalide doit produire une **erreur
|
||||
claire** au démarrage (pas de fallback silencieux).
|
||||
|
||||
## 15. Périmètre du dépôt et backlog de détachement
|
||||
|
||||
Décision utilisateur : à terme, **seul le code** doit rester ici. Sont
|
||||
concernés par un détachement (données et contenu) : fiches de personnages
|
||||
(`fiches_personnages/`), données de monstres/PJ/sorts/cartes (`data/`), et les
|
||||
fichiers de prompts.
|
||||
|
||||
**Pour l'instant : laissé en place** (rattrapage prévu en fin de projet).
|
||||
Contrainte de conception à respecter dès maintenant pour ne pas se peindre dans
|
||||
un coin :
|
||||
|
||||
- **Aucun chemin de données en dur** dans le nouveau code (couche phase 2) :
|
||||
centraliser la résolution des chemins (config/env) pour préparer le
|
||||
déplacement.
|
||||
- Les prompts sont **déjà** conçus comme externes (§14) — pas de régression.
|
||||
- Le détachement final devra fournir un **jeu de données d'exemple** et une
|
||||
documentation de configuration (chemins) pour que le dépôt code reste utilisable
|
||||
seul.
|
||||
|
||||
## 16. Résultats du spike (exécuté)
|
||||
|
||||
**Livré** : D1 (`LLMPolicy(Observation)` + `LegacyLLMPolicy`), `schema_version`
|
||||
pinné, `objectives.py`, `run_encounter(..., collect_runs=True)`, une politique
|
||||
de monstres **focus fire** (spike) et une **carte de danger** de démonstration.
|
||||
|
||||
### Mesures (décision d'architecture)
|
||||
|
||||
| Mesure | Valeur |
|
||||
|---|---|
|
||||
| Observation, 7 combattants | **64 Ko ≈ 16 000 tokens** (max 67 Ko) |
|
||||
| Observation, duel 2v2 | 22 Ko ≈ 5 500 tokens |
|
||||
| Appels de politique / combat | **124** (7 combattants), 12 (duel) |
|
||||
| Appels par cellule de sweep (1 000 runs) | **124 000** |
|
||||
|
||||
→ **Conclusion 1** : un LLM **par tour** en sweep est infaisable (~2 milliards de
|
||||
tokens par cellule) — **D9 validé**, et la **vue compacte est obligatoire**
|
||||
(cible ~2-4 k tokens, contre 16 k pour le snapshot brut).
|
||||
|
||||
### Carte de danger (RUNS=100, seed=11)
|
||||
|
||||
| Scénario | Monstres | Victoire PJ | Survivent | TPK | ≥1 mort | PV perdus | Rounds |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| warband 4v4 | défaut | 90,0 % | 87,0 % | 0,0 % | 13,0 % | 66,1 | 12,3 |
|
||||
| warband 4v4 | focus fire | 98,0 % | 76,0 % | 0,0 % | **24,0 %** | 45,0 | 9,6 |
|
||||
| liche 2v2 | défaut | 0,0 % | 100,0 % | 0,0 % | 0,0 % | 44,0 | 3,7 |
|
||||
| liche 2v2 | focus fire | 0,0 % | 81,0 % | **2,0 %** | **19,0 %** | 44,0 | 5,8 |
|
||||
|
||||
→ **Conclusion 2** : mesuré contre la politique par défaut, le danger est
|
||||
**sous-estimé** (morts 13 %→24 % et 0 %→19 %, TPK 0 %→2 %) — **D12 validé**.
|
||||
→ **Conclusion 3** : le **taux de victoire trompe** (il monte à 98 % pendant que
|
||||
les morts augmentent) → les métriques de mort/PV/**queue** sont nécessaires —
|
||||
**D13 validé**.
|
||||
|
||||
**Limites du spike** : n=100 (bandes larges ±9 %), adversaire de niveau spike
|
||||
(focus fire seulement, sans positionnement ni sorts), 2 scénarios. La
|
||||
cartographie réelle (P4) demandera plus de runs, un adversaire complet
|
||||
(`AdversarySpec`) et un balayage de `ScenarioSpec`.
|
||||
|
||||
### Vue compacte (P0.2, mesures réelles — commit 9da9537)
|
||||
|
||||
`llm_view.py` : `ObservationView` (structuré, déterministe) + `build_view()`
|
||||
+ `render()` vers texte pour prompt. Règles : **grilles sparsifiées** (terrain
|
||||
et élévation = cellules non défaut ; `threatened`/`ranged_threatened` = liste
|
||||
de cellules, omis si vide), **clés décisionnelles seules** par combattant et
|
||||
par paire, `schema_version` inclus, ordre déterministe trié. Mesure sur
|
||||
l'affrontement représentatif (oni/misty/nairda vs lich/thrune_agent/orc/goblin,
|
||||
seed 1) :
|
||||
|
||||
| Vue | Octets | Tokens | % du raw |
|
||||
|---|---|---|---|
|
||||
| brute `to_dict()` | 64 204 | ~16 051 | 100 % |
|
||||
| **compacte `render()`** | **15 691** | **~3 922** | **24,4 %** |
|
||||
|
||||
Ratio 24,4 % = **cible ~2-4k tokens atteinte**. Grilles : terrain 78 cellules
|
||||
sparses, élévation 0, menacées 15, menacées à distance 116 (au lieu de matrices
|
||||
pleines). Contrats de test : réduction < 25 % et < 16 000 octets, grilles
|
||||
sparses, clés supprimées (condition_definitions, etc. — voir le test), ids/
|
||||
render déterministes. Test : `tests/test_phase2_view.py`.
|
||||
|
||||
### Candidats d'action (P0.3 — commit 8b50430)
|
||||
|
||||
`candidates.py` : `candidates(observation) -> tuple[Action, ...]` — actions
|
||||
**plausibles** dérivées des `pairs` et des données par combattant (attack/
|
||||
full_attack si `weapon_for`, charge si `can_charge`, pas de 5 ft + attaque si
|
||||
`can_step5_to_attack`, mouvement vers une cible, sort connu avec slot, canal/
|
||||
hex/deed si ressource présente, défense totale, ready/delay) ; `swift`/`free`/
|
||||
`full_round` exclus ; liste bornée et triée. **Pas de second moteur de
|
||||
légalité** : le moteur reste l'arbitre. Test : `tests/test_phase2_candidates.py`.
|
||||
|
||||
### Runner et rejeu (P0.4 + P0.5 — commits 334f8f9, 6c6de16)
|
||||
|
||||
`run_encounter` accepte désormais `llm_policy=` (observation-only ou legacy,
|
||||
transmise à chaque `CombatEngine`) et `policy_factory=` (constructeur de
|
||||
`Policy` **par run**, appelé une fois par indice dans l'ordre ; `None` pour un
|
||||
run = repli `default_policy`). Les trois entrées (`policy`, `llm_policy`,
|
||||
`policy_factory`) sont mutuellement exclusives (erreur claire). Test :
|
||||
`tests/test_phase2_runner.py` (factory 1 appel/run dans l'ordre, repli,
|
||||
exclusivité, une bataille complète pilotée par `llm_policy`).
|
||||
|
||||
`replay.py` : `ReplayRecord(call_seq, observation, action)` + `ReplayLog`
|
||||
(spec + seed + round_cap + appels) sérialisé en **JSONL** (clé `call_seq`),
|
||||
`observation_key()` (digest déterministe, insensible à la collapse
|
||||
tuple/liste de JSON), et `replay(log) -> LLMPolicy` qui renvoie les actions
|
||||
enregistrées dans l'ordre des `call_seq`, avec gardes : observation
|
||||
divergente (`ReplayMismatchError`), log épuisé (`ReplayExhaustedError`),
|
||||
`call_seq` dupliqué ou hors ordre, JSONL invalide. **Un run enregistré se
|
||||
rejoue à l'identique** (wins/draws/rounds/transcript — aussi depuis un log
|
||||
dégelé du JSONL). Test : `tests/test_phase2_replay.py`. Le rejet de replay
|
||||
par défaut est documenté (D3 : pas de mémoire inter-tours ; `delay`/ready
|
||||
rejoués par les actions enregistrées elles-mêmes, sans ré-invocation).
|
||||
|
||||
### Étage A (P2 — commit dbd1a58)
|
||||
|
||||
`strategy.py` : `StrategySpec` (cible `nearest`/`weakest_ac`/`lowest_hp`,
|
||||
`retreat_hp_fraction`, `spell_policy` `default`/`none`, `use_charges`,
|
||||
`use_class_abilities`) et `compile_strategy(spec) -> Policy` —
|
||||
`compile_strategy(StrategySpec()) is default_policy` (identité) ; les autres
|
||||
specs compilent vers des politiques paramétrées (échelle de mêlée, charge
|
||||
quand la ligne droite existe, retraite sous le seuil de pv, sorts / charges /
|
||||
capacités de classe désactivables), cible choisie par `select_target`
|
||||
(CA la plus basse, pv les plus bas, ou le plus proche). Test :
|
||||
`tests/test_phase2_strategy.py`.
|
||||
|
||||
`sweep.py` : `SweepCell(scenario, strategy, spec, policy, runs, seed)` +
|
||||
`SweepRunner(party=, workers=)` (registry des sorts chargée **une fois**,
|
||||
lecture seule partagée entre threads — D10), `run(cells, max_total_runs=)`
|
||||
(erreur claire nommant `runs`/`budget`, cellules ≥ 1 run) et `run_grid(...)`
|
||||
en ordre **scénario-majeur** — chaque cellule : `run_encounter(collect_runs=True)`
|
||||
+ `danger_metrics`. Une politique `None` = `default_policy` (le runner
|
||||
n'importe pas `strategy.py`). `format_sweep` : rendu texte aligné (ascii).
|
||||
Tests : `tests/test_phase2_sweep.py` (déterminisme bit-à-bit, workers=1 ≡
|
||||
workers=4, cellule `None` ≡ `run_encounter` direct, borne de coût).
|
||||
|
||||
**Mesure représentative (RUNS=100, seed=7, partie `players` = gobelins,
|
||||
adversaire `focus fire` = orcs)** :
|
||||
|
||||
| Scénario | Monstres | Victoire PJ | Survivent | TPK | ≥1 mort | PV perdus | Rounds |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 3 gobelins vs 2 orcs | défaut | 93,0 % | 96,0 % | 0,0 % | 4,0 % | 5,0 | 6,5 |
|
||||
| 3 gobelins vs 2 orcs | focus fire | 98,0 % | 100,0 % | 0,0 % | 0,0 % | 2,4 | 5,6 |
|
||||
| 2 gobelins vs 3 orcs | défaut | 50,0 % | 96,0 % | 0,0 % | 4,0 % | 7,5 | 10,6 |
|
||||
| 2 gobelins vs 3 orcs | focus fire | 54,0 % | 96,0 % | 0,0 % | 4,0 % | 7,3 | 10,1 |
|
||||
|
||||
→ la sortie P2 (« sweep ≥ 2×2 avec métriques, borné en coût ») est atteinte :
|
||||
2×2 = 400 combats en quelques secondes sur 4 workers, métriques de danger
|
||||
complètes par cellule, déterminisme vérifié. Ce 2×2 gobelins/orcs est un
|
||||
**smoke test du runner** (n=100, bande 3σ ±6 % sur les taux) : à ce n, le
|
||||
delta focus fire est dans le bruit sur les morts (4→0 en 3v2, plat en 2v3) —
|
||||
contrairement au spike warband/liche où il sortait nettement de la bande. Le
|
||||
signal le plus net ici est la **queue** (worst-decile HP 1,4 → 6,8 en 3v2 ;
|
||||
0,0 dans les deux cellules 2v3) et les rounds moyens (6,5 → 5,6 en 3v2). La
|
||||
cartographie du danger à grande n, PJ **et** monstres, reste **P4**.
|
||||
|
||||
### Optimiseur (P3 — commit dd64c2e)
|
||||
|
||||
`scenarios.py` : `SideSpec(name, zone, combatant_files, hp_fraction=1.0)` et
|
||||
`ScenarioSpec(name, map_file, sides, round_cap=100)` — `build_encounter`
|
||||
compile un scénario en `EncounterSpec` déterministe (aucun chemin de données
|
||||
en dur dans le src, §15), `scenario_variants(base, hp_fractions=...)` étend
|
||||
grossier le scénario par fractions de PV des côtés (grille de robustesse, sans
|
||||
RNG). Test : `tests/test_phase2_scenarios.py`.
|
||||
|
||||
`search.py` : `evaluate(strategy, scenario, *, party, runs, seed, workers=4)`
|
||||
(`StrategyBenchmark` : métriques de danger du camp `party` + bande 3σ sur le
|
||||
taux de victoire), `PlaybookEntry(rank, strategy, benchmark)`,
|
||||
`strategy_name` (nom canonique de `StrategySpec`) et
|
||||
`optimize(scenario, *, party, runs, seed, workers=4, max_total_runs=None,
|
||||
candidates=())` — recherche en 3 étages : grille grossière de 24 cellules
|
||||
(cible × politique de sorts × charges × capacités de classe, repli à 0),
|
||||
affinage des 3 meilleures cellules sur `retreat_hp_fraction ∈ {0.25, 0.5,
|
||||
0.75}` (9 cellules ; cellules déjà mesurées dédupliquées), puis les candidats
|
||||
LLM (`candidates`, P3.3) dédupliqués et évalués. Classement :
|
||||
`(-P(survive), PV perdus, -pire dcile PV, rounds, nom)`. Le budget
|
||||
`max_total_runs` est vérifié **avant** tout combat (`ValueError` nommant
|
||||
`budget`/`max_total_runs`). `format_playbook` rend le classement en colonnes
|
||||
ASCII avec la bande 3σ. Test : `tests/test_phase2_search.py`.
|
||||
|
||||
**Playbook représentatif (RUNS=200, seed=7, workers=4, camp mesuré =
|
||||
`players` = 3 gobelins vs 2 orcs)** — 33 cellules (grille 24 + affinage 9 +
|
||||
candidats 0) ; les rangs non listés sont des variantes à leviers inertes ou
|
||||
de repli strictement identiques à leur cellule de tête :
|
||||
|
||||
| Rang | Stratégie | Victoire PJ (3σ) | Survivent | ≥1 mort | PV perdus | Pire dcile | Rounds |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | `lowest_hp` (+ inertes, rangs 2-8) | 97,0 % [93,4 ; 100,0] | 99,0 % | 1,0 % | 3,0 | 5,2 | 5,8 |
|
||||
| 9 | `weakest_ac` (+ inertes, rangs 10-16) | 97,0 % [93,4 ; 100,0] | 99,0 % | 1,0 % | 3,0 | 5,2 | 5,8 |
|
||||
| 17 | `lowest_hp, retreat=0.25` (+ inertes) | 96,0 % [91,8 ; 100,0] | 97,5 % | 2,5 % | 3,1 | 3,9 | 6,1 |
|
||||
| 20 | `lowest_hp, retreat=0.5` (+ inertes) | 95,0 % [90,4 ; 99,6] | 97,5 % | 2,5 % | 3,5 | 3,0 | 6,4 |
|
||||
| 23 | `default` (nearest, repli 0) | 95,0 % [90,4 ; 99,6] | 97,0 % | 3,0 % | 5,0 | 2,8 | 6,9 |
|
||||
| 31 | `lowest_hp, retreat=0.75` (+ inertes) | 94,0 % [89,0 ; 99,0] | 95,5 % | 4,5 % | 3,7 | 2,7 | 7,6 |
|
||||
|
||||
TPK 0,0 % dans les 33 cellules. → la sortie P3 (« meilleur candidat du grid +
|
||||
affinage, playbook classé ») est atteinte : 6600 combats en quelques secondes
|
||||
sur 4 workers, classement déterministe (rejoué bit-à-bit), budget vérifié
|
||||
avant exécution.
|
||||
→ **Le ciblage est le levier dominant** : `lowest_hp` et `weakest_ac` battent
|
||||
`default` sur toutes les métriques du camp — PV perdus 5,0 → 3,0, pire dcile
|
||||
2,8 → 5,2, rounds 6,9 → 5,8, survive 97,0 → 99,0 %. Égalité bit-à-bit entre
|
||||
les deux ordres de cibles : les adversaires adverses étant statistiquement
|
||||
identiques (gobelins entre eux, orcs entre eux), les deux heuristiques
|
||||
sélectionnent la même cible au même rang.
|
||||
→ **La retraite agressive est strictement dominée** : seuil 0,25 → 0,5 → 0,75
|
||||
dégrade la survie (99,0 → 97,5 → 97,5 → 95,5 %), le pire dcile (5,2 → 3,9 →
|
||||
3,0 → 2,7) et allonge le combat (5,8 → 6,1 → 6,4 → 7,6 rounds) — une retraite
|
||||
est une action à round complet qui paie le départ sans compenser.
|
||||
→ **Leviers inertes sur cette rencontre** : `spell_policy`, `use_charges` et
|
||||
`use_class_abilities` ne changent rien (gobelins et orcs n'ont ni sorts, ni
|
||||
ressources, ni capacités de classe) — leurs variantes font exactement égalité
|
||||
avec leur cellule de grille (rangs 2-8, 10-16, 24-30). Un scénario avec
|
||||
lanceur est nécessaire pour discriminer ces leviers (rencontres riches en
|
||||
P4/P5).
|
||||
→ **Sémantique de mesure — symétrique** : une politique par cellule, appliquée
|
||||
aux **deux** camps (`CombatEngine` exécute la politique passée au moteur, sans
|
||||
routage par côté ; `compile_strategy` est côté-agnostic) ; le classement porte
|
||||
sur les `DangerMetrics` du camp `party`. Contrairement au spike P2
|
||||
(`focus_fire_policy` routait par côté : orcs uniquement), le playbook mesure
|
||||
donc « la meilleure stratégie quand tout le monde la joue » — l'asymétrie PJ
|
||||
contre adversaire optimisé arrive en P4 (`AdversarySpec`).
|
||||
→ Cohérence : la cellule `default` (95,0 % [90,4 ; 99,6]) contient la ligne P2
|
||||
défaut (93,0 %) dans sa bande 3σ (mêmes seeds, n = 100 → 200).
|
||||
→ Limite : P3.3 (candidats LLM) est câblé et testé (déduplication,
|
||||
0 candidat ici) mais non exercé contre un vrai LLM — aucun fournisseur avant
|
||||
P5. Bande 3σ ±4,6 points à p = 0,97, n = 200 : au sein d'une famille à
|
||||
leviers inertes les écarts sont nuls (bit-identiques), mais un départage plus
|
||||
fin (n plus grand, carte plus riche) reste ouvert.
|
||||
|
||||
### Adversaire et robustesse (P4)
|
||||
|
||||
`adversary.py` : `AdversarySpec(target_priority="nearest" | "lowest_hp" |
|
||||
"weakest_ac" | "healer", spell_policy="default" | "none",
|
||||
retreat_hp_fraction=0.0 | 0.25 | 0.5, use_charges=True | False)` — les
|
||||
leviers adverses (D12 : l'adversaire joue pour gagner) ; un seul
|
||||
`AdversarySpec()` (défaut = `nearest`, la politique focus fire du spike P2)
|
||||
est composé avec la stratégie du camp mesuré en une seule politique de
|
||||
combat par `compile_matchup(party, adversary, party_side=)`. `adversary_name`
|
||||
: nom canonique d'un levier. Test : `tests/test_phase2_adversary.py`.
|
||||
|
||||
`robustness.py` : `matchup_benchmark(scenario, party, adversary, *,
|
||||
party_side, runs, seed) -> MatchupBenchmark` (mesure le danger des DEUX
|
||||
camps — même mesure, les deux sens du §3.2), puis les trois couches :
|
||||
|
||||
- `adversarial_analysis` (P4.2, min-max alterné) : itérations successives —
|
||||
la partie maximise sa survie parmi `party_candidates`, puis l'adversaire
|
||||
minimise le danger de la partie parmi `adversary_candidates` ; chaque
|
||||
itération est rendue par `format_robustness`. Le budget `max_total_runs=`
|
||||
est vérifié **avant** tout combat (`ValueError` nommant
|
||||
`budget`/`max_total_runs`).
|
||||
- `danger_map` (P4.3) : grille `SideSpec.hp_fraction` PJ × adversaire, une
|
||||
cellule par paire de fractions = un `MatchupBenchmark` ; `format_danger_map`
|
||||
rend la grille (survie des deux camps par cellule).
|
||||
- `frontier` (P4.3) : bissection de la fraction de PV PJ où `criterion`
|
||||
croise `target` (convention §3.2), adversaire fixé — le retour est plafonné
|
||||
au pas `precision` (`ceil`, jamais arrondi, pour rester du côté croisant du
|
||||
seuil) ; None quand même `max_fraction` ne croise pas. `format_frontier`
|
||||
rend « no frontier ».
|
||||
- `sensitivity` (P4.3) : classe les 7 leviers adverses par |delta| du
|
||||
critère de la partie vs baseline `AdversarySpec()` (budget exact 8×runs,
|
||||
vérifié avant combat) ; `format_sensitivity` affiche les probabilités en %,
|
||||
`expected_hp_lost` à 1 décimale.
|
||||
|
||||
Tests : `tests/test_phase2_robustness.py`.
|
||||
|
||||
**Mesures représentatives (RUNS=100, seed=7, camp mesuré = `players` =
|
||||
3 gobelins vs 2 orcs, `sample_arena.yaml`)** :
|
||||
|
||||
P4.1 — équilibre du min-max alterné (2 itérations ; l'adversaire best-
|
||||
response converge vers `AdversarySpec()` dès la première itération, les deux
|
||||
itérations sont bit-identiques) :
|
||||
|
||||
| Stratégie PJ | P(survie PJ) | PV perdus PJ | P(survie adv) | PV perdus adv | TPK |
|
||||
|---|---|---|---|---|---|
|
||||
| `default` (nearest) | 96 % | 5,0 | 98 % | 11,6 | 0 % |
|
||||
| `lowest_hp` | 95 % | 6,3 | 99 % | 11,5 | 0 % |
|
||||
| `retreat=0.5` | 97 % | 5,3 | 99 % | 11,4 | 0 % |
|
||||
|
||||
→ la meilleure réponse adverse est le ciblage `nearest` par défaut (le plus
|
||||
dangereux pour la partie, voir sensitivity) ; face à cet équilibre les trois
|
||||
stratégies PJ sont dans le bruit (bande 3σ ±6 points à n=100). Le camp
|
||||
adverse encaisse ~11,5 PV moyens par combat (il perd presque toutes les
|
||||
runs) mais ne **meurt** (hp ≤ −CON) que dans ~2 % des runs (98-99 % de
|
||||
survie adv) : la rencontre bat l'adversaire en le mettant à ≤ 0 pv sans
|
||||
souvent le tuer — cohérent avec la règle de mort `hp ≤ −CON` du moteur.
|
||||
|
||||
P4.2 — danger_map (PJ `default` × adversaire `weakest_ac`, fractions
|
||||
0.5/0.7/1.0, 9 cellules ; grille `Party\Adv` : survie partie/survie
|
||||
adversaire par cellule) :
|
||||
|
||||
| | adv=0.5 | adv=0.7 | adv=1.0 |
|
||||
|---|---|---|---|
|
||||
| party=0.5 | 100 %/100 % | 100 %/99 % | 99 %/99 % |
|
||||
| party=0.7 | 99 %/100 % | 100 %/99 % | 100 %/99 % |
|
||||
| party=1.0 | 100 %/100 % | 100 %/99 % | 100 %/99 % |
|
||||
|
||||
→ la grille est plate (99-100 %) et le TPK nul partout : la mort est un
|
||||
événement rare sur cette rencontre (au plus 1 run sur 100 par cellule) et
|
||||
les écarts observés restent sous la bande 3σ (±6 points) — même décimée à
|
||||
50 % de PV, la partie survit. Une carte discriminante exige une rencontre
|
||||
plus équilibrée (le 2 gobelins vs 3 orcs de P2, où ≥ 1 mort monte à 4 %).
|
||||
|
||||
P4.3 — frontier (`p_survives`, target=0.5, fractions [0.10, 1.00],
|
||||
adversaire `healer`) : **`no frontier`** — la survie de la partie ne
|
||||
descend jamais sous 50 %, même à 10 % de PV initiaux : le plancher de
|
||||
robustesse de la partie est sous la barre des 10 %.
|
||||
|
||||
P4.3 — sensitivity (criterion=`expected_hp_lost`, baseline `default` :
|
||||
PV perdus 4,97, survie 96 %) :
|
||||
|
||||
| Rang | Levier | Value | Delta |
|
||||
|---|---|---|---|
|
||||
| 1 | `healer` | 2,4 | −2,6 |
|
||||
| 2 | `lowest_hp` | 2,4 | −2,6 |
|
||||
| 3 | `weakest_ac` | 2,4 | −2,6 |
|
||||
| 4 | `retreat=0.5` | 3,1 | −1,9 |
|
||||
| 5 | `retreat=0.25` | 4,0 | −1,0 |
|
||||
| 6 | `spell_policy=none` | 5,0 | +0,0 |
|
||||
| 7 | `use_charges=false` | 5,0 | +0,0 |
|
||||
|
||||
→ le ciblage est le levier adverse le plus sensible : quitter `nearest`
|
||||
pour n'importe quel ciblage explicite (healer/lowest_hp/weakest_ac, ex æquo
|
||||
à −2,6) fait tomber l'attrition de la partie de 5,0 à 2,4 PV et remonte sa
|
||||
survie à 100 % — le ciblage de la cible la plus proche (la plus engagée en
|
||||
mêlée) est donc le plus dangereux pour la partie ; des adversaires
|
||||
statistiquement identiques rendent les trois ciblages ex æquo (même cible au
|
||||
même rang, déjà observé en P3). La retraite orc (combat écourté) réduit
|
||||
aussi le danger (−1,9 / −1,0). Leviers inertes : `spell_policy` et
|
||||
`use_charges` (les orcs n'ont ni sorts ni ressources) — delta 0,000.
|
||||
Budget : `adversarial_analysis(max_total_runs=1)` → `ValueError`
|
||||
(« robustness budget exceeded: 800 runs > 1 max_total_runs ») avant tout
|
||||
combat. Déterminisme : deux passes de `matchup_benchmark` bit-identiques
|
||||
(p_survives 1,000 / p_tpk 0,000 / hp_lost 2,4).
|
||||
|
||||
→ la sortie P4 (« adversaire `AdversarySpec` + cartographie de danger
|
||||
complète, bornée en coût ») est atteinte : min-max alterné convergeant,
|
||||
grille de danger 3×3, frontière de survie, classement des leviers adverses,
|
||||
déterminisme et budget vérifiés. Les mesures placent la rencontre de
|
||||
référence 3v2 du côté sûr de l'équilibre : TPK nul, plancher de survie sous
|
||||
10 % de PV, et le seul levier adverse qui y compte est le ciblage par
|
||||
défaut.
|
||||
|
||||
### Fournisseur réel OpenAI-compatible (P5.1)
|
||||
|
||||
Le « ProviderClient » du §10 est **`OpenAILLMClient`** (`provider.py`) : il
|
||||
parle le format wire **chat-completions** d'OpenAI sur la bibliothèque
|
||||
standard (`urllib.request`) — aucune dépendance HTTP dans le dépôt.
|
||||
L'endpoint, le modèle et la clé viennent de l'environnement (§15) :
|
||||
`PF1E_LLM_BASE_URL` (défaut `https://api.openai.com/v1`), `PF1E_LLM_MODEL`
|
||||
(défaut `gpt-4o-mini`) et `PF1E_LLM_API_KEY` (jamais codée en dur) ; un
|
||||
gateway vLLM / llama.cpp / compatible se branche par variables d'environnement,
|
||||
sans toucher au code.
|
||||
|
||||
Taxonomie d'erreurs (partagée avec `client.py`) :
|
||||
|
||||
- `ProviderError` (transitoire) : erreur de transport, HTTP 429 et tout 5xx —
|
||||
rejoués par la boucle de backoff de `LLMClient` et comptés par le
|
||||
coupe-circuit ;
|
||||
- `ProviderRejectedError` (permanent) : tout autre 4xx (clé invalide, modèle
|
||||
inconnu, quota) — levée immédiatement, jamais rejouée ;
|
||||
- `ProviderResponseError` (permanent) : un corps 2xx qui n'est pas un payload
|
||||
chat-completions valide (JSON invalide, `choices`/`message`/`content`
|
||||
absents, `usage` malformé) — levée immédiatement.
|
||||
|
||||
Le corps de requête demande `response_format` JSON (`json_object`, mode
|
||||
JSON/schéma requis, D arbitrées §13) ; l'usage est lu du bloc `usage` du
|
||||
payload et estimé au caractère (`estimate_tokens`) quand le fournisseur
|
||||
l'omet. Les 17 cas de `tests/test_phase2_provider.py` pinent le format wire,
|
||||
la taxonomie, la résolution de clé par environnement, le ledger de tokens et
|
||||
le cache — à travers un transport `urlopen` injectable, jamais le réseau.
|
||||
Gate verte : **2186 tests**.
|
||||
|
||||
### Few-shot et réparation complète (P5.2)
|
||||
|
||||
**Few-shot externe** (`prompt.py` + `prompt_builder.py`) : un pack peut
|
||||
porter, à côté de chaque `{name}.md`, un fichier `{name}.examples.json` —
|
||||
une **liste JSON** de paires d'enseignement `{"observation": {...},
|
||||
"actions": [...]}` (clés strictement validées, toute clé inconnue est un
|
||||
`PromptPackError`). La découverte est optionnelle et triée ; un fichier
|
||||
orphelin (sans `{name}.md` correspondant) est une erreur de pack ; un
|
||||
contenu non-JSON, un non-tableau, un item non-objet, une `observation`
|
||||
non-objet ou des `actions` non-tableau/non-objets sont rejetés à la même
|
||||
taxonomie (`_pack_error`, message citant le chemin et l'index). Le paquet
|
||||
expose les paires via `PromptPack.examples(name)` (tuple de
|
||||
`FewShotExample` frozen) ; le **hash de version couvre désormais aussi le
|
||||
contenu des exemples** : `prompt_version = sha256(content + "\0" +
|
||||
examples_text)[:12]`, **byte-identique** quand aucun fichier d'exemples
|
||||
n'existe (le cache/ledger D6 existant n'est pas invalidé). `PromptBuilder`
|
||||
insère entre le template et l'observation une section `## few-shot examples`
|
||||
dont chaque bloc `### example {n}` rend `observation:` puis `response:` en
|
||||
JSON trié (`sort_keys=True`) — bit-identique sans exemples (garde de
|
||||
régression testée sur l'ordre des sections).
|
||||
|
||||
**Réparation complète** (`nl_policy.py`) : la boucle de réparation de P1.3
|
||||
devient **bornée et tracée**. `NLPolicy` porte `max_repairs` (défaut `1` —
|
||||
comportement d'origine) : chaque réponse invalide est suivie d'un retry dont
|
||||
le prompt accumule le **trail chronologique** des échecs précédents
|
||||
(`-- previous reply invalid (attempt N) --` + message), refermé par
|
||||
`Return a valid JSON action sequence.` ; au-delà de `max_repairs` tentatives
|
||||
échouées, le fallback scripté prend le relais (`fallbacks`). `max_repairs=0`
|
||||
désactive la réparation (fallback immédiat) ; une valeur négative est
|
||||
rejetée au `__post_init__` (`ValueError`). Les compteurs vivent dans
|
||||
`NLPolicyStats` (dataclass mutable, `policy.stats`) : `calls`,
|
||||
`repairs_attempted`, `repairs_succeeded` et `fallbacks` — la matière des
|
||||
rapports de P5.3 et du sweep.
|
||||
|
||||
Fixture `tests/fixtures/prompt_pack_fs/` (pack « few-shot » : version
|
||||
`0.1.0-test`, même prompt texte que le pack source, un micro-exemple
|
||||
goblin→orc). Les 25 nouveaux cas pinent : la validation de
|
||||
`tests/test_phase2_prompt.py` (12), le rendu et l'ordre des sections de
|
||||
`tests/test_phase2_prompt_builder.py` (5, dont le tableau vide → pas de
|
||||
section) et la boucle bornée de `tests/test_phase2_nl_policy.py` (8, via un
|
||||
client enregistreur dérivé de `FakeLLMClient` qui capture chaque prompt
|
||||
envoyé — trail vérifié tentative par tentative). Gate verte : **2211 tests**.
|
||||
|
||||
### État du plan
|
||||
|
||||
- **P0** ✅ · **P1** ✅ (`LLMClient` + `FakeLLMClient` scripté + cache
|
||||
composite, garde-fous coût/tokens, `NLPolicy`, `PromptBuilder` versionné) ·
|
||||
**P2** ✅ (`strategy.py` + `SweepRunner`, sortie mesurée) · **P3** ✅
|
||||
(`scenarios.py` + `search.py`, playbook représentatif mesuré) · **P4** ✅
|
||||
(`adversary.py` + `robustness.py`, adversaire `AdversarySpec` + carte de
|
||||
danger complète) · **P5.1** ✅ (`provider.py`, fournisseur OpenAI-compatible
|
||||
HTTP au format chat-completions) · **P5.2** ✅ (few-shot externe
|
||||
`*.examples.json` hashé dans `prompt_version` + boucle de réparation bornée
|
||||
`max_repairs` tracée par `NLPolicyStats`) — gate verte (**2211 tests**).
|
||||
- Détail P0 : **P0.0 (D1)** ✅ · **P0.1 (`schema_version`)** ✅ · mesures ✅ ·
|
||||
`objectives.py` ✅ · `collect_runs` ✅ · **P0.2** (vue compacte) ✅ ·
|
||||
**P0.3** (`candidates`) ✅ · **P0.4** (`llm_policy` + `policy_factory`) ✅ ·
|
||||
**P0.5** (ReplayLog `call_seq` + `replay()` identique) ✅.
|
||||
- **Frontière P0 atteinte** : gate verte (2022 tests) ; un run enregistré se
|
||||
rejoue à l'identique.
|
||||
- **P5.3** ✅ (rapport FR) — module `report.py` (`ReportContext`, `LLMReportPolicy`,
|
||||
`ReportBudget`, `format_llm_report`), prompt pack FR `report.md`, CLI
|
||||
`--llm-report`/`--output`, budget `ReportBudget` intégré à `CostBudget`,
|
||||
27 tests ajoutés — gate verte (**2238 tests**).
|
||||
@@ -56,6 +56,17 @@ allowed-confusables = ["σ"]
|
||||
"src/pf1e_simulator/combat.py" = ["PLR0913"]
|
||||
# runner.py: run_encounter needs 6 params (spec, runs, seed, policy, spell_registry, capture_first_transcript)
|
||||
"src/pf1e_simulator/runner.py" = ["PLR0913"]
|
||||
# client.py: LLMClient/FakeLLMClient need the full safeguard knobs (cache, budget, pricing, retry, breaker)
|
||||
"src/pf1e_simulator/client.py" = ["PLR0913"]
|
||||
# provider.py: OpenAILLMClient.__init__ needs the full knob set (model, key,
|
||||
# base url, timeout, transport, schema, cache, budget, pricing, retry, breaker)
|
||||
# and the chat-completions URL is built from an env-gated base URL defaulting
|
||||
# to HTTPS (section 15), so the S310 audit and the 14-param constructor are
|
||||
# deliberate
|
||||
"src/pf1e_simulator/provider.py" = ["PLR0913", "S310"]
|
||||
# robustness.py: the cartography API takes the full knob set (scenario, specs,
|
||||
# side, runs, seed, fractions, target, precision, budget) as keyword parameters
|
||||
"src/pf1e_simulator/robustness.py" = ["PLR0913"]
|
||||
|
||||
# ── Basedpyright ──────────────────────────────────────────────────────────────
|
||||
[tool.basedpyright]
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Adversary spec and compilation (docs/phase-2-llm-architecture.md section 10, P4.1).
|
||||
|
||||
The adversary-facing mirror of strategy.py: ``AdversarySpec`` is a
|
||||
field-for-field copy of ``StrategySpec`` (same knobs, same validation) so the
|
||||
enemy side of a matchup carries its own spec type without touching the party
|
||||
search space. ``compile_adversary`` has identity semantics — it rebuilds the
|
||||
mirrored ``StrategySpec`` and delegates to ``compile_strategy``, so the
|
||||
all-default spec compiles to the module-level ``default_policy`` object
|
||||
itself. ``compile_matchup`` glues the two sides into ONE ``Policy`` that
|
||||
routes per combatant side: the engine invokes a single policy for every
|
||||
turn (``CombatEngine._take_turn`` → ``_policy(self, state)``), so the policy
|
||||
branches on ``state.side``, following the ``phase2_spike.focus_fire_policy``
|
||||
pattern. Deterministic throughout: no RNG, no LLM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from pf1e_simulator.combat import (
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Policy,
|
||||
default_policy,
|
||||
)
|
||||
from pf1e_simulator.strategy import (
|
||||
SpellPolicy,
|
||||
StrategySpec,
|
||||
TargetPriority,
|
||||
compile_strategy,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AdversarySpec:
|
||||
"""Field-for-field mirror of ``StrategySpec`` (same defaults, same validation)."""
|
||||
|
||||
name: str = "default"
|
||||
target_priority: TargetPriority = "nearest"
|
||||
retreat_hp_fraction: float = 0.0 # 0.0 = never retreat
|
||||
spell_policy: SpellPolicy = "default"
|
||||
use_charges: bool = True
|
||||
use_class_abilities: bool = True
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
msg = "name must be a non-empty string"
|
||||
raise ValueError(msg)
|
||||
if not 0.0 <= self.retreat_hp_fraction <= 1.0:
|
||||
msg = "retreat_hp_fraction must be within [0, 1]"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def adversary_name(spec: AdversarySpec) -> str:
|
||||
"""A deterministic label for `spec` from its non-default levers.
|
||||
|
||||
Local mirror of ``search.strategy_name`` (same field order: target_priority,
|
||||
spell_policy, use_charges, use_class_abilities, retreat_hp_fraction; joined
|
||||
with ", "; the `name` field never appears; all-default is "default"), kept
|
||||
local so this module never imports search.py.
|
||||
"""
|
||||
levers: list[str] = []
|
||||
if spec.target_priority != "nearest":
|
||||
levers.append(spec.target_priority)
|
||||
if spec.spell_policy != "default":
|
||||
levers.append(f"spell_policy={spec.spell_policy}")
|
||||
if not spec.use_charges:
|
||||
levers.append("use_charges=false")
|
||||
if not spec.use_class_abilities:
|
||||
levers.append("use_class_abilities=false")
|
||||
if spec.retreat_hp_fraction != 0.0:
|
||||
levers.append(f"retreat={spec.retreat_hp_fraction}")
|
||||
if not levers:
|
||||
return "default"
|
||||
return ", ".join(levers)
|
||||
|
||||
|
||||
def compile_adversary(spec: AdversarySpec) -> Policy:
|
||||
"""Compile an ``AdversarySpec`` into a ``Policy`` (identity semantics).
|
||||
|
||||
Rebuilds the mirrored ``StrategySpec`` and delegates to
|
||||
``compile_strategy``, so ``compile_adversary(AdversarySpec()) is
|
||||
default_policy`` holds exactly like the strategy-side identity.
|
||||
"""
|
||||
return compile_strategy(
|
||||
StrategySpec(
|
||||
name=spec.name,
|
||||
target_priority=spec.target_priority,
|
||||
retreat_hp_fraction=spec.retreat_hp_fraction,
|
||||
spell_policy=spec.spell_policy,
|
||||
use_charges=spec.use_charges,
|
||||
use_class_abilities=spec.use_class_abilities,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def compile_matchup(
|
||||
party: StrategySpec, adversary: AdversarySpec, *, party_side: str = "players"
|
||||
) -> Policy:
|
||||
"""One ``Policy`` that routes every turn by ``state.side`` (P4.1).
|
||||
|
||||
Turns of ``party_side`` delegate to ``compile_strategy(party)``; every
|
||||
other side delegates to ``compile_adversary(adversary)`` — the engine
|
||||
invokes ONE policy for all combatants, so side routing lives inside the
|
||||
policy (the ``phase2_spike.focus_fire_policy`` pattern, generalised to
|
||||
per-side compiled strategies). When both specs compile to
|
||||
``default_policy``, the default object itself is returned (identity, no
|
||||
wrapper). An empty ``party_side`` is a ValueError.
|
||||
"""
|
||||
if not party_side:
|
||||
msg = "party_side must be a non-empty string"
|
||||
raise ValueError(msg)
|
||||
party_policy = compile_strategy(party)
|
||||
adversary_policy = compile_adversary(adversary)
|
||||
if party_policy is default_policy and adversary_policy is default_policy:
|
||||
return default_policy
|
||||
|
||||
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
if state.side == party_side:
|
||||
return party_policy(engine, state)
|
||||
return adversary_policy(engine, state)
|
||||
|
||||
return policy
|
||||
@@ -0,0 +1,384 @@
|
||||
"""Plausible candidate actions derived from the observation pairs (P0.3).
|
||||
|
||||
The engine stays the arbiter: ``candidates`` is a pure function of the
|
||||
``Observation`` snapshot (no ``engine.*`` access) that proposes a
|
||||
deterministic menu of plausible actions for the acting combatant. Every
|
||||
candidate is derived from the per-pair decision helpers (``weapon_for``,
|
||||
``can_charge``, ``can_step5_to_attack``, ``distance_ft``,
|
||||
``has_line_of_effect``) and the combatant snapshots (``resources_left``
|
||||
including ``spell_slot_{level}``, the enriched ``spells`` list). A candidate
|
||||
that fails a legality gate at execution is rejected with explicit feedback
|
||||
(``last_turn_results``) — never silently dropped here.
|
||||
|
||||
Candidate categories (docs ``phase-2-llm-architecture.md`` P0.3, D4/D5)
|
||||
-----------------------------------------------------------------------
|
||||
- ``attack`` / ``full_attack`` for every enemy the acting combatant can
|
||||
already hit (``weapon_for`` is set).
|
||||
- ``charge`` for every enemy whose pair says a straight-line charge is
|
||||
possible (``can_charge``).
|
||||
- ``5foot_step`` for every enemy one step brings into reach
|
||||
(``can_step5_to_attack``).
|
||||
- ``move`` toward every enemy (auto-pathed by the engine); skipped while
|
||||
grappled (``grappled_with`` — the engine forbids movement then).
|
||||
- ``cast_spell`` for every known spell with an available slot
|
||||
(``spell_slot_{level}``, cantrips free), a castable ``cast_time`` (not
|
||||
``long``), at every enemy it reaches (``distance_ft <= range_ft`` and
|
||||
``has_line_of_effect`` — the same gate ``_cast_spell`` applies). Harmless
|
||||
spells (cure, buffs) target the caster: a beneficial spell on a hostile
|
||||
target would heal or buff the enemy.
|
||||
- ``channel_energy`` (heal and harm modes) / ``hex`` (evil_eye, slumber per
|
||||
enemy) / ``deed`` (up_close_and_deadly, quick_clear) when the matching
|
||||
resource (``channel_energy`` / ``hexes`` / ``grit``) is present.
|
||||
- ``total_defense`` and ``delay`` (always plausible, no target).
|
||||
- ``ready`` with each trigger, readied attack on the nearest enemy.
|
||||
|
||||
Kinds emitted (D4): ``attack``, ``full_attack``, ``charge``, ``5foot_step``,
|
||||
``move``, ``cast_spell``, ``channel_energy``, ``hex``, ``deed``,
|
||||
``total_defense``, ``ready``, ``delay``. ``swift``/``free``/``full_round``
|
||||
are excluded (``full_attack`` is the supported full-round kind). The menu is
|
||||
capped at ``MAX_CANDIDATES`` (32), trimmed from the tail of the deterministic
|
||||
order so the core options always survive.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, TypeGuard
|
||||
|
||||
from .combat import Action, Observation
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from .combat import ReadyTrigger
|
||||
|
||||
# Ready triggers offered for the ``ready`` candidate (the engine's
|
||||
# ``ReadyTrigger`` literal).
|
||||
_READY_TRIGGERS: tuple[ReadyTrigger, ...] = (
|
||||
"enemy_enters_reach",
|
||||
"enemy_casts",
|
||||
"enemy_attacks",
|
||||
"enemy_hits_ally",
|
||||
)
|
||||
|
||||
# Class-ability resource keys (in ``resources_left``) that gate the
|
||||
# channel/hex/deed candidates.
|
||||
_CHANNEL_RESOURCE = "channel_energy"
|
||||
_HEX_RESOURCE = "hexes"
|
||||
_DEED_RESOURCE = "grit"
|
||||
|
||||
# Candidate kinds the generator may emit (D4). The generic minor kinds
|
||||
# (``swift``/``free``/``full_round``) are excluded: a policy proposes real
|
||||
# actions, not placeholders.
|
||||
_CANDIDATE_KINDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"attack",
|
||||
"full_attack",
|
||||
"charge",
|
||||
"5foot_step",
|
||||
"move",
|
||||
"cast_spell",
|
||||
"channel_energy",
|
||||
"hex",
|
||||
"deed",
|
||||
"total_defense",
|
||||
"ready",
|
||||
"delay",
|
||||
}
|
||||
)
|
||||
|
||||
# Deterministic menu order: most decisive first, so the cap trims the least
|
||||
# useful tail (extra ready triggers, delay) rather than the core options.
|
||||
_KIND_PRIORITY: dict[str, int] = {
|
||||
"full_attack": 0,
|
||||
"attack": 1,
|
||||
"charge": 2,
|
||||
"5foot_step": 3,
|
||||
"move": 4,
|
||||
"cast_spell": 5,
|
||||
"channel_energy": 6,
|
||||
"hex": 7,
|
||||
"deed": 8,
|
||||
"total_defense": 9,
|
||||
"ready": 10,
|
||||
"delay": 11,
|
||||
}
|
||||
|
||||
# Upper bound on the returned menu: keeps the LLM prompt compact while
|
||||
# covering every category.
|
||||
MAX_CANDIDATES = 32
|
||||
|
||||
|
||||
def candidates(observation: Observation) -> tuple[Action, ...]:
|
||||
"""Deterministic menu of plausible actions for the acting combatant.
|
||||
|
||||
Pure read of the snapshot: no engine access, no RNG, no mutation. The
|
||||
engine stays the arbiter — a candidate that fails a legality gate is
|
||||
rejected with explicit feedback, never silently dropped here.
|
||||
"""
|
||||
combatants = observation.combatants
|
||||
if not (0 <= observation.acting_combatant < len(combatants)):
|
||||
return ()
|
||||
acting = combatants[observation.acting_combatant]
|
||||
if acting.get("active") is not True:
|
||||
return ()
|
||||
acting_side = acting.get("side")
|
||||
pairs = tuple(
|
||||
pair
|
||||
for pair in observation.pairs
|
||||
if _pair_target_is_enemy(pair, combatants, acting_side)
|
||||
)
|
||||
result: list[Action] = []
|
||||
_attack_candidates(result, pairs, combatants)
|
||||
_charge_candidates(result, pairs, combatants)
|
||||
_step_candidates(result, pairs, combatants)
|
||||
_move_candidates(result, acting, pairs, combatants)
|
||||
_spell_candidates(result, acting, pairs, combatants)
|
||||
_class_ability_candidates(result, acting, pairs, combatants)
|
||||
result.append(Action(kind="total_defense"))
|
||||
_ready_candidates(result, pairs, combatants)
|
||||
result.append(Action(kind="delay"))
|
||||
result.sort(key=_sort_key)
|
||||
return tuple(result[:MAX_CANDIDATES])
|
||||
|
||||
|
||||
def _attack_candidates(
|
||||
result: list[Action],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Standard and full-round attacks for every enemy already in reach."""
|
||||
for pair in pairs:
|
||||
if pair.get("weapon_for") is not None:
|
||||
target = _target_id(pair, combatants)
|
||||
result.append(Action(kind="attack", target_id=target))
|
||||
result.append(Action(kind="full_attack", target_id=target))
|
||||
|
||||
|
||||
def _charge_candidates(
|
||||
result: list[Action],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Charge for every enemy with a clear straight-line path."""
|
||||
result.extend(
|
||||
Action(kind="charge", target_id=_target_id(pair, combatants))
|
||||
for pair in pairs
|
||||
if pair.get("can_charge") is True
|
||||
)
|
||||
|
||||
|
||||
def _step_candidates(
|
||||
result: list[Action],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""5-foot step for every enemy one step brings into reach."""
|
||||
result.extend(
|
||||
Action(kind="5foot_step", target_id=_target_id(pair, combatants))
|
||||
for pair in pairs
|
||||
if pair.get("can_step5_to_attack") is True
|
||||
)
|
||||
|
||||
|
||||
def _move_candidates(
|
||||
result: list[Action],
|
||||
acting: dict[str, object],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Move toward every enemy; skipped while grappled (movement forbidden)."""
|
||||
if acting.get("grappled_with") is not None:
|
||||
return
|
||||
result.extend(
|
||||
Action(kind="move", target_id=_target_id(pair, combatants)) for pair in pairs
|
||||
)
|
||||
|
||||
|
||||
def _spell_candidates(
|
||||
result: list[Action],
|
||||
acting: dict[str, object],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Cast every known spell with a slot, at every enemy it reaches.
|
||||
|
||||
Harmless spells (cure, buffs) target the caster: a beneficial spell on a
|
||||
hostile target would heal or buff the enemy. Offensive spells are
|
||||
proposed per enemy the spell reaches.
|
||||
"""
|
||||
spells = acting.get("spells")
|
||||
resources = acting.get("resources_left")
|
||||
if not _is_tuple(spells) or not _is_str_object_dict(resources):
|
||||
return
|
||||
acting_id = _as_str(acting["id"])
|
||||
for spell in spells:
|
||||
if not _is_str_object_dict(spell):
|
||||
continue
|
||||
if not _spell_has_slot(spell, resources):
|
||||
continue
|
||||
if _as_str(spell.get("cast_time", "standard")) == "long":
|
||||
continue
|
||||
name = _as_str(spell["name"])
|
||||
if spell.get("harmless") is True:
|
||||
result.append(
|
||||
Action(kind="cast_spell", target_id=acting_id, spell_name=name)
|
||||
)
|
||||
continue
|
||||
result.extend(
|
||||
Action(
|
||||
kind="cast_spell",
|
||||
target_id=_target_id(pair, combatants),
|
||||
spell_name=name,
|
||||
)
|
||||
for pair in pairs
|
||||
if _spell_reaches_pair(spell, pair)
|
||||
)
|
||||
|
||||
|
||||
def _class_ability_candidates(
|
||||
result: list[Action],
|
||||
acting: dict[str, object],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Channel/hex/deed candidates, each gated on its resource being present."""
|
||||
resources = acting.get("resources_left")
|
||||
if not _is_str_object_dict(resources):
|
||||
return
|
||||
if _resource_available(resources, _CHANNEL_RESOURCE):
|
||||
result.append(Action(kind="channel_energy", channel_mode="heal"))
|
||||
result.append(Action(kind="channel_energy", channel_mode="harm"))
|
||||
if _resource_available(resources, _HEX_RESOURCE):
|
||||
for pair in pairs:
|
||||
target = _target_id(pair, combatants)
|
||||
result.append(Action(kind="hex", hex_name="evil_eye", target_id=target))
|
||||
result.append(Action(kind="hex", hex_name="slumber", target_id=target))
|
||||
if _resource_available(resources, _DEED_RESOURCE):
|
||||
result.append(Action(kind="deed", deed_name="up_close_and_deadly"))
|
||||
result.append(Action(kind="deed", deed_name="quick_clear"))
|
||||
|
||||
|
||||
def _ready_candidates(
|
||||
result: list[Action],
|
||||
pairs: tuple[dict[str, object], ...],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> None:
|
||||
"""Ready with each trigger, readied attack on the nearest enemy."""
|
||||
nearest = _nearest_enemy(pairs, combatants)
|
||||
if nearest is None:
|
||||
return
|
||||
result.extend(
|
||||
Action(kind="ready", ready_trigger=trigger, target_id=nearest)
|
||||
for trigger in _READY_TRIGGERS
|
||||
)
|
||||
|
||||
|
||||
def _sort_key(action: Action) -> tuple[int, str, str, str, str, str, str]:
|
||||
"""Deterministic ordering of the candidate menu (decisive first)."""
|
||||
return (
|
||||
_KIND_PRIORITY.get(action.kind, 99),
|
||||
action.target_id or "",
|
||||
action.spell_name or "",
|
||||
action.hex_name or "",
|
||||
action.deed_name or "",
|
||||
action.ready_trigger or "",
|
||||
action.channel_mode or "",
|
||||
)
|
||||
|
||||
|
||||
def _target_id(
|
||||
pair: dict[str, object], combatants: tuple[dict[str, object], ...]
|
||||
) -> str:
|
||||
"""Stable id of the pair's target (falls back to the index)."""
|
||||
index = pair.get("target_index")
|
||||
if isinstance(index, int) and 0 <= index < len(combatants):
|
||||
cid = combatants[index].get("id")
|
||||
if isinstance(cid, str):
|
||||
return cid
|
||||
return str(index)
|
||||
|
||||
|
||||
def _pair_target_is_enemy(
|
||||
pair: dict[str, object],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
acting_side: object,
|
||||
) -> bool:
|
||||
"""True when the pair's target is an active combatant on the other side.
|
||||
|
||||
The observation's pairs cover every other combatant (allies included),
|
||||
so the side check is what keeps the menu hostile-only.
|
||||
"""
|
||||
index = pair.get("target_index")
|
||||
if not isinstance(index, int) or not (0 <= index < len(combatants)):
|
||||
return False
|
||||
target = combatants[index]
|
||||
return target.get("active") is True and target.get("side") != acting_side
|
||||
|
||||
|
||||
def _resource_available(resources: dict[str, object], key: str) -> bool:
|
||||
"""True when the class-ability resource has at least one use left."""
|
||||
count = resources.get(key)
|
||||
return isinstance(count, int) and count > 0
|
||||
|
||||
|
||||
def _spell_has_slot(spell: dict[str, object], resources: dict[str, object]) -> bool:
|
||||
"""True when the caster still has a slot for the spell's level.
|
||||
|
||||
Cantrips (level 0) are cast freely — the snapshot has no level-0 slot
|
||||
key, mirroring ``_has_slot`` in the engine.
|
||||
"""
|
||||
level = spell.get("level")
|
||||
if not isinstance(level, int):
|
||||
return False
|
||||
if level == 0:
|
||||
return True
|
||||
count = resources.get(f"spell_slot_{level}")
|
||||
return isinstance(count, int) and count > 0
|
||||
|
||||
|
||||
def _spell_reaches_pair(spell: dict[str, object], pair: dict[str, object]) -> bool:
|
||||
"""True when the spell's range covers the pair with line of effect.
|
||||
|
||||
Mirrors the engine's ``_spell_reaches`` gate: ``distance_ft`` (the pair's
|
||||
creature distance) within the snapshot's precomputed ``range_ft``, plus
|
||||
the pair's ``has_line_of_effect``.
|
||||
"""
|
||||
range_ft = spell.get("range_ft")
|
||||
distance_ft = pair.get("distance_ft")
|
||||
return (
|
||||
isinstance(range_ft, int)
|
||||
and isinstance(distance_ft, int)
|
||||
and distance_ft <= range_ft
|
||||
and pair.get("has_line_of_effect") is True
|
||||
)
|
||||
|
||||
|
||||
def _nearest_enemy(
|
||||
pairs: tuple[dict[str, object], ...], combatants: tuple[dict[str, object], ...]
|
||||
) -> str | None:
|
||||
"""Id of the nearest enemy (lowest ``distance_ft``), or None."""
|
||||
best: str | None = None
|
||||
best_distance: int | None = None
|
||||
for pair in pairs:
|
||||
distance = pair.get("distance_ft")
|
||||
if not isinstance(distance, int):
|
||||
continue
|
||||
if best_distance is None or distance < best_distance:
|
||||
best_distance = distance
|
||||
best = _target_id(pair, combatants)
|
||||
return best
|
||||
|
||||
|
||||
def _is_tuple(value: object) -> TypeGuard[tuple[object, ...]]:
|
||||
"""Narrow ``object`` to a tuple (snapshot collections are tuples)."""
|
||||
return isinstance(value, tuple)
|
||||
|
||||
|
||||
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
"""Narrow ``object`` to the snapshot's ``dict[str, object]`` shape."""
|
||||
return isinstance(value, dict)
|
||||
|
||||
|
||||
def _as_str(value: object) -> str:
|
||||
if not isinstance(value, str):
|
||||
msg = f"expected str, got {type(value).__name__}"
|
||||
raise TypeError(msg)
|
||||
return value
|
||||
+111
-4
@@ -19,11 +19,24 @@ if TYPE_CHECKING:
|
||||
|
||||
from pf1e_simulator.runner import EncounterReport, EncounterSpec
|
||||
|
||||
from pf1e_simulator.config import ENV_PROMPT_PACK_DIR, env_path
|
||||
from pf1e_simulator.loaders import CombatantLoadError, SheetLoadError, load_combatant
|
||||
from pf1e_simulator.map import MapValidationError, load_map
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
from pf1e_simulator.report import (
|
||||
LLMReportPolicy,
|
||||
build_report_context,
|
||||
format_llm_report,
|
||||
)
|
||||
from pf1e_simulator.runner import EncounterSpec, RunOutcome, Side, run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
|
||||
try:
|
||||
from pf1e_simulator.provider import OpenAILLMClient
|
||||
except ImportError: # pragma: no cover - optional dependency
|
||||
OpenAILLMClient = None # type: ignore[assignment,misc]
|
||||
|
||||
|
||||
class CliError(Exception):
|
||||
"""User-facing CLI error (bad arguments or unusable files)."""
|
||||
@@ -129,7 +142,7 @@ def _build_spec(args: argparse.Namespace) -> EncounterSpec:
|
||||
raise CliError(msg)
|
||||
unknown = sorted(provided_names - set(map_spec.deployment))
|
||||
if unknown:
|
||||
msg = f"--side names not in map deployment: {', '.join(unknown)}"
|
||||
msg = f"--side names not in map deployment: {', '.join(sorted(unknown))}"
|
||||
raise CliError(msg)
|
||||
sides: list[Side] = []
|
||||
for raw in provided:
|
||||
@@ -172,6 +185,56 @@ def format_report(report: EncounterReport, spec: EncounterSpec) -> str:
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _run_llm_report(
|
||||
spec: EncounterSpec,
|
||||
args: argparse.Namespace,
|
||||
registry: dict[str, object] | None,
|
||||
) -> str:
|
||||
"""Generate the LLM-powered report."""
|
||||
if OpenAILLMClient is None:
|
||||
print("pf1e-sim: LLM provider not available", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
pack_dir = env_path(ENV_PROMPT_PACK_DIR)
|
||||
if pack_dir is None:
|
||||
print(f"pf1e-sim: {ENV_PROMPT_PACK_DIR} environment variable not set", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
pack = PromptPack.load(pack_dir)
|
||||
prompt_version = pack.prompt_version("report")
|
||||
builder = PromptBuilder(pack)
|
||||
|
||||
client = OpenAILLMClient(
|
||||
prompt_version=prompt_version,
|
||||
model=args.llm_model,
|
||||
api_key=args.llm_api_key,
|
||||
base_url=args.llm_base_url,
|
||||
)
|
||||
|
||||
report: EncounterReport
|
||||
outcomes: tuple[RunOutcome, ...]
|
||||
report, outcomes = run_encounter( # type: ignore[misc]
|
||||
spec,
|
||||
runs=args.runs,
|
||||
seed=args.seed,
|
||||
spell_registry=registry, # type: ignore[arg-type]
|
||||
capture_first_transcript=args.transcript,
|
||||
monsters_die_at_zero=args.simple_monster_death,
|
||||
collect_runs=True,
|
||||
)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=report, # type: ignore[arg-type]
|
||||
spec_name=spec.map.name,
|
||||
outcomes=outcomes, # type: ignore[arg-type]
|
||||
party="players",
|
||||
)
|
||||
|
||||
policy = LLMReportPolicy(client=client, builder=builder, prompt_name="report")
|
||||
narrative = policy(ctx)
|
||||
return format_llm_report(ctx, narrative)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""CLI entry point; returns the process exit code."""
|
||||
parser = argparse.ArgumentParser(
|
||||
@@ -207,9 +270,39 @@ def main(argv: list[str] | None = None) -> int:
|
||||
"(player characters are never affected)"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--llm-report",
|
||||
action="store_true",
|
||||
help="generate an LLM-powered French narrative report (requires PF1E_LLM_API_KEY)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
help="write the full report (factual + LLM narrative) to this file",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--llm-model",
|
||||
type=str,
|
||||
help="override the LLM model (default: PF1E_LLM_MODEL env or gpt-4o-mini)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--llm-base-url",
|
||||
type=str,
|
||||
help="override the LLM base URL (default: PF1E_LLM_BASE_URL env or https://api.openai.com/v1)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--llm-api-key",
|
||||
type=str,
|
||||
help="override the LLM API key (default: PF1E_LLM_API_KEY env)",
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
if not args.side:
|
||||
parser.error("at least one --side is required")
|
||||
if args.llm_report and args.transcript:
|
||||
parser.error("--llm-report and --transcript are mutually exclusive")
|
||||
if args.output and not args.llm_report:
|
||||
parser.error("--output requires --llm-report")
|
||||
|
||||
try:
|
||||
spec = _build_spec(args)
|
||||
except CliError as exc:
|
||||
@@ -218,8 +311,14 @@ def main(argv: list[str] | None = None) -> int:
|
||||
except (MapValidationError, CombatantLoadError, SheetLoadError, OSError) as exc:
|
||||
print(f"pf1e-sim: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
try:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
|
||||
report = None
|
||||
if args.llm_report:
|
||||
full_report = _run_llm_report(spec, args, registry) # type: ignore[arg-type]
|
||||
else:
|
||||
report = run_encounter(
|
||||
spec,
|
||||
runs=args.runs,
|
||||
@@ -228,12 +327,20 @@ def main(argv: list[str] | None = None) -> int:
|
||||
capture_first_transcript=args.transcript,
|
||||
monsters_die_at_zero=args.simple_monster_death,
|
||||
)
|
||||
full_report = format_report(report, spec)
|
||||
|
||||
except ValueError as exc:
|
||||
print(f"pf1e-sim: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
if args.transcript and report.transcript is not None:
|
||||
|
||||
if args.transcript and report is not None and report.transcript is not None:
|
||||
print("=== Déroulé du premier combat ===")
|
||||
print(render_transcript_table(report.transcript))
|
||||
print()
|
||||
print(format_report(report, spec))
|
||||
|
||||
print(full_report)
|
||||
|
||||
if args.output:
|
||||
args.output.write_text(full_report, encoding="utf-8")
|
||||
|
||||
return 0
|
||||
|
||||
@@ -0,0 +1,536 @@
|
||||
"""Provider-agnostic LLM client for the phase-2 decision layer (P1.1/P1.2).
|
||||
|
||||
One abstraction for every LLM call the simulator makes, so sweeps, the
|
||||
compiler and the per-turn showcase all share the same safeguards:
|
||||
|
||||
- **Composite cache** (decision D6): a ``ResponseCache`` keyed by
|
||||
``(model, params, prompt_version, schema_version, prompt hash)``. The
|
||||
``prompt_version`` (external pack version + per-prompt content hash, see
|
||||
``prompt.py``) makes a campaign reproducible after a prompt edit (section
|
||||
14). The cache is sharded (decision D10) and thread-safe, so parallel
|
||||
sweep workers can share one instance.
|
||||
- **Cost/token guards** (P1.2, section 13): a ``CostBudget`` enforces
|
||||
optional ceilings on provider calls, total tokens and estimated USD cost,
|
||||
raising a typed ``BudgetExceededError`` instead of silently continuing.
|
||||
The ledger counts every attempt plus tokens/cost of successful completions.
|
||||
- **Retry/backoff**: a transient ``ProviderError`` is retried with
|
||||
exponential backoff (injectable ``sleep`` for tests), then a typed
|
||||
``RetryExhaustedError`` is raised.
|
||||
- **Circuit breaker** (section 13): after ``failure_threshold`` consecutive
|
||||
failures the client refuses further calls (``CircuitOpenError``) for
|
||||
``open_refusals`` calls, then allows one probe attempt; a success closes
|
||||
the breaker, a failure reopens it.
|
||||
|
||||
``LLMClient`` is abstract and provider-agnostic (section 13): subclasses
|
||||
implement ``_call_once`` (one attempt, returning text + token usage).
|
||||
``FakeLLMClient`` serves a scripted sequence of responses offline (P1.1),
|
||||
with optional simulated transient failures - this is what drives P1.3's
|
||||
fully offline battle tests.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import time
|
||||
import zlib
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from threading import RLock
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.combat import OBSERVATION_SCHEMA_VERSION
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable, Sequence
|
||||
|
||||
|
||||
class ClientError(RuntimeError):
|
||||
"""Base error for the LLM client layer."""
|
||||
|
||||
|
||||
class ProviderError(ClientError):
|
||||
"""A transient provider failure - safe to retry."""
|
||||
|
||||
|
||||
class RetryExhaustedError(ClientError):
|
||||
"""Every retry attempt failed; the last provider error is the cause."""
|
||||
|
||||
|
||||
class CircuitOpenError(ClientError):
|
||||
"""The circuit breaker is open: calls are refused, not attempted."""
|
||||
|
||||
|
||||
class BudgetExceededError(ClientError):
|
||||
"""A cost/token/call ceiling was hit (P1.2, section 13)."""
|
||||
|
||||
|
||||
class ScriptExhaustedError(ClientError):
|
||||
"""A scripted fake client has no more responses to serve."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClientParams:
|
||||
"""Sampling parameters; part of the composite cache key (D6)."""
|
||||
|
||||
temperature: float = 0.0
|
||||
max_tokens: int = 1024
|
||||
top_p: float = 1.0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Pricing:
|
||||
"""USD per 1000 tokens, used to estimate the cost ledger."""
|
||||
|
||||
prompt_per_1k_usd: float = 0.0
|
||||
completion_per_1k_usd: float = 0.0
|
||||
|
||||
def cost_usd(self, usage: TokenUsage) -> float:
|
||||
return (
|
||||
usage.prompt_tokens / 1000 * self.prompt_per_1k_usd
|
||||
+ usage.completion_tokens / 1000 * self.completion_per_1k_usd
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CostBudget:
|
||||
"""Optional hard ceilings enforced by ``LLMClient`` (section 13).
|
||||
|
||||
Each field defaults to ``None`` = unlimited for that resource.
|
||||
"""
|
||||
|
||||
max_calls: int | None = None
|
||||
max_total_tokens: int | None = None
|
||||
max_cost_usd: float | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TokenUsage:
|
||||
"""Token counts of one completion (measured by the provider, or estimated)."""
|
||||
|
||||
prompt_tokens: int
|
||||
completion_tokens: int
|
||||
|
||||
@property
|
||||
def total(self) -> int:
|
||||
return self.prompt_tokens + self.completion_tokens
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Completion:
|
||||
"""One successful provider response."""
|
||||
|
||||
text: str
|
||||
usage: TokenUsage
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClientStats:
|
||||
"""Read-only snapshot of the client ledger (for reports and sweeps)."""
|
||||
|
||||
calls: int
|
||||
prompt_tokens: int
|
||||
completion_tokens: int
|
||||
cost_usd: float
|
||||
cache_hits: int
|
||||
cache_misses: int
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""Rough token estimate (chars / 4) used when the provider reports none."""
|
||||
return max(1, len(text) // 4)
|
||||
|
||||
|
||||
def response_cache_key(
|
||||
*,
|
||||
model: str,
|
||||
params: ClientParams,
|
||||
prompt_version: str,
|
||||
schema_version: int,
|
||||
prompt: str,
|
||||
) -> str:
|
||||
"""Composite cache key (D6): model, params, prompt version, schema, hash.
|
||||
|
||||
The prompt hash covers the full rendered prompt text, so two different
|
||||
observations never collide even when they share every other component.
|
||||
"""
|
||||
params_component = (
|
||||
f"t={params.temperature:g};mt={params.max_tokens};tp={params.top_p:g}"
|
||||
)
|
||||
digest = hashlib.sha256(prompt.encode("utf-8")).hexdigest()
|
||||
return (
|
||||
f"{model}|{params_component}|{prompt_version}"
|
||||
f"|schema={schema_version}|{digest}"
|
||||
)
|
||||
|
||||
|
||||
class ResponseCache:
|
||||
"""Thread-safe, sharded cache of completion texts (D6 + D10).
|
||||
|
||||
The key is the composite string from :func:`response_cache_key`; the
|
||||
value is the completion text. Shards are selected by ``zlib.crc32`` so
|
||||
parallel sweep workers (D10) never contend on one dict.
|
||||
"""
|
||||
|
||||
def __init__(self, shard_count: int = 8) -> None:
|
||||
if shard_count < 1:
|
||||
msg = f"shard_count must be >= 1, got {shard_count}"
|
||||
raise ValueError(msg)
|
||||
self._shards: tuple[dict[str, str], ...] = tuple({} for _ in range(shard_count))
|
||||
self._locks: tuple[RLock, ...] = tuple(RLock() for _ in range(shard_count))
|
||||
self._stat_lock = RLock()
|
||||
self._hits = 0
|
||||
self._misses = 0
|
||||
|
||||
def get(self, key: str) -> str | None:
|
||||
"""Return the cached completion for ``key``, or ``None``."""
|
||||
shard, lock = self._shard_for(key)
|
||||
with lock:
|
||||
value = shard.get(key)
|
||||
with self._stat_lock:
|
||||
if value is None:
|
||||
self._misses += 1
|
||||
else:
|
||||
self._hits += 1
|
||||
return value
|
||||
|
||||
def put(self, key: str, text: str) -> None:
|
||||
"""Store ``text`` under ``key`` (idempotent)."""
|
||||
shard, lock = self._shard_for(key)
|
||||
with lock:
|
||||
shard[key] = text
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Drop every entry and reset the hit/miss counters (tests only)."""
|
||||
for shard, lock in zip(self._shards, self._locks, strict=True):
|
||||
with lock:
|
||||
shard.clear()
|
||||
with self._stat_lock:
|
||||
self._hits = 0
|
||||
self._misses = 0
|
||||
|
||||
@property
|
||||
def hits(self) -> int:
|
||||
with self._stat_lock:
|
||||
return self._hits
|
||||
|
||||
@property
|
||||
def misses(self) -> int:
|
||||
with self._stat_lock:
|
||||
return self._misses
|
||||
|
||||
def _shard_for(self, key: str) -> tuple[dict[str, str], RLock]:
|
||||
index = zlib.crc32(key.encode("utf-8")) % len(self._shards)
|
||||
return self._shards[index], self._locks[index]
|
||||
|
||||
|
||||
class CircuitBreaker:
|
||||
"""Trips after N consecutive failures; probes after K refusals.
|
||||
|
||||
Not thread-safe by itself: ``LLMClient`` serializes all access with its
|
||||
own lock.
|
||||
"""
|
||||
|
||||
def __init__(self, *, failure_threshold: int = 3, open_refusals: int = 2) -> None:
|
||||
if failure_threshold < 1:
|
||||
msg = f"failure_threshold must be >= 1, got {failure_threshold}"
|
||||
raise ValueError(msg)
|
||||
if open_refusals < 0:
|
||||
msg = f"open_refusals must be >= 0, got {open_refusals}"
|
||||
raise ValueError(msg)
|
||||
self.failure_threshold = failure_threshold
|
||||
self.open_refusals = open_refusals
|
||||
self._consecutive_failures = 0
|
||||
self._refusals = 0
|
||||
|
||||
@property
|
||||
def is_open(self) -> bool:
|
||||
return self._consecutive_failures >= self.failure_threshold
|
||||
|
||||
@property
|
||||
def probe_due(self) -> bool:
|
||||
"""True when an open breaker allows exactly one probe attempt."""
|
||||
return self.is_open and self._refusals >= self.open_refusals
|
||||
|
||||
@property
|
||||
def consecutive_failures(self) -> int:
|
||||
return self._consecutive_failures
|
||||
|
||||
def record_failure(self) -> None:
|
||||
self._consecutive_failures += 1
|
||||
self._refusals = 0
|
||||
|
||||
def record_refusal(self) -> None:
|
||||
self._refusals += 1
|
||||
|
||||
def close(self) -> None:
|
||||
self._consecutive_failures = 0
|
||||
self._refusals = 0
|
||||
|
||||
|
||||
class LLMClient(ABC):
|
||||
"""Provider-agnostic client: caching, budgets, retry/backoff, breaker.
|
||||
|
||||
Subclasses implement :meth:`_call_once` (one provider attempt). The
|
||||
public :meth:`complete` adds the shared safeguards around it.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
model: str,
|
||||
prompt_version: str,
|
||||
schema_version: int = OBSERVATION_SCHEMA_VERSION,
|
||||
cache: ResponseCache | None = None,
|
||||
budget: CostBudget | None = None,
|
||||
pricing: Pricing | None = None,
|
||||
retries: int = 2,
|
||||
backoff_base_s: float = 0.25,
|
||||
sleep: Callable[[float], None] = time.sleep,
|
||||
breaker: CircuitBreaker | None = None,
|
||||
) -> None:
|
||||
if retries < 0:
|
||||
msg = f"retries must be >= 0, got {retries}"
|
||||
raise ValueError(msg)
|
||||
if backoff_base_s < 0:
|
||||
msg = f"backoff_base_s must be >= 0, got {backoff_base_s}"
|
||||
raise ValueError(msg)
|
||||
self.model = model
|
||||
self.prompt_version = prompt_version
|
||||
self.schema_version = schema_version
|
||||
# Cache is mandatory (section 13); a client without an explicit one
|
||||
# gets its own instance, so caching is never silently off.
|
||||
self._cache = cache if cache is not None else ResponseCache()
|
||||
self._budget = budget
|
||||
self._pricing = pricing
|
||||
self._retries = retries
|
||||
self._backoff_base_s = backoff_base_s
|
||||
self._sleep = sleep
|
||||
self._breaker = breaker if breaker is not None else CircuitBreaker()
|
||||
self._lock = RLock()
|
||||
self._calls = 0
|
||||
self._prompt_tokens = 0
|
||||
self._completion_tokens = 0
|
||||
self._cost_usd = 0.0
|
||||
self._cache_hits = 0
|
||||
self._cache_misses = 0
|
||||
|
||||
@abstractmethod
|
||||
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
|
||||
"""One provider attempt; raise ``ProviderError`` on transient failure."""
|
||||
|
||||
def complete(
|
||||
self,
|
||||
prompt: str,
|
||||
*,
|
||||
params: ClientParams | None = None,
|
||||
) -> str:
|
||||
"""Complete ``prompt`` through the cache/budget/breaker safeguards.
|
||||
|
||||
Returns the completion text (from the cache when possible). Raises a
|
||||
typed ``ClientError`` subclass on budget, circuit or retry failure.
|
||||
"""
|
||||
parameters = params if params is not None else ClientParams()
|
||||
key = response_cache_key(
|
||||
model=self.model,
|
||||
params=parameters,
|
||||
prompt_version=self.prompt_version,
|
||||
schema_version=self.schema_version,
|
||||
prompt=prompt,
|
||||
)
|
||||
|
||||
with self._lock:
|
||||
self._check_call_budget()
|
||||
cached = self._cache.get(key)
|
||||
if cached is not None:
|
||||
self._cache_hits += 1
|
||||
return cached
|
||||
self._cache_misses += 1
|
||||
if self._breaker.is_open and not self._breaker.probe_due:
|
||||
self._breaker.record_refusal()
|
||||
msg = (
|
||||
f"circuit breaker open "
|
||||
f"({self._breaker.consecutive_failures} consecutive "
|
||||
f"failures); call refused"
|
||||
)
|
||||
raise CircuitOpenError(msg)
|
||||
|
||||
try:
|
||||
completion = self._complete_with_retry(prompt, parameters)
|
||||
except RetryExhaustedError:
|
||||
with self._lock:
|
||||
self._breaker.record_failure()
|
||||
raise
|
||||
|
||||
with self._lock:
|
||||
self._record_usage(completion.usage)
|
||||
self._breaker.close()
|
||||
self._cache.put(key, completion.text)
|
||||
return completion.text
|
||||
|
||||
def _complete_with_retry(
|
||||
self,
|
||||
prompt: str,
|
||||
params: ClientParams,
|
||||
) -> Completion:
|
||||
"""Attempt ``_call_once`` up to ``retries + 1`` times, backing off."""
|
||||
last_error: ProviderError | None = None
|
||||
for attempt in range(self._retries + 1):
|
||||
with self._lock:
|
||||
self._calls += 1
|
||||
try:
|
||||
return self._call_once(prompt, params=params)
|
||||
except ProviderError as exc:
|
||||
last_error = exc
|
||||
if attempt < self._retries:
|
||||
self._sleep(self._backoff_base_s * (2**attempt))
|
||||
msg = (
|
||||
f"provider failed after {self._retries + 1} attempt(s): {last_error}"
|
||||
)
|
||||
raise RetryExhaustedError(msg) from last_error
|
||||
|
||||
def _check_call_budget(self) -> None:
|
||||
if self._budget is None or self._budget.max_calls is None:
|
||||
return
|
||||
if self._calls >= self._budget.max_calls:
|
||||
msg = (
|
||||
f"call budget exceeded: {self._calls} >= "
|
||||
f"{self._budget.max_calls} calls"
|
||||
)
|
||||
raise BudgetExceededError(msg)
|
||||
|
||||
def _record_usage(self, usage: TokenUsage) -> None:
|
||||
"""Add a completion to the ledger, enforcing the token/cost ceilings."""
|
||||
if self._budget is not None:
|
||||
ceiling = self._budget.max_total_tokens
|
||||
if ceiling is not None:
|
||||
total_after = (
|
||||
self._prompt_tokens + self._completion_tokens + usage.total
|
||||
)
|
||||
if total_after > ceiling:
|
||||
msg = (
|
||||
f"token budget exceeded: {total_after} > {ceiling} "
|
||||
f"total tokens"
|
||||
)
|
||||
raise BudgetExceededError(msg)
|
||||
self._prompt_tokens += usage.prompt_tokens
|
||||
self._completion_tokens += usage.completion_tokens
|
||||
if self._pricing is not None:
|
||||
cost = self._pricing.cost_usd(usage)
|
||||
self._cost_usd += cost
|
||||
ceiling = None if self._budget is None else self._budget.max_cost_usd
|
||||
if ceiling is not None and self._cost_usd > ceiling:
|
||||
msg = (
|
||||
f"cost budget exceeded: {self._cost_usd:.6f} USD > "
|
||||
f"{ceiling:.6f} USD"
|
||||
)
|
||||
raise BudgetExceededError(msg)
|
||||
|
||||
@property
|
||||
def calls(self) -> int:
|
||||
with self._lock:
|
||||
return self._calls
|
||||
|
||||
@property
|
||||
def prompt_tokens(self) -> int:
|
||||
with self._lock:
|
||||
return self._prompt_tokens
|
||||
|
||||
@property
|
||||
def completion_tokens(self) -> int:
|
||||
with self._lock:
|
||||
return self._completion_tokens
|
||||
|
||||
@property
|
||||
def cost_usd(self) -> float:
|
||||
with self._lock:
|
||||
return self._cost_usd
|
||||
|
||||
@property
|
||||
def cache_hits(self) -> int:
|
||||
with self._lock:
|
||||
return self._cache_hits
|
||||
|
||||
@property
|
||||
def cache_misses(self) -> int:
|
||||
with self._lock:
|
||||
return self._cache_misses
|
||||
|
||||
@property
|
||||
def stats(self) -> ClientStats:
|
||||
with self._lock:
|
||||
return ClientStats(
|
||||
calls=self._calls,
|
||||
prompt_tokens=self._prompt_tokens,
|
||||
completion_tokens=self._completion_tokens,
|
||||
cost_usd=self._cost_usd,
|
||||
cache_hits=self._cache_hits,
|
||||
cache_misses=self._cache_misses,
|
||||
)
|
||||
|
||||
|
||||
class FakeLLMClient(LLMClient):
|
||||
"""Scripted, offline LLM client (P1.1) for tests and the showcase.
|
||||
|
||||
Serves ``script`` in order, one response per successful call. With
|
||||
``failures > 0`` the first ``failures`` attempts raise a simulated
|
||||
``ProviderError`` (to exercise retry/backoff and the breaker); scripts
|
||||
are only consumed by successes. A call past the script raises
|
||||
``ScriptExhaustedError`` (a test-authoring bug, not a transient outage).
|
||||
Token usage is estimated from character counts.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
script: Sequence[str],
|
||||
*,
|
||||
model: str = "fake",
|
||||
prompt_version: str = "fixture:fake:0",
|
||||
failures: int = 0,
|
||||
schema_version: int = OBSERVATION_SCHEMA_VERSION,
|
||||
cache: ResponseCache | None = None,
|
||||
budget: CostBudget | None = None,
|
||||
pricing: Pricing | None = None,
|
||||
retries: int = 2,
|
||||
backoff_base_s: float = 0.25,
|
||||
breaker: CircuitBreaker | None = None,
|
||||
) -> None:
|
||||
if failures < 0:
|
||||
msg = f"failures must be >= 0, got {failures}"
|
||||
raise ValueError(msg)
|
||||
super().__init__(
|
||||
model=model,
|
||||
prompt_version=prompt_version,
|
||||
schema_version=schema_version,
|
||||
cache=cache,
|
||||
budget=budget,
|
||||
pricing=pricing,
|
||||
retries=retries,
|
||||
backoff_base_s=backoff_base_s,
|
||||
sleep=lambda _: None, # tests must never actually sleep
|
||||
breaker=breaker,
|
||||
)
|
||||
self._script = tuple(script)
|
||||
self._failures = failures
|
||||
self._attempts = 0
|
||||
self._served = 0
|
||||
|
||||
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
|
||||
del params # a scripted fake ignores sampling parameters
|
||||
with self._lock:
|
||||
self._attempts += 1
|
||||
if self._attempts <= self._failures:
|
||||
msg = f"simulated provider outage (attempt {self._attempts})"
|
||||
raise ProviderError(msg)
|
||||
if self._served >= len(self._script):
|
||||
msg = (
|
||||
f"fake script exhausted after {self._served} served "
|
||||
f"response(s) of {len(self._script)}"
|
||||
)
|
||||
raise ScriptExhaustedError(msg)
|
||||
text = self._script[self._served]
|
||||
self._served += 1
|
||||
usage = TokenUsage(
|
||||
prompt_tokens=estimate_tokens(prompt),
|
||||
completion_tokens=estimate_tokens(text),
|
||||
)
|
||||
return Completion(text=text, usage=usage)
|
||||
@@ -121,6 +121,7 @@ Phase 0 documented deviations from PF1e (conventions):
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import inspect
|
||||
from collections.abc import Callable, Iterable
|
||||
from dataclasses import asdict, dataclass, field, replace
|
||||
from itertools import pairwise
|
||||
@@ -177,19 +178,21 @@ if TYPE_CHECKING:
|
||||
from pf1e_simulator.rng import Rng
|
||||
from pf1e_simulator.spells import SpellSpec
|
||||
|
||||
TrickCondition = Literal[
|
||||
"dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"
|
||||
]
|
||||
|
||||
ReadyTrigger = Literal[
|
||||
"enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"
|
||||
]
|
||||
|
||||
from pf1e_simulator.los import (
|
||||
cover_bonus_cells,
|
||||
has_line_of_effect_cells,
|
||||
)
|
||||
|
||||
# Runtime aliases (not only annotations): Pydantic type adapters over ``Action``
|
||||
# must resolve them, so they cannot live inside the TYPE_CHECKING block.
|
||||
TrickCondition = Literal[
|
||||
"dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"
|
||||
]
|
||||
|
||||
ReadyTrigger = Literal[
|
||||
"enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"
|
||||
]
|
||||
|
||||
_SQUARE_FT = 5 # Phase 0 maps use 5-ft squares
|
||||
_NATURAL_ONE = 1 # PF1e: natural 1 always misses
|
||||
_NATURAL_TWENTY = 20 # PF1e: natural 20 always hits and threatens
|
||||
@@ -1259,6 +1262,12 @@ class ChargeResult:
|
||||
reason: str | None
|
||||
|
||||
|
||||
# Observation schema version (P0.1, docs/phase-2-llm-architecture.md section
|
||||
# 10). Bump only when a top-level key is removed or its meaning changes;
|
||||
# additive keys keep the version.
|
||||
OBSERVATION_SCHEMA_VERSION = 1
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Observation:
|
||||
"""Serializable snapshot of combat state for a policy/LLM.
|
||||
@@ -1269,7 +1278,9 @@ class Observation:
|
||||
collections are copies (tuples/dicts), never read back by the engine, so a
|
||||
policy may mutate them freely without touching combat; the snapshot is
|
||||
JSON-serializable (the top level is a frozen dataclass; the inner dicts are
|
||||
plain mutable copies, not deep-frozen).
|
||||
plain mutable copies, not deep-frozen). ``to_dict()`` carries
|
||||
``schema_version`` (``OBSERVATION_SCHEMA_VERSION``) as its first key so a
|
||||
consumer can detect breaking schema changes.
|
||||
|
||||
``combatants`` holds one dict per combatant in ``self._states`` order;
|
||||
``pairs`` holds one dict per (acting combatant x each other combatant) with
|
||||
@@ -1310,6 +1321,7 @@ class Observation:
|
||||
def to_dict(self) -> dict[str, object]:
|
||||
"""Return a JSON-serializable dict with the pinned top-level keys."""
|
||||
return {
|
||||
"schema_version": OBSERVATION_SCHEMA_VERSION,
|
||||
"round": self.round,
|
||||
"acting_combatant": self.acting_combatant,
|
||||
"combatants": self.combatants,
|
||||
@@ -1514,7 +1526,7 @@ _REASON_CANNOT_COMBINE = "cannot_combine"
|
||||
|
||||
# Action kinds that are declared but have no engine dispatch (U1 surfaces
|
||||
# them as explicit rejections instead of silent no-ops).
|
||||
_UNSUPPORTED_ACTION_KINDS: frozenset[str] = frozenset({"swift", "free", "full_round"})
|
||||
UNSUPPORTED_ACTION_KINDS: frozenset[str] = frozenset({"swift", "free", "full_round"})
|
||||
# Action kinds whose helper is a silent no-op (or worse) without a resolved
|
||||
# target; _execute rejects them up front with missing_target instead.
|
||||
# move / 5foot_step / run are exempt when an explicit destination is provided
|
||||
@@ -1613,7 +1625,7 @@ class CombatEngine:
|
||||
*,
|
||||
round_cap: int = 100,
|
||||
policy: Policy | None = None,
|
||||
llm_policy: LLMPolicy | None = None,
|
||||
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
|
||||
spell_registry: dict[str, SpellSpec] | None = None,
|
||||
reaction_policy: ReactionPolicy | None = None,
|
||||
monsters_die_at_zero: bool = False,
|
||||
@@ -1685,18 +1697,24 @@ class CombatEngine:
|
||||
if self._monsters_die_at_zero and not _is_pc_combatant(state.combatant):
|
||||
state.dies_at_zero = True
|
||||
|
||||
def _llm_adapter(self, llm_policy: LLMPolicy) -> Policy:
|
||||
"""Wrap an ``LLMPolicy`` into a ``Policy``.
|
||||
def _llm_adapter(self, llm_policy: LLMPolicy | LegacyLLMPolicy) -> Policy:
|
||||
"""Wrap an ``LLMPolicy`` (or a pre-D1 two-argument callable) into a ``Policy``.
|
||||
|
||||
At the point of invocation (``actions = self._policy(self, state)`` in
|
||||
``_take_turn``), the Observation is built for the current state and
|
||||
delegated to ``llm_policy(observation, state)``. The adapter consumes
|
||||
no RNG and mutates no state.
|
||||
delegated. New-style policies get ``llm_policy(observation)`` only;
|
||||
a legacy ``llm_policy(observation, state)`` callable is detected ONCE
|
||||
here (signature inspection, not per turn) and keeps receiving the live
|
||||
state. The adapter consumes no RNG and mutates no state.
|
||||
"""
|
||||
two_arg = _accepts_state(llm_policy)
|
||||
call: Callable[..., tuple[Action, ...]] = llm_policy
|
||||
|
||||
def adapter(_engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
observation = self._build_observation(state)
|
||||
return llm_policy(observation, state)
|
||||
if two_arg:
|
||||
return call(observation, state)
|
||||
return call(observation)
|
||||
|
||||
return adapter
|
||||
|
||||
@@ -5079,7 +5097,7 @@ class CombatEngine:
|
||||
reports ok.
|
||||
"""
|
||||
kind = action.kind
|
||||
if kind in _UNSUPPORTED_ACTION_KINDS:
|
||||
if kind in UNSUPPORTED_ACTION_KINDS:
|
||||
self._log(
|
||||
f"round {self._current_round} {state.combatant.id}: "
|
||||
f"{kind} rejected ({_REASON_UNSUPPORTED_KIND})"
|
||||
@@ -10572,7 +10590,36 @@ def _ready_payload(action: Action) -> Action:
|
||||
|
||||
Policy = Callable[[CombatEngine, CombatantState], tuple[Action, ...]]
|
||||
|
||||
LLMPolicy = Callable[[Observation, CombatantState], tuple[Action, ...]]
|
||||
# D1 (docs/phase-2-llm-architecture.md section 13): the LLM policy entry is
|
||||
# observation-only -- the snapshot is the policy's sole information source and
|
||||
# the live CombatantState is never handed to a new-style policy. Pre-D1
|
||||
# two-argument callables remain accepted via CombatEngine._llm_adapter.
|
||||
LLMPolicy = Callable[[Observation], tuple[Action, ...]]
|
||||
|
||||
LegacyLLMPolicy = Callable[[Observation, CombatantState], tuple[Action, ...]]
|
||||
|
||||
_LEGACY_LLM_POLICY_ARITY = 2
|
||||
|
||||
|
||||
def _accepts_state(llm_policy: LLMPolicy | LegacyLLMPolicy) -> bool:
|
||||
"""True when ``llm_policy`` is a pre-D1 callable taking (observation, state).
|
||||
|
||||
Counts REQUIRED positional parameters (a defaulted second parameter is a
|
||||
new-style policy that tolerates the 1-argument call). Callables whose
|
||||
signature cannot be inspected default to the new-style 1-argument call.
|
||||
"""
|
||||
try:
|
||||
parameters = list(inspect.signature(llm_policy).parameters.values())
|
||||
except (TypeError, ValueError):
|
||||
return False
|
||||
required_positional = [
|
||||
p
|
||||
for p in parameters
|
||||
if p.default is inspect.Parameter.empty
|
||||
and p.kind
|
||||
in (inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.POSITIONAL_OR_KEYWORD)
|
||||
]
|
||||
return len(required_positional) >= _LEGACY_LLM_POLICY_ARITY
|
||||
|
||||
ReactionPolicy = Callable[
|
||||
[CombatEngine, CombatantState, ReactionEvent], Action | None
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Environment-driven configuration for the phase-2 LLM layer (P1/P5.1).
|
||||
|
||||
Architecture doc section 15: the phase-2 code never hard-codes data or
|
||||
configuration paths -- every externally supplied value is resolved here from
|
||||
the environment, so the repository can keep shipping pure code once the data
|
||||
sets and the prompt pack detach. The prompt-pack contract is documented in
|
||||
section 14 (see :mod:`pf1e_simulator.prompt`).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
# Environment variable names (documented in the architecture doc, sections
|
||||
# 14-15). ``PF1E_PROMPT_PACK_DIR`` points at the external, versioned prompt
|
||||
# pack; ``PF1E_LLM_API_KEY`` feeds the remote provider client (P5.1);
|
||||
# ``PF1E_LLM_BASE_URL`` and ``PF1E_LLM_MODEL`` select the provider endpoint
|
||||
# and model, so a self-hosted OpenAI-compatible gateway (vLLM, llama.cpp...)
|
||||
# is configured without touching the code.
|
||||
ENV_PROMPT_PACK_DIR = "PF1E_PROMPT_PACK_DIR"
|
||||
ENV_LLM_API_KEY = "PF1E_LLM_API_KEY"
|
||||
ENV_LLM_BASE_URL = "PF1E_LLM_BASE_URL"
|
||||
ENV_LLM_MODEL = "PF1E_LLM_MODEL"
|
||||
|
||||
# Sensible offline defaults: the OpenAI-compatible chat endpoint and a small,
|
||||
# cheap model. Nothing here is a secret -- the API key stays environment-only.
|
||||
DEFAULT_LLM_BASE_URL = "https://api.openai.com/v1"
|
||||
DEFAULT_LLM_MODEL = "gpt-4o-mini"
|
||||
|
||||
|
||||
def env_str(name: str, *, default: str | None = None) -> str | None:
|
||||
"""Read a configuration string from the environment, if set."""
|
||||
return os.environ.get(name, default)
|
||||
|
||||
|
||||
def env_path(name: str) -> Path | None:
|
||||
"""Read a configuration path from the environment, if set."""
|
||||
raw = os.environ.get(name)
|
||||
return Path(raw) if raw else None
|
||||
|
||||
|
||||
def llm_api_key() -> str | None:
|
||||
"""The provider API key, or None when the caller runs offline."""
|
||||
return env_str(ENV_LLM_API_KEY)
|
||||
|
||||
|
||||
def llm_base_url() -> str:
|
||||
"""The OpenAI-compatible base URL (``PF1E_LLM_BASE_URL``, else default).
|
||||
|
||||
The returned value has no trailing slash; ``OpenAILLMClient`` appends
|
||||
``/chat/completions`` to it (P5.1).
|
||||
"""
|
||||
raw = env_str(ENV_LLM_BASE_URL)
|
||||
return raw.rstrip("/") if raw else DEFAULT_LLM_BASE_URL
|
||||
|
||||
|
||||
def llm_model() -> str:
|
||||
"""The provider model name (``PF1E_LLM_MODEL``, else default)."""
|
||||
raw = env_str(ENV_LLM_MODEL)
|
||||
return raw or DEFAULT_LLM_MODEL
|
||||
|
||||
@@ -0,0 +1,545 @@
|
||||
"""Compact, decision-sufficient observation view for LLM policies (P0.2).
|
||||
|
||||
The raw ``Observation.to_dict()`` snapshot is ~64 KB (~16k tokens) for a
|
||||
7-combatant encounter - too large to feed an LLM every turn (docs
|
||||
``phase-2-llm-architecture.md`` section 16, target ~2-4k tokens). This module
|
||||
builds a compact ``ObservationView`` that keeps every tactically relevant
|
||||
fact while dropping duplicates, derivables and verbose expansions, and
|
||||
renders it as a deterministic, line-based prompt string.
|
||||
|
||||
Compaction rules
|
||||
----------------
|
||||
- ``terrain``: sparse list of ``(row, col, type)`` for non-default cells
|
||||
only. A cell is default when its semantics are ``move_cost == 1``,
|
||||
``blocks_los == False``, ``cover == False`` and ``cover_bonus == 0``.
|
||||
- ``elevation``: sparse list of ``(row, col, height)`` for non-zero cells.
|
||||
- ``threatened`` / ``ranged_threatened``: sparse list of ``(row, col)`` for
|
||||
True cells (empty when no cell is covered).
|
||||
- per-combatant: only decision-relevant keys are kept (see
|
||||
``_COMPACT_COMBATANT_KEYS``). Dropped: ``occupied_cells`` (derivable from
|
||||
``pos`` + ``size``), ``abilities``/``skills`` (raw scores, not used by the
|
||||
candidate generator), ``initiative_mod``/``initiative_total`` (the
|
||||
top-level order suffices), ``temp_hp_rounds_left`` (``temp_hp`` is the
|
||||
current value), ``features``/``feature_details`` (the resources and
|
||||
``sneak_dice`` already expose the class abilities), ``carried_items``,
|
||||
``weapon_damage``, ``aware``, ``alignment``, ``armor_spell_failure``,
|
||||
``arcane_caster`` (all zero in the data) and ``run_rounds``.
|
||||
- attacks: compacted to ``{id, name, kind, attack_bonus, damage (strings
|
||||
like "1d6 slashing"), damage_bonus, crit_range, crit_mult, reach_ft,
|
||||
range_increment_ft, count, firearm}`` - sunder-only object stats
|
||||
(``hardness``/``hit_points``) and ``splash`` are dropped.
|
||||
- spells: compacted to ``{name, level, range_ft, save, dc, cast_time,
|
||||
area}`` - ``summary`` (the bulk of the raw size), ``school``,
|
||||
``half_on_save``, ``save_negates``, ``harmless`` and ``components`` are
|
||||
dropped (no component failure in the engine's data).
|
||||
- per-pair: all decision helpers are kept unchanged, except ``target_index``
|
||||
is replaced by the stable combatant ``target`` id.
|
||||
- ``acting_combatant`` is rendered as the combatant id (not the index).
|
||||
- ``condition_definitions`` is dropped (condition names suffice).
|
||||
- every collection is deterministically ordered (fixed key order, row-major
|
||||
or the raw snapshot's own sorted order).
|
||||
|
||||
Render format
|
||||
-------------
|
||||
The render is line-based ``key=value`` text. The nested ``attacks`` and
|
||||
``spells`` expansions are rendered as positional tuples (the field order is
|
||||
declared once in the ``attack_fields`` / ``spell_fields`` legend lines) with
|
||||
``-`` standing for a missing value (``None`` save/dc/area, no range
|
||||
increment); this is what brings the view under the ~4k-token ceiling.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import TypeGuard
|
||||
|
||||
from .combat import OBSERVATION_SCHEMA_VERSION, Observation
|
||||
|
||||
# Per-combatant keys kept in the compact view, in deterministic order.
|
||||
# Everything else in the raw snapshot is derivable, verbose, or not
|
||||
# decision-relevant for the candidate generator and the LLM.
|
||||
_COMPACT_COMBATANT_KEYS: tuple[str, ...] = (
|
||||
"id",
|
||||
"side",
|
||||
"pos",
|
||||
"hp",
|
||||
"hp_max",
|
||||
"nonlethal",
|
||||
"active",
|
||||
"dead",
|
||||
"dying",
|
||||
"stabilized",
|
||||
"conditions",
|
||||
"resources_left",
|
||||
"weapon_broken",
|
||||
"weapon_destroyed",
|
||||
"weapon_stolen",
|
||||
"weapon_dropped",
|
||||
"grappled_with",
|
||||
"hidden_from",
|
||||
"moved_this_turn",
|
||||
"power_attack",
|
||||
"defensive_ac_bonus",
|
||||
"fight_defensively",
|
||||
"aid_attack_bonus",
|
||||
"aid_ac_bonus",
|
||||
"running",
|
||||
"run_rest_rounds",
|
||||
"bleed_damage",
|
||||
"recurring_damage",
|
||||
"aoo_remaining",
|
||||
"name",
|
||||
"level",
|
||||
"size",
|
||||
"natural_reach_ft",
|
||||
"creature_type",
|
||||
"bab",
|
||||
"casting_ability",
|
||||
"dr",
|
||||
"metamagic_feats",
|
||||
"summon_duration",
|
||||
"immediate_used",
|
||||
"deed_up_close_and_deadly",
|
||||
"active_buffs",
|
||||
"one_shot_bonuses",
|
||||
"temp_hp",
|
||||
"speed_land_ft",
|
||||
"speed_fly_ft",
|
||||
"speed_effective_ft",
|
||||
"ac",
|
||||
"saves",
|
||||
"cmb",
|
||||
"cmd",
|
||||
"attacks",
|
||||
"spells",
|
||||
"spell_resistance",
|
||||
"resistances",
|
||||
"immunities",
|
||||
"sneak_dice",
|
||||
"engaged_in_melee",
|
||||
"last_turn_results",
|
||||
"readied",
|
||||
"pending_cast",
|
||||
)
|
||||
|
||||
# Per-pair keys kept (all decision helpers), in the raw snapshot's order.
|
||||
# ``target_index`` is replaced by the stable combatant ``target`` id.
|
||||
_PAIR_KEYS: tuple[str, ...] = (
|
||||
"distance_ft",
|
||||
"weapon_for",
|
||||
"weapon_reason",
|
||||
"range_penalty",
|
||||
"can_charge",
|
||||
"charge_reason",
|
||||
"can_step5_to_attack",
|
||||
"has_line_of_effect",
|
||||
"flanking",
|
||||
"movement_cost_ft",
|
||||
"target_threatens_me",
|
||||
"enemies_threatening_me",
|
||||
"cover_bonus",
|
||||
)
|
||||
|
||||
# Attack keys kept in the compact expansion (sunder-only object stats and
|
||||
# the splash marker are dropped).
|
||||
_ATTACK_KEYS: tuple[str, ...] = (
|
||||
"id",
|
||||
"name",
|
||||
"kind",
|
||||
"attack_bonus",
|
||||
"damage",
|
||||
"damage_bonus",
|
||||
"crit_range",
|
||||
"crit_mult",
|
||||
"reach_ft",
|
||||
"range_increment_ft",
|
||||
"count",
|
||||
"firearm",
|
||||
)
|
||||
|
||||
# Spell keys kept in the compact expansion (the summary is the bulk of the
|
||||
# raw size; school/components/save flags are not decision-relevant).
|
||||
_SPELL_KEYS: tuple[str, ...] = (
|
||||
"name",
|
||||
"level",
|
||||
"range_ft",
|
||||
"save",
|
||||
"dc",
|
||||
"cast_time",
|
||||
"area",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ObservationView:
|
||||
"""Compact, decision-sufficient view of one combat observation.
|
||||
|
||||
All collections are deterministically ordered; ``terrain``/``elevation``/
|
||||
``threatened``/``ranged_threatened`` are sparse (see module docstring).
|
||||
``acting_combatant`` is the acting combatant's id.
|
||||
"""
|
||||
|
||||
schema_version: int
|
||||
round: int
|
||||
acting_combatant: str
|
||||
dimensions: tuple[int, int]
|
||||
terrain: tuple[tuple[int, int, str], ...]
|
||||
elevation: tuple[tuple[int, int, int], ...]
|
||||
threatened: tuple[tuple[int, int], ...]
|
||||
ranged_threatened: tuple[tuple[int, int], ...]
|
||||
terrain_semantics: tuple[dict[str, object], ...]
|
||||
initiative_order: tuple[str, ...]
|
||||
combatants: tuple[dict[str, object], ...]
|
||||
pairs: tuple[dict[str, object], ...]
|
||||
|
||||
|
||||
def build_view(observation: Observation) -> ObservationView:
|
||||
"""Build a compact ``ObservationView`` from a raw ``Observation``."""
|
||||
height = len(observation.terrain)
|
||||
width = len(observation.terrain[0]) if height else 0
|
||||
semantics_by_type = {
|
||||
str(sem["type"]): sem for sem in observation.terrain_semantics
|
||||
}
|
||||
terrain = tuple(
|
||||
(r, c, cell_type)
|
||||
for r, row in enumerate(observation.terrain)
|
||||
for c, cell_type in enumerate(row)
|
||||
if cell_type is not None
|
||||
and not _is_default_terrain(cell_type, semantics_by_type)
|
||||
)
|
||||
elevation = tuple(
|
||||
(r, c, height_at)
|
||||
for r, row in enumerate(observation.elevation)
|
||||
for c, height_at in enumerate(row)
|
||||
if height_at != 0
|
||||
)
|
||||
combatants = tuple(
|
||||
_compact_combatant(combatant) for combatant in observation.combatants
|
||||
)
|
||||
return ObservationView(
|
||||
schema_version=OBSERVATION_SCHEMA_VERSION,
|
||||
round=observation.round,
|
||||
acting_combatant=_combatant_id(combatants, observation.acting_combatant),
|
||||
dimensions=(height, width),
|
||||
terrain=terrain,
|
||||
elevation=elevation,
|
||||
threatened=_sparse_bool_grid(observation.threatened),
|
||||
ranged_threatened=_sparse_bool_grid(observation.ranged_threatened),
|
||||
terrain_semantics=observation.terrain_semantics,
|
||||
initiative_order=observation.initiative_order,
|
||||
combatants=combatants,
|
||||
pairs=tuple(_compact_pair(pair, combatants) for pair in observation.pairs),
|
||||
)
|
||||
|
||||
|
||||
def render(view: ObservationView) -> str:
|
||||
"""Render a compact view as a deterministic, line-based prompt string."""
|
||||
lines: list[str] = [
|
||||
f"schema_version={view.schema_version}",
|
||||
f"round={view.round}",
|
||||
f"acting={view.acting_combatant}",
|
||||
f"map={view.dimensions[0]}x{view.dimensions[1]}",
|
||||
f"terrain={_fmt_cells(view.terrain)}",
|
||||
f"elevation={_fmt_cells(view.elevation)}",
|
||||
f"threatened={_fmt_cells2(view.threatened)}",
|
||||
f"ranged_threatened={_fmt_cells2(view.ranged_threatened)}",
|
||||
f"terrain_semantics={_fmt_semantics(view.terrain_semantics)}",
|
||||
f"initiative={','.join(view.initiative_order)}",
|
||||
f"attack_fields={','.join(_ATTACK_KEYS)}",
|
||||
f"spell_fields={','.join(_SPELL_KEYS)}",
|
||||
"combatants:",
|
||||
]
|
||||
for index, combatant in enumerate(view.combatants):
|
||||
lines.append(f" {index} " + _fmt_combatant(combatant))
|
||||
lines.append("pairs:")
|
||||
lines.extend(" " + _fmt_kv(pair) for pair in view.pairs)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def compact_observation(observation: Observation) -> str:
|
||||
"""One-shot: build the compact view of an observation and render it."""
|
||||
return render(build_view(observation))
|
||||
|
||||
|
||||
def _is_default_terrain(
|
||||
cell_type: str,
|
||||
semantics_by_type: dict[str, dict[str, object]],
|
||||
) -> bool:
|
||||
"""True when a cell carries no tactical information worth listing."""
|
||||
sem = semantics_by_type.get(cell_type)
|
||||
if sem is None:
|
||||
return False
|
||||
return (
|
||||
sem.get("move_cost") == 1
|
||||
and not sem.get("blocks_los")
|
||||
and not sem.get("cover")
|
||||
and sem.get("cover_bonus") == 0
|
||||
)
|
||||
|
||||
|
||||
def _sparse_bool_grid(
|
||||
grid: tuple[tuple[bool, ...], ...],
|
||||
) -> tuple[tuple[int, int], ...]:
|
||||
"""Sparse ``(row, col)`` list of True cells, row-major."""
|
||||
return tuple(
|
||||
(r, c)
|
||||
for r, row in enumerate(grid)
|
||||
for c, value in enumerate(row)
|
||||
if value
|
||||
)
|
||||
|
||||
|
||||
def _compact_combatant(combatant: dict[str, object]) -> dict[str, object]:
|
||||
"""Select the decision-relevant keys and compact nested expansions."""
|
||||
compact = {
|
||||
key: combatant[key] for key in _COMPACT_COMBATANT_KEYS if key in combatant
|
||||
}
|
||||
attacks = compact.get("attacks")
|
||||
if _is_tuple(attacks):
|
||||
compact["attacks"] = tuple(
|
||||
_compact_attack(attack) for attack in attacks if _is_str_object_dict(attack)
|
||||
)
|
||||
spells = compact.get("spells")
|
||||
if _is_tuple(spells):
|
||||
compact["spells"] = tuple(
|
||||
_compact_spell(spell) for spell in spells if _is_str_object_dict(spell)
|
||||
)
|
||||
return compact
|
||||
|
||||
|
||||
def _compact_attack(attack: dict[str, object]) -> dict[str, object]:
|
||||
"""Compact one attack: damage becomes formula+type strings."""
|
||||
return {
|
||||
"id": _as_str(attack["id"]),
|
||||
"name": _as_str(attack["name"]),
|
||||
"kind": _as_str(attack["kind"]),
|
||||
"attack_bonus": _as_int(attack["attack_bonus"]),
|
||||
"damage": _damage_strings(attack.get("damage")),
|
||||
"damage_bonus": _as_int(attack["damage_bonus"]),
|
||||
"crit_range": _as_int(attack["crit_range"]),
|
||||
"crit_mult": _as_int(attack["crit_mult"]),
|
||||
"reach_ft": _as_int(attack["reach_ft"]),
|
||||
"range_increment_ft": _as_optional_int(attack["range_increment_ft"]),
|
||||
"count": _as_int(attack["count"]),
|
||||
"firearm": _as_bool(attack["firearm"]),
|
||||
}
|
||||
|
||||
|
||||
def _compact_spell(spell: dict[str, object]) -> dict[str, object]:
|
||||
"""Compact one spell: keep only the decision-relevant fields."""
|
||||
return {
|
||||
"name": _as_str(spell["name"]),
|
||||
"level": _as_optional_int(spell["level"]),
|
||||
"range_ft": _as_int(spell["range_ft"]),
|
||||
"save": _as_str(spell["save"]),
|
||||
"dc": _as_optional_int(spell["dc"]),
|
||||
"cast_time": _as_str(spell["cast_time"]),
|
||||
"area": spell.get("area"),
|
||||
}
|
||||
|
||||
|
||||
def _compact_pair(
|
||||
pair: dict[str, object],
|
||||
combatants: tuple[dict[str, object], ...],
|
||||
) -> dict[str, object]:
|
||||
"""Select the decision-helper keys of one pair, in fixed order."""
|
||||
compact = {key: pair[key] for key in _PAIR_KEYS if key in pair}
|
||||
compact["target"] = _combatant_id(combatants, _as_int(pair["target_index"]))
|
||||
return compact
|
||||
|
||||
|
||||
def _combatant_id(combatants: tuple[dict[str, object], ...], index: int) -> str:
|
||||
"""Stable id of the combatant at ``index`` (falls back to the index)."""
|
||||
if 0 <= index < len(combatants):
|
||||
cid = combatants[index].get("id")
|
||||
if isinstance(cid, str):
|
||||
return cid
|
||||
return str(index)
|
||||
|
||||
|
||||
def _damage_strings(value: object) -> tuple[str, ...]:
|
||||
"""Render a damage component list as ``"1d6 slashing"`` strings."""
|
||||
if not _is_tuple(value):
|
||||
return ()
|
||||
strings: list[str] = []
|
||||
for component in value:
|
||||
if not _is_str_object_dict(component):
|
||||
continue
|
||||
formula = component.get("formula")
|
||||
if not isinstance(formula, str):
|
||||
continue
|
||||
types = component.get("types")
|
||||
if _is_tuple(types):
|
||||
parts = [formula, *(t for t in types if isinstance(t, str))]
|
||||
else:
|
||||
parts = [formula]
|
||||
strings.append(" ".join(parts))
|
||||
return tuple(strings)
|
||||
|
||||
|
||||
def _fmt_combatant(combatant: dict[str, object]) -> str:
|
||||
"""Render one compact combatant, attacks/spells positionally."""
|
||||
parts: list[str] = []
|
||||
for key, value in combatant.items():
|
||||
if key == "attacks":
|
||||
if _is_tuple(value):
|
||||
parts.append(
|
||||
"attacks=("
|
||||
+ ",".join(
|
||||
_fmt_attack(attack)
|
||||
for attack in value
|
||||
if _is_str_object_dict(attack)
|
||||
)
|
||||
+ ")"
|
||||
)
|
||||
continue
|
||||
if key == "spells":
|
||||
if _is_tuple(value):
|
||||
parts.append(
|
||||
"spells=("
|
||||
+ ",".join(
|
||||
_fmt_spell(spell)
|
||||
for spell in value
|
||||
if _is_str_object_dict(spell)
|
||||
)
|
||||
+ ")"
|
||||
)
|
||||
continue
|
||||
parts.append(f"{key}={_fmt_value(value)}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def _fmt_attack(attack: dict[str, object]) -> str:
|
||||
"""Positional attack tuple; ``-`` for a missing range increment."""
|
||||
damage = attack.get("damage")
|
||||
damage_str = (
|
||||
"+".join(d for d in damage if isinstance(d, str)) if _is_tuple(damage) else ""
|
||||
)
|
||||
if not damage_str:
|
||||
damage_str = "-"
|
||||
range_increment = _as_optional_int(attack["range_increment_ft"])
|
||||
range_str = "-" if range_increment is None else str(range_increment)
|
||||
return (
|
||||
f"({_as_str(attack['id'])},{_as_str(attack['name'])},"
|
||||
f"{_as_str(attack['kind'])},{_as_int(attack['attack_bonus'])},"
|
||||
f"{damage_str},{_as_int(attack['damage_bonus'])},"
|
||||
f"{_as_int(attack['crit_range'])},{_as_int(attack['crit_mult'])},"
|
||||
f"{_as_int(attack['reach_ft'])},{range_str},"
|
||||
f"{_as_int(attack['count'])},{1 if _as_bool(attack['firearm']) else 0})"
|
||||
)
|
||||
|
||||
|
||||
def _fmt_spell(spell: dict[str, object]) -> str:
|
||||
"""Positional spell tuple; ``-`` for a missing save/dc/area."""
|
||||
save = _as_str(spell["save"])
|
||||
save_str = "-" if save == "none" else save
|
||||
dc = _as_optional_int(spell["dc"])
|
||||
dc_str = "-" if dc is None else str(dc)
|
||||
area = spell.get("area")
|
||||
if area is None:
|
||||
area_str = "-"
|
||||
elif _is_str_object_dict(area):
|
||||
shape = _as_str(area["shape"])
|
||||
radius = _as_int(area["radius_ft"])
|
||||
centered = _as_bool(area["centered_on_caster"])
|
||||
area_str = f"{shape}:{radius}" + (":c" if centered else "")
|
||||
else:
|
||||
area_str = "-"
|
||||
level = _as_optional_int(spell["level"])
|
||||
level_str = "-" if level is None else str(level)
|
||||
return (
|
||||
f"({_as_str(spell['name'])},{level_str},"
|
||||
f"{_as_int(spell['range_ft'])},{save_str},{dc_str},"
|
||||
f"{_as_str(spell['cast_time'])},{area_str})"
|
||||
)
|
||||
|
||||
|
||||
def _fmt_cells(cells: tuple[tuple[int, int, object], ...]) -> str:
|
||||
return "".join(f"({r},{c},{_fmt_value(v)})" for r, c, v in cells)
|
||||
|
||||
|
||||
def _fmt_cells2(cells: tuple[tuple[int, int], ...]) -> str:
|
||||
return "".join(f"({r},{c})" for r, c in cells)
|
||||
|
||||
|
||||
def _fmt_semantics(semantics: tuple[dict[str, object], ...]) -> str:
|
||||
parts: list[str] = []
|
||||
for sem in semantics:
|
||||
name = str(sem.get("type", "?"))
|
||||
move_cost = sem.get("move_cost")
|
||||
blocks_los = sem.get("blocks_los")
|
||||
cover = sem.get("cover")
|
||||
cover_bonus = sem.get("cover_bonus")
|
||||
parts.append(
|
||||
f"{name}{{mc={move_cost},los={1 if blocks_los else 0},"
|
||||
f"cov={1 if cover else 0},cb={cover_bonus}}}"
|
||||
)
|
||||
return ",".join(parts)
|
||||
|
||||
|
||||
def _fmt_kv(mapping: dict[str, object]) -> str:
|
||||
return " ".join(f"{key}={_fmt_value(value)}" for key, value in mapping.items())
|
||||
|
||||
|
||||
def _fmt_value(value: object) -> str:
|
||||
"""Compact, unambiguous rendering of one snapshot value.
|
||||
|
||||
``None`` falls through to ``str(value)`` (already ``"None"``).
|
||||
"""
|
||||
if isinstance(value, bool):
|
||||
return "1" if value else "0"
|
||||
if isinstance(value, (int, float)):
|
||||
return str(value)
|
||||
if isinstance(value, str):
|
||||
return (
|
||||
value
|
||||
if value and all(c.isalnum() or c in "_-" for c in value)
|
||||
else json.dumps(value)
|
||||
)
|
||||
if _is_tuple(value):
|
||||
return "(" + ",".join(_fmt_value(item) for item in value) + ")"
|
||||
if _is_str_object_dict(value):
|
||||
return (
|
||||
"{"
|
||||
+ ",".join(f"{key}:{_fmt_value(item)}" for key, item in value.items())
|
||||
+ "}"
|
||||
)
|
||||
return str(value)
|
||||
|
||||
|
||||
def _as_str(value: object) -> str:
|
||||
if not isinstance(value, str):
|
||||
msg = f"expected str, got {type(value).__name__}"
|
||||
raise TypeError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def _as_int(value: object) -> int:
|
||||
if not isinstance(value, int):
|
||||
msg = f"expected int, got {type(value).__name__}"
|
||||
raise TypeError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def _as_bool(value: object) -> bool:
|
||||
if not isinstance(value, bool):
|
||||
msg = f"expected bool, got {type(value).__name__}"
|
||||
raise TypeError(msg)
|
||||
return value
|
||||
|
||||
|
||||
def _as_optional_int(value: object) -> int | None:
|
||||
if value is None:
|
||||
return None
|
||||
return _as_int(value)
|
||||
|
||||
|
||||
def _is_tuple(value: object) -> TypeGuard[tuple[object, ...]]:
|
||||
"""Narrow ``object`` to a tuple (snapshot collections are tuples)."""
|
||||
return isinstance(value, tuple)
|
||||
|
||||
|
||||
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
"""Narrow ``object`` to the snapshot's ``dict[str, object]`` shape."""
|
||||
return isinstance(value, dict)
|
||||
@@ -0,0 +1,313 @@
|
||||
"""Natural-language policy (P1.3): parses LLM JSON replies into Action sequences.
|
||||
|
||||
P5.2 adds a bounded repair loop: on a parse failure the policy re-asks the
|
||||
model with an accumulated error trail up to ``max_repairs`` times, then
|
||||
falls back to the scripted policy; ``NLPolicyStats`` exposes the call and
|
||||
repair counters for the sweep and the phase-3 report.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field, fields
|
||||
from typing import TYPE_CHECKING, Literal, TypeGuard, get_args
|
||||
|
||||
from pf1e_simulator.candidates import candidates
|
||||
from pf1e_simulator.combat import Action, ReadyTrigger, TrickCondition
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.client import LLMClient
|
||||
from pf1e_simulator.combat import Observation
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
|
||||
|
||||
# ---- Type aliases for the 27 supported kinds (filtered from Action.kind) ----
|
||||
KindValue = Literal[
|
||||
"attack", "move", "wait", "immediate", "full_attack", "charge", "withdraw",
|
||||
"5foot_step", "cast_spell", "maneuver", "channel_energy", "hex", "deed",
|
||||
"spell_combat", "inspire_courage", "retrieve_weapon", "repair_weapon",
|
||||
"activate_power_attack", "delay", "ready", "total_defense",
|
||||
"fight_defensively", "stand_up", "drop_prone", "aid_another", "run", "hide",
|
||||
]
|
||||
_KIND_VALUES: frozenset[str] = frozenset(get_args(KindValue))
|
||||
_FIELD_NAMES: frozenset[str] = frozenset(f.name for f in fields(Action))
|
||||
_PAIR_LEN = 2
|
||||
_CHANNEL_MODES: frozenset[str] = frozenset(get_args(Literal["heal", "harm"]))
|
||||
_AID_MODES: frozenset[str] = frozenset(get_args(Literal["attack", "ac"]))
|
||||
_TRICK_CONDITIONS: frozenset[str] = frozenset(get_args(TrickCondition))
|
||||
_READY_TRIGGERS: frozenset[str] = frozenset(get_args(ReadyTrigger))
|
||||
|
||||
|
||||
class NLPolicyError(ValueError):
|
||||
"""The LLM reply is not a valid JSON action sequence."""
|
||||
|
||||
|
||||
# ---- Type guards -------------------------------------------------------------
|
||||
|
||||
def _is_kind(value: object) -> TypeGuard[KindValue]:
|
||||
return value in _KIND_VALUES
|
||||
|
||||
|
||||
def _is_channel_mode(value: object) -> TypeGuard[Literal["heal", "harm"]]:
|
||||
return value in _CHANNEL_MODES
|
||||
|
||||
|
||||
def _is_aid_mode(value: object) -> TypeGuard[Literal["attack", "ac"]]:
|
||||
return value in _AID_MODES
|
||||
|
||||
|
||||
def _is_trick_condition(value: object) -> TypeGuard[TrickCondition]:
|
||||
return value in _TRICK_CONDITIONS
|
||||
|
||||
|
||||
def _is_ready_trigger(value: object) -> TypeGuard[ReadyTrigger]:
|
||||
return value in _READY_TRIGGERS
|
||||
|
||||
|
||||
def _is_object_list(value: object) -> TypeGuard[list[object]]:
|
||||
return isinstance(value, list)
|
||||
|
||||
|
||||
def _is_object_seq(value: object) -> TypeGuard[list[object] | tuple[object, ...]]:
|
||||
return isinstance(value, (list, tuple))
|
||||
|
||||
|
||||
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
return isinstance(value, dict)
|
||||
|
||||
|
||||
# ---- Coercion helpers --------------------------------------------------------
|
||||
|
||||
def _opt_str(value: object, name: str) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
msg = f"{name!r} must be a string"
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
def _str_tuple(value: object, name: str) -> tuple[str, ...]:
|
||||
if value is None:
|
||||
return ()
|
||||
if not _is_object_seq(value):
|
||||
msg = f"{name!r} must be a list of strings"
|
||||
raise NLPolicyError(msg)
|
||||
out: list[str] = []
|
||||
for item in value:
|
||||
if not isinstance(item, str):
|
||||
msg = f"{name!r} must be a list of strings"
|
||||
raise NLPolicyError(msg)
|
||||
out.append(item)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _int_pair(value: object, name: str) -> tuple[int, int] | None:
|
||||
if value is None:
|
||||
return None
|
||||
if not _is_object_seq(value):
|
||||
msg = f"{name!r} must be a [row, col] pair"
|
||||
raise NLPolicyError(msg)
|
||||
if len(value) != _PAIR_LEN:
|
||||
msg = f"{name!r} must be a [row, col] pair"
|
||||
raise NLPolicyError(msg)
|
||||
first, second = value[0], value[1]
|
||||
if (
|
||||
isinstance(first, bool)
|
||||
or isinstance(second, bool)
|
||||
or not isinstance(first, int)
|
||||
or not isinstance(second, int)
|
||||
):
|
||||
msg = f"{name!r} must be a [row, col] pair"
|
||||
raise NLPolicyError(msg)
|
||||
return (first, second)
|
||||
|
||||
|
||||
def _opt_bool(value: object, name: str) -> bool:
|
||||
if value is None:
|
||||
return False
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
msg = f"{name!r} must be a boolean"
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
def _opt_channel_mode(value: object) -> Literal["heal", "harm"] | None:
|
||||
if value is None:
|
||||
return None
|
||||
if _is_channel_mode(value):
|
||||
return value
|
||||
msg = "'channel_mode' must be one of: heal, harm"
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
def _opt_aid_mode(value: object) -> Literal["attack", "ac"] | None:
|
||||
if value is None:
|
||||
return None
|
||||
if _is_aid_mode(value):
|
||||
return value
|
||||
msg = "'aid_mode' must be one of: attack, ac"
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
def _opt_trick_condition(value: object) -> TrickCondition | None:
|
||||
if value is None:
|
||||
return None
|
||||
if _is_trick_condition(value):
|
||||
return value
|
||||
msg = "'trick_condition' must be one of: " + ", ".join(get_args(TrickCondition))
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
def _opt_ready_trigger(value: object) -> ReadyTrigger | None:
|
||||
if value is None:
|
||||
return None
|
||||
if _is_ready_trigger(value):
|
||||
return value
|
||||
msg = "'ready_trigger' must be one of: " + ", ".join(get_args(ReadyTrigger))
|
||||
raise NLPolicyError(msg)
|
||||
|
||||
|
||||
# ---- Action construction -----------------------------------------------------
|
||||
|
||||
def _action_from_item(item: object) -> Action:
|
||||
if not _is_object_dict(item):
|
||||
msg = "each action must be a JSON object"
|
||||
raise NLPolicyError(msg)
|
||||
fields_dict = item
|
||||
unknown = sorted(set(fields_dict) - _FIELD_NAMES)
|
||||
if unknown:
|
||||
msg = f"unknown action field(s): {', '.join(unknown)}"
|
||||
raise NLPolicyError(msg)
|
||||
kind = fields_dict.get("kind")
|
||||
if not _is_kind(kind):
|
||||
msg = "missing or unknown 'kind'"
|
||||
raise NLPolicyError(msg)
|
||||
return Action(
|
||||
kind=kind,
|
||||
target_id=_opt_str(fields_dict.get("target_id"), "target_id"),
|
||||
spell_name=_opt_str(fields_dict.get("spell_name"), "spell_name"),
|
||||
maneuver=_opt_str(fields_dict.get("maneuver"), "maneuver"),
|
||||
hex_name=_opt_str(fields_dict.get("hex_name"), "hex_name"),
|
||||
deed_name=_opt_str(fields_dict.get("deed_name"), "deed_name"),
|
||||
class_ability=_opt_str(fields_dict.get("class_ability"), "class_ability"),
|
||||
channel_mode=_opt_channel_mode(fields_dict.get("channel_mode")),
|
||||
channel_exclude=_str_tuple(fields_dict.get("channel_exclude"), "channel_exclude"),
|
||||
direction=_int_pair(fields_dict.get("direction"), "direction"),
|
||||
trick_condition=_opt_trick_condition(fields_dict.get("trick_condition")),
|
||||
weapon_id=_opt_str(fields_dict.get("weapon_id"), "weapon_id"),
|
||||
ready_trigger=_opt_ready_trigger(fields_dict.get("ready_trigger")),
|
||||
destination=_int_pair(fields_dict.get("destination"), "destination"),
|
||||
target_pos=_int_pair(fields_dict.get("target_pos"), "target_pos"),
|
||||
aid_mode=_opt_aid_mode(fields_dict.get("aid_mode")),
|
||||
metamagic=_str_tuple(fields_dict.get("metamagic"), "metamagic"),
|
||||
cast_defensively=_opt_bool(fields_dict.get("cast_defensively"), "cast_defensively"),
|
||||
tumble=_opt_bool(fields_dict.get("tumble"), "tumble"),
|
||||
)
|
||||
|
||||
|
||||
# ---- Public parse API --------------------------------------------------------
|
||||
|
||||
def parse_actions(reply: str) -> tuple[Action, ...]:
|
||||
"""Parse an LLM reply into a tuple of Action objects."""
|
||||
try:
|
||||
parsed = json.loads(reply)
|
||||
except json.JSONDecodeError as exc:
|
||||
msg = f"reply is not valid JSON: {exc}"
|
||||
raise NLPolicyError(msg) from exc
|
||||
if not _is_object_list(parsed):
|
||||
msg = "reply must be a JSON array of action objects"
|
||||
raise NLPolicyError(msg)
|
||||
items = parsed
|
||||
return tuple(_action_from_item(item) for item in items)
|
||||
|
||||
|
||||
# ---- Fallback policy ---------------------------------------------------------
|
||||
|
||||
Fallback = Callable[[tuple[Action, ...]], tuple[Action, ...]]
|
||||
|
||||
|
||||
def fallback_default(menu: tuple[Action, ...]) -> tuple[Action, ...]:
|
||||
"""Deterministic fallback: first decisive action, then a defensive hold."""
|
||||
for kind in ("full_attack", "attack", "charge", "move", "5foot_step"):
|
||||
for action in menu:
|
||||
if action.kind == kind:
|
||||
return (action,)
|
||||
if menu:
|
||||
return (menu[0],)
|
||||
return (Action(kind="wait"),)
|
||||
|
||||
|
||||
# ---- Repair protocol ---------------------------------------------------------
|
||||
|
||||
_REPAIR_INSTRUCTION = "Return a valid JSON action sequence."
|
||||
_REPAIR_HEADER_TEMPLATE = "-- previous reply invalid (attempt {}) --\n{}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class NLPolicyStats:
|
||||
"""Mutable counters for the LLM decision loop (repair observability).
|
||||
|
||||
The phase-3 report and the sweep pipeline read these counters to
|
||||
quantify how often the LLM layer repairs invalid replies or falls back
|
||||
to the scripted policy.
|
||||
"""
|
||||
|
||||
calls: int = 0
|
||||
repairs_attempted: int = 0
|
||||
repairs_succeeded: int = 0
|
||||
fallbacks: int = 0
|
||||
|
||||
|
||||
# ---- NLPolicy ----------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NLPolicy:
|
||||
"""Turn LLM completions into Action sequences, with a scripted fallback."""
|
||||
|
||||
client: LLMClient
|
||||
builder: PromptBuilder
|
||||
prompt_name: str = "nl_default"
|
||||
fallback: Fallback = fallback_default
|
||||
max_repairs: int = 1
|
||||
stats: NLPolicyStats = field(default_factory=NLPolicyStats)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.max_repairs < 0:
|
||||
msg = f"max_repairs must be >= 0, got {self.max_repairs}"
|
||||
raise ValueError(msg)
|
||||
expected = self.builder.prompt_version(self.prompt_name)
|
||||
if self.client.prompt_version != expected:
|
||||
msg = (
|
||||
f"client prompt_version {self.client.prompt_version!r} does not "
|
||||
f"match pack {expected!r} for {self.prompt_name!r}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
def __call__(self, observation: Observation) -> tuple[Action, ...]:
|
||||
menu = candidates(observation)
|
||||
prompt = self.builder.build(observation, prompt_name=self.prompt_name)
|
||||
reply = self.client.complete(prompt)
|
||||
self.stats.calls += 1
|
||||
errors: list[NLPolicyError] = []
|
||||
while True:
|
||||
try:
|
||||
actions = parse_actions(reply)
|
||||
break
|
||||
except NLPolicyError as exc:
|
||||
if len(errors) >= self.max_repairs:
|
||||
self.stats.fallbacks += 1
|
||||
return self.fallback(menu)
|
||||
errors.append(exc)
|
||||
self.stats.repairs_attempted += 1
|
||||
trail = "\n\n".join(
|
||||
_REPAIR_HEADER_TEMPLATE.format(index, error)
|
||||
for index, error in enumerate(errors, start=1)
|
||||
)
|
||||
retry_prompt = f"{prompt}\n\n{trail}\n{_REPAIR_INSTRUCTION}"
|
||||
reply = self.client.complete(retry_prompt)
|
||||
self.stats.calls += 1
|
||||
if errors:
|
||||
self.stats.repairs_succeeded += 1
|
||||
return actions
|
||||
@@ -0,0 +1,135 @@
|
||||
"""Objective primitives for the Phase-2 optimizer and danger cartography.
|
||||
|
||||
Pure functions over the per-run summaries produced by
|
||||
``run_encounter(..., collect_runs=True)`` (docs/phase-2-llm-architecture.md
|
||||
sections 3.1 and 3.2). These are the metrics an optimizer maximizes (party
|
||||
survival) and a danger map reports per scenario - no engine access, no
|
||||
state, no RNG: given the same ``RunOutcome`` tuple they always return the
|
||||
same numbers, which is what makes them safe as a search objective.
|
||||
|
||||
Definitions (all probabilities are empirical frequencies over the runs;
|
||||
``party`` is the side name being measured, e.g. ``"players"``):
|
||||
|
||||
- ``p_survives`` : P(party survives all) - no party member dead.
|
||||
- ``p_tpk`` : P(total party kill) - every party member dead.
|
||||
- ``p_any_death`` : P(at least one party member death).
|
||||
- ``expected_hp_lost``: mean total party hit points lost (hp_max - hp,
|
||||
clamped to [0, hp_max] per member: healing above max never counts
|
||||
negative, and overkill on a corpse is not counted as extra loss).
|
||||
- ``expected_rounds`` : mean battle length.
|
||||
- ``worst_decile_party_hp`` : TAIL metric - the mean party hp *remaining*
|
||||
over the worst decile of runs (the 10% with the largest hp loss, at
|
||||
least 1 run): a CVaR-style measure so rare disasters are not averaged
|
||||
away. A dead or dying member contributes 0 hp remaining.
|
||||
|
||||
A side with no members in a run (should not happen in a normal encounter)
|
||||
is treated vacuously: survives all, no TPK, 0 hp lost.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pf1e_simulator.runner import CombatantOutcome, RunOutcome
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DangerMetrics:
|
||||
"""The full danger profile of one party over a batch of run outcomes."""
|
||||
|
||||
runs: int
|
||||
p_survives: float
|
||||
p_tpk: float
|
||||
p_any_death: float
|
||||
expected_hp_lost: float
|
||||
expected_rounds: float
|
||||
worst_decile_party_hp: float
|
||||
|
||||
|
||||
def _party_members(outcome: RunOutcome, party: str) -> tuple[CombatantOutcome, ...]:
|
||||
return tuple(c for c in outcome.combatants if c.side == party)
|
||||
|
||||
|
||||
def _hp_lost(outcome: RunOutcome, party: str) -> int:
|
||||
return sum(
|
||||
min(c.hp_max, max(0, c.hp_max - c.hp))
|
||||
for c in _party_members(outcome, party)
|
||||
)
|
||||
|
||||
|
||||
def _hp_remaining(outcome: RunOutcome, party: str) -> int:
|
||||
return sum(max(0, c.hp) for c in _party_members(outcome, party))
|
||||
|
||||
|
||||
def p_survives(outcomes: Sequence[RunOutcome], party: str) -> float:
|
||||
"""P(no member of `party` is dead at the end of a run)."""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
hits = sum(1 for o in outcomes if not any(c.dead for c in _party_members(o, party)))
|
||||
return hits / len(outcomes)
|
||||
|
||||
|
||||
def p_tpk(outcomes: Sequence[RunOutcome], party: str) -> float:
|
||||
"""P(every member of `party` is dead at the end of a run)."""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
hits = 0
|
||||
for o in outcomes:
|
||||
members = _party_members(o, party)
|
||||
if members and all(c.dead for c in members):
|
||||
hits += 1
|
||||
return hits / len(outcomes)
|
||||
|
||||
|
||||
def p_any_death(outcomes: Sequence[RunOutcome], party: str) -> float:
|
||||
"""P(at least one member of `party` is dead at the end of a run)."""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
hits = sum(1 for o in outcomes if any(c.dead for c in _party_members(o, party)))
|
||||
return hits / len(outcomes)
|
||||
|
||||
|
||||
def expected_hp_lost(outcomes: Sequence[RunOutcome], party: str) -> float:
|
||||
"""Mean total `party` hit points lost per run (clamped to [0, hp_max] per member)."""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
return sum(_hp_lost(o, party) for o in outcomes) / len(outcomes)
|
||||
|
||||
|
||||
def expected_rounds(outcomes: Sequence[RunOutcome]) -> float:
|
||||
"""Mean number of rounds per run (draws count their full round cap)."""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
return sum(o.rounds for o in outcomes) / len(outcomes)
|
||||
|
||||
|
||||
def worst_decile_party_hp(outcomes: Sequence[RunOutcome], party: str) -> float:
|
||||
"""Mean `party` hp remaining over the worst-decile runs (tail/CVaR metric).
|
||||
|
||||
The worst decile is the ``max(1, ceil(10%))`` runs with the largest hp
|
||||
loss; ties keep list order. Complements ``expected_hp_lost``: the mean
|
||||
hides tail disasters, this number does not.
|
||||
"""
|
||||
if not outcomes:
|
||||
return 0.0
|
||||
tail = max(1, math.ceil(len(outcomes) / 10))
|
||||
worst = sorted(outcomes, key=lambda o: _hp_lost(o, party), reverse=True)[:tail]
|
||||
return sum(_hp_remaining(o, party) for o in worst) / tail
|
||||
|
||||
|
||||
def danger_metrics(outcomes: Sequence[RunOutcome], party: str) -> DangerMetrics:
|
||||
"""Compute the whole danger profile in one call (the optimizer's objective)."""
|
||||
return DangerMetrics(
|
||||
runs=len(outcomes),
|
||||
p_survives=p_survives(outcomes, party),
|
||||
p_tpk=p_tpk(outcomes, party),
|
||||
p_any_death=p_any_death(outcomes, party),
|
||||
expected_hp_lost=expected_hp_lost(outcomes, party),
|
||||
expected_rounds=expected_rounds(outcomes),
|
||||
worst_decile_party_hp=worst_decile_party_hp(outcomes, party),
|
||||
)
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Phase-2 spike: a strengthened monster policy for danger cartography.
|
||||
|
||||
Spike-level, not the final ``adversary.py`` (docs/phase-2-llm-architecture.md
|
||||
sections 3.2 and 10, P4.1): the danger a party faces only exists against an
|
||||
adversary that plays to win, and ``default_policy`` is a weak baseline - each
|
||||
monster attacks its own nearest enemy, so the party never concentrates
|
||||
punishment on one member. This module provides the minimal upgrade that
|
||||
proves the point: FOCUS FIRE. Every monster on the designated side
|
||||
concentrates on the same party member - the active enemy with the lowest AC
|
||||
(ties: lowest current hp, then initiative list order) - instead of its
|
||||
nearest one.
|
||||
|
||||
The party side (any side other than the designated monster side) delegates
|
||||
to ``default_policy`` unchanged, so one policy object drives the whole
|
||||
battle and the danger-map comparison isolates exactly the monster-side
|
||||
change. Spellcasting, class abilities and maneuvers stay on the default
|
||||
flow: the spike measures what target selection alone is worth.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pf1e_simulator.combat import (
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Policy,
|
||||
default_policy,
|
||||
)
|
||||
|
||||
|
||||
def focus_target(engine: CombatEngine, state: CombatantState) -> CombatantState | None:
|
||||
"""The enemy the whole monster side concentrates on: lowest AC, then hp, then order.
|
||||
|
||||
AC is the combatant's current total AC as exposed on the frozen profile
|
||||
(the spike does not re-derive buff/condition modifiers); hp is current
|
||||
hit points (a dying target ranks below a healthy one). Deterministic:
|
||||
ties break on the engine's initiative list order.
|
||||
"""
|
||||
enemies = [s for s in engine.states if s.side != state.side and s.active]
|
||||
if not enemies:
|
||||
return None
|
||||
return min(enemies, key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)))
|
||||
|
||||
|
||||
def _strike(
|
||||
engine: CombatEngine, state: CombatantState, target: CombatantState
|
||||
) -> tuple[Action, ...]:
|
||||
"""Attack the focus target with the default melee/ranged action ladder."""
|
||||
if engine.weapon_for(state, target) is not None:
|
||||
return (Action(kind="full_attack", target_id=target.combatant.id),)
|
||||
charge = engine.can_charge(state, target)
|
||||
if charge is not None and charge.weapon is not None:
|
||||
return (Action(kind="charge", target_id=target.combatant.id),)
|
||||
if engine.can_step5_to_attack(state, target):
|
||||
return (
|
||||
Action(kind="5foot_step", target_id=target.combatant.id),
|
||||
Action(kind="full_attack", target_id=target.combatant.id),
|
||||
)
|
||||
return (
|
||||
Action(kind="move", target_id=target.combatant.id),
|
||||
Action(kind="attack", target_id=target.combatant.id),
|
||||
)
|
||||
|
||||
|
||||
def focus_fire_policy(monster_side: str = "monsters") -> Policy:
|
||||
"""A ``Policy`` where the monster side focus-fires; every other side plays default.
|
||||
|
||||
Pass to ``run_encounter(policy=focus_fire_policy("monsters"))``. The
|
||||
party side delegates to ``default_policy``, so the only behavioural
|
||||
change is monster target concentration.
|
||||
"""
|
||||
|
||||
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
if state.side != monster_side:
|
||||
return default_policy(engine, state)
|
||||
if engine.has_condition_flag(state, "must_flee"):
|
||||
target = engine.nearest_enemy(state)
|
||||
if target is None:
|
||||
return (Action(kind="wait"),)
|
||||
return (Action(kind="withdraw", target_id=target.combatant.id),)
|
||||
target = focus_target(engine, state)
|
||||
if target is None:
|
||||
return (Action(kind="wait"),)
|
||||
return _strike(engine, state, target)
|
||||
|
||||
return policy
|
||||
@@ -0,0 +1,234 @@
|
||||
"""Externally-loaded, versioned prompt pack for the phase-2 LLM layer (P1).
|
||||
|
||||
Prompts do NOT live in this repository (architecture doc section 14 -- the
|
||||
repo is code only). They live in a separate, versioned pack directory,
|
||||
resolved through configuration (env var ``PF1E_PROMPT_PACK_DIR``, never
|
||||
hard-coded -- section 15). This module loads a pack once per resolved path,
|
||||
exposes each prompt by name and computes the ``prompt_version`` that must
|
||||
enter the response-cache composite key (decision D6), the ``ReplayRecord``
|
||||
and the sweep reports, so a campaign stays reproducible after a prompt edit
|
||||
(traceability rule, section 14).
|
||||
|
||||
Pack layout
|
||||
-----------
|
||||
<pack_dir>/
|
||||
version.txt # single line: semver or commit hash (mandatory)
|
||||
<name>.md # one file per prompt name, e.g. ``nl_default.md``
|
||||
<name>.examples.json # optional few-shot examples for that prompt: a
|
||||
# JSON array of ``{"observation": {...},
|
||||
# "actions": [{...}, ...]}`` teaching pairs
|
||||
|
||||
Loading is eager and strict. A missing pack, a missing ``version.txt`` or a
|
||||
missing ``<name>.md`` raises a typed ``PromptPackError`` -- there is no
|
||||
silent fallback (validation rule, section 14). An optional
|
||||
``<name>.examples.json`` file carries few-shot examples (P5.2): it comes
|
||||
from the external pack, never from this repo (section 14), and is validated
|
||||
at load time -- an unreadable, malformed or orphaned examples file is a
|
||||
pack error, not a silent skip. ``prompt_version`` covers the examples
|
||||
content alongside the prompt text, so the response-cache key (D6) stays in
|
||||
sync with any pack edit. Unit tests use the minimal local fixtures under
|
||||
``tests/fixtures/prompt_pack/`` so CI never depends on the external prompt
|
||||
repository (pinning rule, section 14).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from functools import cache
|
||||
from typing import TYPE_CHECKING, NoReturn, TypeGuard
|
||||
|
||||
from pf1e_simulator.config import ENV_PROMPT_PACK_DIR, env_path
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
_PROMPT_SUFFIX = ".md"
|
||||
_EXAMPLES_SUFFIX = ".examples.json"
|
||||
_VERSION_FILE = "version.txt"
|
||||
_HASH_PREFIX_LEN = 12
|
||||
_HASH_SEPARATOR = "\0"
|
||||
_EXAMPLE_KEYS = frozenset({"observation", "actions"})
|
||||
|
||||
|
||||
class PromptPackError(RuntimeError):
|
||||
"""A prompt pack is missing, invalid, or lacks a required prompt."""
|
||||
|
||||
|
||||
def _pack_error(kind: str, message: str) -> NoReturn:
|
||||
msg = f"{kind}: {message}"
|
||||
raise PromptPackError(msg)
|
||||
|
||||
|
||||
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
return isinstance(value, dict)
|
||||
|
||||
|
||||
def _is_object_list(value: object) -> TypeGuard[list[object]]:
|
||||
return isinstance(value, list)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FewShotExample:
|
||||
"""One structured few-shot pair: an example observation and a valid reply.
|
||||
|
||||
``observation`` is a plain JSON object (the pack author composes it, the
|
||||
builder renders it verbatim); ``actions`` is a JSON array of Action-shaped
|
||||
objects teaching the exact reply format the parser accepts.
|
||||
"""
|
||||
|
||||
observation: dict[str, object]
|
||||
actions: tuple[dict[str, object], ...]
|
||||
|
||||
|
||||
def _parse_examples(path: Path, text: str) -> tuple[FewShotExample, ...]:
|
||||
"""Parse and strictly validate one few-shot examples file."""
|
||||
try:
|
||||
raw = json.loads(text)
|
||||
except json.JSONDecodeError as exc:
|
||||
msg = f"{path.name} is not valid JSON: {exc}"
|
||||
_pack_error("invalid pack", msg)
|
||||
if not _is_object_list(raw):
|
||||
_pack_error("invalid pack", f"{path.name} must be a JSON array of examples")
|
||||
examples: list[FewShotExample] = []
|
||||
for index, item in enumerate(raw):
|
||||
if not _is_object_dict(item):
|
||||
msg = f"{path.name}[{index}] must be an object with 'observation' and 'actions'"
|
||||
_pack_error("invalid pack", msg)
|
||||
unknown = sorted(set(item) - _EXAMPLE_KEYS)
|
||||
if unknown:
|
||||
msg = f"{path.name}[{index}] has unknown key(s): {', '.join(unknown)}"
|
||||
_pack_error("invalid pack", msg)
|
||||
observation = item.get("observation")
|
||||
actions = item.get("actions")
|
||||
if not _is_object_dict(observation):
|
||||
msg = f"{path.name}[{index}] 'observation' must be a JSON object"
|
||||
_pack_error("invalid pack", msg)
|
||||
if not _is_object_list(actions):
|
||||
msg = f"{path.name}[{index}] 'actions' must be a JSON array of objects"
|
||||
_pack_error("invalid pack", msg)
|
||||
action_dicts: list[dict[str, object]] = []
|
||||
for action in actions:
|
||||
if not _is_object_dict(action):
|
||||
msg = f"{path.name}[{index}] 'actions' must contain only JSON objects"
|
||||
_pack_error("invalid pack", msg)
|
||||
action_dicts.append(action)
|
||||
examples.append(FewShotExample(observation=observation, actions=tuple(action_dicts)))
|
||||
return tuple(examples)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptPack:
|
||||
"""A versioned set of prompts loaded from one external directory.
|
||||
|
||||
Instances are immutable (frozen), so a loaded pack can be shared by
|
||||
parallel sweep workers without drift. ``_prompts`` maps prompt names
|
||||
(file stems) to their full text, read at load time; ``_examples`` maps
|
||||
prompt names to their parsed few-shot pairs (empty when the pack ships
|
||||
none) and ``_examples_text`` keeps the raw file bytes for hashing.
|
||||
"""
|
||||
|
||||
root: Path
|
||||
version: str
|
||||
_prompts: dict[str, str] = field(repr=False)
|
||||
_examples: dict[str, tuple[FewShotExample, ...]] = field(repr=False)
|
||||
_examples_text: dict[str, str] = field(repr=False)
|
||||
|
||||
def get(self, name: str) -> str:
|
||||
"""Return the prompt ``name``'s text, or raise ``PromptPackError``."""
|
||||
try:
|
||||
return self._prompts[name]
|
||||
except KeyError as exc:
|
||||
msg = (
|
||||
f"prompt '{name}' missing from pack at {self.root} "
|
||||
f"(expected file '{name}{_PROMPT_SUFFIX}')"
|
||||
)
|
||||
raise PromptPackError(msg) from exc
|
||||
|
||||
def examples(self, name: str) -> tuple[FewShotExample, ...]:
|
||||
"""Return the few-shot pairs for ``name`` (empty when none)."""
|
||||
return self._examples.get(name, ())
|
||||
|
||||
def prompt_version(self, name: str) -> str:
|
||||
"""Pack version plus per-prompt content hash (cache-key component, D6).
|
||||
|
||||
The digest covers the prompt text and, when the pack ships a
|
||||
``<name>.examples.json`` file, that file's bytes too -- any pack edit
|
||||
invalidates the cache key. Without an examples file the digest is
|
||||
byte-identical to the pre-few-shot scheme.
|
||||
"""
|
||||
content = self.get(name)
|
||||
examples_text = self._examples_text.get(name)
|
||||
digest_input = content
|
||||
if examples_text is not None:
|
||||
digest_input = f"{content}{_HASH_SEPARATOR}{examples_text}"
|
||||
digest = hashlib.sha256(digest_input.encode("utf-8")).hexdigest()[:_HASH_PREFIX_LEN]
|
||||
return f"{self.version}:{name}:{digest}"
|
||||
|
||||
@classmethod
|
||||
def load(cls, root: Path) -> PromptPack:
|
||||
"""Load and validate the pack rooted at ``root`` (strict, no fallback)."""
|
||||
if not root.is_dir():
|
||||
_pack_error(
|
||||
"pack directory not found",
|
||||
f"{root} does not exist or is not a directory",
|
||||
)
|
||||
version_file = root / _VERSION_FILE
|
||||
if not version_file.is_file():
|
||||
_pack_error("invalid pack", f"{root} has no {_VERSION_FILE} file")
|
||||
version = version_file.read_text(encoding="utf-8").strip()
|
||||
if not version:
|
||||
_pack_error("invalid pack", f"{root / _VERSION_FILE} is empty")
|
||||
prompts: dict[str, str] = {}
|
||||
for path in sorted(root.glob(f"*{_PROMPT_SUFFIX}")):
|
||||
prompts[path.stem] = path.read_text(encoding="utf-8")
|
||||
examples: dict[str, tuple[FewShotExample, ...]] = {}
|
||||
examples_text: dict[str, str] = {}
|
||||
for path in sorted(root.glob(f"*{_EXAMPLES_SUFFIX}")):
|
||||
stem = path.name.removesuffix(_EXAMPLES_SUFFIX)
|
||||
if stem not in prompts:
|
||||
msg = f"{path.name} has no matching '{stem}{_PROMPT_SUFFIX}' prompt"
|
||||
_pack_error("invalid pack", msg)
|
||||
text = path.read_text(encoding="utf-8")
|
||||
examples[stem] = _parse_examples(path, text)
|
||||
examples_text[stem] = text
|
||||
return cls(
|
||||
root=root,
|
||||
version=version,
|
||||
_prompts=prompts,
|
||||
_examples=examples,
|
||||
_examples_text=examples_text,
|
||||
)
|
||||
|
||||
|
||||
@cache
|
||||
def _load_cached(root: Path) -> PromptPack:
|
||||
"""Load once per resolved path (thread-safe; parallel sweeps share it)."""
|
||||
return PromptPack.load(root)
|
||||
|
||||
|
||||
def load_prompt_pack(pack_dir: Path | None = None) -> PromptPack:
|
||||
"""Load the prompt pack from ``pack_dir`` or the configured directory.
|
||||
|
||||
With ``pack_dir`` given, load that directory directly (tests use this
|
||||
with their local fixture). Otherwise resolve
|
||||
``PF1E_PROMPT_PACK_DIR``; when unset, raise a clear error instead of
|
||||
falling back to a bundled (non-existent) pack.
|
||||
"""
|
||||
if pack_dir is not None:
|
||||
return _load_cached(pack_dir)
|
||||
configured = env_path(ENV_PROMPT_PACK_DIR)
|
||||
if configured is None:
|
||||
_pack_error(
|
||||
"prompt pack not configured",
|
||||
f"set {ENV_PROMPT_PACK_DIR} to the external prompt pack directory "
|
||||
"(architecture doc section 14)",
|
||||
)
|
||||
return _load_cached(configured)
|
||||
|
||||
|
||||
def clear_pack_cache() -> None:
|
||||
"""Drop the per-path pack cache (used by tests that relocate fixtures)."""
|
||||
_load_cached.cache_clear()
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Prompt builder for the phase-2 LLM layer (P1.4).
|
||||
|
||||
Builds a deterministic prompt from an ``Observation`` plus the external
|
||||
prompt pack: template + optional few-shot examples + compact observation +
|
||||
plausible actions + JSON action schema. The examples (P5.2) come from the
|
||||
pack's ``<name>.examples.json`` file when present and are mounted between
|
||||
the template and the live observation, so the query stays the last section
|
||||
before the reply (few-shot recency) and packs without examples keep the
|
||||
original section order bit-for-bit. The schema is derived from the
|
||||
``Action`` dataclass via ``typing.get_type_hints`` so it stays in sync with
|
||||
the engine.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import MISSING, asdict, dataclass, fields
|
||||
from types import UnionType
|
||||
from typing import TYPE_CHECKING, Literal, Union, get_args, get_origin, get_type_hints
|
||||
|
||||
from pf1e_simulator.candidates import candidates
|
||||
from pf1e_simulator.combat import UNSUPPORTED_ACTION_KINDS, Action, Observation
|
||||
from pf1e_simulator.llm_view import compact_observation
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.prompt import FewShotExample, PromptPack
|
||||
|
||||
|
||||
def _supported_kinds() -> tuple[str, ...]:
|
||||
"""Return the 27 action kinds supported by the LLM layer (declaration order)."""
|
||||
kind_literal = get_type_hints(Action)["kind"]
|
||||
all_kinds = get_args(kind_literal)
|
||||
return tuple(k for k in all_kinds if k not in UNSUPPORTED_ACTION_KINDS)
|
||||
|
||||
|
||||
_EMPTY_LITERAL_MSG = "empty Literal"
|
||||
_UNSUPPORTED_LITERAL_MSG = "unsupported Literal values: {}"
|
||||
_MULTI_UNION_MSG = "multi-type union not supported in Action: {}"
|
||||
_UNSUPPORTED_TYPE_MSG = "unsupported type in Action schema: {}"
|
||||
_TUPLE_ELLIPSIS_LEN = 2
|
||||
|
||||
|
||||
def _json_schema_base(tp: object) -> dict[str, object] | None:
|
||||
"""Return the JSON Schema fragment for a base type, else None."""
|
||||
if tp is type(None):
|
||||
return {"type": "null"}
|
||||
if tp is str:
|
||||
return {"type": "string"}
|
||||
if tp is int:
|
||||
return {"type": "integer"}
|
||||
if tp is bool:
|
||||
return {"type": "boolean"}
|
||||
return None
|
||||
|
||||
|
||||
def _json_schema_literal(args: tuple[object, ...]) -> dict[str, object]:
|
||||
"""Return the JSON Schema fragment for a Literal type."""
|
||||
if not args:
|
||||
raise ValueError(_EMPTY_LITERAL_MSG)
|
||||
first = args[0]
|
||||
if isinstance(first, str):
|
||||
base_type = "string"
|
||||
elif isinstance(first, int):
|
||||
base_type = "integer"
|
||||
else:
|
||||
msg = _UNSUPPORTED_LITERAL_MSG.format(args)
|
||||
raise TypeError(msg)
|
||||
return {"type": base_type, "enum": list(args)}
|
||||
|
||||
|
||||
def _json_schema_tuple(args: tuple[object, ...]) -> dict[str, object]:
|
||||
"""Return the JSON Schema fragment for a tuple type."""
|
||||
if not args:
|
||||
return {"type": "array"}
|
||||
if len(args) == _TUPLE_ELLIPSIS_LEN and args[1] is Ellipsis:
|
||||
# tuple[T, ...] → homogeneous array
|
||||
item_schema = _json_schema_for(args[0])
|
||||
return {"type": "array", "items": item_schema}
|
||||
# Fixed-length tuple → prefixItems + minItems/maxItems
|
||||
prefix = [_json_schema_for(arg) for arg in args]
|
||||
return {
|
||||
"type": "array",
|
||||
"prefixItems": prefix,
|
||||
"minItems": len(args),
|
||||
"maxItems": len(args),
|
||||
}
|
||||
|
||||
|
||||
def _json_schema_union(args: tuple[object, ...]) -> dict[str, object]:
|
||||
"""Return the JSON Schema fragment for a nullable single-member union."""
|
||||
non_none = [a for a in args if a is not type(None)]
|
||||
if len(non_none) == 1:
|
||||
member_schema = _json_schema_for(non_none[0])
|
||||
# Merge "null" into the type array
|
||||
member_type = member_schema.get("type")
|
||||
if isinstance(member_type, str):
|
||||
member_schema["type"] = [member_type, "null"]
|
||||
# If member already has an enum (e.g. nullable Literal), keep it.
|
||||
return member_schema
|
||||
msg = _MULTI_UNION_MSG.format(args)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _json_schema_for(tp: object) -> dict[str, object]:
|
||||
"""Convert a Python type annotation to a JSON Schema fragment."""
|
||||
base_schema = _json_schema_base(tp)
|
||||
if base_schema is not None:
|
||||
return base_schema
|
||||
|
||||
origin = get_origin(tp)
|
||||
args = get_args(tp)
|
||||
|
||||
if origin is Literal:
|
||||
return _json_schema_literal(args)
|
||||
|
||||
if origin is tuple:
|
||||
return _json_schema_tuple(args)
|
||||
|
||||
if origin is Union or origin is UnionType:
|
||||
return _json_schema_union(args)
|
||||
|
||||
msg = _UNSUPPORTED_TYPE_MSG.format(tp)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def action_schema() -> dict[str, object]:
|
||||
"""JSON Schema for a single Action object."""
|
||||
hints = get_type_hints(Action)
|
||||
properties = {name: _json_schema_for(tp) for name, tp in hints.items()}
|
||||
# Override kind with the filtered 27-kind enum (source Literal order)
|
||||
properties["kind"] = {"type": "string", "enum": list(_supported_kinds())}
|
||||
# Only 'kind' lacks a default
|
||||
required = [
|
||||
f.name
|
||||
for f in fields(Action)
|
||||
if f.default is MISSING and f.default_factory is MISSING
|
||||
]
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
"required": required,
|
||||
"additionalProperties": False,
|
||||
}
|
||||
|
||||
|
||||
def render_candidates(actions: tuple[Action, ...]) -> tuple[str, ...]:
|
||||
"""Render the plausible-action menu as numbered lines with compact JSON."""
|
||||
lines: list[str] = []
|
||||
for index, action in enumerate(actions):
|
||||
payload = {
|
||||
key: value
|
||||
for key, value in asdict(action).items()
|
||||
if key != "kind" and value is not None and value != () and value is not False
|
||||
}
|
||||
suffix = f" {json.dumps(payload, sort_keys=True, separators=(',', ':'))}" if payload else ""
|
||||
lines.append(f"{index}. {action.kind}{suffix}")
|
||||
return tuple(lines)
|
||||
|
||||
|
||||
def _render_examples(examples: tuple[FewShotExample, ...]) -> str:
|
||||
"""Render few-shot pairs as observation-to-response teaching blocks."""
|
||||
blocks: list[str] = []
|
||||
for index, example in enumerate(examples):
|
||||
observation_json = json.dumps(example.observation, sort_keys=True)
|
||||
actions_json = json.dumps(example.actions, sort_keys=True)
|
||||
blocks.append(
|
||||
f"### example {index + 1}\n"
|
||||
f"observation:\n{observation_json}\n\n"
|
||||
f"response:\n{actions_json}"
|
||||
)
|
||||
return "## few-shot examples\n" + "\n\n".join(blocks)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PromptBuilder:
|
||||
"""Builds the complete LLM prompt from a prompt pack and an observation."""
|
||||
|
||||
pack: PromptPack
|
||||
|
||||
def prompt_version(self, name: str) -> str:
|
||||
"""Pack version plus per-prompt content hash (cache-key component, D6)."""
|
||||
return self.pack.prompt_version(name)
|
||||
|
||||
def build(self, observation: object, *, prompt_name: str = "nl_default") -> str:
|
||||
"""Assemble the full prompt for the given observation."""
|
||||
if not isinstance(observation, Observation):
|
||||
msg = "observation must be an Observation instance"
|
||||
raise TypeError(msg)
|
||||
|
||||
template = self.pack.get(prompt_name)
|
||||
obs_text = compact_observation(observation).rstrip("\n")
|
||||
menu = candidates(observation)
|
||||
actions_text = "\n".join(render_candidates(menu))
|
||||
schema_text = json.dumps(action_schema(), sort_keys=True, indent=2)
|
||||
sections: list[str] = [template]
|
||||
examples = self.pack.examples(prompt_name)
|
||||
if examples:
|
||||
sections.append(_render_examples(examples))
|
||||
sections.extend([
|
||||
"## observation\n" + obs_text,
|
||||
"## plausible actions\n" + actions_text,
|
||||
"## action schema\n" + schema_text,
|
||||
])
|
||||
return "\n\n".join(sections)
|
||||
@@ -0,0 +1,227 @@
|
||||
"""Real OpenAI-compatible chat client for the phase-2 LLM layer (P5.1).
|
||||
|
||||
``OpenAILLMClient`` speaks the OpenAI chat-completions wire format over HTTPS
|
||||
using only the standard library (``urllib.request``), so the repository ships
|
||||
no HTTP dependency. The base URL, model and API key are resolved from the
|
||||
environment (:mod:`pf1e_simulator.config`, section 15); the key is never
|
||||
embedded in code.
|
||||
|
||||
Error taxonomy (shared with :mod:`pf1e_simulator.client`):
|
||||
|
||||
- ``ProviderError`` for transient failures (transport errors, HTTP 429, any
|
||||
5xx) -- retried by ``LLMClient``'s backoff loop and counted by the breaker;
|
||||
- ``ProviderRejectedError`` for permanent 4xx rejections (bad key, unknown
|
||||
model, quota) -- raised immediately, never retried;
|
||||
- ``ProviderResponseError`` for a 2xx body that is not a valid
|
||||
chat-completions payload -- raised immediately, never retried.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import urllib.request
|
||||
from typing import TYPE_CHECKING, Protocol, Self, TypeGuard
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import Request
|
||||
|
||||
from pf1e_simulator.client import (
|
||||
CircuitBreaker,
|
||||
ClientError,
|
||||
ClientParams,
|
||||
Completion,
|
||||
CostBudget,
|
||||
LLMClient,
|
||||
Pricing,
|
||||
ProviderError,
|
||||
ResponseCache,
|
||||
TokenUsage,
|
||||
estimate_tokens,
|
||||
)
|
||||
from pf1e_simulator.combat import OBSERVATION_SCHEMA_VERSION
|
||||
from pf1e_simulator.config import (
|
||||
ENV_LLM_API_KEY,
|
||||
llm_api_key,
|
||||
llm_base_url,
|
||||
llm_model,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
# Status-code buckets of the chat-completions wire (RFC 9110): 2xx is a
|
||||
# success, 429 and every 5xx mean "try again later" (the only statuses safe
|
||||
# to retry), and any other 4xx is a permanent rejection.
|
||||
_OK_MIN = 200
|
||||
_OK_EXCLUSIVE = 300
|
||||
_RATE_LIMITED = 429
|
||||
_SERVER_ERROR_MIN = 500
|
||||
|
||||
|
||||
class ProviderRejectedError(ClientError):
|
||||
"""A permanent 4xx provider rejection - never retried."""
|
||||
|
||||
|
||||
class ProviderResponseError(ClientError):
|
||||
"""A 2xx body that is not a valid chat-completions payload."""
|
||||
|
||||
|
||||
class HttpResponse(Protocol):
|
||||
"""The minimal ``urllib.response`` surface the wire call uses."""
|
||||
|
||||
status: int
|
||||
|
||||
def read(self) -> bytes: ...
|
||||
|
||||
def __enter__(self) -> Self: ...
|
||||
|
||||
def __exit__(self, _exc_type: object, _exc_value: object, _traceback: object) -> None: ...
|
||||
|
||||
|
||||
def _is_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
return isinstance(value, dict)
|
||||
|
||||
|
||||
def _is_object_list(value: object) -> TypeGuard[list[object]]:
|
||||
return isinstance(value, list)
|
||||
|
||||
|
||||
def _parse_completion(payload: dict[str, object]) -> str:
|
||||
"""Extract the completion text from a chat-completions payload."""
|
||||
choices_raw = payload.get("choices")
|
||||
if not _is_object_list(choices_raw) or not choices_raw:
|
||||
msg = "provider response has no choices"
|
||||
raise ProviderResponseError(msg)
|
||||
choice = choices_raw[0]
|
||||
if not _is_object_dict(choice):
|
||||
msg = "provider response choice is not a JSON object"
|
||||
raise ProviderResponseError(msg)
|
||||
message_raw = choice.get("message")
|
||||
if not _is_object_dict(message_raw):
|
||||
msg = "provider response message is not a JSON object"
|
||||
raise ProviderResponseError(msg)
|
||||
content = message_raw.get("content")
|
||||
if not isinstance(content, str):
|
||||
msg = "provider response content is not a string"
|
||||
raise ProviderResponseError(msg)
|
||||
return content
|
||||
|
||||
|
||||
def _parse_usage(payload: dict[str, object], prompt: str, text: str) -> TokenUsage:
|
||||
"""Read the usage block, or estimate tokens when the provider omits it."""
|
||||
usage_raw = payload.get("usage")
|
||||
if not _is_object_dict(usage_raw):
|
||||
return TokenUsage(
|
||||
prompt_tokens=estimate_tokens(prompt),
|
||||
completion_tokens=estimate_tokens(text),
|
||||
)
|
||||
prompt_tokens = usage_raw.get("prompt_tokens")
|
||||
completion_tokens = usage_raw.get("completion_tokens")
|
||||
if not isinstance(prompt_tokens, int) or not isinstance(completion_tokens, int):
|
||||
msg = "provider usage block is malformed"
|
||||
raise ProviderResponseError(msg)
|
||||
return TokenUsage(prompt_tokens=prompt_tokens, completion_tokens=completion_tokens)
|
||||
|
||||
|
||||
class OpenAILLMClient(LLMClient):
|
||||
"""Real chat-completions client (OpenAI or any compatible gateway).
|
||||
|
||||
Endpoint, model and key default to the environment (section 15); all may
|
||||
be overridden explicitly. ``urlopen`` and ``sleep`` are injectable so
|
||||
tests exercise the full retry/breaker machinery offline.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
prompt_version: str,
|
||||
model: str | None = None,
|
||||
api_key: str | None = None,
|
||||
base_url: str | None = None,
|
||||
timeout_s: float = 30.0,
|
||||
urlopen: Callable[..., HttpResponse] = urllib.request.urlopen,
|
||||
schema_version: int = OBSERVATION_SCHEMA_VERSION,
|
||||
cache: ResponseCache | None = None,
|
||||
budget: CostBudget | None = None,
|
||||
pricing: Pricing | None = None,
|
||||
retries: int = 2,
|
||||
backoff_base_s: float = 0.25,
|
||||
breaker: CircuitBreaker | None = None,
|
||||
sleep: Callable[[float], None] | None = None,
|
||||
) -> None:
|
||||
key = api_key if api_key is not None else llm_api_key()
|
||||
if key is None or key == "":
|
||||
msg = (
|
||||
f"no LLM API key: set the {ENV_LLM_API_KEY} environment "
|
||||
"variable or pass an explicit api_key"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if timeout_s <= 0:
|
||||
msg = f"timeout_s must be > 0, got {timeout_s}"
|
||||
raise ValueError(msg)
|
||||
super().__init__(
|
||||
model=model if model is not None else llm_model(),
|
||||
prompt_version=prompt_version,
|
||||
schema_version=schema_version,
|
||||
cache=cache,
|
||||
budget=budget,
|
||||
pricing=pricing,
|
||||
retries=retries,
|
||||
backoff_base_s=backoff_base_s,
|
||||
sleep=sleep if sleep is not None else time.sleep,
|
||||
breaker=breaker,
|
||||
)
|
||||
resolved_base_url = base_url if base_url is not None else llm_base_url()
|
||||
self._api_key = key
|
||||
self._base_url = resolved_base_url.rstrip("/")
|
||||
self._timeout_s = timeout_s
|
||||
self._urlopen = urlopen
|
||||
|
||||
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
|
||||
"""One chat-completions attempt; raise ``ProviderError`` when retryable."""
|
||||
body = {
|
||||
"model": self.model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": params.temperature,
|
||||
"max_tokens": params.max_tokens,
|
||||
"top_p": params.top_p,
|
||||
"response_format": {"type": "json_object"},
|
||||
}
|
||||
request = Request(
|
||||
f"{self._base_url}/chat/completions",
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers={
|
||||
"Authorization": f"Bearer {self._api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with self._urlopen(request, timeout=self._timeout_s) as response:
|
||||
status = response.status
|
||||
data = response.read()
|
||||
except HTTPError as exc:
|
||||
status = exc.code
|
||||
data = exc.read()
|
||||
except OSError as exc:
|
||||
msg = f"provider unreachable: {exc}"
|
||||
raise ProviderError(msg) from exc
|
||||
|
||||
if status == _RATE_LIMITED or status >= _SERVER_ERROR_MIN:
|
||||
msg = f"provider transient failure: HTTP {status}"
|
||||
raise ProviderError(msg)
|
||||
if not _OK_MIN <= status < _OK_EXCLUSIVE:
|
||||
msg = f"provider rejected request: HTTP {status}"
|
||||
raise ProviderRejectedError(msg)
|
||||
|
||||
try:
|
||||
payload = json.loads(data.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
msg = f"provider response is not valid JSON: {exc}"
|
||||
raise ProviderResponseError(msg) from exc
|
||||
if not _is_object_dict(payload):
|
||||
msg = "provider payload is not a JSON object"
|
||||
raise ProviderResponseError(msg)
|
||||
text = _parse_completion(payload)
|
||||
usage = _parse_usage(payload, prompt, text)
|
||||
return Completion(text=text, usage=usage)
|
||||
@@ -0,0 +1,250 @@
|
||||
"""Replay log for observation-only policies (docs/phase-2 P0.5).
|
||||
|
||||
A ``ReplayLog`` records every ``(observation, action)`` pair an
|
||||
observation-only policy produced during one battle, keyed by a monotonic
|
||||
``call_seq`` assigned in execution order. It also carries the ``EncounterSpec``
|
||||
plus the battle ``seed`` and ``round_cap`` so a recorded battle can be re-run
|
||||
standalone. ``replay(log)`` turns a log back into an ``LLMPolicy`` that returns
|
||||
the recorded action for each observation, raising a typed ``ReplayError`` (a
|
||||
``ValueError``) on an out-of-sequence observation or once the log is exhausted.
|
||||
|
||||
Why ``call_seq`` and not the observation as the key: the same observation can
|
||||
legitimately recur (a stalled battle), and delayed turns / summons can revisit
|
||||
an earlier position — the sequence position is the only unambiguous key.
|
||||
|
||||
The JSONL serialization is self-contained (a header line with the spec, seed
|
||||
and round cap, then one record per line) so ``ReplayLog.from_jsonl(text)``
|
||||
round-trips without any out-of-band state. ``Observation`` uses the engine's
|
||||
canonical ``to_dict()`` (``schema_version`` included); ``Action`` and the spec
|
||||
are dumped through Pydantic type adapters so their tuple fields survive as
|
||||
JSON arrays and are coerced back to tuples on load. Observations are matched
|
||||
during playback by :func:`observation_key` (their canonical JSON), not raw
|
||||
dataclass equality: JSON collapses tuples and lists, and the snapshot's inner
|
||||
``dict[str, object]`` holds both.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Self, TypeGuard
|
||||
|
||||
from pydantic import TypeAdapter
|
||||
|
||||
from pf1e_simulator.combat import Action, Observation
|
||||
from pf1e_simulator.map import MapSpec
|
||||
from pf1e_simulator.models import Combatant
|
||||
from pf1e_simulator.runner import EncounterSpec
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.combat import LLMPolicy
|
||||
|
||||
_HEADER_KIND = "header"
|
||||
_RECORD_KIND = "record"
|
||||
|
||||
# ``EncounterSpec`` is a stdlib dataclass whose annotations reference
|
||||
# ``MapSpec``/``Combatant``; ``runner`` imports those only under TYPE_CHECKING,
|
||||
# so the adapter's default namespace cannot resolve them. Rebuild it with the
|
||||
# two runtime classes in scope.
|
||||
_SPEC_ADAPTER = TypeAdapter(EncounterSpec)
|
||||
_SPEC_ADAPTER.rebuild(_types_namespace={"MapSpec": MapSpec, "Combatant": Combatant})
|
||||
_OBSERVATION_ADAPTER = TypeAdapter(Observation)
|
||||
_ACTION_ADAPTER = TypeAdapter(Action)
|
||||
|
||||
|
||||
class ReplayError(ValueError):
|
||||
"""Base error for replay-log construction and playback."""
|
||||
|
||||
|
||||
class ReplayMismatchError(ReplayError):
|
||||
"""A replay policy received an observation that differs from the recorded one."""
|
||||
|
||||
|
||||
class ReplayExhaustedError(ReplayError):
|
||||
"""A replay policy was called past the last recorded ``call_seq``."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReplayRecord:
|
||||
"""One recorded policy call: the snapshot shown and the action chosen.
|
||||
|
||||
``call_seq`` is the position of this call in the execution order of the
|
||||
battle (0-based, contiguous), assigned by the recorder.
|
||||
"""
|
||||
|
||||
call_seq: int
|
||||
observation: Observation
|
||||
action: Action
|
||||
|
||||
|
||||
class ReplayLog:
|
||||
"""A recorded battle: spec + seed + round cap + ordered policy calls."""
|
||||
|
||||
def __init__(self, *, spec: EncounterSpec, seed: int, round_cap: int) -> None:
|
||||
self.spec = spec
|
||||
self.seed = seed
|
||||
self.round_cap = round_cap
|
||||
self._records: list[ReplayRecord] = []
|
||||
|
||||
def add(self, record: ReplayRecord) -> None:
|
||||
"""Append one call; require a strictly increasing ``call_seq``.
|
||||
|
||||
The log is a sequence: a duplicate or out-of-order ``call_seq`` would
|
||||
make the recorded execution order ambiguous, so both are rejected.
|
||||
"""
|
||||
if self._records:
|
||||
last = self._records[-1].call_seq
|
||||
if record.call_seq == last:
|
||||
msg = f"duplicate call_seq {record.call_seq}"
|
||||
raise ReplayError(msg)
|
||||
if record.call_seq < last:
|
||||
msg = f"call_seq {record.call_seq} is not after {last}"
|
||||
raise ReplayError(msg)
|
||||
self._records.append(record)
|
||||
|
||||
@property
|
||||
def records(self) -> tuple[ReplayRecord, ...]:
|
||||
"""The recorded calls ordered by ``call_seq`` (replay order)."""
|
||||
return tuple(sorted(self._records, key=lambda record: record.call_seq))
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._records)
|
||||
|
||||
def to_jsonl(self) -> str:
|
||||
"""Serialize to newline-delimited JSON (header line, then one per call)."""
|
||||
lines = [self._header_json()]
|
||||
lines.extend(self._record_json(record) for record in self.records)
|
||||
return "".join(line + "\n" for line in lines)
|
||||
|
||||
@classmethod
|
||||
def from_jsonl(cls, text: str) -> Self:
|
||||
"""Rebuild a log from ``to_jsonl`` output; raise ``ReplayError`` if invalid."""
|
||||
lines = [line for line in text.splitlines() if line.strip()]
|
||||
if not lines:
|
||||
msg = "empty replay log"
|
||||
raise ReplayError(msg)
|
||||
header = _as_object(_loads(lines[0]), "header")
|
||||
if _as_str(header.get("kind"), "kind") != _HEADER_KIND:
|
||||
msg = "first line is not a replay header"
|
||||
raise ReplayError(msg)
|
||||
log = cls(
|
||||
spec=_SPEC_ADAPTER.validate_python(header["spec"]),
|
||||
seed=_as_int(header.get("seed"), "seed"),
|
||||
round_cap=_as_int(header.get("round_cap"), "round_cap"),
|
||||
)
|
||||
for line in lines[1:]:
|
||||
data = _as_object(_loads(line), "record")
|
||||
if _as_str(data.get("kind"), "kind") != _RECORD_KIND:
|
||||
msg = "record line is not a replay record"
|
||||
raise ReplayError(msg)
|
||||
log.add(
|
||||
ReplayRecord(
|
||||
call_seq=_as_int(data.get("call_seq"), "call_seq"),
|
||||
observation=_OBSERVATION_ADAPTER.validate_python(
|
||||
data["observation"]
|
||||
),
|
||||
action=_ACTION_ADAPTER.validate_python(data["action"]),
|
||||
)
|
||||
)
|
||||
return log
|
||||
|
||||
def _header_json(self) -> str:
|
||||
payload: dict[str, object] = {
|
||||
"kind": _HEADER_KIND,
|
||||
"seed": self.seed,
|
||||
"round_cap": self.round_cap,
|
||||
"spec": _SPEC_ADAPTER.dump_python(self.spec, mode="json"),
|
||||
}
|
||||
return json.dumps(
|
||||
payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
)
|
||||
|
||||
def _record_json(self, record: ReplayRecord) -> str:
|
||||
payload: dict[str, object] = {
|
||||
"kind": _RECORD_KIND,
|
||||
"call_seq": record.call_seq,
|
||||
"observation": record.observation.to_dict(),
|
||||
"action": _ACTION_ADAPTER.dump_python(
|
||||
record.action, mode="json", exclude_defaults=True
|
||||
),
|
||||
}
|
||||
return json.dumps(
|
||||
payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
)
|
||||
|
||||
|
||||
def observation_key(observation: Observation) -> str:
|
||||
"""Return the canonical identity of an observation for replay matching.
|
||||
|
||||
``Observation.to_dict()`` is already JSON-shaped, so dumping it with sorted
|
||||
keys yields a value that is insensitive to dict insertion order and treats
|
||||
tuples and lists alike (JSON has no tuple type).
|
||||
"""
|
||||
return json.dumps(
|
||||
observation.to_dict(), sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
)
|
||||
|
||||
|
||||
def replay(log: ReplayLog) -> LLMPolicy:
|
||||
"""Build an observation-only policy that replays ``log`` in ``call_seq`` order.
|
||||
|
||||
The returned policy walks the log once, comparing the observation it is
|
||||
handed to the recorded one before returning that call's action (wrapped in
|
||||
a one-action tuple). An out-of-sequence observation raises
|
||||
``ReplayMismatchError``; a call past the end raises ``ReplayExhaustedError``.
|
||||
"""
|
||||
ordered = log.records
|
||||
keys = [observation_key(record.observation) for record in ordered]
|
||||
cursor = 0
|
||||
|
||||
def policy(observation: Observation) -> tuple[Action, ...]:
|
||||
nonlocal cursor
|
||||
if cursor >= len(ordered):
|
||||
msg = f"replay log exhausted after {len(ordered)} calls"
|
||||
raise ReplayExhaustedError(msg)
|
||||
record = ordered[cursor]
|
||||
if keys[cursor] != observation_key(observation):
|
||||
msg = (
|
||||
f"replay mismatch at call_seq {record.call_seq}: "
|
||||
"the observation differs from the recorded one"
|
||||
)
|
||||
raise ReplayMismatchError(msg)
|
||||
cursor += 1
|
||||
return (record.action,)
|
||||
|
||||
return policy
|
||||
|
||||
|
||||
def _loads(text: str) -> object:
|
||||
"""Parse one JSONL line, mapping a JSON error to a ``ReplayError``."""
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError as exc:
|
||||
msg = f"invalid JSON in replay log: {exc}"
|
||||
raise ReplayError(msg) from exc
|
||||
|
||||
|
||||
def _as_object(value: object, what: str) -> dict[str, object]:
|
||||
if _is_str_object_dict(value):
|
||||
return value
|
||||
msg = f"expected a JSON object for {what}"
|
||||
raise ReplayError(msg)
|
||||
|
||||
|
||||
def _as_str(value: object, what: str) -> str:
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
msg = f"expected a string for {what}"
|
||||
raise ReplayError(msg)
|
||||
|
||||
|
||||
def _as_int(value: object, what: str) -> int:
|
||||
# bool is an int subclass but never a valid call_seq/seed/round_cap.
|
||||
if isinstance(value, int) and not isinstance(value, bool):
|
||||
return value
|
||||
msg = f"expected an integer for {what}"
|
||||
raise ReplayError(msg)
|
||||
|
||||
|
||||
def _is_str_object_dict(value: object) -> TypeGuard[dict[str, object]]:
|
||||
return isinstance(value, dict)
|
||||
@@ -0,0 +1,297 @@
|
||||
"""LLM report writer (Phase 3) — aggregator + French narrative generator.
|
||||
|
||||
This module provides the data structures and pure functions to build a
|
||||
report context from Monte Carlo outcomes and LLM observability counters,
|
||||
then format the final French report combining the factual balance report
|
||||
with an LLM-generated narrative section.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.client import BudgetExceededError, CostBudget
|
||||
from pf1e_simulator.objectives import danger_metrics
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pf1e_simulator.client import ClientStats, LLMClient
|
||||
from pf1e_simulator.nl_policy import NLPolicyStats
|
||||
from pf1e_simulator.objectives import DangerMetrics
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
from pf1e_simulator.robustness import (
|
||||
DangerCell,
|
||||
MatchupBenchmark,
|
||||
SensitivityEntry,
|
||||
)
|
||||
from pf1e_simulator.runner import EncounterReport, RunOutcome
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReportContext:
|
||||
"""Complete context for the LLM report writer.
|
||||
|
||||
Fields marked as optional are filled when the corresponding analysis
|
||||
was executed (e.g. `collect_runs=True` for DangerMetrics, robustness
|
||||
calls for MatchupBenchmark/SensitivityEntry/DangerCell).
|
||||
"""
|
||||
|
||||
encounter: EncounterReport
|
||||
spec_name: str
|
||||
|
||||
danger: DangerMetrics | None = None
|
||||
|
||||
nl_stats: NLPolicyStats | None = None
|
||||
client_stats: ClientStats | None = None
|
||||
|
||||
matchup: MatchupBenchmark | None = None
|
||||
danger_map: tuple[DangerCell, ...] | None = None
|
||||
frontier_value: float | None = None
|
||||
sensitivity: tuple[SensitivityEntry, ...] | None = None
|
||||
|
||||
|
||||
def build_report_context( # noqa: PLR0913 - intentional builder with many optional params
|
||||
*,
|
||||
encounter: EncounterReport,
|
||||
spec_name: str,
|
||||
outcomes: Sequence[RunOutcome] | None = None,
|
||||
party: str = "players",
|
||||
nl_stats: NLPolicyStats | None = None,
|
||||
client_stats: ClientStats | None = None,
|
||||
matchup: MatchupBenchmark | None = None,
|
||||
danger_map: tuple[DangerCell, ...] | None = None,
|
||||
frontier_value: float | None = None,
|
||||
sensitivity: tuple[SensitivityEntry, ...] | None = None,
|
||||
) -> ReportContext:
|
||||
"""Build a `ReportContext` from raw Monte Carlo outputs."""
|
||||
danger: DangerMetrics | None = None
|
||||
if outcomes is not None:
|
||||
danger = danger_metrics(outcomes, party)
|
||||
|
||||
return ReportContext(
|
||||
encounter=encounter,
|
||||
spec_name=spec_name,
|
||||
danger=danger,
|
||||
nl_stats=nl_stats,
|
||||
client_stats=client_stats,
|
||||
matchup=matchup,
|
||||
danger_map=danger_map,
|
||||
frontier_value=frontier_value,
|
||||
sensitivity=sensitivity,
|
||||
)
|
||||
|
||||
|
||||
def _render_encounter(ctx: ReportContext) -> str:
|
||||
"""Render the EncounterReport in the same format as cli.format_report."""
|
||||
r = ctx.encounter
|
||||
lines = ["=== Rapport d'équilibrage ===", f"Carte : {ctx.spec_name}"]
|
||||
lines.extend(f"{side_name} : {r.wins.get(side_name, 0)} combats" for side_name in r.wins)
|
||||
lines.append(f"{r.runs} combats simulés (seed {r.seed}, plafond ? rounds)")
|
||||
lines.append("")
|
||||
for side_name in r.wins:
|
||||
wins = r.wins.get(side_name, 0)
|
||||
rate = r.win_rate(side_name)
|
||||
low, high = r.win_rate_band(side_name)
|
||||
lines.append(
|
||||
f"Victoires {side_name} : {rate:.1%} ({wins}) — "
|
||||
f"bande 3σ : [{low:.1%}, {high:.1%}]"
|
||||
)
|
||||
lines.append(f"Nuls : {r.draws / r.runs:.1%} ({r.draws})")
|
||||
lines.append(f"Rounds moyens : {r.avg_rounds:.1f}")
|
||||
lines.append("")
|
||||
lines.append("Attrition moyenne par combatant :")
|
||||
for cid, attr in r.attrition.items():
|
||||
lines.append(
|
||||
f" {cid} : {attr.hits:.2f} touches · {attr.crits:.2f} critiques · "
|
||||
f"{attr.damage_dealt:.2f} dégâts infligés · {attr.damage_taken:.2f} subis"
|
||||
f" · {attr.healing_done:.2f} soins prodigués"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _render_danger(danger: DangerMetrics) -> str:
|
||||
"""Render DangerMetrics section."""
|
||||
lines = ["=== Métriques de danger ==="]
|
||||
lines.append(f"runs: {danger.runs}")
|
||||
lines.append(f"p_survives: {danger.p_survives:.2f}")
|
||||
lines.append(f"p_tpk: {danger.p_tpk:.2f}")
|
||||
lines.append(f"p_any_death: {danger.p_any_death:.2f}")
|
||||
lines.append(f"expected_hp_lost: {danger.expected_hp_lost:.1f}")
|
||||
lines.append(f"expected_rounds: {danger.expected_rounds:.1f}")
|
||||
lines.append(f"worst_decile_party_hp: {danger.worst_decile_party_hp:.1f}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _render_nl_stats(stats: NLPolicyStats) -> str:
|
||||
lines = ["=== Observabilité LLM ==="]
|
||||
lines.append(f"calls: {stats.calls}")
|
||||
lines.append(f"repairs_attempted: {stats.repairs_attempted}")
|
||||
lines.append(f"repairs_succeeded: {stats.repairs_succeeded}")
|
||||
lines.append(f"fallbacks: {stats.fallbacks}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _render_client_stats(stats: ClientStats) -> str:
|
||||
lines = ["=== Coût LLM ==="]
|
||||
lines.append(f"prompt_tokens: {stats.prompt_tokens}")
|
||||
lines.append(f"completion_tokens: {stats.completion_tokens}")
|
||||
lines.append(f"cost_usd: {stats.cost_usd:.3f}")
|
||||
lines.append(f"cache_hits: {stats.cache_hits}")
|
||||
lines.append(f"cache_misses: {stats.cache_misses}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _render_matchup(matchup: MatchupBenchmark) -> str:
|
||||
lines = ["=== Robustesse (matchup) ==="]
|
||||
lines.append(f"party: {matchup.party}")
|
||||
lines.append(f"adversary: {matchup.adversary}")
|
||||
lines.append(f"runs: {matchup.runs}")
|
||||
lines.append(f"seed: {matchup.seed}")
|
||||
lines.append(f"party_danger.p_survives: {matchup.party_danger.p_survives:.2f}")
|
||||
lines.append(f"party_danger.p_tpk: {matchup.party_danger.p_tpk:.2f}")
|
||||
lines.append(f"adversary_danger.p_survives: {matchup.adversary_danger.p_survives:.2f}")
|
||||
lines.append(f"adversary_danger.p_tpk: {matchup.adversary_danger.p_tpk:.2f}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_report_prompt(ctx: ReportContext, template: str) -> str:
|
||||
"""Build the complete LLM prompt for the report writer."""
|
||||
sections: list[str] = [template, ""]
|
||||
|
||||
sections.append(_render_encounter(ctx))
|
||||
sections.append("")
|
||||
|
||||
if ctx.danger is not None:
|
||||
sections.append(_render_danger(ctx.danger))
|
||||
sections.append("")
|
||||
|
||||
if ctx.matchup is not None:
|
||||
sections.append(_render_matchup(ctx.matchup))
|
||||
sections.append("")
|
||||
|
||||
if ctx.nl_stats is not None:
|
||||
sections.append(_render_nl_stats(ctx.nl_stats))
|
||||
sections.append("")
|
||||
|
||||
if ctx.client_stats is not None:
|
||||
sections.append(_render_client_stats(ctx.client_stats))
|
||||
sections.append("")
|
||||
|
||||
if sections[-1] == "":
|
||||
sections.pop()
|
||||
|
||||
return "\n".join(sections)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReportBudget:
|
||||
"""Budget limits for a single report generation (integrates with CostBudget).
|
||||
|
||||
Mirrors the ceilings from the architecture doc (section 13):
|
||||
small caps for showcase/vitrine mode — max 1 call per report.
|
||||
"""
|
||||
|
||||
max_calls: int = 1
|
||||
max_total_tokens: int | None = None
|
||||
max_cost_usd: float | None = None
|
||||
|
||||
def to_cost_budget(self) -> CostBudget:
|
||||
"""Convert to the client's CostBudget type."""
|
||||
return CostBudget(
|
||||
max_calls=self.max_calls,
|
||||
max_total_tokens=self.max_total_tokens,
|
||||
max_cost_usd=self.max_cost_usd,
|
||||
)
|
||||
|
||||
def for_retries(self, max_retries: int) -> ReportBudget:
|
||||
"""Return a budget allowing for retries (max_calls = max_retries + 1)."""
|
||||
if self.max_calls == 1:
|
||||
return ReportBudget(
|
||||
max_calls=max_retries + 1,
|
||||
max_total_tokens=self.max_total_tokens,
|
||||
max_cost_usd=self.max_cost_usd,
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReportStats:
|
||||
"""Mutable counters for the LLM report writer."""
|
||||
|
||||
calls: int = 0
|
||||
retries: int = 0
|
||||
fallbacks: int = 0
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LLMReportPolicy:
|
||||
"""Generate a French narrative report via LLM completion.
|
||||
|
||||
Distinct from NLPolicy: produces free-text narrative (not JSON actions),
|
||||
uses simple retry loop (no repair trail), and tracks separate stats.
|
||||
"""
|
||||
|
||||
client: LLMClient
|
||||
builder: PromptBuilder
|
||||
budget: ReportBudget = field(default_factory=ReportBudget)
|
||||
prompt_name: str = "report"
|
||||
max_retries: int = 1
|
||||
stats: ReportStats = field(default_factory=ReportStats)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.max_retries < 0:
|
||||
msg = f"max_retries must be >= 0, got {self.max_retries}"
|
||||
raise ValueError(msg)
|
||||
expected = self.builder.prompt_version(self.prompt_name)
|
||||
if self.client.prompt_version != expected:
|
||||
msg = (
|
||||
f"client prompt_version {self.client.prompt_version!r} does not "
|
||||
f"match pack {expected!r} for {self.prompt_name!r}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
# Apply the report budget (adjusted for retries) to the client
|
||||
if self.client._budget is None: # noqa: SLF001 - private access required for budget injection # pyright: ignore[reportPrivateUsage]
|
||||
adjusted_budget = self.budget.for_retries(self.max_retries)
|
||||
self.client._budget = adjusted_budget.to_cost_budget() # noqa: SLF001 - private access required for budget injection # pyright: ignore[reportPrivateUsage]
|
||||
|
||||
def __call__(self, ctx: ReportContext) -> str:
|
||||
prompt = build_report_prompt(ctx, self.builder.pack.get(self.prompt_name))
|
||||
retries = 0
|
||||
while True:
|
||||
self.stats.calls += 1
|
||||
try:
|
||||
return self.client.complete(prompt)
|
||||
except BudgetExceededError:
|
||||
# Budget exceeded during retry - treat as fallback
|
||||
self.stats.fallbacks += 1
|
||||
return ""
|
||||
except Exception: # noqa: BLE001 - client may raise various exceptions
|
||||
if retries >= self.max_retries:
|
||||
self.stats.fallbacks += 1
|
||||
return ""
|
||||
retries += 1
|
||||
self.stats.retries += 1
|
||||
|
||||
|
||||
def format_llm_report(ctx: ReportContext, llm_text: str) -> str:
|
||||
"""Combine the factual balance report with the LLM narrative section.
|
||||
|
||||
The result is the complete French report printed to stdout and/or written
|
||||
to a file via `--output`.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
ctx: The populated `ReportContext` with all available data.
|
||||
llm_text: The narrative text returned by `LLMReportPolicy` (may be empty).
|
||||
|
||||
Returns
|
||||
-------
|
||||
Complete French report as a single string.
|
||||
"""
|
||||
factual = _render_encounter(ctx)
|
||||
if not llm_text.strip():
|
||||
return factual
|
||||
|
||||
return f"{factual}\n\n=== Récit d'équilibrage (LLM) ===\n{llm_text.strip()}"
|
||||
@@ -0,0 +1,682 @@
|
||||
"""Robustness layer: alternate min-max and danger cartography
|
||||
(docs/phase-2-llm-architecture.md sections 3.2 and 10; P4.2 and P4.3).
|
||||
|
||||
A party's danger is only meaningful against an adversary that plays to
|
||||
win (D12). This module implements the two robustness layers on top of
|
||||
the compiled Etage-A pipeline:
|
||||
|
||||
- P4.2 - alternate min-max (section 3.2: max_PJ . min_monstres):
|
||||
``adversarial_analysis`` alternately fixes one side and best-responds
|
||||
the other (the party maximizes its own ``DangerMetrics``, then the
|
||||
adversary minimizes the party's), every evaluation being a
|
||||
``matchup_benchmark`` of one party ``StrategySpec`` vs one adversary
|
||||
``AdversarySpec`` over a ``ScenarioSpec``. The same measure gives both
|
||||
senses: danger for the party AND danger for the monsters.
|
||||
- P4.3 - danger cartography (non exhaustive, D14): ``danger_map`` sweeps
|
||||
a grid of per-side initial-HP fractions (``SideSpec.hp_fraction``) and
|
||||
reports both sides' danger per cell, ``frontier`` bisects the party HP
|
||||
fraction where a danger criterion crosses a threshold, and
|
||||
``sensitivity`` ranks which single adversary lever moves the party
|
||||
danger the most.
|
||||
|
||||
Deterministic throughout: no RNG, no LLM, no network. The engine's
|
||||
``SeededRng`` under ``run_encounter`` is the only randomness and it is
|
||||
seeded, so identical inputs produce bit-identical frozen results.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pf1e_simulator.objectives import DangerMetrics
|
||||
from pf1e_simulator.runner import EncounterReport
|
||||
from pf1e_simulator.strategy import StrategySpec
|
||||
|
||||
from pf1e_simulator.adversary import AdversarySpec, adversary_name, compile_matchup
|
||||
from pf1e_simulator.objectives import danger_metrics
|
||||
from pf1e_simulator.runner import run_encounter
|
||||
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec, build_encounter
|
||||
|
||||
_CRITERIA = ("p_survives", "p_tpk", "p_any_death", "expected_hp_lost")
|
||||
_MATCHUP_SIDES = 2 # a matchup is exactly one arena pair (two sides)
|
||||
|
||||
|
||||
def _check_criterion(criterion: str) -> None:
|
||||
if criterion not in _CRITERIA:
|
||||
msg = (
|
||||
"criterion must be one of ('p_survives', 'p_tpk', 'p_any_death', "
|
||||
f"'expected_hp_lost'), got {criterion!r}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _danger_value(danger: DangerMetrics, criterion: str) -> float:
|
||||
"""The `criterion` value of a danger profile (explicit dispatch, no getattr)."""
|
||||
if criterion == "p_survives":
|
||||
return danger.p_survives
|
||||
if criterion == "p_tpk":
|
||||
return danger.p_tpk
|
||||
if criterion == "p_any_death":
|
||||
return danger.p_any_death
|
||||
return danger.expected_hp_lost
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MatchupBenchmark:
|
||||
"""One (party strategy x adversary spec) matchup over a scenario.
|
||||
|
||||
``report`` aggregates all runs; ``party_danger`` is the danger of the
|
||||
``party_side`` side and ``adversary_danger`` the danger of the other
|
||||
side (same measure, both senses: PJ and monsters).
|
||||
"""
|
||||
|
||||
scenario: str
|
||||
party: str
|
||||
adversary: str
|
||||
runs: int
|
||||
seed: int
|
||||
report: EncounterReport
|
||||
party_danger: DangerMetrics
|
||||
adversary_danger: DangerMetrics
|
||||
party_side: str
|
||||
|
||||
|
||||
def _other_side(scenario: ScenarioSpec, party_side: str) -> str:
|
||||
"""The unique side different from ``party_side`` (exactly-2-sides contract)."""
|
||||
names = tuple(side.name for side in scenario.sides)
|
||||
if len(names) != _MATCHUP_SIDES:
|
||||
msg = f"matchup requires exactly 2 sides, got {len(names)}"
|
||||
raise ValueError(msg)
|
||||
if party_side not in names:
|
||||
msg = f"party_side {party_side!r} is not a side of the scenario"
|
||||
raise ValueError(msg)
|
||||
(other,) = (name for name in names if name != party_side)
|
||||
return other
|
||||
|
||||
|
||||
def matchup_benchmark(
|
||||
scenario: ScenarioSpec,
|
||||
party: StrategySpec,
|
||||
adversary: AdversarySpec,
|
||||
*,
|
||||
party_side: str = "players",
|
||||
runs: int = 100,
|
||||
seed: int = 1,
|
||||
) -> MatchupBenchmark:
|
||||
"""Compile one matchup and measure both sides' danger over `runs` battles.
|
||||
|
||||
Deterministic: same inputs -> bit-identical ``MatchupBenchmark``
|
||||
(frozen). Exactly two sides are required (one arena pair), the party
|
||||
side name must be non-empty and be one of them, and ``runs`` >= 1.
|
||||
"""
|
||||
if not party_side:
|
||||
msg = "party_side must be a non-empty string"
|
||||
raise ValueError(msg)
|
||||
adversary_side = _other_side(scenario, party_side)
|
||||
if runs < 1:
|
||||
msg = "runs must be >= 1"
|
||||
raise ValueError(msg)
|
||||
encounter = build_encounter(scenario)
|
||||
policy = compile_matchup(party, adversary, party_side=party_side)
|
||||
report, outcomes = run_encounter(
|
||||
encounter, runs, seed, policy=policy, collect_runs=True
|
||||
)
|
||||
return MatchupBenchmark(
|
||||
scenario=scenario.name,
|
||||
party=party.name,
|
||||
adversary=adversary.name,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
report=report,
|
||||
party_danger=danger_metrics(outcomes, party_side),
|
||||
adversary_danger=danger_metrics(outcomes, adversary_side),
|
||||
party_side=party_side,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RobustnessEntry:
|
||||
"""One turn of the alternating min-max: the pair kept after that turn."""
|
||||
|
||||
iteration: int
|
||||
party: StrategySpec
|
||||
adversary: AdversarySpec
|
||||
benchmark: MatchupBenchmark
|
||||
|
||||
|
||||
def _party_rank(benchmark: MatchupBenchmark) -> tuple[float, float, str]:
|
||||
"""Key: the party best-response prefers high survival, low HP lost, then name."""
|
||||
danger = benchmark.party_danger
|
||||
return (danger.p_survives, -danger.expected_hp_lost, benchmark.party)
|
||||
|
||||
|
||||
def _adversary_rank(benchmark: MatchupBenchmark) -> tuple[float, float, str]:
|
||||
"""Key: the adversary best-response prefers low party survival, high HP lost."""
|
||||
danger = benchmark.party_danger
|
||||
return (danger.p_survives, -danger.expected_hp_lost, benchmark.adversary)
|
||||
|
||||
|
||||
def adversarial_analysis(
|
||||
scenario: ScenarioSpec,
|
||||
*,
|
||||
party_candidates: Sequence[StrategySpec],
|
||||
adversary_candidates: Sequence[AdversarySpec],
|
||||
party_side: str = "players",
|
||||
runs: int = 100,
|
||||
seed: int = 1,
|
||||
iterations: int = 3,
|
||||
max_total_runs: int | None = None,
|
||||
) -> tuple[RobustnessEntry, ...]:
|
||||
"""Alternate (party, adversary) best responses over `iterations` turns.
|
||||
|
||||
Each turn re-evaluates every party candidate against the current
|
||||
adversary and keeps the best for the party (max own survival), then
|
||||
every adversary candidate against the kept party and keeps the worst
|
||||
for the party (min own survival, max party HP lost) -- the min-max
|
||||
alternation of section 3.2. The kept pair is reported with its
|
||||
benchmark.
|
||||
|
||||
Budget: at most ``iterations * (party + adversary candidates) * runs``
|
||||
battles. When ``max_total_runs`` is set and the budget exceeds it, no
|
||||
battle is fought and a ValueError is raised.
|
||||
"""
|
||||
if not party_candidates:
|
||||
msg = "party_candidates must be non-empty"
|
||||
raise ValueError(msg)
|
||||
if not adversary_candidates:
|
||||
msg = "adversary_candidates must be non-empty"
|
||||
raise ValueError(msg)
|
||||
if iterations < 1:
|
||||
msg = f"iterations must be >= 1, got {iterations}"
|
||||
raise ValueError(msg)
|
||||
if runs < 1:
|
||||
msg = f"runs must be >= 1, got {runs}"
|
||||
raise ValueError(msg)
|
||||
cost = iterations * (len(party_candidates) + len(adversary_candidates)) * runs
|
||||
if max_total_runs is not None and cost > max_total_runs:
|
||||
msg = (
|
||||
f"robustness budget exceeded: {cost} runs > "
|
||||
f"{max_total_runs} max_total_runs"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
party = party_candidates[0]
|
||||
adversary = adversary_candidates[0]
|
||||
entries: list[RobustnessEntry] = []
|
||||
for iteration in range(1, iterations + 1):
|
||||
party_pairs: list[tuple[StrategySpec, MatchupBenchmark]] = [
|
||||
(
|
||||
candidate,
|
||||
matchup_benchmark(
|
||||
scenario,
|
||||
candidate,
|
||||
adversary,
|
||||
party_side=party_side,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
),
|
||||
)
|
||||
for candidate in party_candidates
|
||||
]
|
||||
party = max(party_pairs, key=lambda pair: _party_rank(pair[1]))[0]
|
||||
adversary_pairs: list[tuple[AdversarySpec, MatchupBenchmark]] = [
|
||||
(
|
||||
candidate,
|
||||
matchup_benchmark(
|
||||
scenario,
|
||||
party,
|
||||
candidate,
|
||||
party_side=party_side,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
),
|
||||
)
|
||||
for candidate in adversary_candidates
|
||||
]
|
||||
adversary, benchmark = min(
|
||||
adversary_pairs, key=lambda pair: _adversary_rank(pair[1])
|
||||
)
|
||||
entries.append(
|
||||
RobustnessEntry(
|
||||
iteration=iteration,
|
||||
party=party,
|
||||
adversary=adversary,
|
||||
benchmark=benchmark,
|
||||
)
|
||||
)
|
||||
return tuple(entries)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DangerCell:
|
||||
"""One (party HP x adversary HP) cell of the danger map."""
|
||||
|
||||
party_hp_fraction: float
|
||||
adversary_hp_fraction: float
|
||||
benchmark: MatchupBenchmark
|
||||
|
||||
|
||||
def _side_variant(
|
||||
scenario: ScenarioSpec,
|
||||
*,
|
||||
party_side: str,
|
||||
party_hp_fraction: float,
|
||||
adversary_hp_fraction: float,
|
||||
) -> ScenarioSpec:
|
||||
"""A deterministic clone of `scenario` with per-side initial-HP fractions.
|
||||
|
||||
The `party_side` side carries `party_hp_fraction`; every other side
|
||||
carries `adversary_hp_fraction` (the matchup contract is exactly 2
|
||||
sides). Name: "<base> party_hp=<f> adv_hp=<g>" so cells stay distinct.
|
||||
"""
|
||||
sides = tuple(
|
||||
SideSpec(
|
||||
name=side.name,
|
||||
zone=side.zone,
|
||||
combatant_files=side.combatant_files,
|
||||
hp_fraction=(
|
||||
party_hp_fraction if side.name == party_side else adversary_hp_fraction
|
||||
),
|
||||
aware=side.aware,
|
||||
)
|
||||
for side in scenario.sides
|
||||
)
|
||||
return ScenarioSpec(
|
||||
name=(
|
||||
f"{scenario.name} party_hp={party_hp_fraction} "
|
||||
f"adv_hp={adversary_hp_fraction}"
|
||||
),
|
||||
map_file=scenario.map_file,
|
||||
sides=sides,
|
||||
round_cap=scenario.round_cap,
|
||||
)
|
||||
|
||||
|
||||
def danger_map(
|
||||
scenario: ScenarioSpec,
|
||||
party: StrategySpec,
|
||||
adversary: AdversarySpec,
|
||||
*,
|
||||
party_side: str = "players",
|
||||
fractions: Sequence[float] = (0.5, 0.7, 1.0),
|
||||
runs: int = 100,
|
||||
seed: int = 1,
|
||||
) -> tuple[DangerCell, ...]:
|
||||
"""The grid of party HP x adversary HP danger cells (non exhaustive, D14).
|
||||
|
||||
One ``DangerCell`` per combination, party-major (party fraction
|
||||
outermost), each with a distinct scenario name
|
||||
"<base> party_hp=<f> adv_hp=<g>". Every fraction must lie in (0, 1].
|
||||
"""
|
||||
if not fractions:
|
||||
msg = "fractions must be non-empty"
|
||||
raise ValueError(msg)
|
||||
for fraction in fractions:
|
||||
if not 0.0 < fraction <= 1.0:
|
||||
msg = f"fraction must be in (0, 1], got {fraction}"
|
||||
raise ValueError(msg)
|
||||
return tuple(
|
||||
DangerCell(
|
||||
party_hp_fraction=party_frac,
|
||||
adversary_hp_fraction=adv_frac,
|
||||
benchmark=matchup_benchmark(
|
||||
_side_variant(
|
||||
scenario,
|
||||
party_side=party_side,
|
||||
party_hp_fraction=party_frac,
|
||||
adversary_hp_fraction=adv_frac,
|
||||
),
|
||||
party,
|
||||
adversary,
|
||||
party_side=party_side,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
),
|
||||
)
|
||||
for party_frac in fractions
|
||||
for adv_frac in fractions
|
||||
)
|
||||
|
||||
|
||||
def _validate_frontier(
|
||||
criterion: str,
|
||||
target: float,
|
||||
min_fraction: float,
|
||||
max_fraction: float,
|
||||
precision: float,
|
||||
) -> None:
|
||||
"""Validate frontier parameters (keeps ``frontier`` itself flat)."""
|
||||
_check_criterion(criterion)
|
||||
if criterion == "expected_hp_lost":
|
||||
if target < 0:
|
||||
msg = f"target must be >= 0, got {target}"
|
||||
raise ValueError(msg)
|
||||
elif not 0.0 < target < 1.0:
|
||||
msg = f"target must be in (0, 1), got {target}"
|
||||
raise ValueError(msg)
|
||||
if min_fraction <= 0:
|
||||
msg = f"min_fraction must be > 0, got {min_fraction}"
|
||||
raise ValueError(msg)
|
||||
if max_fraction > 1:
|
||||
msg = f"max_fraction must be <= 1, got {max_fraction}"
|
||||
raise ValueError(msg)
|
||||
if min_fraction > max_fraction:
|
||||
msg = (
|
||||
f"min_fraction must be <= max_fraction, "
|
||||
f"got min={min_fraction} max={max_fraction}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if precision <= 0:
|
||||
msg = f"precision must be > 0, got {precision}"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def frontier(
|
||||
scenario: ScenarioSpec,
|
||||
party: StrategySpec,
|
||||
adversary: AdversarySpec,
|
||||
*,
|
||||
party_side: str = "players",
|
||||
criterion: str = "p_tpk",
|
||||
target: float = 0.2,
|
||||
min_fraction: float = 0.25,
|
||||
max_fraction: float = 1.0,
|
||||
precision: float = 0.05,
|
||||
runs: int = 100,
|
||||
seed: int = 1,
|
||||
) -> float | None:
|
||||
"""The smallest party HP fraction where `criterion` crosses `target`.
|
||||
|
||||
The adversary HP fraction stays at 1.0. Crossing convention (section
|
||||
3.2): ``value >= target`` for p_tpk, p_any_death and expected_hp_lost;
|
||||
``value <= target`` for p_survives. Monotone danger (less party HP is
|
||||
strictly worse): a crossing at ``min_fraction`` already yields the
|
||||
smallest probed fraction, ceiled to the ``precision`` grid; otherwise
|
||||
bisection converges to the smallest crossing fraction and returns
|
||||
``ceil(hi / precision) * precision`` (ceiling keeps the fraction on
|
||||
the crossing side of the threshold). None when not even
|
||||
``max_fraction`` crosses. Probabilities require ``0 < target < 1``;
|
||||
expected_hp_lost requires ``target >= 0``.
|
||||
"""
|
||||
_validate_frontier(criterion, target, min_fraction, max_fraction, precision)
|
||||
|
||||
def _crossed(fraction: float) -> bool:
|
||||
cell = matchup_benchmark(
|
||||
_side_variant(
|
||||
scenario,
|
||||
party_side=party_side,
|
||||
party_hp_fraction=fraction,
|
||||
adversary_hp_fraction=1.0,
|
||||
),
|
||||
party,
|
||||
adversary,
|
||||
party_side=party_side,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
)
|
||||
value = _danger_value(cell.party_danger, criterion)
|
||||
if criterion == "p_survives":
|
||||
return value <= target
|
||||
return value >= target
|
||||
|
||||
if _crossed(min_fraction):
|
||||
return math.ceil(min_fraction / precision) * precision
|
||||
if not _crossed(max_fraction):
|
||||
return None
|
||||
lo = min_fraction
|
||||
hi = max_fraction
|
||||
while hi - lo > precision:
|
||||
mid = (lo + hi) / 2.0
|
||||
if _crossed(mid):
|
||||
hi = mid
|
||||
else:
|
||||
lo = mid
|
||||
return math.ceil(hi / precision) * precision
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SensitivityEntry:
|
||||
"""One adversary lever raised from the baseline, with its danger delta."""
|
||||
|
||||
levers: str
|
||||
spec: AdversarySpec
|
||||
benchmark: MatchupBenchmark
|
||||
delta: float
|
||||
|
||||
|
||||
def sensitivity(
|
||||
scenario: ScenarioSpec,
|
||||
party: StrategySpec,
|
||||
*,
|
||||
party_side: str = "players",
|
||||
criterion: str = "expected_hp_lost",
|
||||
runs: int = 100,
|
||||
seed: int = 1,
|
||||
max_total_runs: int | None = None,
|
||||
) -> tuple[SensitivityEntry, ...]:
|
||||
"""Rank which single adversary lever moves the party danger the most.
|
||||
|
||||
Baseline is the all-default ``AdversarySpec()``; each of the 7
|
||||
candidate levers varies exactly one knob (target_priority x3,
|
||||
spell_policy=none, retreat x2, use_charges=false). ``delta`` is the
|
||||
signed change of the party ``criterion`` value vs the baseline
|
||||
(positive = more dangerous). Sorted by |delta| descending, ties by
|
||||
lever label ascending. Budget is exactly 8 x runs (baseline + 7
|
||||
levers), enforced before any battle.
|
||||
"""
|
||||
_check_criterion(criterion)
|
||||
cost = 8 * runs
|
||||
if max_total_runs is not None and cost > max_total_runs:
|
||||
msg = (
|
||||
f"robustness budget exceeded: {cost} runs > "
|
||||
f"{max_total_runs} max_total_runs"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
levers = (
|
||||
AdversarySpec(target_priority="weakest_ac"),
|
||||
AdversarySpec(target_priority="lowest_hp"),
|
||||
AdversarySpec(target_priority="healer"),
|
||||
AdversarySpec(spell_policy="none"),
|
||||
AdversarySpec(retreat_hp_fraction=0.25),
|
||||
AdversarySpec(retreat_hp_fraction=0.5),
|
||||
AdversarySpec(use_charges=False),
|
||||
)
|
||||
baseline = matchup_benchmark(
|
||||
scenario,
|
||||
party,
|
||||
AdversarySpec(),
|
||||
party_side=party_side,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
)
|
||||
base = _danger_value(baseline.party_danger, criterion)
|
||||
entries: list[SensitivityEntry] = []
|
||||
for spec in levers:
|
||||
benchmark = matchup_benchmark(
|
||||
scenario, party, spec, party_side=party_side, runs=runs, seed=seed
|
||||
)
|
||||
entries.append(
|
||||
SensitivityEntry(
|
||||
levers=adversary_name(spec),
|
||||
spec=spec,
|
||||
benchmark=benchmark,
|
||||
delta=_danger_value(benchmark.party_danger, criterion) - base,
|
||||
)
|
||||
)
|
||||
entries.sort(key=lambda entry: (-abs(entry.delta), entry.levers))
|
||||
return tuple(entries)
|
||||
|
||||
|
||||
def _render_line(values: Sequence[str], widths: Sequence[int]) -> str:
|
||||
"""One aligned row: every cell ljust-padded to its column width."""
|
||||
return " ".join(value.ljust(width) for value, width in zip(values, widths, strict=True))
|
||||
|
||||
|
||||
def _render_table(headers: Sequence[str], rows: Sequence[Sequence[str]]) -> str:
|
||||
"""A plain-text grid: header, a full-width dash separator, data rows.
|
||||
|
||||
Column widths are the max width over header and rows (same algorithm
|
||||
as ``sweep.format_sweep``), so every line has identical length and the
|
||||
separator spans the whole header line.
|
||||
"""
|
||||
columns: tuple[Sequence[str], ...] = (headers, *rows)
|
||||
widths = tuple(
|
||||
max(len(row[column]) for row in columns) for column in range(len(headers))
|
||||
)
|
||||
header = _render_line(headers, widths)
|
||||
lines = [header, "-" * len(header)]
|
||||
lines.extend(_render_line(row, widths) for row in rows)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_matchup(benchmark: MatchupBenchmark) -> str:
|
||||
"""Render one matchup: per-side win rate, danger profile and rounds."""
|
||||
headers = (
|
||||
"Scenario",
|
||||
"Runs",
|
||||
"Side",
|
||||
"Win%",
|
||||
"Survive",
|
||||
"TPK",
|
||||
">=1-death",
|
||||
"HP-lost",
|
||||
"Rounds",
|
||||
)
|
||||
adversary_side = next(
|
||||
side for side in benchmark.report.wins if side != benchmark.party_side
|
||||
)
|
||||
party_row = (
|
||||
benchmark.scenario,
|
||||
str(benchmark.runs),
|
||||
"party",
|
||||
f"{benchmark.report.win_rate(benchmark.party_side):.0%}",
|
||||
f"{benchmark.party_danger.p_survives:.0%}",
|
||||
f"{benchmark.party_danger.p_tpk:.0%}",
|
||||
f"{benchmark.party_danger.p_any_death:.0%}",
|
||||
f"{benchmark.party_danger.expected_hp_lost:.1f}",
|
||||
f"{benchmark.report.avg_rounds:.1f}",
|
||||
)
|
||||
adversary_row = (
|
||||
"",
|
||||
"",
|
||||
"adversary",
|
||||
f"{benchmark.report.win_rate(adversary_side):.0%}",
|
||||
f"{benchmark.adversary_danger.p_survives:.0%}",
|
||||
f"{benchmark.adversary_danger.p_tpk:.0%}",
|
||||
f"{benchmark.adversary_danger.p_any_death:.0%}",
|
||||
f"{benchmark.adversary_danger.expected_hp_lost:.1f}",
|
||||
f"{benchmark.report.avg_rounds:.1f}",
|
||||
)
|
||||
return _render_table(headers, (party_row, adversary_row))
|
||||
|
||||
|
||||
def format_robustness(entries: Sequence[RobustnessEntry]) -> str:
|
||||
"""Render the alternating min-max log, one aligned line per turn."""
|
||||
headers = (
|
||||
"Iter",
|
||||
"Party",
|
||||
"Adversary",
|
||||
"P(party-survive)",
|
||||
"P(party-TPK)",
|
||||
"Party-HP",
|
||||
"P(adv-survive)",
|
||||
"P(adv-TPK)",
|
||||
"Adv-HP",
|
||||
)
|
||||
rows = tuple(
|
||||
(
|
||||
str(entry.iteration),
|
||||
entry.party.name,
|
||||
entry.adversary.name,
|
||||
f"{entry.benchmark.party_danger.p_survives:.0%}",
|
||||
f"{entry.benchmark.party_danger.p_tpk:.0%}",
|
||||
f"{entry.benchmark.party_danger.expected_hp_lost:.1f}",
|
||||
f"{entry.benchmark.adversary_danger.p_survives:.0%}",
|
||||
f"{entry.benchmark.adversary_danger.p_tpk:.0%}",
|
||||
f"{entry.benchmark.adversary_danger.expected_hp_lost:.1f}",
|
||||
)
|
||||
for entry in entries
|
||||
)
|
||||
return _render_table(headers, rows)
|
||||
|
||||
|
||||
def format_danger_map(cells: Sequence[DangerCell]) -> str:
|
||||
"""Render the danger grid: rows = party HP, columns = adversary HP.
|
||||
|
||||
Each cell shows party survive% / adversary survive%, both as 0%
|
||||
percentages (order preserved from ``danger_map``, hence the
|
||||
dict.fromkeys dedupe instead of a set).
|
||||
"""
|
||||
party_fractions: tuple[float, ...] = tuple(
|
||||
dict.fromkeys(cell.party_hp_fraction for cell in cells)
|
||||
)
|
||||
adversary_fractions: tuple[float, ...] = tuple(
|
||||
dict.fromkeys(cell.adversary_hp_fraction for cell in cells)
|
||||
)
|
||||
by_cell: dict[tuple[float, float], DangerCell] = {
|
||||
(cell.party_hp_fraction, cell.adversary_hp_fraction): cell for cell in cells
|
||||
}
|
||||
headers = (
|
||||
"Party\\Adv",
|
||||
*(f"adv={fraction}" for fraction in adversary_fractions),
|
||||
)
|
||||
rows: list[tuple[str, ...]] = []
|
||||
for fraction in party_fractions:
|
||||
row: list[str] = [f"party={fraction}"]
|
||||
for adv in adversary_fractions:
|
||||
cell = by_cell[(fraction, adv)]
|
||||
row.append(
|
||||
f"{cell.benchmark.party_danger.p_survives:.0%}/"
|
||||
f"{cell.benchmark.adversary_danger.p_survives:.0%}"
|
||||
)
|
||||
rows.append(tuple(row))
|
||||
return _render_table(headers, rows)
|
||||
|
||||
|
||||
def format_frontier(
|
||||
criterion: str,
|
||||
target: float,
|
||||
fraction: float | None,
|
||||
*,
|
||||
min_fraction: float,
|
||||
max_fraction: float,
|
||||
) -> str:
|
||||
"""Render a frontier result (None -> "no frontier" message)."""
|
||||
if fraction is None:
|
||||
return (
|
||||
f"no frontier for {criterion} {target} "
|
||||
f"in [{min_fraction:.2f}, {max_fraction:.2f}]"
|
||||
)
|
||||
return (
|
||||
f"frontier {criterion} {target}: "
|
||||
f"party hp_fraction = {fraction:.2f}"
|
||||
)
|
||||
|
||||
|
||||
def _format_criterion_value(danger: DangerMetrics, criterion: str) -> str:
|
||||
"""One danger criterion as text: 0% for probabilities, 1 decimal for HP."""
|
||||
value = _danger_value(danger, criterion)
|
||||
if criterion == "expected_hp_lost":
|
||||
return f"{value:.1f}"
|
||||
return f"{value:.0%}"
|
||||
|
||||
|
||||
def format_sensitivity(entries: Sequence[SensitivityEntry], criterion: str) -> str:
|
||||
"""Render the lever ranking: rank, levers, party criterion value, delta."""
|
||||
headers = ("Rank", "Levers", "Value", "Delta")
|
||||
probability = criterion != "expected_hp_lost"
|
||||
rows = tuple(
|
||||
(
|
||||
str(index),
|
||||
entry.levers,
|
||||
_format_criterion_value(entry.benchmark.party_danger, criterion),
|
||||
f"{entry.delta:+.0%}" if probability else f"{entry.delta:+.1f}",
|
||||
)
|
||||
for index, entry in enumerate(entries, start=1)
|
||||
)
|
||||
return _render_table(headers, rows)
|
||||
+206
-62
@@ -10,10 +10,12 @@ attrition averages. All statistical helpers live in metrics.py.
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Literal, overload
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.combat import Policy
|
||||
from collections.abc import Callable
|
||||
|
||||
from pf1e_simulator.combat import LegacyLLMPolicy, LLMPolicy, Policy
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.map import MapSpec
|
||||
from pf1e_simulator.models import Combatant
|
||||
@@ -24,6 +26,7 @@ from pf1e_simulator.combat import (
|
||||
CombatantState,
|
||||
CombatantStats,
|
||||
CombatEngine,
|
||||
CombatResult,
|
||||
)
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.map import Pos, zone_cells
|
||||
@@ -61,6 +64,39 @@ class CombatantAttrition:
|
||||
healing_done: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CombatantOutcome:
|
||||
"""Final state of one combatant at the end of one battle (per-run record).
|
||||
|
||||
``dead`` is the engine's own death condition (``CombatantState.dead``:
|
||||
hp at or below the combatant's death threshold - CON for living PCs, 0
|
||||
for undead/constructs and under the simple monster-death rule). A dying
|
||||
PC (hp < 0 but above -CON) is NOT dead here. ``hp`` may be negative;
|
||||
``hp_max`` is carried so objectives can compute hit points lost.
|
||||
"""
|
||||
|
||||
id: str
|
||||
side: str
|
||||
hp: int
|
||||
hp_max: int
|
||||
dead: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RunOutcome:
|
||||
"""Minimal per-battle outcome for the danger/optimization objectives.
|
||||
|
||||
Collected only when ``run_encounter`` is called with
|
||||
``collect_runs=True``; the aggregate ``EncounterReport`` stays the
|
||||
default (cheap) output. ``combatants`` covers every creature that
|
||||
existed at battle end, including summons.
|
||||
"""
|
||||
|
||||
winner: str | None
|
||||
rounds: int
|
||||
combatants: tuple[CombatantOutcome, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EncounterReport:
|
||||
"""Aggregated results of a Monte Carlo encounter.
|
||||
@@ -169,67 +205,17 @@ def build_states(spec: EncounterSpec) -> list[CombatantState]:
|
||||
return states
|
||||
|
||||
|
||||
def run_encounter(
|
||||
spec: EncounterSpec,
|
||||
runs: int,
|
||||
seed: int,
|
||||
*,
|
||||
policy: Policy | None = None,
|
||||
spell_registry: dict[str, SpellSpec] | None = None,
|
||||
capture_first_transcript: bool = False,
|
||||
monsters_die_at_zero: bool = False,
|
||||
) -> EncounterReport:
|
||||
"""Run `runs` battles (one SeededRng per run) and aggregate balance metrics.
|
||||
def _accumulate_stats(totals: dict[str, CombatantStats], result: CombatResult) -> None:
|
||||
"""Add one battle's per-combatant stats into the running totals, in place.
|
||||
|
||||
The spell registry is loaded once (from ``data/spells/`` unless overridden)
|
||||
and passed into every ``CombatEngine`` so caster PCs actually cast.
|
||||
|
||||
When ``capture_first_transcript`` is True, the turn-by-turn log of the
|
||||
first battle (``seed + 0``) is stored in ``report.transcript``.
|
||||
|
||||
``monsters_die_at_zero`` enables the optional simple monster-death rule
|
||||
(CLI ``--simple-monster-death``): monsters die at hp <= 0; PCs keep the
|
||||
PF1e hp <= -CON rule. Default False keeps behavior bit-identical.
|
||||
Accumulate whatever ids the engine reports: summoned creatures act in the
|
||||
battle and appear in result.stats, so seeding from the initial states
|
||||
only would KeyError on them. Ids seen for the first time get a fresh
|
||||
accumulator. Duplicate summons of the same creature are disambiguated by
|
||||
the engine the same way build_states does (celestial_wolf,
|
||||
celestial_wolf-2, ...), so each summoned creature owns its own slot and
|
||||
their attrition is split, not merged.
|
||||
"""
|
||||
if runs <= 0:
|
||||
msg = "runs must be positive"
|
||||
raise ValueError(msg)
|
||||
grid = Grid.from_spec(spec.map)
|
||||
states = build_states(spec)
|
||||
if spell_registry is None:
|
||||
spell_registry = load_spell_registry(SPELLS_DIR)
|
||||
wins = {side.name: 0 for side in spec.sides}
|
||||
draws = 0
|
||||
total_rounds = 0
|
||||
# Accumulate whatever ids the engine reports: summoned creatures act in the
|
||||
# battle and appear in result.stats, so seeding from the initial states
|
||||
# only would KeyError on them. Ids seen for the first time get a fresh
|
||||
# accumulator. Duplicate summons of the same creature are disambiguated by
|
||||
# the engine the same way build_states does (celestial_wolf,
|
||||
# celestial_wolf-2, ...), so each summoned creature owns its own slot and
|
||||
# their attrition is split, not merged.
|
||||
totals: dict[str, CombatantStats] = {}
|
||||
transcript: tuple[str, ...] | None = None
|
||||
for index in range(runs):
|
||||
if index > 0:
|
||||
states = build_states(spec)
|
||||
rng = SeededRng(seed + index)
|
||||
result = CombatEngine(
|
||||
rng,
|
||||
grid,
|
||||
states,
|
||||
round_cap=spec.round_cap,
|
||||
policy=policy,
|
||||
spell_registry=spell_registry,
|
||||
monsters_die_at_zero=monsters_die_at_zero,
|
||||
).run()
|
||||
if capture_first_transcript and index == 0:
|
||||
transcript = result.transcript
|
||||
if result.winner is None:
|
||||
draws += 1
|
||||
else:
|
||||
wins[result.winner] = wins.get(result.winner, 0) + 1
|
||||
total_rounds += result.rounds
|
||||
for cid, stats in result.stats.items():
|
||||
current = totals.get(cid)
|
||||
if current is None:
|
||||
@@ -242,6 +228,161 @@ def run_encounter(
|
||||
damage_taken=current.damage_taken + stats.damage_taken,
|
||||
healing_done=current.healing_done + stats.healing_done,
|
||||
)
|
||||
|
||||
|
||||
def _run_outcome(engine: CombatEngine, result: CombatResult) -> RunOutcome:
|
||||
"""One battle's per-run summary, read from the engine's final live states."""
|
||||
return RunOutcome(
|
||||
winner=result.winner,
|
||||
rounds=result.rounds,
|
||||
combatants=tuple(
|
||||
CombatantOutcome(
|
||||
id=state.combatant.id,
|
||||
side=state.side,
|
||||
hp=state.hp,
|
||||
hp_max=state.combatant.hp_max,
|
||||
dead=state.dead,
|
||||
)
|
||||
for state in engine.states
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _check_policy_args(
|
||||
policy: Policy | None,
|
||||
llm_policy: LLMPolicy | LegacyLLMPolicy | None,
|
||||
policy_factory: Callable[[int], Policy | None] | None,
|
||||
) -> None:
|
||||
"""Reject conflicting policy entry points (P0.4).
|
||||
|
||||
``policy_factory`` owns the per-run policy slot, so it cannot be combined
|
||||
with ``policy`` or ``llm_policy``. ``policy`` + ``llm_policy`` is left to
|
||||
``CombatEngine`` (single source of truth for the mutual-exclusion error).
|
||||
"""
|
||||
if policy_factory is not None and policy is not None:
|
||||
msg = "policy_factory and policy are mutually exclusive"
|
||||
raise ValueError(msg)
|
||||
if policy_factory is not None and llm_policy is not None:
|
||||
msg = "policy_factory and llm_policy are mutually exclusive"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
@overload
|
||||
def run_encounter(
|
||||
spec: EncounterSpec,
|
||||
runs: int,
|
||||
seed: int,
|
||||
*,
|
||||
policy: Policy | None = None,
|
||||
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
|
||||
policy_factory: Callable[[int], Policy | None] | None = None,
|
||||
spell_registry: dict[str, SpellSpec] | None = None,
|
||||
capture_first_transcript: bool = False,
|
||||
monsters_die_at_zero: bool = False,
|
||||
collect_runs: Literal[False] = False,
|
||||
) -> EncounterReport: ...
|
||||
|
||||
|
||||
@overload
|
||||
def run_encounter(
|
||||
spec: EncounterSpec,
|
||||
runs: int,
|
||||
seed: int,
|
||||
*,
|
||||
policy: Policy | None = None,
|
||||
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
|
||||
policy_factory: Callable[[int], Policy | None] | None = None,
|
||||
spell_registry: dict[str, SpellSpec] | None = None,
|
||||
capture_first_transcript: bool = False,
|
||||
monsters_die_at_zero: bool = False,
|
||||
collect_runs: Literal[True],
|
||||
) -> tuple[EncounterReport, tuple[RunOutcome, ...]]: ...
|
||||
|
||||
|
||||
def run_encounter(
|
||||
spec: EncounterSpec,
|
||||
runs: int,
|
||||
seed: int,
|
||||
*,
|
||||
policy: Policy | None = None,
|
||||
llm_policy: LLMPolicy | LegacyLLMPolicy | None = None,
|
||||
policy_factory: Callable[[int], Policy | None] | None = None,
|
||||
spell_registry: dict[str, SpellSpec] | None = None,
|
||||
capture_first_transcript: bool = False,
|
||||
monsters_die_at_zero: bool = False,
|
||||
collect_runs: bool = False,
|
||||
) -> EncounterReport | tuple[EncounterReport, tuple[RunOutcome, ...]]:
|
||||
"""Run `runs` battles (one SeededRng per run) and aggregate balance metrics.
|
||||
|
||||
The spell registry is loaded once (from ``data/spells/`` unless overridden)
|
||||
and passed into every ``CombatEngine`` so caster PCs actually cast.
|
||||
|
||||
When ``capture_first_transcript`` is True, the turn-by-turn log of the
|
||||
first battle (``seed + 0``) is stored in ``report.transcript``.
|
||||
|
||||
``monsters_die_at_zero`` enables the optional simple monster-death rule
|
||||
(CLI ``--simple-monster-death``): monsters die at hp <= 0; PCs keep the
|
||||
PF1e hp <= -CON rule. Default False keeps behavior bit-identical.
|
||||
|
||||
``collect_runs`` (Phase 2, danger cartography) additionally returns a
|
||||
tuple ``(report, run_outcomes)`` where ``run_outcomes`` holds one
|
||||
``RunOutcome`` per battle (winner, rounds, final per-combatant
|
||||
id/side/hp/dead). Default False keeps the signature and behavior of
|
||||
every existing caller unchanged (the report alone).
|
||||
|
||||
``llm_policy`` (Phase 2, P0.4) forwards an observation-only (or legacy
|
||||
two-argument) policy straight to every ``CombatEngine``. It is mutually
|
||||
exclusive with ``policy``; the engine raises ``ValueError`` when both are
|
||||
given, so that conflict surfaces here.
|
||||
|
||||
``policy_factory`` (Phase 2, P0.4) builds the engine-style ``Policy`` per
|
||||
run: it is called once per battle with the run index (``0..runs-1``) and
|
||||
its result is passed as that run's ``policy`` (``None`` falls back to
|
||||
``default_policy``). It is mutually exclusive with both ``policy`` and
|
||||
``llm_policy``. Note the per-run callable is a ``Policy`` (it receives the
|
||||
live engine), not an ``LLMPolicy``; use ``llm_policy`` for the
|
||||
observation-only entry point.
|
||||
"""
|
||||
if runs <= 0:
|
||||
msg = "runs must be positive"
|
||||
raise ValueError(msg)
|
||||
_check_policy_args(policy, llm_policy, policy_factory)
|
||||
grid = Grid.from_spec(spec.map)
|
||||
states = build_states(spec)
|
||||
if spell_registry is None:
|
||||
spell_registry = load_spell_registry(SPELLS_DIR)
|
||||
wins = {side.name: 0 for side in spec.sides}
|
||||
draws = 0
|
||||
total_rounds = 0
|
||||
totals: dict[str, CombatantStats] = {}
|
||||
transcript: tuple[str, ...] | None = None
|
||||
outcomes: list[RunOutcome] = []
|
||||
for index in range(runs):
|
||||
if index > 0:
|
||||
states = build_states(spec)
|
||||
rng = SeededRng(seed + index)
|
||||
run_policy = policy if policy_factory is None else policy_factory(index)
|
||||
engine = CombatEngine(
|
||||
rng,
|
||||
grid,
|
||||
states,
|
||||
round_cap=spec.round_cap,
|
||||
policy=run_policy,
|
||||
llm_policy=llm_policy,
|
||||
spell_registry=spell_registry,
|
||||
monsters_die_at_zero=monsters_die_at_zero,
|
||||
)
|
||||
result = engine.run()
|
||||
if collect_runs:
|
||||
outcomes.append(_run_outcome(engine, result))
|
||||
if capture_first_transcript and index == 0:
|
||||
transcript = result.transcript
|
||||
if result.winner is None:
|
||||
draws += 1
|
||||
else:
|
||||
wins[result.winner] = wins.get(result.winner, 0) + 1
|
||||
total_rounds += result.rounds
|
||||
_accumulate_stats(totals, result)
|
||||
attrition = {
|
||||
cid: CombatantAttrition(
|
||||
hits=stats.hits / runs,
|
||||
@@ -252,7 +393,7 @@ def run_encounter(
|
||||
)
|
||||
for cid, stats in totals.items()
|
||||
}
|
||||
return EncounterReport(
|
||||
report = EncounterReport(
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
wins=wins,
|
||||
@@ -261,3 +402,6 @@ def run_encounter(
|
||||
attrition=attrition,
|
||||
transcript=transcript,
|
||||
)
|
||||
if collect_runs:
|
||||
return report, tuple(outcomes)
|
||||
return report
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Scenario search space and deterministic sampling (docs/phase-2-llm-architecture.md
|
||||
sections 3.1, 6, 10; P3.1).
|
||||
|
||||
``ScenarioSpec`` is the reusable scenario search space of section 6: a map
|
||||
file, the sides that fill its deployment zones, and a battle round cap. It is
|
||||
compiled deterministically into a runner ``EncounterSpec`` by
|
||||
``build_encounter``, which loads the map and every combatant file and applies
|
||||
each side's initial-HP fraction. ``scenario_variants`` implements the coarse
|
||||
sampling of section 3.1 ("echantillonnage latin"): deterministic per-side HP
|
||||
fraction variants of a base scenario, with no RNG.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.runner import EncounterSpec, Side
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SideSpec:
|
||||
"""One side of a scenario: a deployment zone letter and the combatant
|
||||
files that fill it, with optional initial-hp scaling."""
|
||||
|
||||
name: str # side name, e.g. "players"
|
||||
zone: str # deployment zone letter, e.g. "A"
|
||||
combatant_files: tuple[Path, ...] # monster/PJ JSON files, loaded via load_combatant
|
||||
hp_fraction: float = 1.0 # initial HP as a fraction of hp_max, in (0, 1]
|
||||
aware: bool | None = None # SRD "Surprise" override; None keeps Combatant.aware (True)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ScenarioSpec:
|
||||
"""A reusable scenario: map file, sides, and a battle round cap (docs
|
||||
section 6 -- the ScenarioSpec search space). Deterministically compiled
|
||||
to an EncounterSpec by build_encounter."""
|
||||
|
||||
name: str
|
||||
map_file: Path
|
||||
sides: tuple[SideSpec, ...]
|
||||
round_cap: int = 100
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
msg = f"scenario name must be non-empty, got {self.name!r}"
|
||||
raise ValueError(msg)
|
||||
if not self.sides:
|
||||
msg = "scenario sides must be non-empty"
|
||||
raise ValueError(msg)
|
||||
for side in self.sides:
|
||||
if not side.zone:
|
||||
msg = (
|
||||
f"side {side.name!r}: zone must be a single non-empty "
|
||||
f"string, got {side.zone!r}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if not 0.0 < side.hp_fraction <= 1.0:
|
||||
msg = (
|
||||
f"side {side.name!r}: hp_fraction must be in (0, 1], "
|
||||
f"got {side.hp_fraction}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if self.round_cap < 1:
|
||||
msg = f"round_cap must be >= 1, got {self.round_cap}"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def _apply_hp_fraction(combatant: Combatant, fraction: float) -> Combatant:
|
||||
"""Scale initial HP via model_copy, the same pattern as runner.build_states."""
|
||||
if fraction == 1.0:
|
||||
return combatant
|
||||
return combatant.model_copy(
|
||||
update={"hp_max": max(1, round(combatant.hp_max * fraction))}
|
||||
)
|
||||
|
||||
|
||||
def _apply_aware(combatant: Combatant, *, aware: bool | None) -> Combatant:
|
||||
"""Apply a side's SRD "Surprise" awareness override via model_copy.
|
||||
|
||||
None keeps the loaded combatant unchanged (Combatant.aware defaults to
|
||||
True, models.py:416-420). SRD "Surprise": only some combatants aware
|
||||
means a surprise round precedes round 1; unaware combatants do not act
|
||||
in it.
|
||||
"""
|
||||
if aware is None:
|
||||
return combatant
|
||||
return combatant.model_copy(update={"aware": aware})
|
||||
|
||||
|
||||
def build_encounter(scenario: ScenarioSpec) -> EncounterSpec:
|
||||
"""Compile a ScenarioSpec into a runner EncounterSpec (deterministic).
|
||||
|
||||
- load_map(scenario.map_file); for each SideSpec, load each combatant
|
||||
file with load_combatant, then apply hp_fraction to hp_max via
|
||||
pydantic model_copy(update={"hp_max": max(1, round(hp_max * fraction))})
|
||||
(same model_copy pattern as runner.build_states).
|
||||
- When a side's aware is not None, it is applied to that side's
|
||||
combatants via model_copy(update={"aware": ...}) at the same point
|
||||
where hp_fraction is applied (SRD "Surprise" round semantics: only
|
||||
some aware -> surprise round; Combatant.aware defaults to True, so
|
||||
aware=None keeps the loaded default and is bit-identical to the
|
||||
behaviour without the field).
|
||||
- Duplicate combatant ids are NOT disambiguated here: build_states
|
||||
(called later by run_encounter) handles that. Just list each loaded
|
||||
combatant once per occurrence in combatant_files.
|
||||
- Returns EncounterSpec(map=..., sides=tuple(Side[...]), round_cap=...).
|
||||
"""
|
||||
map_spec = load_map(scenario.map_file)
|
||||
if map_spec.zones is not None and not map_spec.deployment:
|
||||
msg = (
|
||||
"map has a zones layer but no deployment section (each zone "
|
||||
"letter needs a side)"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
for side in scenario.sides:
|
||||
if not side.combatant_files:
|
||||
msg = f"side {side.name!r}: at least one combatant file is required"
|
||||
raise ValueError(msg)
|
||||
letters = [side.zone for side in scenario.sides]
|
||||
shared = sorted({letter for letter in letters if letters.count(letter) > 1})
|
||||
if shared:
|
||||
msg = (
|
||||
"sides share zone letters (each side needs its own distinct "
|
||||
f"zone, or they deploy on top of each other): {shared}"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
if map_spec.deployment:
|
||||
declared = set(map_spec.deployment.values())
|
||||
for side in scenario.sides:
|
||||
if side.zone not in declared:
|
||||
msg = (
|
||||
f"side {side.name!r} zone {side.zone!r} not declared in "
|
||||
"map deployment"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
sides = tuple(
|
||||
Side(
|
||||
name=side.name,
|
||||
combatants=tuple(
|
||||
_apply_aware(
|
||||
_apply_hp_fraction(load_combatant(path), side.hp_fraction),
|
||||
aware=side.aware,
|
||||
)
|
||||
for path in side.combatant_files
|
||||
),
|
||||
zone=side.zone,
|
||||
)
|
||||
for side in scenario.sides
|
||||
)
|
||||
return EncounterSpec(map=map_spec, sides=sides, round_cap=scenario.round_cap)
|
||||
|
||||
|
||||
def scenario_variants(
|
||||
base: ScenarioSpec,
|
||||
*,
|
||||
hp_fractions: Sequence[float] = (1.0,),
|
||||
) -> tuple[ScenarioSpec, ...]:
|
||||
"""Deterministic variants of `base` over per-side HP fractions.
|
||||
|
||||
Returns one ScenarioSpec per hp_fraction in `hp_fractions`, each with
|
||||
every side's hp_fraction set to that value (scenario names suffixed
|
||||
" <name> hp=<fraction>", e.g. "goblins-vs-orcs hp=0.5"; the identity
|
||||
fraction 1.0 keeps the base name, so the default call returns base
|
||||
itself). Same deps as __post_init__ validation (each fraction in
|
||||
(0, 1]); empty tuple of fractions -> ValueError. Deterministic given
|
||||
the same inputs: no RNG.
|
||||
"""
|
||||
if not hp_fractions:
|
||||
msg = "hp_fractions must be non-empty"
|
||||
raise ValueError(msg)
|
||||
for fraction in hp_fractions:
|
||||
if not 0.0 < fraction <= 1.0:
|
||||
msg = f"hp_fraction must be in (0, 1], got {fraction}"
|
||||
raise ValueError(msg)
|
||||
variants: list[ScenarioSpec] = []
|
||||
for fraction in hp_fractions:
|
||||
name = base.name if fraction == 1.0 else f"{base.name} hp={fraction}"
|
||||
variants.append(
|
||||
ScenarioSpec(
|
||||
name=name,
|
||||
map_file=base.map_file,
|
||||
sides=tuple(
|
||||
SideSpec(
|
||||
name=side.name,
|
||||
zone=side.zone,
|
||||
combatant_files=side.combatant_files,
|
||||
hp_fraction=fraction,
|
||||
)
|
||||
for side in base.sides
|
||||
),
|
||||
round_cap=base.round_cap,
|
||||
)
|
||||
)
|
||||
return tuple(variants)
|
||||
@@ -0,0 +1,318 @@
|
||||
"""Strategy search and playbook optimization (docs/phase-2-llm-architecture.md
|
||||
sections 3.1, 7, 10; P3.2/P3.3).
|
||||
|
||||
``evaluate`` measures one strategy against one scenario and returns its
|
||||
``StrategyBenchmark`` (danger metrics, win counts, and the party's 3-sigma
|
||||
win-rate band). ``optimize`` searches the strategy space in three stages - a
|
||||
coarse 24-cell grid over the target/spell/charge/class-ability levers, a
|
||||
retreat-threshold refinement of the top grid cells, then any LLM
|
||||
``candidates`` - and returns a ranked ``PlaybookEntry`` tuple.
|
||||
``strategy_name`` renders a deterministic label for a spec and
|
||||
``format_playbook`` renders the ranked playbook as aligned ASCII columns.
|
||||
Deterministic: the only randomness comes from the fixed ``seed`` passed to
|
||||
the sweep runner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, replace
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pf1e_simulator.objectives import DangerMetrics
|
||||
from pf1e_simulator.runner import EncounterSpec
|
||||
from pf1e_simulator.scenarios import ScenarioSpec
|
||||
from pf1e_simulator.sweep import SweepCellResult
|
||||
|
||||
from pf1e_simulator.scenarios import build_encounter
|
||||
from pf1e_simulator.strategy import StrategySpec, compile_strategy
|
||||
from pf1e_simulator.sweep import SweepCell, SweepRunner
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StrategyBenchmark:
|
||||
"""The measured outcome of one strategy on one scenario (P3.2)."""
|
||||
|
||||
strategy: StrategySpec
|
||||
scenario_name: str
|
||||
danger: DangerMetrics
|
||||
wins: dict[str, int]
|
||||
draws: int
|
||||
avg_rounds: float
|
||||
win_rate_band: tuple[float, float]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PlaybookEntry:
|
||||
"""One ranked row of an optimized playbook (P3.2/P3.3)."""
|
||||
|
||||
rank: int
|
||||
strategy: StrategySpec
|
||||
benchmark: StrategyBenchmark
|
||||
|
||||
|
||||
def strategy_name(strategy: StrategySpec) -> str:
|
||||
"""A deterministic label for `strategy` from its non-default levers.
|
||||
|
||||
Levers render in field order (target_priority, spell_policy, use_charges,
|
||||
use_class_abilities, retreat_hp_fraction), joined with ", "; the `name`
|
||||
field is never part of the label. An all-default spec is "default".
|
||||
"""
|
||||
levers: list[str] = []
|
||||
if strategy.target_priority != "nearest":
|
||||
levers.append(strategy.target_priority)
|
||||
if strategy.spell_policy != "default":
|
||||
levers.append(f"spell_policy={strategy.spell_policy}")
|
||||
if not strategy.use_charges:
|
||||
levers.append("use_charges=false")
|
||||
if not strategy.use_class_abilities:
|
||||
levers.append("use_class_abilities=false")
|
||||
if strategy.retreat_hp_fraction != 0.0:
|
||||
levers.append(f"retreat={strategy.retreat_hp_fraction}")
|
||||
if not levers:
|
||||
return "default"
|
||||
return ", ".join(levers)
|
||||
|
||||
|
||||
def _grid_specs() -> tuple[StrategySpec, ...]:
|
||||
"""The coarse 24-cell grid: 3 targets x 2 spell policies x 2 x 2, retreat 0."""
|
||||
specs: list[StrategySpec] = []
|
||||
for target_priority in ("nearest", "weakest_ac", "lowest_hp"):
|
||||
for spell_policy in ("default", "none"):
|
||||
for use_charges in (True, False):
|
||||
for use_class_abilities in (True, False):
|
||||
raw = StrategySpec(
|
||||
target_priority=target_priority,
|
||||
spell_policy=spell_policy,
|
||||
use_charges=use_charges,
|
||||
use_class_abilities=use_class_abilities,
|
||||
)
|
||||
specs.append(replace(raw, name=strategy_name(raw)))
|
||||
return tuple(specs)
|
||||
|
||||
|
||||
def _refine_specs(top: Sequence[StrategySpec]) -> tuple[StrategySpec, ...]:
|
||||
"""Retreat-threshold refinements of the top grid specs (3 per spec)."""
|
||||
refined: list[StrategySpec] = []
|
||||
for spec in top:
|
||||
for fraction in (0.25, 0.5, 0.75):
|
||||
raw = replace(spec, retreat_hp_fraction=fraction)
|
||||
refined.append(replace(raw, name=strategy_name(raw)))
|
||||
return tuple(refined)
|
||||
|
||||
|
||||
def _rank_key(
|
||||
strategy: StrategySpec, danger: DangerMetrics
|
||||
) -> tuple[float, float, float, float, str]:
|
||||
"""Sort key: best first (survive high, hp lost low, tail high, rounds low)."""
|
||||
return (
|
||||
-danger.p_survives,
|
||||
danger.expected_hp_lost,
|
||||
-danger.worst_decile_party_hp,
|
||||
danger.expected_rounds,
|
||||
strategy_name(strategy),
|
||||
)
|
||||
|
||||
|
||||
def _cell(
|
||||
scenario: ScenarioSpec,
|
||||
encounter: EncounterSpec,
|
||||
spec: StrategySpec,
|
||||
runs: int,
|
||||
seed: int,
|
||||
) -> SweepCell:
|
||||
"""One sweep cell for `spec` under its compiled strategy."""
|
||||
return SweepCell(
|
||||
scenario=scenario.name,
|
||||
strategy=strategy_name(spec),
|
||||
spec=encounter,
|
||||
policy=compile_strategy(spec),
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
|
||||
def _cells(
|
||||
scenario: ScenarioSpec,
|
||||
encounter: EncounterSpec,
|
||||
specs: Sequence[StrategySpec],
|
||||
runs: int,
|
||||
seed: int,
|
||||
) -> tuple[SweepCell, ...]:
|
||||
"""One cell per spec, in spec order."""
|
||||
return tuple(_cell(scenario, encounter, spec, runs, seed) for spec in specs)
|
||||
|
||||
|
||||
def _benchmark(
|
||||
strategy: StrategySpec,
|
||||
scenario_name: str,
|
||||
result: SweepCellResult,
|
||||
party: str,
|
||||
) -> StrategyBenchmark:
|
||||
"""Build the benchmark for one strategy from its sweep cell result."""
|
||||
return StrategyBenchmark(
|
||||
strategy=strategy,
|
||||
scenario_name=scenario_name,
|
||||
danger=result.danger,
|
||||
wins=result.report.wins,
|
||||
draws=result.report.draws,
|
||||
avg_rounds=result.report.avg_rounds,
|
||||
win_rate_band=result.report.win_rate_band(party),
|
||||
)
|
||||
|
||||
|
||||
def evaluate( # noqa: PLR0913 - pinned Phase-2 API (6 knobs)
|
||||
strategy: StrategySpec,
|
||||
scenario: ScenarioSpec,
|
||||
*,
|
||||
party: str,
|
||||
runs: int,
|
||||
seed: int,
|
||||
workers: int = 4,
|
||||
) -> StrategyBenchmark:
|
||||
"""Measure one strategy on one scenario (P3.2).
|
||||
|
||||
Compiles `strategy` and runs a single sweep cell of `runs` seeded battles;
|
||||
the benchmark carries the party's danger metrics, win counts, and 3-sigma
|
||||
win-rate band.
|
||||
"""
|
||||
encounter = build_encounter(scenario)
|
||||
runner = SweepRunner(party=party, workers=workers)
|
||||
(result,) = runner.run((_cell(scenario, encounter, strategy, runs, seed),))
|
||||
return _benchmark(strategy, scenario.name, result, party)
|
||||
|
||||
|
||||
def optimize( # noqa: PLR0913 - pinned Phase-2 API (7 knobs)
|
||||
scenario: ScenarioSpec,
|
||||
*,
|
||||
party: str,
|
||||
runs: int,
|
||||
seed: int,
|
||||
workers: int = 4,
|
||||
max_total_runs: int | None = None,
|
||||
candidates: Sequence[StrategySpec] = (),
|
||||
) -> tuple[PlaybookEntry, ...]:
|
||||
"""Search the strategy space and return a ranked playbook (P3.2/P3.3).
|
||||
|
||||
Stage 1 sweeps the coarse 24-cell grid (target priority x spell policy x
|
||||
charges x class abilities, retreat 0); stage 2 refines the top 3 grid
|
||||
cells over retreat thresholds 0.25/0.5/0.75; stage 3 adds the LLM
|
||||
`candidates`, deduplicated by field equality against every spec already
|
||||
run. Entries are ranked by (-p_survives, +expected_hp_lost,
|
||||
-worst_decile_party_hp, +avg_rounds, strategy_name). `max_total_runs`
|
||||
caps the total Monte Carlo cost before anything executes (mirrors
|
||||
SweepRunner.run). The signature is pinned by the Phase-2 API, hence the
|
||||
PLR0913 noqa.
|
||||
"""
|
||||
encounter = build_encounter(scenario)
|
||||
runner = SweepRunner(party=party, workers=workers)
|
||||
|
||||
stage1_specs = _grid_specs()
|
||||
seen: set[StrategySpec] = set(stage1_specs)
|
||||
stage3_specs: list[StrategySpec] = []
|
||||
for spec in candidates:
|
||||
if spec not in seen:
|
||||
seen.add(spec)
|
||||
stage3_specs.append(spec)
|
||||
|
||||
total_runs = (len(stage1_specs) + 9 + len(stage3_specs)) * runs
|
||||
if max_total_runs is not None and total_runs > max_total_runs:
|
||||
msg = (
|
||||
f"optimize budget exceeded: {total_runs} runs > "
|
||||
f"{max_total_runs} max_total_runs"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
|
||||
stage1_results = runner.run(_cells(scenario, encounter, stage1_specs, runs, seed))
|
||||
stage1_ranked = sorted(
|
||||
zip(stage1_specs, stage1_results, strict=True),
|
||||
key=lambda pair: _rank_key(pair[0], pair[1].danger),
|
||||
)
|
||||
top3 = [spec for spec, _ in stage1_ranked[:3]]
|
||||
stage2_specs = _refine_specs(top3)
|
||||
stage2_results = runner.run(_cells(scenario, encounter, stage2_specs, runs, seed))
|
||||
|
||||
seen = set(stage1_specs) | set(stage2_specs)
|
||||
stage3_specs = []
|
||||
for spec in candidates:
|
||||
if spec not in seen:
|
||||
seen.add(spec)
|
||||
stage3_specs.append(spec)
|
||||
stage3_results = (
|
||||
runner.run(_cells(scenario, encounter, stage3_specs, runs, seed))
|
||||
if stage3_specs
|
||||
else ()
|
||||
)
|
||||
|
||||
pairs = [
|
||||
*zip(stage1_specs, stage1_results, strict=True),
|
||||
*zip(stage2_specs, stage2_results, strict=True),
|
||||
*zip(stage3_specs, stage3_results, strict=True),
|
||||
]
|
||||
benchmarks = [
|
||||
_benchmark(spec, scenario.name, result, party) for spec, result in pairs
|
||||
]
|
||||
ordered = sorted(benchmarks, key=lambda b: _rank_key(b.strategy, b.danger))
|
||||
return tuple(
|
||||
PlaybookEntry(rank=index, strategy=benchmark.strategy, benchmark=benchmark)
|
||||
for index, benchmark in enumerate(ordered, start=1)
|
||||
)
|
||||
|
||||
|
||||
def _win_rate_text(benchmark: StrategyBenchmark, party: str) -> str:
|
||||
"""The party's win rate with its 3-sigma band, e.g. "93.0 [89.5, 96.5]"."""
|
||||
rate = benchmark.wins.get(party, 0) / benchmark.danger.runs
|
||||
low, high = benchmark.win_rate_band
|
||||
return f"{rate * 100:.1f} [{low * 100:.1f}, {high * 100:.1f}]"
|
||||
|
||||
|
||||
def format_playbook(entries: Sequence[PlaybookEntry], party: str) -> str:
|
||||
"""Render a ranked playbook as aligned plain-text columns (ASCII only).
|
||||
|
||||
One row per entry in order, under a header line and a separator row of
|
||||
dashes. Percentages and HP values use one decimal; the Win% column shows
|
||||
the party's win rate with its 3-sigma band. Column widths are computed
|
||||
from the content; columns are separated by a single space.
|
||||
"""
|
||||
headers = (
|
||||
"Rank",
|
||||
"Strategy",
|
||||
"Runs",
|
||||
"P(survive)",
|
||||
"P(TPK)",
|
||||
"P(>=1 death)",
|
||||
"HP lost",
|
||||
"Worst-decile HP",
|
||||
"Rounds",
|
||||
"Win% (3σ)",
|
||||
)
|
||||
rows = [
|
||||
(
|
||||
str(entry.rank),
|
||||
entry.benchmark.strategy.name,
|
||||
str(entry.benchmark.danger.runs),
|
||||
f"{entry.benchmark.danger.p_survives:.1%}",
|
||||
f"{entry.benchmark.danger.p_tpk:.1%}",
|
||||
f"{entry.benchmark.danger.p_any_death:.1%}",
|
||||
f"{entry.benchmark.danger.expected_hp_lost:.1f}",
|
||||
f"{entry.benchmark.danger.worst_decile_party_hp:.1f}",
|
||||
f"{entry.benchmark.danger.expected_rounds:.1f}",
|
||||
_win_rate_text(entry.benchmark, party),
|
||||
)
|
||||
for entry in entries
|
||||
]
|
||||
widths = [
|
||||
max(len(header), *(len(row[index]) for row in rows))
|
||||
for index, header in enumerate(headers)
|
||||
]
|
||||
|
||||
def _line(values: Sequence[str]) -> str:
|
||||
return " ".join(
|
||||
value.ljust(width) for value, width in zip(values, widths, strict=True)
|
||||
)
|
||||
|
||||
lines = [_line(headers), _line(["-" * width for width in widths])]
|
||||
lines.extend(_line(row) for row in rows)
|
||||
return "\n".join(lines)
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Etage A strategy compiler (docs/phase-2-llm-architecture.md sections 2 and 10, P2.1).
|
||||
|
||||
Deterministic, additive: the engine (combat.py) is never modified. A
|
||||
``StrategySpec`` is a small set of frozen knobs (target priority, retreat
|
||||
threshold, spell policy, charge/class-ability toggles) compiled by
|
||||
``compile_strategy`` into a ``Policy`` callable. The all-default spec compiles
|
||||
to the module-level ``default_policy`` object itself (identity), so the common
|
||||
case is bit-identical and zero-cost. Non-default specs mirror
|
||||
``default_policy``'s decision flow with the spec's values substituted.
|
||||
|
||||
``StrategySpec`` feeds the SweepRunner (sweep.py, later batch) and the LLM
|
||||
NL-to-spec translation (later batch): the optimizer searches over specs, the
|
||||
LLM produces specs from natural language, and this module is the single
|
||||
deterministic executor both rely on.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal
|
||||
|
||||
from pf1e_simulator.combat import (
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Policy,
|
||||
# The decision helpers below are module-private by convention but stable:
|
||||
# the engine stays unchanged, and duplicating their spell/hex/channel
|
||||
# logic here would drift.
|
||||
_choose_spell_action, # pyright: ignore[reportPrivateUsage]
|
||||
_emit_channel_energy, # pyright: ignore[reportPrivateUsage]
|
||||
_emit_hex, # pyright: ignore[reportPrivateUsage]
|
||||
_spell_combat_actions, # pyright: ignore[reportPrivateUsage]
|
||||
_swift_bonus_actions, # pyright: ignore[reportPrivateUsage]
|
||||
default_policy,
|
||||
)
|
||||
|
||||
TargetPriority = Literal["nearest", "weakest_ac", "lowest_hp", "healer"]
|
||||
SpellPolicy = Literal["default", "none"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StrategySpec:
|
||||
"""Deterministic strategy knobs compiled to a ``Policy`` (Etage A).
|
||||
|
||||
All fields default to the behaviour of ``default_policy``; a spec built
|
||||
with every default compiles to ``default_policy`` itself (identity).
|
||||
"""
|
||||
|
||||
name: str = "default"
|
||||
target_priority: TargetPriority = "nearest"
|
||||
retreat_hp_fraction: float = 0.0 # 0.0 = never retreat
|
||||
spell_policy: SpellPolicy = "default"
|
||||
use_charges: bool = True
|
||||
use_class_abilities: bool = True
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
msg = "name must be a non-empty string"
|
||||
raise ValueError(msg)
|
||||
if not 0.0 <= self.retreat_hp_fraction <= 1.0:
|
||||
msg = "retreat_hp_fraction must be within [0, 1]"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
def select_target(
|
||||
engine: CombatEngine, state: CombatantState, priority: TargetPriority
|
||||
) -> CombatantState | None:
|
||||
"""The active enemy selected by ``priority``; None when no enemy is active.
|
||||
|
||||
Deterministic: ties break on ``engine.states`` index order.
|
||||
"""
|
||||
if priority == "nearest":
|
||||
return engine.nearest_enemy(state)
|
||||
enemies = [s for s in engine.states if s.side != state.side and s.active]
|
||||
if not enemies:
|
||||
return None
|
||||
match priority:
|
||||
case "weakest_ac":
|
||||
return min(
|
||||
enemies,
|
||||
key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)),
|
||||
)
|
||||
case "lowest_hp":
|
||||
return min(enemies, key=lambda s: (s.hp, engine.states.index(s)))
|
||||
case "healer":
|
||||
# Prefer the caster/healer (non-empty combatant.spells); among
|
||||
# several, min by (ac, hp, list order). No active caster on the
|
||||
# enemy side -> identical to weakest_ac.
|
||||
casters = [s for s in enemies if s.combatant.spells]
|
||||
pool = casters or enemies
|
||||
return min(
|
||||
pool,
|
||||
key=lambda s: (s.combatant.ac.total, s.hp, engine.states.index(s)),
|
||||
)
|
||||
|
||||
|
||||
def _melee_actions(
|
||||
engine: CombatEngine,
|
||||
state: CombatantState,
|
||||
target: CombatantState,
|
||||
spec: StrategySpec,
|
||||
) -> tuple[Action, ...]:
|
||||
"""The strike ladder: full-attack, charge (if enabled), step, or approach."""
|
||||
if engine.weapon_for(state, target) is not None:
|
||||
return (Action(kind="full_attack", target_id=target.combatant.id),)
|
||||
if spec.use_charges:
|
||||
charge = engine.can_charge(state, target)
|
||||
if charge is not None and charge.weapon is not None:
|
||||
return (Action(kind="charge", target_id=target.combatant.id),)
|
||||
if engine.can_step5_to_attack(state, target):
|
||||
return (
|
||||
Action(kind="5foot_step", target_id=target.combatant.id),
|
||||
Action(kind="full_attack", target_id=target.combatant.id),
|
||||
)
|
||||
return (
|
||||
Action(kind="move", target_id=target.combatant.id),
|
||||
Action(kind="attack", target_id=target.combatant.id),
|
||||
)
|
||||
|
||||
|
||||
def _normal_actions(
|
||||
engine: CombatEngine,
|
||||
state: CombatantState,
|
||||
target: CombatantState,
|
||||
spec: StrategySpec,
|
||||
) -> tuple[Action, ...]:
|
||||
"""The spell/melee action sequence (without swift/free class-ability add-ons)."""
|
||||
if spec.spell_policy != "none" and state.combatant.spells:
|
||||
spell_action = _choose_spell_action(engine, state, target)
|
||||
if spell_action is not None:
|
||||
return spell_action
|
||||
return _melee_actions(engine, state, target, spec)
|
||||
|
||||
|
||||
def _class_ability_action(
|
||||
engine: CombatEngine, state: CombatantState, target: CombatantState
|
||||
) -> tuple[Action, ...] | None:
|
||||
"""The channel/hex replacement action when a class ability fires."""
|
||||
channel_mode = _emit_channel_energy(engine, state)
|
||||
if channel_mode is not None:
|
||||
return (
|
||||
Action(
|
||||
kind="channel_energy",
|
||||
target_id=target.combatant.id,
|
||||
channel_mode=channel_mode,
|
||||
),
|
||||
)
|
||||
if _emit_hex(engine, state, target):
|
||||
return (Action(kind="hex", target_id=target.combatant.id, hex_name="evil_eye"),)
|
||||
return None
|
||||
|
||||
|
||||
def _compiled_policy(spec: StrategySpec) -> Policy:
|
||||
"""Build the ``Policy`` that executes ``spec`` (non-default specs only)."""
|
||||
|
||||
def policy(engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
target = select_target(engine, state, spec.target_priority)
|
||||
if target is None:
|
||||
return (Action(kind="wait"),)
|
||||
if engine.has_condition_flag(state, "must_flee") or (
|
||||
spec.retreat_hp_fraction > 0.0
|
||||
and state.hp < spec.retreat_hp_fraction * state.combatant.hp_max
|
||||
):
|
||||
return (Action(kind="withdraw", target_id=target.combatant.id),)
|
||||
if spec.use_class_abilities:
|
||||
class_action = _class_ability_action(engine, state, target)
|
||||
if class_action is not None:
|
||||
return class_action
|
||||
normal = _normal_actions(engine, state, target, spec)
|
||||
if not spec.use_class_abilities:
|
||||
return normal
|
||||
spell_combat = _spell_combat_actions(engine, state, target)
|
||||
if spell_combat is not None:
|
||||
return spell_combat
|
||||
swift_bonus = _swift_bonus_actions(engine, state, target, normal)
|
||||
return (*swift_bonus, *normal) if swift_bonus else normal
|
||||
|
||||
return policy
|
||||
|
||||
|
||||
_DEFAULT_SPEC = StrategySpec()
|
||||
|
||||
|
||||
def compile_strategy(spec: StrategySpec) -> Policy:
|
||||
"""Compile a ``StrategySpec`` into a ``Policy`` callable.
|
||||
|
||||
The all-default spec returns the module-level ``default_policy`` object
|
||||
itself (identity), so the common case is bit-identical and zero-cost.
|
||||
"""
|
||||
if spec == _DEFAULT_SPEC:
|
||||
return default_policy
|
||||
return _compiled_policy(spec)
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Parallel, cost-bounded sweep runner for Phase-2 danger cartography.
|
||||
|
||||
Executes a grid of (scenario x strategy) cells (docs/phase-2-llm-architecture.md
|
||||
section 10, P2.3; decisions D9/D10): each cell runs ``run_encounter(...,
|
||||
collect_runs=True)`` for a fixed number of seeded runs under one policy and
|
||||
reports the aggregate ``EncounterReport`` plus the party's ``DangerMetrics``
|
||||
(objectives.danger_metrics). Cells are independent, so they run on a thread
|
||||
pool (D10); the spell registry is loaded once and shared read-only across
|
||||
threads. The runner is strategy-agnostic: policies are passed in as callables
|
||||
(``None`` = ``run_encounter``'s own ``default_policy``) and no strategy module
|
||||
is imported here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from pf1e_simulator.combat import Policy
|
||||
from pf1e_simulator.runner import EncounterReport, EncounterSpec
|
||||
from pf1e_simulator.spells import SpellSpec
|
||||
|
||||
from pf1e_simulator.objectives import DangerMetrics, danger_metrics
|
||||
from pf1e_simulator.runner import run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SweepCell:
|
||||
"""One (scenario, strategy) cell of a sweep: N seeded runs with one policy."""
|
||||
|
||||
scenario: str
|
||||
strategy: str
|
||||
spec: EncounterSpec
|
||||
policy: Policy | None # None = default_policy (run_encounter's own default)
|
||||
runs: int
|
||||
seed: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SweepCellResult:
|
||||
"""Aggregate + danger metrics for one cell."""
|
||||
|
||||
cell: SweepCell
|
||||
report: EncounterReport
|
||||
danger: DangerMetrics
|
||||
|
||||
|
||||
class SweepRunner:
|
||||
"""Run a grid of (scenario, strategy) cells in parallel, bounded by cost.
|
||||
|
||||
``party`` is the side name whose danger is measured (passed to
|
||||
``danger_metrics``); ``workers`` bounds the thread pool. The spell
|
||||
registry is loaded once in ``__init__`` and shared read-only across all
|
||||
cells (reloading it per cell would be wasteful; a read-only dict is safe
|
||||
to share between threads).
|
||||
"""
|
||||
|
||||
def __init__(self, *, party: str, workers: int = 4) -> None:
|
||||
if not party:
|
||||
msg = "party must be a non-empty side name"
|
||||
raise ValueError(msg)
|
||||
if workers < 1:
|
||||
msg = "workers must be >= 1"
|
||||
raise ValueError(msg)
|
||||
self._party: str = party
|
||||
self._workers: int = workers
|
||||
self._registry: dict[str, SpellSpec] = load_spell_registry(SPELLS_DIR)
|
||||
|
||||
def run(
|
||||
self,
|
||||
cells: Sequence[SweepCell],
|
||||
*,
|
||||
max_total_runs: int | None = None,
|
||||
) -> tuple[SweepCellResult, ...]:
|
||||
"""Run every cell on the thread pool and return results in input order.
|
||||
|
||||
``max_total_runs`` caps the total Monte Carlo cost (docs section 9,
|
||||
D9/D14 "Budget - plafonds petits"): when the cells' runs sum exceeds
|
||||
it, a ``ValueError`` is raised before anything executes. Each cell
|
||||
must run at least once. Exceptions from a cell propagate loudly - a
|
||||
failing cell fails the whole sweep.
|
||||
"""
|
||||
total_runs = sum(cell.runs for cell in cells)
|
||||
if max_total_runs is not None and total_runs > max_total_runs:
|
||||
msg = (
|
||||
f"sweep budget exceeded: {total_runs} runs > "
|
||||
f"{max_total_runs} max_total_runs"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
for cell in cells:
|
||||
if cell.runs < 1:
|
||||
msg = (
|
||||
f"cell runs must be >= 1 "
|
||||
f"(scenario={cell.scenario!r}, strategy={cell.strategy!r})"
|
||||
)
|
||||
raise ValueError(msg)
|
||||
with ThreadPoolExecutor(max_workers=self._workers) as pool:
|
||||
futures = [pool.submit(self._run_cell, cell) for cell in cells]
|
||||
results = [future.result() for future in futures]
|
||||
return tuple(results)
|
||||
|
||||
def _run_cell(self, cell: SweepCell) -> SweepCellResult:
|
||||
"""One cell's work: run the encounter and measure the party's danger."""
|
||||
report, outcomes = run_encounter(
|
||||
cell.spec,
|
||||
cell.runs,
|
||||
cell.seed,
|
||||
policy=cell.policy,
|
||||
spell_registry=self._registry,
|
||||
collect_runs=True,
|
||||
)
|
||||
danger = danger_metrics(outcomes, self._party)
|
||||
return SweepCellResult(cell=cell, report=report, danger=danger)
|
||||
|
||||
def run_grid(
|
||||
self,
|
||||
scenarios: Sequence[tuple[str, EncounterSpec]],
|
||||
strategies: Sequence[tuple[str, Policy | None]],
|
||||
*,
|
||||
runs: int,
|
||||
seed: int,
|
||||
max_total_runs: int | None = None,
|
||||
) -> tuple[SweepCellResult, ...]:
|
||||
"""Build the scenario-major (scenario x strategy) grid and run it.
|
||||
|
||||
Cells are built in SCENARIO-major order (for each scenario, for each
|
||||
strategy - mirrors the danger-map table of docs section 16), every
|
||||
cell sharing the same ``runs`` and ``seed``. Names come from the
|
||||
tuples; a ``None`` policy stays ``None`` (default_policy).
|
||||
"""
|
||||
cells = [
|
||||
SweepCell(
|
||||
scenario=scenario_name,
|
||||
strategy=strategy_name,
|
||||
spec=spec,
|
||||
policy=policy,
|
||||
runs=runs,
|
||||
seed=seed,
|
||||
)
|
||||
for scenario_name, spec in scenarios
|
||||
for strategy_name, policy in strategies
|
||||
]
|
||||
return self.run(cells, max_total_runs=max_total_runs)
|
||||
|
||||
|
||||
def format_sweep(results: Sequence[SweepCellResult], party: str) -> str:
|
||||
"""Render a sweep as aligned plain-text columns (ASCII only).
|
||||
|
||||
One row per result, in input order, under a header line and a separator
|
||||
row of dashes. ``party`` is the side whose danger was measured: the Win%
|
||||
column shows that side's win rate (``report.wins.get(party, 0) /
|
||||
report.runs``) as a percentage with one decimal. Column widths are
|
||||
computed from the content; columns are separated by a single space.
|
||||
"""
|
||||
headers = (
|
||||
"Scenario",
|
||||
"Strategy",
|
||||
"Runs",
|
||||
"Win%",
|
||||
"Draws",
|
||||
"Rounds",
|
||||
"P(survive)",
|
||||
"P(TPK)",
|
||||
"P(>=1 death)",
|
||||
"HP lost",
|
||||
"Worst-decile HP",
|
||||
)
|
||||
rows = [
|
||||
(
|
||||
result.cell.scenario,
|
||||
result.cell.strategy,
|
||||
str(result.cell.runs),
|
||||
f"{result.report.wins.get(party, 0) / result.report.runs * 100:.1f}",
|
||||
str(result.report.draws),
|
||||
f"{result.report.avg_rounds:.1f}",
|
||||
f"{result.danger.p_survives:.1%}",
|
||||
f"{result.danger.p_tpk:.1%}",
|
||||
f"{result.danger.p_any_death:.1%}",
|
||||
f"{result.danger.expected_hp_lost:.1f}",
|
||||
f"{result.danger.worst_decile_party_hp:.1f}",
|
||||
)
|
||||
for result in results
|
||||
]
|
||||
widths = [
|
||||
max(len(header), *(len(row[index]) for row in rows))
|
||||
for index, header in enumerate(headers)
|
||||
]
|
||||
|
||||
def _line(values: Sequence[str]) -> str:
|
||||
return " ".join(
|
||||
value.ljust(width) for value, width in zip(values, widths, strict=True)
|
||||
)
|
||||
|
||||
lines = [_line(headers), _line(["-" * width for width in widths])]
|
||||
lines.extend(_line(row) for row in rows)
|
||||
return "\n".join(lines)
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
You are the tactical director for a Pathfinder 1e battle.
|
||||
|
||||
You see the current battle state as a compact observation view, followed by
|
||||
the list of plausible actions the acting combatant could take.
|
||||
Respond with the JSON action sequence described in the schema.
|
||||
+1
@@ -0,0 +1 @@
|
||||
0.1.0-test
|
||||
@@ -0,0 +1,6 @@
|
||||
[
|
||||
{
|
||||
"observation": {"round": 1, "combatant": "goblin", "enemy": "orc", "distance_ft": 10},
|
||||
"actions": [{"kind": "attack", "target_id": "orc"}]
|
||||
}
|
||||
]
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
You are the tactical director for a Pathfinder 1e battle.
|
||||
|
||||
You see the current battle state as a compact observation view, followed by
|
||||
the list of plausible actions the acting combatant could take.
|
||||
Respond with the JSON action sequence described in the schema.
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
Vous êtes le narrateur pour un simulateur de combat Pathfinder 1e.
|
||||
|
||||
Vous recevez un rapport de bataille structuré contenant :
|
||||
- Le rapport d'équilibrage factuel (taux de victoire, bandes de confiance 3σ, attrition)
|
||||
- Les métriques de danger (probabilité de survie, risque TPK, perte de PV attendue, queue pire décile)
|
||||
- La cartographie de robustesse optionnelle (matchups, carte de danger, frontière, sensibilité)
|
||||
- L'observabilité LLM (appels, réparations, fallbacks, usage tokens, coût)
|
||||
|
||||
Rédigez un résumé narratif concis en français pour le maître de jeu. Ton : analytique, évocateur, ancré dans les chiffres. N'inventez PAS de chiffres — citez les statistiques fournies textuellement. PAS de formatage markdown. Texte brut, français uniquement.
|
||||
|
||||
Structurez votre réponse ainsi :
|
||||
1. **Bilan d'équilibrage** — 2-3 phrases synthétisant les taux de victoire et leur signification.
|
||||
2. **Danger et attrition** — 2-3 phrases sur survie/TPK/perte PV/pire décile.
|
||||
3. **Robustesse** (si données présentes) — 1-2 phrases sur matchup/carte de danger/frontière/sensibilité.
|
||||
4. **Observabilité LLM** (si données présentes) — 1 phrase sur appels/réparations/coût.
|
||||
+1
@@ -0,0 +1 @@
|
||||
0.1.0-test
|
||||
@@ -229,7 +229,7 @@ def test_results_visible_in_next_turn_observation() -> None:
|
||||
# the second invocation must already see the first turn's rejection.
|
||||
seen: list[Observation] = []
|
||||
|
||||
def llm_policy(obs: Observation, state: CombatantState) -> tuple[Action, ...]:
|
||||
def llm_policy(obs: Observation) -> tuple[Action, ...]:
|
||||
seen.append(obs)
|
||||
if len(seen) == 1:
|
||||
return (Action(kind="move"),)
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
"""CLI integration test for --llm-report flag (P5.3 T5)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.cli import main
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
|
||||
def _argv_common() -> list[str]:
|
||||
return [
|
||||
"--map", str(MAP),
|
||||
"--side", "players", str(ROOT / "data" / "monsters" / "goblin.json"),
|
||||
"--side", "monsters", str(ROOT / "data" / "monsters" / "orc.json"),
|
||||
]
|
||||
|
||||
|
||||
def test_cli_llm_report_mutually_exclusive_with_transcript() -> None:
|
||||
"""--llm-report and --transcript are mutually exclusive."""
|
||||
argv = [*_argv_common(), "--llm-report", "--transcript"]
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main(argv)
|
||||
assert exc_info.value.code == 2
|
||||
|
||||
|
||||
def test_cli_output_requires_llm_report() -> None:
|
||||
"""--output requires --llm-report."""
|
||||
argv = [*_argv_common(), "--output", "report.md"]
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main(argv)
|
||||
assert exc_info.value.code == 2
|
||||
|
||||
|
||||
def test_cli_llm_report_requires_api_key_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""--llm-report fails gracefully when PF1E_LLM_API_KEY not set."""
|
||||
monkeypatch.delenv("PF1E_LLM_API_KEY", raising=False)
|
||||
monkeypatch.delenv("PF1E_PROMPT_PACK_DIR", raising=False)
|
||||
|
||||
argv = [*_argv_common(), "--llm-report"]
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main(argv)
|
||||
assert exc_info.value.code == 2
|
||||
|
||||
|
||||
def test_cli_llm_report_requires_prompt_pack_dir(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""--llm-report fails gracefully when PF1E_PROMPT_PACK_DIR not set."""
|
||||
monkeypatch.setenv("PF1E_LLM_API_KEY", "test-key")
|
||||
monkeypatch.delenv("PF1E_PROMPT_PACK_DIR", raising=False)
|
||||
|
||||
argv = [*_argv_common(), "--llm-report"]
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main(argv)
|
||||
assert exc_info.value.code == 2
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Tests for format_llm_report (P5.3 T4)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pf1e_simulator.report import build_report_context, format_llm_report
|
||||
from pf1e_simulator.runner import CombatantAttrition, EncounterReport
|
||||
|
||||
|
||||
def _make_encounter() -> EncounterReport:
|
||||
return EncounterReport(
|
||||
runs=100,
|
||||
seed=1,
|
||||
wins={"players": 65, "monsters": 33},
|
||||
draws=2,
|
||||
avg_rounds=7.2,
|
||||
attrition={
|
||||
"goblin": CombatantAttrition(2.45, 0.12, 18.30, 4.20, 0.00),
|
||||
"orc": CombatantAttrition(3.20, 0.25, 28.40, 15.60, 0.00),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_format_llm_report_empty_narrative() -> None:
|
||||
"""Empty LLM text returns only the factual report."""
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
result = format_llm_report(ctx, "")
|
||||
|
||||
assert "=== Rapport d'équilibrage ===" in result
|
||||
assert "Victoires players : 65.0%" in result
|
||||
assert "Récit d'équilibrage" not in result
|
||||
|
||||
|
||||
def test_format_llm_report_whitespace_only_narrative() -> None:
|
||||
"""Whitespace-only LLM text returns only the factual report."""
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
result = format_llm_report(ctx, " \n \t ")
|
||||
|
||||
assert "=== Rapport d'équilibrage ===" in result
|
||||
assert "Récit d'équilibrage" not in result
|
||||
|
||||
|
||||
def test_format_llm_report_with_narrative() -> None:
|
||||
"""Non-empty LLM text appends narrative section after factual report."""
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
narrative = "Le gobelin a surpris l'orc par sa vitesse."
|
||||
result = format_llm_report(ctx, narrative)
|
||||
|
||||
assert "=== Rapport d'équilibrage ===" in result
|
||||
assert "Victoires players : 65.0%" in result
|
||||
assert "=== Récit d'équilibrage (LLM) ===" in result
|
||||
assert narrative in result
|
||||
# factual comes before narrative
|
||||
factual_end = result.index("=== Récit d'équilibrage (LLM) ===")
|
||||
assert "Victoires players" in result[:factual_end]
|
||||
|
||||
|
||||
def test_format_llm_report_strips_narrative_whitespace() -> None:
|
||||
"""Leading/trailing whitespace in narrative is stripped."""
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
result = format_llm_report(ctx, " \n Texte narratif. \n ")
|
||||
|
||||
assert result.endswith("Texte narratif.")
|
||||
assert "=== Récit d'équilibrage (LLM) ===\nTexte narratif." in result
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Package U8 — final validation: an observation-only LLM policy drives the
|
||||
Phase-2 capabilities (U1-U7b) end-to-end through the real engine.
|
||||
|
||||
Each test wires a ``llm_policy`` (``LLMPolicy(observation, state)``) that
|
||||
Each test wires a ``llm_policy`` (``LLMPolicy(observation)``) that
|
||||
reads ONLY the frozen Observation snapshot — never ``engine.*`` — and emits
|
||||
one newly-unblocked tactical decision per capability: ``ready`` (U4),
|
||||
``delay`` (U3), ``total_defense`` (U5a), ``move`` with an explicit
|
||||
@@ -121,7 +121,7 @@ def _counting(decide: Decide) -> LLMPolicy:
|
||||
"""LLMPolicy calling ``decide(id, call_no, obs)`` per turn (obs only)."""
|
||||
calls: dict[str, int] = {}
|
||||
|
||||
def policy(obs: Observation, _state: CombatantState) -> tuple[Action, ...]:
|
||||
def policy(obs: Observation) -> tuple[Action, ...]:
|
||||
cid = str(_me(obs)["id"])
|
||||
n = calls.get(cid, 0)
|
||||
calls[cid] = n + 1
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
"""Tests for LLMReportPolicy (P5.3 T3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.client import FakeLLMClient
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
from pf1e_simulator.report import LLMReportPolicy, build_report_context
|
||||
from pf1e_simulator.runner import CombatantAttrition, EncounterReport
|
||||
|
||||
FIXTURE_DIR = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
|
||||
|
||||
def _make_encounter() -> EncounterReport:
|
||||
return EncounterReport(
|
||||
runs=100,
|
||||
seed=1,
|
||||
wins={"players": 65, "monsters": 33},
|
||||
draws=2,
|
||||
avg_rounds=7.2,
|
||||
attrition={
|
||||
"goblin": CombatantAttrition(2.45, 0.12, 18.30, 4.20, 0.00),
|
||||
"orc": CombatantAttrition(3.20, 0.25, 28.40, 15.60, 0.00),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _make_policy(
|
||||
script: list[str],
|
||||
*,
|
||||
max_retries: int = 1,
|
||||
) -> LLMReportPolicy:
|
||||
pack = PromptPack.load(FIXTURE_DIR)
|
||||
builder = PromptBuilder(pack)
|
||||
expected_version = pack.prompt_version("report")
|
||||
client = FakeLLMClient(script=script, prompt_version=expected_version) # type: ignore[arg-type]
|
||||
return LLMReportPolicy(
|
||||
client=client, builder=builder, prompt_name="report", max_retries=max_retries
|
||||
)
|
||||
|
||||
|
||||
def test_llm_report_policy_returns_text() -> None:
|
||||
"""Policy returns the scripted text from FakeLLMClient."""
|
||||
policy = _make_policy(script=["Rapport narratif de test."])
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == "Rapport narratif de test."
|
||||
assert policy.stats.calls == 1
|
||||
assert policy.stats.retries == 0
|
||||
assert policy.stats.fallbacks == 0
|
||||
|
||||
|
||||
def test_llm_report_policy_retry_on_exception() -> None:
|
||||
"""Policy retries on exception and increments retries counter."""
|
||||
policy = _make_policy(
|
||||
script=[Exception("boom"), "Rapport après retry."], # type: ignore[list-item]
|
||||
max_retries=1,
|
||||
)
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == "Rapport après retry."
|
||||
assert policy.stats.calls == 2
|
||||
assert policy.stats.retries == 1
|
||||
assert policy.stats.fallbacks == 0
|
||||
|
||||
|
||||
def test_llm_report_policy_fallback_after_max_retries() -> None:
|
||||
"""Policy returns empty string and increments fallbacks after max retries."""
|
||||
policy = _make_policy(
|
||||
script=[Exception("boom"), Exception("boom2")], # type: ignore[list-item]
|
||||
max_retries=1,
|
||||
)
|
||||
ctx = build_report_context(encounter=_make_encounter(), spec_name="test")
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == ""
|
||||
assert policy.stats.calls == 2
|
||||
assert policy.stats.retries == 1
|
||||
assert policy.stats.fallbacks == 1
|
||||
|
||||
|
||||
def test_llm_report_policy_prompt_version_mismatch_raises() -> None:
|
||||
"""Policy rejects client with mismatched prompt_version."""
|
||||
pack = PromptPack.load(FIXTURE_DIR)
|
||||
builder = PromptBuilder(pack)
|
||||
expected_version = pack.prompt_version("report")
|
||||
client = FakeLLMClient(script=["x"], prompt_version=expected_version)
|
||||
client.prompt_version = "wrong-version"
|
||||
|
||||
with pytest.raises(ValueError, match="prompt_version"):
|
||||
LLMReportPolicy(client=client, builder=builder, prompt_name="report")
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Unit tests for the Phase-2 objective primitives (objectives.py).
|
||||
|
||||
Synthetic RunOutcome tuples with hand-computed expectations: the metrics
|
||||
are pure frequencies/means over per-run summaries, so every number below is
|
||||
verifiable by eye. No engine, no RNG.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pf1e_simulator.objectives import (
|
||||
danger_metrics,
|
||||
expected_hp_lost,
|
||||
expected_rounds,
|
||||
p_any_death,
|
||||
p_survives,
|
||||
p_tpk,
|
||||
worst_decile_party_hp,
|
||||
)
|
||||
from pf1e_simulator.runner import CombatantOutcome, RunOutcome
|
||||
|
||||
|
||||
def _pc(cid: str, hp: int, hp_max: int = 10) -> CombatantOutcome:
|
||||
dead = hp <= -12
|
||||
return CombatantOutcome(id=cid, side="players", hp=hp, hp_max=hp_max, dead=dead)
|
||||
|
||||
|
||||
def _mob(cid: str, hp: int, hp_max: int = 10) -> CombatantOutcome:
|
||||
dead = hp <= -12
|
||||
return CombatantOutcome(id=cid, side="monsters", hp=hp, hp_max=hp_max, dead=dead)
|
||||
|
||||
|
||||
def _run(rounds: int, *combatants: CombatantOutcome) -> RunOutcome:
|
||||
return RunOutcome(winner=None, rounds=rounds, combatants=combatants)
|
||||
|
||||
|
||||
def test_empty_batch_returns_zero() -> None:
|
||||
assert p_survives((), "players") == 0.0
|
||||
assert p_tpk((), "players") == 0.0
|
||||
assert p_any_death((), "players") == 0.0
|
||||
assert expected_hp_lost((), "players") == 0.0
|
||||
assert expected_rounds(()) == 0.0
|
||||
assert worst_decile_party_hp((), "players") == 0.0
|
||||
|
||||
|
||||
def test_survival_tpk_and_death_rates() -> None:
|
||||
# run 1: both players clean (hp 10/10, 8/10) -> survives, no death
|
||||
# run 2: one player dying (hp -1, NOT dead at -CON 12), one at 3 hp
|
||||
# -> survives (nobody dead), no death counted
|
||||
# run 3: one player dead (hp -12), one at 0 hp -> any death, not TPK
|
||||
# run 4: both dead (hp -13, -12) -> TPK
|
||||
runs = (
|
||||
_run(2, _pc("a", 10), _pc("b", 8), _mob("m", 5)),
|
||||
_run(3, _pc("a", -1), _pc("b", 3)),
|
||||
_run(4, _pc("a", -12), _pc("b", 0)),
|
||||
_run(5, _pc("a", -13), _pc("b", -12)),
|
||||
)
|
||||
assert p_survives(runs, "players") == 2 / 4
|
||||
assert p_any_death(runs, "players") == 2 / 4
|
||||
assert p_tpk(runs, "players") == 1 / 4
|
||||
# monsters absent from a run: vacuously survived, never TPK
|
||||
assert p_survives(runs, "monsters") == 1.0
|
||||
assert p_tpk(runs, "monsters") == 0.0
|
||||
|
||||
|
||||
def test_expected_hp_lost_floors_per_member() -> None:
|
||||
# run 1: a lost 2, b lost 7 (total 9). Run 2: a at -12 counts as the
|
||||
# full 10 lost (clamped), b at 10 lost nothing. Mean of the two totals
|
||||
# is 9.5.
|
||||
runs = (
|
||||
_run(1, _pc("a", 8), _pc("b", 3)),
|
||||
_run(2, _pc("a", -12), _pc("b", 10)),
|
||||
)
|
||||
assert expected_hp_lost(runs, "players") == 9.5
|
||||
assert expected_rounds(runs) == 1.5
|
||||
|
||||
|
||||
def test_worst_decile_takes_the_tail_not_the_mean() -> None:
|
||||
# 20 runs: 19 clean (hp lost 0) + 1 disaster (both dead, 20 lost).
|
||||
# Mean hp lost = 1.0 but the worst decile (2 runs: disaster + one clean)
|
||||
# averages 10 hp remaining - the tail is visible.
|
||||
clean = _run(1, _pc("a", 10), _pc("b", 10))
|
||||
disaster = _run(9, _pc("a", -12), _pc("b", -13))
|
||||
runs = (disaster, *([clean] * 19))
|
||||
assert expected_hp_lost(runs, "players") == 1.0
|
||||
assert worst_decile_party_hp(runs, "players") == (0 + 20) / 2
|
||||
|
||||
|
||||
def test_worst_decile_minimum_one_run() -> None:
|
||||
runs = (_run(1, _pc("a", 4)),)
|
||||
assert worst_decile_party_hp(runs, "players") == 4
|
||||
|
||||
|
||||
def test_danger_metrics_aggregates_every_primitive() -> None:
|
||||
runs = (
|
||||
_run(2, _pc("a", 10), _mob("m", 0)),
|
||||
_run(4, _pc("a", -12), _mob("m", -13)),
|
||||
)
|
||||
m = danger_metrics(runs, "players")
|
||||
assert m.runs == 2
|
||||
assert m.p_survives == 0.5
|
||||
assert m.p_tpk == 0.5
|
||||
assert m.p_any_death == 0.5
|
||||
assert m.expected_hp_lost == 5.0
|
||||
assert m.expected_rounds == 3.0
|
||||
assert m.worst_decile_party_hp == 0.0
|
||||
@@ -9,9 +9,12 @@ Covers the pinned schema:
|
||||
- Terrain/elevation layers.
|
||||
- Frozen copies: mutating live state after construction does not change the
|
||||
already-built Observation.
|
||||
- ``to_dict()`` is JSON-serializable and exposes no ``engine.*`` callables.
|
||||
- ``to_dict()`` is JSON-serializable, carries the pinned ``schema_version``
|
||||
key, and exposes no ``engine.*`` callables.
|
||||
- The ``llm_policy`` adapter builds the Observation at the point of policy
|
||||
invocation and delegates to ``llm_policy(observation, state)``.
|
||||
invocation and delegates to ``llm_policy(observation)`` (D1: observation
|
||||
only); a legacy ``llm_policy(observation, state)`` callable keeps working
|
||||
through the compatibility adapter.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -20,7 +23,13 @@ import json
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.combat import Action, CombatantState, CombatEngine, Observation
|
||||
from pf1e_simulator.combat import (
|
||||
OBSERVATION_SCHEMA_VERSION,
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Observation,
|
||||
)
|
||||
from pf1e_simulator.conditions import SHAKEN
|
||||
from pf1e_simulator.dice import parse_dice
|
||||
from pf1e_simulator.effects import ActiveCondition
|
||||
@@ -441,11 +450,11 @@ def test_observation_frozen_copies() -> None:
|
||||
|
||||
|
||||
def test_llm_policy_wrapper_e2e() -> None:
|
||||
"""Given: a CombatEngine with an llm_policy that reads ONLY the observation
|
||||
"""Given: a CombatEngine with an observation-only llm_policy (D1 signature)
|
||||
When: the battle runs
|
||||
Then: the policy's decisions are honored and the transcript is coherent."""
|
||||
|
||||
def llm_policy(obs: Observation, state: CombatantState) -> tuple:
|
||||
def llm_policy(obs: Observation) -> tuple:
|
||||
# Read only the observation: full_attack if the target is adjacent,
|
||||
# otherwise move toward it.
|
||||
target_id = obs.combatants[obs.pairs[0]["target_index"]]["id"]
|
||||
@@ -470,6 +479,65 @@ def test_llm_policy_wrapper_e2e() -> None:
|
||||
assert any("move" in line for line in result.transcript)
|
||||
|
||||
|
||||
def test_legacy_two_arg_llm_policy_still_works() -> None:
|
||||
"""Given: a pre-D1 llm_policy taking (observation, state)
|
||||
When: the battle runs through the compatibility adapter
|
||||
Then: the policy receives BOTH arguments and its decisions are honored."""
|
||||
seen_states: list[CombatantState] = []
|
||||
|
||||
def legacy_policy(obs: Observation, state: CombatantState) -> tuple:
|
||||
seen_states.append(state)
|
||||
assert state.combatant.id == obs.combatants[obs.acting_combatant]["id"]
|
||||
target_id = obs.combatants[obs.pairs[0]["target_index"]]["id"]
|
||||
return (Action(kind="move", target_id=target_id),)
|
||||
|
||||
hero = make_combatant("hero")
|
||||
foe = make_combatant("foe")
|
||||
hero_state = make_state(hero, "players", (0, 0))
|
||||
foe_state = make_state(foe, "monsters", (0, 3))
|
||||
engine = CombatEngine(
|
||||
ScriptedRng([10, 9, 8, 15, 3, 5]),
|
||||
make_grid(),
|
||||
[hero_state, foe_state],
|
||||
round_cap=1,
|
||||
llm_policy=legacy_policy,
|
||||
)
|
||||
|
||||
result = engine.run()
|
||||
assert seen_states, "legacy policy was never invoked"
|
||||
assert any("move" in line for line in result.transcript)
|
||||
|
||||
|
||||
def test_observation_schema_version_pinned() -> None:
|
||||
"""Given: a built Observation
|
||||
When: to_dict() is called
|
||||
Then: the top-level key set is exactly the pinned keys and
|
||||
schema_version == OBSERVATION_SCHEMA_VERSION, JSON-serializable."""
|
||||
hero = make_combatant("hero")
|
||||
foe = make_combatant("foe")
|
||||
hero_state = make_state(hero, "players", (0, 0))
|
||||
foe_state = make_state(foe, "monsters", (0, 3))
|
||||
engine = make_engine([hero_state, foe_state])
|
||||
|
||||
data = engine._build_observation(hero_state).to_dict()
|
||||
assert data["schema_version"] == OBSERVATION_SCHEMA_VERSION
|
||||
assert set(data) == {
|
||||
"schema_version",
|
||||
"round",
|
||||
"acting_combatant",
|
||||
"combatants",
|
||||
"pairs",
|
||||
"terrain",
|
||||
"elevation",
|
||||
"threatened",
|
||||
"ranged_threatened",
|
||||
"terrain_semantics",
|
||||
"condition_definitions",
|
||||
"initiative_order",
|
||||
}
|
||||
json.dumps(data) # must not raise
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 12. policy + llm_policy → ValueError
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1051,7 +1051,8 @@ def test_observation_enriched_terrain_semantics_no_callable_no_share() -> None:
|
||||
def test_observation_enriched_e5_existing_keys_preserved() -> None:
|
||||
"""Given: an Observation built for the L5 fields
|
||||
When: the top-level dict is inspected
|
||||
Then: all pre-existing top-level keys remain (round, acting_combatant,
|
||||
Then: all pre-existing top-level keys remain (schema_version, round,
|
||||
acting_combatant,
|
||||
combatants, pairs, terrain, elevation) and the new keys
|
||||
(threatened, ranged_threatened, terrain_semantics,
|
||||
condition_definitions, initiative_order) are added — exact key set."""
|
||||
@@ -1063,6 +1064,7 @@ def test_observation_enriched_e5_existing_keys_preserved() -> None:
|
||||
|
||||
obs = _snapshot(engine, hero_state)
|
||||
assert set(obs) == {
|
||||
"schema_version",
|
||||
"round",
|
||||
"acting_combatant",
|
||||
"combatants",
|
||||
@@ -1138,7 +1140,7 @@ def _best_usable_attack(obs: Observation) -> dict[str, object] | None:
|
||||
return best
|
||||
|
||||
|
||||
def _enriched_llm_policy(obs: Observation, state: CombatantState) -> tuple[Action, ...]:
|
||||
def _enriched_llm_policy(obs: Observation) -> tuple[Action, ...]:
|
||||
"""Read ONLY the observation — never state.combatant.* nor engine.*."""
|
||||
cast = _castable_spell_action(obs)
|
||||
if cast is not None:
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
"""Phase-2 adversary compilation tests (docs/phase-2-llm-architecture.md 10 P4.1).
|
||||
|
||||
Deterministic, offline: no network, no LLM. Pins the ``AdversarySpec`` /
|
||||
``compile_adversary`` / ``compile_matchup`` contract (default-policy identity,
|
||||
per-side routing on a real ``CombatEngine``, empty ``party_side`` rejection),
|
||||
the additive ``"healer"`` target priority (prefer the enemy caster, fall back
|
||||
to ``weakest_ac``), and ``SideSpec.aware`` landing on the right side's
|
||||
combatants plus its SRD "Surprise" round effect. Conventions follow
|
||||
tests/test_phase2_search.py: small n (runs <= 5), fixed seed, ROOT-relative
|
||||
path constants.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.adversary import (
|
||||
AdversarySpec,
|
||||
adversary_name,
|
||||
compile_adversary,
|
||||
compile_matchup,
|
||||
)
|
||||
from pf1e_simulator.combat import CombatantState, CombatEngine, default_policy
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import build_states, run_encounter
|
||||
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec, build_encounter
|
||||
from pf1e_simulator.strategy import StrategySpec, compile_strategy, select_target
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.models import Combatant
|
||||
from pf1e_simulator.runner import EncounterSpec
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
|
||||
ORC = ROOT / "data" / "monsters" / "orc.json"
|
||||
NAIRDA = ROOT / "data" / "pcs" / "nairda.json"
|
||||
|
||||
RUNS = 5
|
||||
SEED = 7
|
||||
ROUND_CAP = 20
|
||||
PARTY = "players"
|
||||
|
||||
_MAP_SPEC = load_map(MAP)
|
||||
|
||||
|
||||
def _c(path: Path) -> Combatant:
|
||||
return load_combatant(path)
|
||||
|
||||
|
||||
def _scenario(*, monsters_aware: bool | None = None) -> ScenarioSpec:
|
||||
"""3 goblins (zone A) vs 2 orcs (zone B), optionally with side awareness."""
|
||||
return ScenarioSpec(
|
||||
name="goblins-vs-orcs",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(
|
||||
name=PARTY,
|
||||
zone="A",
|
||||
combatant_files=(GOBLIN, GOBLIN, GOBLIN),
|
||||
),
|
||||
SideSpec(
|
||||
name="monsters",
|
||||
zone="B",
|
||||
combatant_files=(ORC, ORC),
|
||||
aware=monsters_aware,
|
||||
),
|
||||
),
|
||||
round_cap=ROUND_CAP,
|
||||
)
|
||||
|
||||
|
||||
def _encounter() -> EncounterSpec:
|
||||
return build_encounter(_scenario())
|
||||
|
||||
|
||||
def _state(
|
||||
combatant: Combatant, side: str, pos: tuple[int, int], *, hp: int | None = None
|
||||
) -> CombatantState:
|
||||
"""A bare runtime state over a loaded combatant (no battle run)."""
|
||||
return CombatantState(
|
||||
combatant=combatant,
|
||||
side=side,
|
||||
pos=pos,
|
||||
hp=combatant.hp_max if hp is None else hp,
|
||||
resources_left={},
|
||||
)
|
||||
|
||||
|
||||
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
|
||||
"""A live engine over the sample arena with the given states (no battle run)."""
|
||||
return CombatEngine(SeededRng(SEED), Grid.from_spec(_MAP_SPEC), states)
|
||||
|
||||
|
||||
# --- identity ---------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compile_adversary_default_is_default_policy() -> None:
|
||||
"""The all-default AdversarySpec compiles to default_policy by identity."""
|
||||
assert compile_adversary(AdversarySpec()) is default_policy
|
||||
|
||||
|
||||
def test_compile_matchup_defaults_is_default_policy() -> None:
|
||||
"""Two all-default specs return default_policy itself, not a wrapper."""
|
||||
assert compile_matchup(StrategySpec(), AdversarySpec()) is default_policy
|
||||
|
||||
|
||||
def test_adversary_name_mirrors_strategy_name() -> None:
|
||||
"""Labels render non-default levers in field order; the name field is excluded."""
|
||||
assert adversary_name(AdversarySpec()) == "default"
|
||||
assert adversary_name(AdversarySpec(target_priority="healer")) == "healer"
|
||||
assert (
|
||||
adversary_name(AdversarySpec(name="custom", spell_policy="none"))
|
||||
== "spell_policy=none"
|
||||
)
|
||||
assert (
|
||||
adversary_name(
|
||||
AdversarySpec(target_priority="weakest_ac", retreat_hp_fraction=0.5)
|
||||
)
|
||||
== "weakest_ac, retreat=0.5"
|
||||
)
|
||||
|
||||
|
||||
# --- validation ------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compile_matchup_empty_party_side_raises() -> None:
|
||||
"""An empty party_side is a ValueError (pre-assigned msg, EM101/EM102)."""
|
||||
with pytest.raises(ValueError, match="party_side"):
|
||||
compile_matchup(StrategySpec(), AdversarySpec(), party_side="")
|
||||
|
||||
|
||||
# --- routing ----------------------------------------------------------------
|
||||
|
||||
|
||||
def test_compile_matchup_routes_by_side() -> None:
|
||||
"""On a real engine, each side of the matchup delegates to its own policy."""
|
||||
party_spec = StrategySpec(name="party-focus", target_priority="weakest_ac")
|
||||
adversary_spec = AdversarySpec(name="adv-focus", target_priority="lowest_hp")
|
||||
matchup = compile_matchup(party_spec, adversary_spec)
|
||||
assert matchup is not default_policy
|
||||
|
||||
encounter = _encounter()
|
||||
states = build_states(encounter)
|
||||
engine = CombatEngine(SeededRng(SEED), Grid.from_spec(encounter.map), states)
|
||||
party_state = next(s for s in states if s.side == PARTY)
|
||||
monster_state = next(s for s in states if s.side == "monsters")
|
||||
|
||||
assert matchup(engine, party_state) == compile_strategy(party_spec)(
|
||||
engine, party_state
|
||||
)
|
||||
assert matchup(engine, monster_state) == compile_adversary(adversary_spec)(
|
||||
engine, monster_state
|
||||
)
|
||||
|
||||
|
||||
def test_matchup_and_healer_strategy_run_deterministically() -> None:
|
||||
"""A healer spec compiles and runs evaluate-style cells bit-identically."""
|
||||
healer = StrategySpec(name="healer-party", target_priority="healer")
|
||||
policy = compile_strategy(healer)
|
||||
assert policy is not default_policy
|
||||
matchup = compile_matchup(healer, AdversarySpec(target_priority="weakest_ac"))
|
||||
|
||||
spec = _encounter()
|
||||
_, first = run_encounter(spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True)
|
||||
_, again = run_encounter(spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True)
|
||||
assert first == again
|
||||
_, matchup_a = run_encounter(
|
||||
spec, runs=RUNS, seed=SEED, policy=matchup, collect_runs=True
|
||||
)
|
||||
_, matchup_b = run_encounter(
|
||||
spec, runs=RUNS, seed=SEED, policy=matchup, collect_runs=True
|
||||
)
|
||||
assert matchup_a == matchup_b
|
||||
|
||||
|
||||
# --- healer targeting -------------------------------------------------------
|
||||
|
||||
|
||||
def test_healer_prefers_the_enemy_caster() -> None:
|
||||
"""With a monster-side caster (nairda) among non-casters, healer picks her."""
|
||||
actor = _state(_c(GOBLIN), PARTY, (1, 1))
|
||||
nairda = _state(_c(NAIRDA), "monsters", (1, 8))
|
||||
orc = _state(_c(ORC), "monsters", (1, 16))
|
||||
assert nairda.combatant.spells # caster precondition (nairda.json)
|
||||
engine = _mini_engine([actor, nairda, orc])
|
||||
|
||||
assert select_target(engine, actor, "healer") is nairda
|
||||
# Discriminator: weakest_ac sees equal AC 13 and prefers the wounded orc.
|
||||
assert select_target(engine, actor, "weakest_ac") is orc
|
||||
|
||||
|
||||
def test_healer_falls_back_to_weakest_ac_without_casters() -> None:
|
||||
"""No active caster on the enemy side -> identical to weakest_ac."""
|
||||
actor = _state(_c(GOBLIN), PARTY, (1, 1))
|
||||
orc = _state(_c(ORC), "monsters", (1, 8))
|
||||
goblin_enemy = _state(_c(GOBLIN), "monsters", (1, 16))
|
||||
assert not orc.combatant.spells
|
||||
assert not goblin_enemy.combatant.spells
|
||||
engine = _mini_engine([actor, orc, goblin_enemy])
|
||||
|
||||
healer = select_target(engine, actor, "healer")
|
||||
assert healer is select_target(engine, actor, "weakest_ac")
|
||||
assert healer is orc # AC 13 < goblin AC 16
|
||||
|
||||
|
||||
# --- SideSpec.aware (SRD "Surprise") ---------------------------------------
|
||||
|
||||
|
||||
def test_side_spec_aware_applies_per_side() -> None:
|
||||
"""aware=False lands on that side's combatants only; the other side keeps True."""
|
||||
encounter = build_encounter(_scenario(monsters_aware=False))
|
||||
players, monsters = encounter.sides
|
||||
assert all(c.aware is True for c in players.combatants)
|
||||
assert all(c.aware is False for c in monsters.combatants)
|
||||
|
||||
states = build_states(encounter)
|
||||
assert all(s.combatant.aware is True for s in states if s.side == PARTY)
|
||||
assert all(s.combatant.aware is False for s in states if s.side == "monsters")
|
||||
|
||||
untouched = build_encounter(_scenario())
|
||||
assert all(c.aware is True for side in untouched.sides for c in side.combatants)
|
||||
|
||||
|
||||
def test_side_spec_aware_triggers_surprise_round() -> None:
|
||||
"""Only some aware -> a surprise round runs; the default (aware=None) has none."""
|
||||
surprise = _encounter_states_and_run(_scenario(monsters_aware=False))
|
||||
assert "surprise round" in surprise
|
||||
|
||||
default = _encounter_states_and_run(_scenario())
|
||||
assert "surprise round" not in default
|
||||
|
||||
|
||||
def _encounter_states_and_run(scenario: ScenarioSpec) -> tuple[str, ...]:
|
||||
"""Run one seeded battle of `scenario` and return its transcript."""
|
||||
encounter = build_encounter(scenario)
|
||||
engine = CombatEngine(
|
||||
SeededRng(SEED),
|
||||
Grid.from_spec(encounter.map),
|
||||
build_states(encounter),
|
||||
round_cap=ROUND_CAP,
|
||||
)
|
||||
return engine.run().transcript
|
||||
@@ -0,0 +1,198 @@
|
||||
"""Phase-2 candidate actions (docs/phase-2-llm-architecture.md P0.3).
|
||||
|
||||
``candidates.candidates`` derives a deterministic menu of plausible
|
||||
``Action`` proposals from the ``Observation`` snapshot's pairs. This module
|
||||
pins the contract: a normal turn yields a non-empty, capped menu whose kinds
|
||||
stay within the declared set; the menu is deterministic; no candidate ever
|
||||
targets an ally (the pairs cover every other combatant, so the side filter is
|
||||
what keeps the menu hostile-only); a weaponless combatant still gets
|
||||
movement/defensive options; an enemy in weapon reach gets attack/full_attack;
|
||||
and harmless spells (cure, buffs) target the caster rather than an enemy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.candidates import _CANDIDATE_KINDS, MAX_CANDIDATES, candidates
|
||||
from pf1e_simulator.combat import (
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Observation,
|
||||
default_policy,
|
||||
)
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, build_states
|
||||
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
|
||||
class _CaptureAll:
|
||||
"""Policy wrapper that snapshots every observation, then delegates."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.observations: dict[str, list[Observation]] = {}
|
||||
|
||||
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
obs = engine._build_observation(state)
|
||||
self.observations.setdefault(state.combatant.id, []).append(obs)
|
||||
return default_policy(engine, state)
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _representative_spec() -> EncounterSpec:
|
||||
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars)."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(
|
||||
name="players",
|
||||
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
|
||||
zone="A",
|
||||
),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=(
|
||||
_c("monsters/lich.json"),
|
||||
_c("monsters/thrune_agent.json"),
|
||||
_c("monsters/orc.json"),
|
||||
_c("monsters/goblin.json"),
|
||||
),
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _weaponless_spec() -> EncounterSpec:
|
||||
"""1v1: a goblin with no usable weapons against an orc."""
|
||||
goblin = _c("monsters/goblin.json").model_copy(update={"attacks": ()})
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(name="players", combatants=(goblin,), zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _run_observations(
|
||||
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
|
||||
) -> dict[str, list[Observation]]:
|
||||
grid = Grid.from_spec(spec.map)
|
||||
states = build_states(spec)
|
||||
cap = _CaptureAll()
|
||||
CombatEngine(
|
||||
SeededRng(seed),
|
||||
grid,
|
||||
states,
|
||||
round_cap=spec.round_cap,
|
||||
policy=cap.policy,
|
||||
spell_registry=registry,
|
||||
).run()
|
||||
return cap.observations
|
||||
|
||||
|
||||
def _pair_target_side(pair: dict[str, object], obs: Observation) -> object:
|
||||
"""Side of the pair's target combatant, or None when out of range."""
|
||||
index = pair.get("target_index")
|
||||
if not isinstance(index, int) or not (0 <= index < len(obs.combatants)):
|
||||
return None
|
||||
return obs.combatants[index].get("side")
|
||||
|
||||
|
||||
def test_candidates_normal_turn_non_empty_and_supported_kinds() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_representative_spec(), 1, registry)
|
||||
obs = next(obs_list[0] for obs_list in observations.values())
|
||||
menu = candidates(obs)
|
||||
assert menu
|
||||
assert len(menu) <= MAX_CANDIDATES
|
||||
kinds = {a.kind for a in menu}
|
||||
assert kinds <= _CANDIDATE_KINDS
|
||||
assert {"move", "total_defense", "delay"} <= kinds
|
||||
|
||||
|
||||
def test_candidates_deterministic() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_representative_spec(), 1, registry)
|
||||
obs = next(obs_list[0] for obs_list in observations.values())
|
||||
assert candidates(obs) == candidates(obs)
|
||||
|
||||
|
||||
def test_candidates_no_allies_targeted() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_representative_spec(), 1, registry)
|
||||
for obs_list in observations.values():
|
||||
for obs in obs_list:
|
||||
acting = obs.combatants[obs.acting_combatant]
|
||||
acting_side = acting["side"]
|
||||
acting_id = acting["id"]
|
||||
for action in candidates(obs):
|
||||
if action.target_id is None or action.target_id == acting_id:
|
||||
continue
|
||||
target = next(
|
||||
(c for c in obs.combatants if c["id"] == action.target_id), None
|
||||
)
|
||||
assert target is not None
|
||||
assert target["side"] != acting_side
|
||||
|
||||
|
||||
def test_candidates_weaponless_gets_movement_and_defense() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_weaponless_spec(), 1, registry)
|
||||
obs = observations["goblin"][0]
|
||||
kinds = {a.kind for a in candidates(obs)}
|
||||
assert {"move", "total_defense", "ready", "delay"} <= kinds
|
||||
assert not ({"attack", "full_attack", "charge"} & kinds)
|
||||
|
||||
|
||||
def test_candidates_in_range_gets_attack() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_representative_spec(), 1, registry)
|
||||
for obs_list in observations.values():
|
||||
for obs in obs_list:
|
||||
acting = obs.combatants[obs.acting_combatant]
|
||||
acting_side = acting["side"]
|
||||
enemy_pairs = [
|
||||
p for p in obs.pairs if _pair_target_side(p, obs) != acting_side
|
||||
]
|
||||
if not any(p.get("weapon_for") is not None for p in enemy_pairs):
|
||||
continue
|
||||
kinds = {a.kind for a in candidates(obs)}
|
||||
assert "attack" in kinds
|
||||
assert "full_attack" in kinds
|
||||
return
|
||||
msg = "no observation with an enemy in weapon reach found"
|
||||
raise AssertionError(msg)
|
||||
|
||||
|
||||
def test_candidates_harmless_spells_target_self() -> None:
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
observations = _run_observations(_representative_spec(), 1, registry)
|
||||
obs = observations["oni"][0]
|
||||
acting = obs.combatants[obs.acting_combatant]
|
||||
acting_id = acting["id"]
|
||||
harmless_casts = [
|
||||
a
|
||||
for a in candidates(obs)
|
||||
if a.kind == "cast_spell"
|
||||
and a.spell_name in {"Guidance", "Shield of Faith", "Stabilize"}
|
||||
]
|
||||
assert harmless_casts
|
||||
assert all(a.target_id == acting_id for a in harmless_casts)
|
||||
@@ -0,0 +1,378 @@
|
||||
"""Phase-2 LLM client layer (docs/phase-2 P1.1/P1.2, decisions D6/D10).
|
||||
|
||||
``client.py`` wraps every LLM call with the shared safeguards: composite
|
||||
cache (D6: model, params, prompt_version, schema_version, prompt hash),
|
||||
cost/token budgets + circuit breaker (P1.2, section 13), retry/backoff on
|
||||
transient provider failures, and ``FakeLLMClient`` (P1.1) to drive battles
|
||||
fully offline. These tests pin the ledger, the budgets and the breaker
|
||||
behaviour without any network.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.client import (
|
||||
BudgetExceededError,
|
||||
CircuitBreaker,
|
||||
CircuitOpenError,
|
||||
ClientParams,
|
||||
ClientStats,
|
||||
Completion,
|
||||
CostBudget,
|
||||
FakeLLMClient,
|
||||
LLMClient,
|
||||
Pricing,
|
||||
ResponseCache,
|
||||
RetryExhaustedError,
|
||||
ScriptExhaustedError,
|
||||
TokenUsage,
|
||||
estimate_tokens,
|
||||
response_cache_key,
|
||||
)
|
||||
|
||||
_PARAMS = ClientParams()
|
||||
_PROMPT = "observation rendered prompt v1"
|
||||
|
||||
|
||||
class _UnusedClient(LLMClient):
|
||||
"""Concrete client whose provider is never invoked (validation tests)."""
|
||||
|
||||
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
|
||||
del prompt, params # validation-only: the provider never runs
|
||||
msg = "provider must never be called"
|
||||
raise AssertionError(msg)
|
||||
|
||||
|
||||
# ── composite cache key (D6) ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_cache_key_is_composite_and_deterministic() -> None:
|
||||
"""Each D6 component changes the key; identical inputs are stable."""
|
||||
base = response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
== base
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="other",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
!= base
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="m",
|
||||
params=ClientParams(temperature=0.7),
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
!= base
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:9.9.9:nl:xyz",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
!= base
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=2,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
!= base
|
||||
)
|
||||
assert (
|
||||
response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt="a different observation",
|
||||
)
|
||||
!= base
|
||||
)
|
||||
|
||||
|
||||
def test_response_cache_round_trip_and_counters() -> None:
|
||||
"""get/put round-trip; hits and misses are counted separately."""
|
||||
cache = ResponseCache(shard_count=4)
|
||||
key = response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="pack:1.0.0:nl:abc",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
assert cache.get(key) is None
|
||||
assert cache.hits == 0
|
||||
assert cache.misses == 1
|
||||
cache.put(key, "action json 1")
|
||||
assert cache.get(key) == "action json 1"
|
||||
assert cache.hits == 1
|
||||
assert cache.misses == 1
|
||||
|
||||
|
||||
def test_response_cache_clear_drops_entries_and_counters() -> None:
|
||||
"""clear() empties every shard and resets the counters (tests only)."""
|
||||
cache = ResponseCache(shard_count=2)
|
||||
key = response_cache_key(
|
||||
model="m",
|
||||
params=_PARAMS,
|
||||
prompt_version="p:1:nl:a",
|
||||
schema_version=1,
|
||||
prompt=_PROMPT,
|
||||
)
|
||||
cache.put(key, "action json 1")
|
||||
assert cache.get(key) == "action json 1"
|
||||
cache.clear()
|
||||
assert cache.get(key) is None
|
||||
assert cache.hits == 0
|
||||
assert cache.misses == 1
|
||||
|
||||
|
||||
def test_response_cache_requires_positive_shard_count() -> None:
|
||||
"""A zero shard count is a data error, not a silent default."""
|
||||
with pytest.raises(ValueError, match="shard_count"):
|
||||
ResponseCache(shard_count=0)
|
||||
|
||||
|
||||
# ── FakeLLMClient: scripted offline behaviour (P1.1) ───────────────────────
|
||||
|
||||
|
||||
def test_fake_serves_script_in_order() -> None:
|
||||
"""The fake returns scripted responses one per successful call."""
|
||||
client = FakeLLMClient(script=("action a", "action b"))
|
||||
assert client.complete("observation one") == "action a"
|
||||
assert client.complete("observation two") == "action b"
|
||||
assert client.stats.calls == 2
|
||||
|
||||
|
||||
def test_fake_script_exhaustion_raises_typed_error() -> None:
|
||||
"""A call past the script is a test-authoring bug, loudly reported."""
|
||||
client = FakeLLMClient(script=("action a",))
|
||||
assert client.complete("observation one") == "action a"
|
||||
with pytest.raises(ScriptExhaustedError, match="script exhausted"):
|
||||
client.complete("observation two")
|
||||
|
||||
|
||||
def test_fake_reports_estimated_token_usage() -> None:
|
||||
"""Token usage is estimated from character counts (no provider metric)."""
|
||||
client = FakeLLMClient(script=("0123456789",))
|
||||
client.complete(_PROMPT)
|
||||
stats = client.stats
|
||||
assert stats.prompt_tokens == estimate_tokens(_PROMPT)
|
||||
assert stats.completion_tokens == estimate_tokens("0123456789")
|
||||
assert isinstance(stats, ClientStats)
|
||||
|
||||
|
||||
# ── cache integration on the client (D6) ───────────────────────────────────
|
||||
|
||||
|
||||
def test_client_cache_hit_consumes_no_script() -> None:
|
||||
"""A cached response is returned without a provider call."""
|
||||
client = FakeLLMClient(script=("action a", "action b"))
|
||||
first = client.complete(_PROMPT)
|
||||
second = client.complete(_PROMPT)
|
||||
assert first == second == "action a"
|
||||
assert client.calls == 1 # the second call came from the cache
|
||||
assert client.cache_misses == 1
|
||||
assert client.cache_hits == 1
|
||||
|
||||
|
||||
def test_client_cache_miss_on_different_prompt() -> None:
|
||||
"""A different prompt is a different cache key: script advances."""
|
||||
client = FakeLLMClient(script=("action a", "action b"))
|
||||
assert client.complete(_PROMPT) == "action a"
|
||||
assert client.complete("another observation") == "action b"
|
||||
assert client.calls == 2
|
||||
|
||||
|
||||
def test_client_cache_respects_prompt_version() -> None:
|
||||
"""The composite key includes prompt_version: packs do not collide."""
|
||||
cache = ResponseCache()
|
||||
a = FakeLLMClient(
|
||||
script=("action a",),
|
||||
prompt_version="pack:1.0.0:nl:aaa",
|
||||
cache=cache,
|
||||
)
|
||||
b = FakeLLMClient(
|
||||
script=("action b",),
|
||||
prompt_version="pack:1.0.0:nl:bbb",
|
||||
cache=cache,
|
||||
)
|
||||
assert a.complete(_PROMPT) == "action a"
|
||||
assert b.complete(_PROMPT) == "action b"
|
||||
assert a.calls == 1
|
||||
assert b.calls == 1
|
||||
|
||||
|
||||
def test_client_cache_respects_model() -> None:
|
||||
"""The composite key includes the model name."""
|
||||
cache = ResponseCache()
|
||||
a = FakeLLMClient(script=("action a",), model="model-a", cache=cache)
|
||||
b = FakeLLMClient(script=("action b",), model="model-b", cache=cache)
|
||||
assert a.complete(_PROMPT) == "action a"
|
||||
assert b.complete(_PROMPT) == "action b"
|
||||
assert a.calls == 1
|
||||
assert b.calls == 1
|
||||
|
||||
|
||||
# ── retry/backoff (P1.2) ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_retry_recovers_after_transient_failures() -> None:
|
||||
"""ProviderError is retried; the script is only consumed by successes."""
|
||||
client = FakeLLMClient(
|
||||
script=("action a",),
|
||||
failures=2,
|
||||
retries=3,
|
||||
)
|
||||
assert client.complete(_PROMPT) == "action a"
|
||||
assert client.calls == 3 # two failures + one success
|
||||
|
||||
|
||||
def test_retry_exhausted_after_too_many_failures() -> None:
|
||||
"""Past the retry budget, a typed RetryExhaustedError is raised."""
|
||||
client = FakeLLMClient(script=("action a",), failures=5, retries=2)
|
||||
with pytest.raises(RetryExhaustedError, match="after 3 attempt"):
|
||||
client.complete(_PROMPT)
|
||||
|
||||
|
||||
def test_retry_counter_requires_positive_value() -> None:
|
||||
"""A negative retry count is a data error."""
|
||||
with pytest.raises(ValueError, match="retries"):
|
||||
FakeLLMClient(script=("a",), retries=-1)
|
||||
|
||||
|
||||
# ── circuit breaker (P1.2, section 13) ─────────────────────────────────────
|
||||
|
||||
|
||||
def test_breaker_opens_refuses_probes_and_recovers() -> None:
|
||||
"""Open after N failures: refusals, then a probe that closes on success."""
|
||||
breaker = CircuitBreaker(failure_threshold=2, open_refusals=1)
|
||||
client = FakeLLMClient(
|
||||
script=("action a",),
|
||||
failures=2, # needs two attempts before real service
|
||||
retries=0,
|
||||
breaker=breaker,
|
||||
)
|
||||
with pytest.raises(RetryExhaustedError):
|
||||
client.complete(_PROMPT) # failure 1: not open yet
|
||||
with pytest.raises(RetryExhaustedError):
|
||||
client.complete(_PROMPT) # failure 2: threshold reached, opens
|
||||
assert breaker.is_open
|
||||
with pytest.raises(CircuitOpenError, match="circuit breaker open"):
|
||||
client.complete(_PROMPT) # refused
|
||||
assert client.complete(_PROMPT) == "action a" # probe succeeds now
|
||||
assert not breaker.is_open
|
||||
|
||||
|
||||
def test_breaker_probe_failure_reopens() -> None:
|
||||
"""A failed probe reopens the breaker for another refusal window."""
|
||||
breaker = CircuitBreaker(failure_threshold=1, open_refusals=1)
|
||||
client = FakeLLMClient(
|
||||
script=("action a",),
|
||||
failures=10, # every attempt fails, probe included
|
||||
retries=0,
|
||||
breaker=breaker,
|
||||
)
|
||||
with pytest.raises(RetryExhaustedError):
|
||||
client.complete(_PROMPT)
|
||||
assert breaker.is_open
|
||||
with pytest.raises(CircuitOpenError):
|
||||
client.complete(_PROMPT)
|
||||
with pytest.raises(RetryExhaustedError):
|
||||
client.complete(_PROMPT) # the probe fails
|
||||
assert breaker.is_open
|
||||
with pytest.raises(CircuitOpenError):
|
||||
client.complete(_PROMPT) # reopened: refuses again
|
||||
|
||||
|
||||
def test_breaker_validation() -> None:
|
||||
"""Breaker thresholds must be sensible numbers."""
|
||||
with pytest.raises(ValueError, match="failure_threshold"):
|
||||
CircuitBreaker(failure_threshold=0)
|
||||
with pytest.raises(ValueError, match="open_refusals"):
|
||||
CircuitBreaker(open_refusals=-1)
|
||||
|
||||
|
||||
# ── cost/token budgets (P1.2) ──────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_call_budget_ceiling() -> None:
|
||||
"""max_calls refuses further calls with a typed error."""
|
||||
client = FakeLLMClient(
|
||||
script=("a", "b", "c"),
|
||||
budget=CostBudget(max_calls=1),
|
||||
)
|
||||
assert client.complete(_PROMPT) == "a"
|
||||
with pytest.raises(BudgetExceededError, match="call budget exceeded"):
|
||||
client.complete("a different observation")
|
||||
|
||||
|
||||
def test_token_budget_ceiling() -> None:
|
||||
"""A completion over max_total_tokens is rejected and not cached."""
|
||||
client = FakeLLMClient(
|
||||
script=("x" * 400,), # ~100 estimated completion tokens
|
||||
budget=CostBudget(max_total_tokens=50),
|
||||
)
|
||||
with pytest.raises(BudgetExceededError, match="token budget exceeded"):
|
||||
client.complete(_PROMPT)
|
||||
|
||||
|
||||
def test_cost_budget_ceiling_and_ledger() -> None:
|
||||
"""Estimated USD cost is accumulated and capped by max_cost_usd."""
|
||||
pricing = Pricing(prompt_per_1k_usd=1.0, completion_per_1k_usd=1.0)
|
||||
long_prompt = "p" * 2000 # ~500 prompt tokens ≈ 0.5 USD
|
||||
client = FakeLLMClient(
|
||||
script=("c" * 2000, "c" * 2000), # ~500 completion tokens each ≈ 0.5 USD
|
||||
pricing=pricing,
|
||||
budget=CostBudget(max_cost_usd=1.0),
|
||||
)
|
||||
assert client.complete(long_prompt) == "c" * 2000
|
||||
assert client.cost_usd == pytest.approx(1.0, abs=1e-6)
|
||||
with pytest.raises(BudgetExceededError, match="cost budget exceeded"):
|
||||
client.complete("q" * 2000)
|
||||
assert client.cost_usd == pytest.approx(2.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_backoff_must_be_positive() -> None:
|
||||
"""A negative backoff is a data error (validation-only client)."""
|
||||
with pytest.raises(ValueError, match="backoff_base_s"):
|
||||
_UnusedClient(
|
||||
model="m",
|
||||
prompt_version="p:1:nl:a",
|
||||
backoff_base_s=-1.0,
|
||||
)
|
||||
|
||||
|
||||
def test_token_usage_total() -> None:
|
||||
"""TokenUsage.total is the sum of both counters."""
|
||||
usage = TokenUsage(prompt_tokens=3, completion_tokens=4)
|
||||
assert usage.total == 7
|
||||
assert estimate_tokens("abcd") >= 1
|
||||
@@ -0,0 +1,193 @@
|
||||
"""Phase-2 danger-cartography spike (docs/phase-2-llm-architecture.md 3.1/3.2, 10 P3/P4).
|
||||
|
||||
Proof of value WITHOUT any LLM: the same party measured against the default
|
||||
monster policy and against the strengthened focus-fire policy (spike module
|
||||
pf1e_simulator.phase2_spike). A danger only exists against an adversary that
|
||||
plays to win, so the DELTA between the two columns is the danger the default
|
||||
Monte Carlo report hides. Prints a compact per-scenario block (party win
|
||||
rate +/- 3 sigma, P(TPK), P(>=1 death), expected hp lost, rounds, worst
|
||||
decile hp) and pins loose assertions: everything is deterministic under a
|
||||
fixed seed, so the pinned inequalities are exact reproductions, not noise.
|
||||
|
||||
Scenarios (sample arena, 100 runs, fixed seed):
|
||||
warband - 4 level-4 PCs vs a thrune agent + 2 orcs + a goblin: the party
|
||||
"wins" either way, but focus fire nearly doubles P(>=1 death).
|
||||
lich - 2 level-2/4 PCs vs the lich + an orc: the party loses both
|
||||
ways, but against the default policy nobody ever actually
|
||||
DIES (downed at hp<0, above -CON); focus fire converts defeat
|
||||
into real deaths and a nonzero TPK rate.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.combat import CombatantState, CombatEngine, default_policy
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.objectives import DangerMetrics, danger_metrics
|
||||
from pf1e_simulator.phase2_spike import focus_fire_policy
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import EncounterSpec, RunOutcome, Side, run_encounter
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
RUNS = 100
|
||||
SEED = 11
|
||||
ROUND_CAP = 20
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _spec(pcs: list[str], monsters: list[str]) -> EncounterSpec:
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=ROUND_CAP,
|
||||
sides=(
|
||||
Side(name="players", combatants=tuple(_c(p) for p in pcs), zone="A"),
|
||||
Side(name="monsters", combatants=tuple(_c(p) for p in monsters), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
WARBAND = _spec(
|
||||
["pcs/jeanne.json", "pcs/tammara.json", "pcs/harvie.json", "pcs/ierlieth.json"],
|
||||
["monsters/thrune_agent.json", "monsters/orc.json", "monsters/orc.json",
|
||||
"monsters/goblin.json"],
|
||||
)
|
||||
LICH = _spec(
|
||||
["pcs/harvie.json", "pcs/ierlieth.json"],
|
||||
["monsters/lich.json", "monsters/orc.json"],
|
||||
)
|
||||
|
||||
|
||||
def _emit(line: str) -> None:
|
||||
sys.stdout.write(line + "\n")
|
||||
|
||||
|
||||
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
|
||||
return CombatEngine(SeededRng(1), Grid.from_spec(WARBAND.map), states)
|
||||
|
||||
|
||||
def _state(cid: str, side: str, pos: tuple[int, int], hp: int = 10) -> CombatantState:
|
||||
c = _c("monsters/orc.json").model_copy(update={"id": cid})
|
||||
return CombatantState(combatant=c, side=side, pos=pos, hp=hp, resources_left={})
|
||||
|
||||
|
||||
def test_focus_fire_concentrates_on_lowest_ac_not_nearest() -> None:
|
||||
"""The monster ignores the adjacent tough target and converges on the squishy one."""
|
||||
policy = focus_fire_policy("monsters")
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
tough = _c("pcs/jeanne.json") # AC 23, adjacent to the orc
|
||||
squishy = _c("pcs/nairda.json") # AC 13, far away
|
||||
tough_state = CombatantState(
|
||||
combatant=tough, side="players", pos=(1, 2), hp=tough.hp_max, resources_left={}
|
||||
)
|
||||
squishy_state = CombatantState(
|
||||
combatant=squishy, side="players", pos=(1, 16), hp=squishy.hp_max, resources_left={}
|
||||
)
|
||||
engine = _mini_engine([orc, tough_state, squishy_state])
|
||||
actions = policy(engine, orc)
|
||||
assert all(a.target_id == "nairda" for a in actions), actions
|
||||
|
||||
|
||||
def test_focus_fire_tie_break_prefers_wounded() -> None:
|
||||
"""Equal AC: the monster focuses the member with the lower current hp."""
|
||||
policy = focus_fire_policy("monsters")
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
fresh = _c("monsters/goblin.json").model_copy(update={"id": "fresh"})
|
||||
hurt = _c("monsters/goblin.json").model_copy(update={"id": "hurt"})
|
||||
fresh_state = CombatantState(
|
||||
combatant=fresh, side="players", pos=(5, 5), hp=fresh.hp_max, resources_left={}
|
||||
)
|
||||
hurt_state = CombatantState(
|
||||
combatant=hurt, side="players", pos=(5, 10), hp=1, resources_left={}
|
||||
)
|
||||
engine = _mini_engine([orc, fresh_state, hurt_state])
|
||||
actions = policy(engine, orc)
|
||||
assert all(a.target_id == "hurt" for a in actions), actions
|
||||
|
||||
|
||||
def test_focus_fire_leaves_the_party_side_on_default_policy() -> None:
|
||||
"""Only the monster side changes: a party-side turn gets default_policy verbatim."""
|
||||
policy = focus_fire_policy("monsters")
|
||||
pc = _state("p", "players", (1, 1))
|
||||
mob = _state("m", "monsters", (1, 8))
|
||||
engine = _mini_engine([pc, mob])
|
||||
assert policy(engine, pc) == default_policy(engine, pc)
|
||||
|
||||
|
||||
def _report(name: str, m: DangerMetrics, win: float, sigma: float) -> None:
|
||||
_emit(
|
||||
f" {name:<9} win={win:6.1%} +/-{3 * sigma:4.1%} surv={m.p_survives:5.1%} "
|
||||
f"tpk={m.p_tpk:5.1%} >=1-death={m.p_any_death:5.1%} "
|
||||
f"hp-lost={m.expected_hp_lost:6.1f} rounds={m.expected_rounds:4.1f} "
|
||||
f"tail-hp={m.worst_decile_party_hp:5.1f}"
|
||||
)
|
||||
|
||||
|
||||
def _measure(
|
||||
spec: EncounterSpec,
|
||||
*,
|
||||
strengthened: bool
|
||||
) -> tuple[float, float, DangerMetrics, tuple[RunOutcome, ...]]:
|
||||
policy = focus_fire_policy("monsters") if strengthened else None
|
||||
report, outcomes = run_encounter(
|
||||
spec, runs=RUNS, seed=SEED, policy=policy, collect_runs=True
|
||||
)
|
||||
win = report.win_rate("players")
|
||||
return win, report.win_rate_sigma("players"), danger_metrics(outcomes, "players"), outcomes
|
||||
|
||||
|
||||
def test_danger_map_default_vs_strengthened(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
"""Print the danger map sketch and pin the delta (deterministic, fixed seed)."""
|
||||
_emit("=== PHASE2 SUMMARY (danger cartography spike) ===")
|
||||
deltas: dict[str, tuple[float, float]] = {}
|
||||
metrics: dict[str, tuple[DangerMetrics, DangerMetrics]] = {}
|
||||
for name, spec in (("warband", WARBAND), ("lich", LICH)):
|
||||
_emit(f"scenario {name}: {len(spec.sides[0].combatants)} PCs vs "
|
||||
f"{len(spec.sides[1].combatants)} monsters, runs={RUNS}, seed={SEED}")
|
||||
w0, s0, m0, _ = _measure(spec, strengthened=False)
|
||||
w1, s1, m1, _ = _measure(spec, strengthened=True)
|
||||
_report("default", m0, w0, s0)
|
||||
_report("focusfire", m1, w1, s1)
|
||||
_emit(
|
||||
f" delta death-rate {m1.p_any_death - m0.p_any_death:+.1%} "
|
||||
f"hp-lost {m1.expected_hp_lost - m0.expected_hp_lost:+.1f}"
|
||||
)
|
||||
deltas[name] = (m0.p_any_death, m1.p_any_death)
|
||||
metrics[name] = (m0, m1)
|
||||
|
||||
out = capsys.readouterr().out
|
||||
assert "PHASE2 SUMMARY" in out
|
||||
|
||||
# Loose, deterministic pins: the strengthened adversary never hides danger.
|
||||
for m0, m1 in metrics.values():
|
||||
assert 0.0 <= m1.p_any_death <= 1.0
|
||||
assert m1.p_tpk >= m0.p_tpk - 0.05
|
||||
assert m1.p_survives <= m0.p_survives + 0.05
|
||||
# warband: focus fire must strictly raise the death rate
|
||||
d0, d1 = deltas["warband"]
|
||||
assert d1 > d0
|
||||
# lich: default policy shows ZERO real deaths; focus fire shows some
|
||||
assert metrics["lich"][0].p_any_death == 0.0
|
||||
assert metrics["lich"][1].p_any_death > 0.0
|
||||
assert metrics["lich"][1].p_tpk > 0.0
|
||||
|
||||
|
||||
def test_danger_map_is_reproducible() -> None:
|
||||
"""Same seed, same policy -> byte-identical per-run outcome tuples."""
|
||||
a = _measure(LICH, strengthened=True)[3]
|
||||
b = _measure(LICH, strengthened=True)[3]
|
||||
assert a == b
|
||||
@@ -0,0 +1,183 @@
|
||||
"""Phase-2 measurement spike (docs/phase-2-llm-architecture.md section 12.b).
|
||||
|
||||
Two numbers decide whether a per-turn LLM is viable for the Monte Carlo
|
||||
sweep: the OBSERVATION SIZE (bytes / rough tokens) and the POLICY CALLS PER
|
||||
BATTLE (hence per sweep cell). This module measures both through the real
|
||||
engine path on a representative caster-heavy encounter, prints a summary
|
||||
block for the orchestrator, and pins LOOSE upper bounds so a regression
|
||||
(bigger snapshot schema, policy called twice per turn, ...) fails loudly.
|
||||
No engine behaviour is changed here: the recorder only reads
|
||||
``engine._build_observation`` (pure read, consumes no RNG) at each policy
|
||||
invocation and delegates to ``default_policy``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.combat import Action, CombatantState, CombatEngine, default_policy
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, build_states
|
||||
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
# Loose ceilings: a regression past these fails loudly; the real values are
|
||||
# far below (printed by the tests for the orchestrator to quote).
|
||||
MAX_OBS_BYTES = 200_000
|
||||
MAX_CALLS_PER_BATTLE = 200
|
||||
|
||||
# Round cap for the measurement battles: generous enough to exercise buffs,
|
||||
# conditions and summons, small enough to bound the calls/battle figure.
|
||||
MEASURE_ROUND_CAP = 20
|
||||
|
||||
|
||||
class _Recorder:
|
||||
"""Counting policy wrapper: counts invocations, snapshots observation size.
|
||||
|
||||
Delegates every decision to ``default_policy`` so the battle trajectory
|
||||
(and therefore the call count) is the real engine path. The observation
|
||||
is rebuilt at each call via ``engine._build_observation(state)`` - the
|
||||
same snapshot the engine hands to an ``LLMPolicy`` - and its JSON size
|
||||
is tracked (first call = start of battle, max = fullest mid-battle
|
||||
snapshot with buffs/conditions active).
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.calls = 0
|
||||
self.first_bytes: int | None = None
|
||||
self.max_bytes = 0
|
||||
|
||||
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
self.calls += 1
|
||||
obs = engine._build_observation(state).to_dict()
|
||||
size = len(json.dumps(obs))
|
||||
if self.first_bytes is None:
|
||||
self.first_bytes = size
|
||||
self.max_bytes = max(self.max_bytes, size)
|
||||
return default_policy(engine, state)
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _representative_spec() -> EncounterSpec:
|
||||
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars).
|
||||
|
||||
Oni (channel + heal + summon), misty (spell combat), nairda (hexes) vs
|
||||
the lich (high-level caster, fear aura, paralysis touch), a thrune
|
||||
agent, an orc and a goblin: buffs, conditions, spells and pairs all
|
||||
load the snapshot.
|
||||
"""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=MEASURE_ROUND_CAP,
|
||||
sides=(
|
||||
Side(
|
||||
name="players",
|
||||
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
|
||||
zone="A",
|
||||
),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=(
|
||||
_c("monsters/lich.json"),
|
||||
_c("monsters/thrune_agent.json"),
|
||||
_c("monsters/orc.json"),
|
||||
_c("monsters/goblin.json"),
|
||||
),
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _duel_spec() -> EncounterSpec:
|
||||
"""Small 2v2 baseline to show how calls/battle scales with combatants."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=MEASURE_ROUND_CAP,
|
||||
sides=(
|
||||
Side(name="players", combatants=(_c("monsters/goblin.json"),) * 2, zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),) * 2, zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _run_battle(
|
||||
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
|
||||
) -> _Recorder:
|
||||
grid = Grid.from_spec(spec.map)
|
||||
states = build_states(spec)
|
||||
rec = _Recorder()
|
||||
CombatEngine(
|
||||
SeededRng(seed),
|
||||
grid,
|
||||
states,
|
||||
round_cap=spec.round_cap,
|
||||
policy=rec.policy,
|
||||
spell_registry=registry,
|
||||
).run()
|
||||
return rec
|
||||
|
||||
|
||||
def _emit(line: str) -> None:
|
||||
sys.stdout.write(line + "\n")
|
||||
|
||||
|
||||
def test_observation_size_and_policy_calls(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
"""Measure both numbers on the representative battle; pin loose bounds."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
spec = _representative_spec()
|
||||
rec = _run_battle(spec, seed=1, registry=registry)
|
||||
assert rec.first_bytes is not None
|
||||
|
||||
# Determinism: same seed -> same trajectory -> same numbers.
|
||||
rec2 = _run_battle(spec, seed=1, registry=registry)
|
||||
assert rec2.calls == rec.calls
|
||||
assert rec2.first_bytes == rec.first_bytes
|
||||
assert rec2.max_bytes == rec.max_bytes
|
||||
|
||||
# Loose bounds (regression guards, not exact pins).
|
||||
assert rec.first_bytes < MAX_OBS_BYTES
|
||||
assert rec.max_bytes < MAX_OBS_BYTES
|
||||
assert rec.calls < MAX_CALLS_PER_BATTLE
|
||||
|
||||
duel = _run_battle(_duel_spec(), seed=1, registry=registry)
|
||||
assert duel.calls < MAX_CALLS_PER_BATTLE
|
||||
assert duel.first_bytes is not None
|
||||
|
||||
first_tokens = rec.first_bytes // 4
|
||||
max_tokens = rec.max_bytes // 4
|
||||
_emit("=== PHASE2 SUMMARY (measurement spike) ===")
|
||||
_emit(f"encounter: 7 combatants, round_cap={MEASURE_ROUND_CAP}, seed=1")
|
||||
_emit(f"observation bytes (first call): {rec.first_bytes} (~{first_tokens} tokens)")
|
||||
_emit(f"observation bytes (max in battle): {rec.max_bytes} (~{max_tokens} tokens)")
|
||||
_emit(f"policy calls/battle (7 combatants): {rec.calls}")
|
||||
_emit(f"policy calls/battle (2v2 duel): {duel.calls}")
|
||||
_emit(f"observation 2v2 duel bytes: {duel.first_bytes} (~{duel.first_bytes // 4} tokens)")
|
||||
for runs in (1, 100, 1000):
|
||||
_emit(f"projected calls per sweep cell at runs={runs}: {rec.calls * runs}")
|
||||
out = capsys.readouterr().out
|
||||
assert "PHASE2 SUMMARY" in out
|
||||
|
||||
|
||||
def test_projection_is_linear_in_runs() -> None:
|
||||
"""Calls per sweep cell = calls/battle x runs (fixed per-battle count)."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
rec = _run_battle(_representative_spec(), seed=7, registry=registry)
|
||||
assert rec.calls > 0
|
||||
assert rec.calls * 1000 == (rec.calls * 100) * 10
|
||||
@@ -0,0 +1,414 @@
|
||||
"""Tests for the natural-language policy (P1.3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.client import (
|
||||
ClientParams,
|
||||
Completion,
|
||||
FakeLLMClient,
|
||||
ResponseCache,
|
||||
ScriptExhaustedError,
|
||||
)
|
||||
from pf1e_simulator.combat import Action, Observation
|
||||
from pf1e_simulator.nl_policy import (
|
||||
NLPolicy,
|
||||
NLPolicyError,
|
||||
NLPolicyStats,
|
||||
fallback_default,
|
||||
parse_actions,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
|
||||
_FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
|
||||
|
||||
|
||||
def _obs() -> Observation:
|
||||
"""Minimal valid observation for deterministic tests."""
|
||||
return Observation(
|
||||
round=1,
|
||||
acting_combatant=0,
|
||||
combatants=(
|
||||
{"id": "goblin", "side": "players", "active": True, "pos": (1, 1)},
|
||||
),
|
||||
pairs=(),
|
||||
terrain=((None,),),
|
||||
elevation=((0,),),
|
||||
)
|
||||
|
||||
|
||||
def _pack() -> PromptPack:
|
||||
return PromptPack.load(_FIXTURE_PACK)
|
||||
|
||||
|
||||
def _builder() -> PromptBuilder:
|
||||
return PromptBuilder(pack=_pack())
|
||||
|
||||
|
||||
def _make_client(script: list[str], *, builder: PromptBuilder, **kwargs) -> FakeLLMClient:
|
||||
"""Create a FakeLLMClient with the correct prompt_version from the builder."""
|
||||
return FakeLLMClient(
|
||||
script=script,
|
||||
prompt_version=builder.prompt_version("nl_default"),
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
|
||||
def _make_policy(
|
||||
client: FakeLLMClient, builder: PromptBuilder, *, max_repairs: int = 1
|
||||
) -> NLPolicy:
|
||||
return NLPolicy(client=client, builder=builder, max_repairs=max_repairs)
|
||||
|
||||
|
||||
class _RecordingClient(FakeLLMClient):
|
||||
"""Scripted client that records every prompt sent to the model.
|
||||
|
||||
The repair-loop tests assert on the accumulated error trail, so the
|
||||
prompts themselves must be observable.
|
||||
"""
|
||||
|
||||
def __init__(self, script: Sequence[str], *, prompt_version: str) -> None:
|
||||
super().__init__(script=script, prompt_version=prompt_version)
|
||||
self.prompts: list[str] = []
|
||||
|
||||
def _call_once(self, prompt: str, *, params: ClientParams) -> Completion:
|
||||
self.prompts.append(prompt)
|
||||
return super()._call_once(prompt, params=params)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def builder() -> PromptBuilder:
|
||||
"""A builder over the fixture pack, shared by the NLPolicy integration tests."""
|
||||
return _builder()
|
||||
|
||||
|
||||
# ---- parse_actions tests -----------------------------------------------------
|
||||
|
||||
def test_parse_actions_ok() -> None:
|
||||
reply = '[{"kind": "move", "destination": [2, 3]}]'
|
||||
actions = parse_actions(reply)
|
||||
assert actions == (Action(kind="move", destination=(2, 3)),)
|
||||
|
||||
|
||||
def test_parse_actions_empty_array() -> None:
|
||||
assert parse_actions("[]") == ()
|
||||
|
||||
|
||||
def test_parse_actions_invalid_json() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"reply is not valid JSON"):
|
||||
parse_actions("not json")
|
||||
|
||||
|
||||
def test_parse_actions_not_a_list() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"reply must be a JSON array of action objects"):
|
||||
parse_actions('{"kind": "attack"}')
|
||||
|
||||
|
||||
def test_parse_actions_unknown_field() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"unknown action field\(s\): bogus"):
|
||||
parse_actions('[{"kind": "attack", "bogus": 1}]')
|
||||
|
||||
|
||||
def test_parse_actions_missing_kind() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"missing or unknown 'kind'"):
|
||||
parse_actions('[{"target_id": "orc"}]')
|
||||
|
||||
|
||||
def test_parse_actions_unknown_kind() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"missing or unknown 'kind'"):
|
||||
parse_actions('[{"kind": "dance"}]')
|
||||
|
||||
|
||||
def test_parse_actions_channel_mode_invalid() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'channel_mode' must be one of: heal, harm"):
|
||||
parse_actions('[{"kind": "channel_energy", "channel_mode": "loud"}]')
|
||||
|
||||
|
||||
def test_parse_actions_destination_invalid_not_pair() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'destination' must be a \[row, col\] pair"):
|
||||
parse_actions('[{"kind": "move", "destination": [2]}]')
|
||||
|
||||
|
||||
def test_parse_actions_destination_invalid_strings() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'destination' must be a \[row, col\] pair"):
|
||||
parse_actions('[{"kind": "move", "destination": ["2", "3"]}]')
|
||||
|
||||
|
||||
def test_parse_actions_aid_mode_invalid() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'aid_mode' must be one of: attack, ac"):
|
||||
parse_actions('[{"kind": "aid_another", "aid_mode": "damage"}]')
|
||||
|
||||
|
||||
def test_parse_actions_trick_condition_invalid() -> None:
|
||||
with pytest.raises(
|
||||
NLPolicyError,
|
||||
match=(
|
||||
r"'trick_condition' must be one of: dazzled, blinded, "
|
||||
r"deafened, entangled, shaken, sickened"
|
||||
),
|
||||
):
|
||||
parse_actions('[{"kind": "cast_spell", "trick_condition": "nauseated"}]')
|
||||
|
||||
|
||||
def test_parse_actions_ready_trigger_invalid() -> None:
|
||||
with pytest.raises(
|
||||
NLPolicyError,
|
||||
match=(
|
||||
r"'ready_trigger' must be one of: enemy_enters_reach, "
|
||||
r"enemy_casts, enemy_attacks, enemy_hits_ally"
|
||||
),
|
||||
):
|
||||
parse_actions('[{"kind": "ready", "ready_trigger": "enemy_sneezed"}]')
|
||||
|
||||
|
||||
def test_parse_actions_channel_exclude_not_list() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'channel_exclude' must be a list of strings"):
|
||||
parse_actions('[{"kind": "channel_energy", "channel_exclude": "orc"}]')
|
||||
|
||||
|
||||
def test_parse_actions_channel_exclude_ok() -> None:
|
||||
actions = parse_actions('[{"kind": "channel_energy", "channel_exclude": ["orc"]}]')
|
||||
assert actions[0].channel_exclude == ("orc",)
|
||||
|
||||
|
||||
def test_parse_actions_tumble_not_bool() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'tumble' must be a boolean"):
|
||||
parse_actions('[{"kind": "attack", "tumble": "yes"}]')
|
||||
|
||||
|
||||
def test_parse_actions_cast_defensively_not_bool() -> None:
|
||||
with pytest.raises(NLPolicyError, match=r"'cast_defensively' must be a boolean"):
|
||||
parse_actions('[{"kind": "cast_spell", "cast_defensively": 1}]')
|
||||
|
||||
|
||||
def test_parse_actions_metamagic_ok() -> None:
|
||||
actions = parse_actions('[{"kind": "cast_spell", "metamagic": ["Empower", "Maximize"]}]')
|
||||
assert actions[0].metamagic == ("Empower", "Maximize")
|
||||
|
||||
|
||||
# ---- fallback_default tests --------------------------------------------------
|
||||
|
||||
def test_fallback_default_prefers_full_attack() -> None:
|
||||
menu = (
|
||||
Action(kind="total_defense"),
|
||||
Action(kind="full_attack", target_id="orc"),
|
||||
Action(kind="move"),
|
||||
)
|
||||
assert fallback_default(menu) == (Action(kind="full_attack", target_id="orc"),)
|
||||
|
||||
|
||||
def test_fallback_default_prefers_attack_over_charge() -> None:
|
||||
menu = (
|
||||
Action(kind="charge", target_id="orc"),
|
||||
Action(kind="attack", target_id="orc"),
|
||||
)
|
||||
assert fallback_default(menu) == (Action(kind="attack", target_id="orc"),)
|
||||
|
||||
|
||||
def test_fallback_default_prefers_charge_over_move() -> None:
|
||||
menu = (
|
||||
Action(kind="move", target_id="orc"),
|
||||
Action(kind="charge", target_id="orc"),
|
||||
)
|
||||
assert fallback_default(menu) == (Action(kind="charge", target_id="orc"),)
|
||||
|
||||
|
||||
def test_fallback_default_falls_back_to_first() -> None:
|
||||
menu = (Action(kind="total_defense"), Action(kind="delay"))
|
||||
assert fallback_default(menu) == (Action(kind="total_defense"),)
|
||||
|
||||
|
||||
def test_fallback_default_empty_menu_returns_wait() -> None:
|
||||
assert fallback_default(()) == (Action(kind="wait"),)
|
||||
|
||||
|
||||
# ---- NLPolicy integration tests ---------------------------------------------
|
||||
|
||||
def test_nl_policy_parse_ok(builder: PromptBuilder) -> None:
|
||||
"""Parse OK path with valid JSON from FakeLLMClient."""
|
||||
script = ['[{"kind": "move", "destination": [2, 3]}]']
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="move", destination=(2, 3)),)
|
||||
assert client.stats.calls == 1
|
||||
|
||||
|
||||
def test_nl_policy_retry_then_succeeds() -> None:
|
||||
"""One bad reply, then a good one → parsed successfully."""
|
||||
builder = _builder()
|
||||
script = ["not json", '[{"kind": "move", "destination": [2, 3]}]']
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="move", destination=(2, 3)),)
|
||||
assert client.stats.calls == 2
|
||||
|
||||
|
||||
def test_nl_policy_retry_then_fallback() -> None:
|
||||
"""Two bad replies → fallback used."""
|
||||
builder = _builder()
|
||||
script = ["not json", "not json"]
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
|
||||
# Minimal obs menu is (total_defense, delay)
|
||||
actions = policy(_obs())
|
||||
# fallback_default picks first decisive action, but menu has only defensive
|
||||
assert actions == (Action(kind="total_defense"),)
|
||||
assert client.stats.calls == 2
|
||||
|
||||
|
||||
def test_nl_policy_prompt_version_mismatch() -> None:
|
||||
"""Client default prompt_version vs builder pack version → ValueError."""
|
||||
builder = _builder()
|
||||
# Use default prompt_version ("fixture:fake:0") which differs from pack
|
||||
client = FakeLLMClient(script=['[{"kind": "wait"}]'])
|
||||
with pytest.raises(ValueError, match=r"prompt_version"):
|
||||
_make_policy(client, builder)
|
||||
|
||||
|
||||
def test_nl_policy_script_exhausted_propagates() -> None:
|
||||
"""ScriptExhaustedError from FakeLLMClient propagates (not caught)."""
|
||||
builder = _builder()
|
||||
client = _make_client([], builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
|
||||
with pytest.raises(ScriptExhaustedError):
|
||||
policy(_obs())
|
||||
|
||||
|
||||
def test_nl_policy_shared_cache_hit() -> None:
|
||||
"""Two clients sharing a ResponseCache: second call hits cache."""
|
||||
builder = _builder()
|
||||
cache = ResponseCache()
|
||||
# First client has one scripted response
|
||||
client1 = _make_client(['[{"kind": "wait"}]'], builder=builder, cache=cache)
|
||||
# Second client has NO script; relies on cache hit
|
||||
client2 = _make_client([], builder=builder, cache=cache)
|
||||
policy1 = _make_policy(client1, builder)
|
||||
policy2 = _make_policy(client2, builder)
|
||||
|
||||
obs = _obs()
|
||||
actions1 = policy1(obs)
|
||||
actions2 = policy2(obs)
|
||||
|
||||
assert actions2 == actions1 == (Action(kind="wait"),)
|
||||
# Second call was a cache hit (no ScriptExhaustedError)
|
||||
assert client2.stats.cache_hits == 1
|
||||
assert client1.stats.cache_misses == 1
|
||||
|
||||
|
||||
# ---- P5.2 repair loop --------------------------------------------------------
|
||||
|
||||
def test_nl_policy_stats_defaults_to_zero() -> None:
|
||||
"""A fresh NLPolicy reports zero calls until the first decision."""
|
||||
builder = _builder()
|
||||
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
assert policy.stats == NLPolicyStats()
|
||||
assert policy.stats.calls == 0
|
||||
|
||||
|
||||
def test_nl_policy_max_repairs_zero_falls_back_immediately() -> None:
|
||||
"""max_repairs=0 disables repair: first bad reply triggers fallback."""
|
||||
builder = _builder()
|
||||
client = _make_client(["bad one"], builder=builder)
|
||||
policy = _make_policy(client, builder, max_repairs=0)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="total_defense"),)
|
||||
assert policy.stats.calls == 1
|
||||
assert policy.stats.repairs_attempted == 0
|
||||
assert policy.stats.repairs_succeeded == 0
|
||||
assert policy.stats.fallbacks == 1
|
||||
|
||||
|
||||
def test_nl_policy_max_repairs_two_repairs_then_succeeds() -> None:
|
||||
"""Two invalid replies are repaired, then the valid reply is parsed."""
|
||||
builder = _builder()
|
||||
script = ["bad one", "bad two", '[{"kind": "move", "destination": [2, 3]}]']
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = _make_policy(client, builder, max_repairs=2)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="move", destination=(2, 3)),)
|
||||
assert policy.stats.calls == 3
|
||||
assert policy.stats.repairs_attempted == 2
|
||||
assert policy.stats.repairs_succeeded == 1
|
||||
assert policy.stats.fallbacks == 0
|
||||
|
||||
|
||||
def test_nl_policy_max_repairs_two_then_falls_back() -> None:
|
||||
"""Past max_repairs the policy stops repairing and falls back."""
|
||||
builder = _builder()
|
||||
script = ["bad one", "bad two", "bad three"]
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = _make_policy(client, builder, max_repairs=2)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="total_defense"),)
|
||||
assert policy.stats.calls == 3
|
||||
assert policy.stats.repairs_attempted == 2
|
||||
assert policy.stats.repairs_succeeded == 0
|
||||
assert policy.stats.fallbacks == 1
|
||||
|
||||
|
||||
def test_nl_policy_repair_trail_accumulates() -> None:
|
||||
"""Each retry prompt carries the full accumulated error trail."""
|
||||
builder = _builder()
|
||||
script = ["bad one", "bad two", '[{"kind": "wait"}]']
|
||||
client = _RecordingClient(
|
||||
script=script,
|
||||
prompt_version=builder.prompt_version("nl_default"),
|
||||
)
|
||||
policy = _make_policy(client, builder, max_repairs=2)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="wait"),)
|
||||
|
||||
# First call: bare prompt with no trail.
|
||||
assert len(client.prompts) == 3
|
||||
assert "-- previous reply invalid" not in client.prompts[0]
|
||||
# First retry: trail carries attempt 1 only.
|
||||
assert "attempt 1" in client.prompts[1]
|
||||
assert "attempt 2" not in client.prompts[1]
|
||||
assert "Return a valid JSON action sequence." in client.prompts[1]
|
||||
# Second retry: trail carries attempts 1 AND 2, in order.
|
||||
attempt1 = client.prompts[2].index("-- previous reply invalid (attempt 1) --")
|
||||
attempt2 = client.prompts[2].index("-- previous reply invalid (attempt 2) --")
|
||||
assert attempt1 < attempt2
|
||||
# The base prompt is preserved at the head of every retry.
|
||||
assert client.prompts[1].startswith(client.prompts[0])
|
||||
|
||||
|
||||
def test_nl_policy_max_repairs_negative_rejected() -> None:
|
||||
"""A negative max_repairs is a configuration error, not a fallback."""
|
||||
builder = _builder()
|
||||
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
|
||||
with pytest.raises(ValueError, match=r"max_repairs must be >= 0, got -1"):
|
||||
_make_policy(client, builder, max_repairs=-1)
|
||||
|
||||
|
||||
def test_nl_policy_stats_after_direct_success() -> None:
|
||||
"""A clean first-try success leaves the repair counters untouched."""
|
||||
builder = _builder()
|
||||
client = _make_client(['[{"kind": "wait"}]'], builder=builder)
|
||||
policy = _make_policy(client, builder)
|
||||
|
||||
actions = policy(_obs())
|
||||
assert actions == (Action(kind="wait"),)
|
||||
assert policy.stats.calls == 1
|
||||
assert policy.stats.repairs_attempted == 0
|
||||
assert policy.stats.repairs_succeeded == 0
|
||||
assert policy.stats.fallbacks == 0
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Offline battles driven by FakeLLMClient through the NL policy (P1.3/P1.4)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from pf1e_simulator.client import FakeLLMClient
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.nl_policy import NLPolicy
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
|
||||
def _c(path: str):
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _duel_spec() -> EncounterSpec:
|
||||
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _pack() -> PromptPack:
|
||||
return PromptPack.load(ROOT / "tests" / "fixtures" / "prompt_pack")
|
||||
|
||||
|
||||
def _builder() -> PromptBuilder:
|
||||
return PromptBuilder(pack=_pack())
|
||||
|
||||
|
||||
def _make_client(script: list[str], *, builder: PromptBuilder) -> FakeLLMClient:
|
||||
return FakeLLMClient(
|
||||
script=script,
|
||||
prompt_version=builder.prompt_version("nl_default"),
|
||||
)
|
||||
|
||||
|
||||
def test_offline_battle_total_defense_draw() -> None:
|
||||
"""Both sides play total_defense → draw at round_cap (20 rounds, 40 calls)."""
|
||||
spec = _duel_spec()
|
||||
builder = _builder()
|
||||
# 60 responses is plenty for 40 turns (20 rounds x 2 combatants)
|
||||
script = ['[{"kind": "total_defense"}]'] * 60
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = NLPolicy(client=client, builder=builder)
|
||||
|
||||
report, _outcomes = run_encounter(spec, runs=1, seed=1, llm_policy=policy, collect_runs=True)
|
||||
|
||||
# Draw at round_cap
|
||||
assert report.draws == 1
|
||||
assert report.wins.get("players", 0) == 0
|
||||
assert report.wins.get("monsters", 0) == 0
|
||||
assert report.avg_rounds == 20
|
||||
# 20 rounds x 2 combatants = 40 policy invocations
|
||||
assert client.stats.calls == 40
|
||||
|
||||
|
||||
def test_offline_battle_fallback() -> None:
|
||||
"""Fallback-driven battle: script of invalid JSON forces fallback on every turn."""
|
||||
spec = _duel_spec()
|
||||
builder = _builder()
|
||||
# Each turn: 2 completes (first parse fail, retry parse fail → fallback).
|
||||
# Max 20 rounds x 2 combatants x 2 = 80 completes. Use 100 for margin.
|
||||
script = ["not json"] * 100
|
||||
client = _make_client(script, builder=builder)
|
||||
policy = NLPolicy(client=client, builder=builder)
|
||||
|
||||
report, _outcomes = run_encounter(spec, runs=1, seed=1, llm_policy=policy, collect_runs=True)
|
||||
|
||||
# The battle should complete (not exhaust script)
|
||||
assert report.runs == 1
|
||||
assert report.draws + sum(report.wins.values()) == 1
|
||||
# At least some calls happened; script not exhausted
|
||||
assert client.stats.calls > 0
|
||||
assert client.stats.calls <= 100
|
||||
@@ -0,0 +1,321 @@
|
||||
"""Phase-2 config and external prompt pack (docs/phase-2 P1a, sections 14-15).
|
||||
|
||||
``config.py`` centralizes every phase-2 path/env read (no hard-coded data
|
||||
paths - section 15). ``prompt.py`` loads the EXTERNAL, versioned prompt pack
|
||||
(section 14): prompts never live in this repo; the pack carries its own
|
||||
version and each prompt has an individual content hash, so ``prompt_version``
|
||||
can enter the response-cache composite key (D6). The minimal pack under
|
||||
``tests/fixtures/prompt_pack/`` is the local fixture so CI never depends on
|
||||
the external prompt repository, and missing/invalid packs raise a clear
|
||||
``PromptPackError`` instead of falling back silently.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.config import (
|
||||
ENV_LLM_API_KEY,
|
||||
ENV_PROMPT_PACK_DIR,
|
||||
env_path,
|
||||
env_str,
|
||||
llm_api_key,
|
||||
)
|
||||
from pf1e_simulator.prompt import (
|
||||
FewShotExample,
|
||||
PromptPack,
|
||||
PromptPackError,
|
||||
clear_pack_cache,
|
||||
load_prompt_pack,
|
||||
)
|
||||
|
||||
FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
|
||||
_FIXTURE_PACK_FS = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
_FIXTURE_VERSION = "0.1.0-test"
|
||||
_FIXTURE_PROMPT_NAME = "nl_default"
|
||||
|
||||
|
||||
def _write_pack(root: Path, *, version: str, prompts: dict[str, str]) -> None:
|
||||
"""Write a minimal prompt pack directory (version.txt + <name>.md)."""
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
(root / "version.txt").write_text(version, encoding="utf-8")
|
||||
for name, text in prompts.items():
|
||||
(root / f"{name}.md").write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
def _write_examples(root: Path, examples: dict[str, str]) -> None:
|
||||
"""Write <name>.examples.json files into an existing pack directory."""
|
||||
for name, text in examples.items():
|
||||
(root / f"{name}.examples.json").write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
# ── config.py ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_env_str_and_path_round_trip(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""env_str/env_path read the environment and respect deletes."""
|
||||
monkeypatch.setenv("PF1E_SIM_TEST_STR", "abc")
|
||||
monkeypatch.setenv("PF1E_SIM_TEST_PATH", str(tmp_path / "x"))
|
||||
assert env_str("PF1E_SIM_TEST_STR") == "abc"
|
||||
assert env_str("PF1E_SIM_TEST_UNSET") is None
|
||||
assert env_str("PF1E_SIM_TEST_UNSET", default="d") == "d"
|
||||
assert env_path("PF1E_SIM_TEST_PATH") == tmp_path / "x"
|
||||
assert env_path("PF1E_SIM_TEST_UNSET") is None
|
||||
monkeypatch.delenv("PF1E_SIM_TEST_STR")
|
||||
assert env_str("PF1E_SIM_TEST_STR") is None
|
||||
|
||||
|
||||
def test_llm_api_key_reads_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""llm_api_key surfaces the dedicated env var (None when unset)."""
|
||||
assert llm_api_key() is None
|
||||
monkeypatch.setenv(ENV_LLM_API_KEY, "sk-test")
|
||||
assert llm_api_key() == "sk-test"
|
||||
|
||||
|
||||
# ── prompt.py — loading and validation ─────────────────────────────────────
|
||||
|
||||
|
||||
def test_load_fixture_pack() -> None:
|
||||
"""The local fixture loads: version pinned, prompts readable by name."""
|
||||
pack = load_prompt_pack(FIXTURE_PACK)
|
||||
assert pack.version == _FIXTURE_VERSION
|
||||
assert pack.root == FIXTURE_PACK
|
||||
text = pack.get(_FIXTURE_PROMPT_NAME)
|
||||
assert "tactical director" in text
|
||||
# The pack has exactly the fixture prompts (only <name>.md files count).
|
||||
assert pack.get("nl_default") == (FIXTURE_PACK / "nl_default.md").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def test_prompt_version_stable_and_content_sensitive(tmp_path: Path) -> None:
|
||||
"""prompt_version carries pack version + per-prompt content hash (D6)."""
|
||||
pack = load_prompt_pack(FIXTURE_PACK)
|
||||
a = pack.prompt_version(_FIXTURE_PROMPT_NAME)
|
||||
b = pack.prompt_version(_FIXTURE_PROMPT_NAME)
|
||||
assert a == b # deterministic within a loaded pack
|
||||
assert a.startswith(f"{_FIXTURE_VERSION}:{_FIXTURE_PROMPT_NAME}:")
|
||||
|
||||
other = tmp_path / "other"
|
||||
_write_pack(other, version=_FIXTURE_VERSION, prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
assert load_prompt_pack(other).prompt_version(_FIXTURE_PROMPT_NAME) != a
|
||||
|
||||
bumped = tmp_path / "bumped"
|
||||
_write_pack(bumped, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
assert load_prompt_pack(bumped).prompt_version(_FIXTURE_PROMPT_NAME).startswith(
|
||||
"1.0.0:"
|
||||
)
|
||||
|
||||
|
||||
def test_missing_prompt_raises_clear_error(tmp_path: Path) -> None:
|
||||
"""A missing prompt name fails loudly, naming the expected file."""
|
||||
other = tmp_path / "other"
|
||||
_write_pack(other, version="1.0.0", prompts={"other": "x"})
|
||||
with pytest.raises(PromptPackError, match=r"missing.*nl_default\.md"):
|
||||
load_prompt_pack(other).get("nl_default")
|
||||
|
||||
|
||||
def test_missing_pack_dir_raises(tmp_path: Path) -> None:
|
||||
"""A nonexistent pack path is a clear error, not a silent fallback."""
|
||||
with pytest.raises(PromptPackError, match="not found"):
|
||||
load_prompt_pack(tmp_path / "nope")
|
||||
|
||||
|
||||
def test_missing_version_file_raises(tmp_path: Path) -> None:
|
||||
"""A pack without version.txt is invalid (traceability, section 14)."""
|
||||
root = tmp_path / "noversion"
|
||||
_write_pack(root, version="", prompts={})
|
||||
(root / "version.txt").unlink()
|
||||
with pytest.raises(PromptPackError, match="version\\.txt"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_empty_version_file_raises(tmp_path: Path) -> None:
|
||||
"""An empty version.txt is invalid (no untraceable pack)."""
|
||||
root = tmp_path / "emptyversion"
|
||||
_write_pack(root, version="", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
with pytest.raises(PromptPackError, match="empty"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_unset_pack_dir_raises(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Without PF1E_PROMPT_PACK_DIR, loading raises instead of falling back."""
|
||||
monkeypatch.delenv(ENV_PROMPT_PACK_DIR, raising=False)
|
||||
with pytest.raises(PromptPackError, match=ENV_PROMPT_PACK_DIR):
|
||||
load_prompt_pack()
|
||||
|
||||
|
||||
def test_pack_dir_from_env(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""load_prompt_pack() resolves the pack through the env var (section 15)."""
|
||||
monkeypatch.setenv(ENV_PROMPT_PACK_DIR, str(FIXTURE_PACK))
|
||||
assert load_prompt_pack().version == _FIXTURE_VERSION
|
||||
monkeypatch.delenv(ENV_PROMPT_PACK_DIR)
|
||||
|
||||
|
||||
def test_cache_and_clear(tmp_path: Path) -> None:
|
||||
"""Repeated loads share the cached instance; clear_pack_cache drops it."""
|
||||
root = tmp_path / "cached"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
first = load_prompt_pack(root)
|
||||
assert load_prompt_pack(root) is first
|
||||
clear_pack_cache()
|
||||
assert load_prompt_pack(root) is not first
|
||||
|
||||
|
||||
def test_load_returns_prompt_pack_instance() -> None:
|
||||
"""The loaded object is a PromptPack (typed API for the client layer)."""
|
||||
pack = load_prompt_pack(FIXTURE_PACK)
|
||||
assert isinstance(pack, PromptPack)
|
||||
assert pack.version == _FIXTURE_VERSION
|
||||
|
||||
|
||||
# ── prompt.py — few-shot examples (P5.2) ────────────────────────────────────
|
||||
|
||||
|
||||
def test_examples_absent_returns_empty() -> None:
|
||||
"""A prompt without an examples file exposes an empty example tuple."""
|
||||
pack = load_prompt_pack(FIXTURE_PACK)
|
||||
assert pack.examples(_FIXTURE_PROMPT_NAME) == ()
|
||||
|
||||
|
||||
def test_few_shot_fixture_pack_loads_examples() -> None:
|
||||
"""prompt_pack_fs ships one teaching pair, parsed into FewShotExample."""
|
||||
pack = load_prompt_pack(_FIXTURE_PACK_FS)
|
||||
examples = pack.examples(_FIXTURE_PROMPT_NAME)
|
||||
assert len(examples) == 1
|
||||
example = examples[0]
|
||||
assert isinstance(example, FewShotExample)
|
||||
assert example.observation == {
|
||||
"round": 1,
|
||||
"combatant": "goblin",
|
||||
"enemy": "orc",
|
||||
"distance_ft": 10,
|
||||
}
|
||||
assert example.actions == ({"kind": "attack", "target_id": "orc"},)
|
||||
|
||||
|
||||
def test_prompt_version_includes_examples_content() -> None:
|
||||
"""With an examples file, the content hash covers it (cache key, D6)."""
|
||||
plain = load_prompt_pack(FIXTURE_PACK)
|
||||
with_examples = load_prompt_pack(_FIXTURE_PACK_FS)
|
||||
# Same prompt text, same pack version: the digest must differ because the
|
||||
# few-shot file is hashed alongside the prompt.
|
||||
assert plain.get(_FIXTURE_PROMPT_NAME) == with_examples.get(_FIXTURE_PROMPT_NAME)
|
||||
assert plain.prompt_version(_FIXTURE_PROMPT_NAME) != with_examples.prompt_version(
|
||||
_FIXTURE_PROMPT_NAME
|
||||
)
|
||||
|
||||
|
||||
def test_prompt_version_byte_identical_without_examples(tmp_path: Path) -> None:
|
||||
"""No examples file → digest is byte-identical to the pre-few-shot scheme."""
|
||||
root = tmp_path / "plain"
|
||||
_write_pack(root, version=_FIXTURE_VERSION, prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
pack = load_prompt_pack(root)
|
||||
digest = pack.prompt_version(_FIXTURE_PROMPT_NAME).split(":")[-1]
|
||||
assert len(digest) == 12
|
||||
|
||||
|
||||
def test_examples_invalid_json_raises(tmp_path: Path) -> None:
|
||||
"""A malformed examples file is a pack error, not a silent skip."""
|
||||
root = tmp_path / "badjson"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(root, {_FIXTURE_PROMPT_NAME: "not json"})
|
||||
with pytest.raises(PromptPackError, match="is not valid JSON"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_not_an_array_raises(tmp_path: Path) -> None:
|
||||
"""The examples file must be a JSON array at the top level."""
|
||||
root = tmp_path / "notarray"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(root, {_FIXTURE_PROMPT_NAME: '{"observation": {}}'})
|
||||
with pytest.raises(PromptPackError, match="must be a JSON array of examples"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_item_not_object_raises(tmp_path: Path) -> None:
|
||||
"""Each array entry must be an object with observation and actions."""
|
||||
root = tmp_path / "itemnotobject"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(root, {_FIXTURE_PROMPT_NAME: "[42]"})
|
||||
with pytest.raises(
|
||||
PromptPackError,
|
||||
match="must be an object with 'observation' and 'actions'",
|
||||
):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_unknown_key_raises(tmp_path: Path) -> None:
|
||||
"""Unexpected keys inside an example are rejected (strict validation)."""
|
||||
root = tmp_path / "unknownkey"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(
|
||||
root,
|
||||
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": [], "hint": 1}]'},
|
||||
)
|
||||
with pytest.raises(PromptPackError, match="unknown key\\(s\\): hint"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_observation_not_object_raises(tmp_path: Path) -> None:
|
||||
"""'observation' must be a JSON object."""
|
||||
root = tmp_path / "badobs"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(
|
||||
root,
|
||||
{_FIXTURE_PROMPT_NAME: '[{"observation": "goblin", "actions": []}]'},
|
||||
)
|
||||
with pytest.raises(PromptPackError, match="'observation' must be a JSON object"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_actions_not_array_raises(tmp_path: Path) -> None:
|
||||
"""'actions' must be a JSON array."""
|
||||
root = tmp_path / "badactions"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(
|
||||
root,
|
||||
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": {"kind": "wait"}}]'},
|
||||
)
|
||||
with pytest.raises(PromptPackError, match="'actions' must be a JSON array"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_action_entry_not_object_raises(tmp_path: Path) -> None:
|
||||
"""Each action inside 'actions' must be an object."""
|
||||
root = tmp_path / "badactionentry"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(
|
||||
root,
|
||||
{_FIXTURE_PROMPT_NAME: '[{"observation": {}, "actions": ["jump"]}]'},
|
||||
)
|
||||
with pytest.raises(PromptPackError, match="must contain only JSON objects"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
|
||||
def test_examples_empty_array_accepted(tmp_path: Path) -> None:
|
||||
"""An empty examples array is valid and yields no teaching pairs."""
|
||||
root = tmp_path / "emptyexamples"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(root, {_FIXTURE_PROMPT_NAME: "[]"})
|
||||
pack = load_prompt_pack(root)
|
||||
assert pack.examples(_FIXTURE_PROMPT_NAME) == ()
|
||||
|
||||
|
||||
def test_orphan_examples_file_raises(tmp_path: Path) -> None:
|
||||
"""An .examples.json without a matching <name>.md is a pack error."""
|
||||
root = tmp_path / "orphan"
|
||||
_write_pack(root, version="1.0.0", prompts={_FIXTURE_PROMPT_NAME: "x"})
|
||||
_write_examples(root, {"stray": "[]"})
|
||||
with pytest.raises(PromptPackError, match="has no matching 'stray\\.md' prompt"):
|
||||
load_prompt_pack(root)
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""Tests for the phase-2 prompt builder (P1.4)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from dataclasses import fields
|
||||
from pathlib import Path
|
||||
|
||||
from pf1e_simulator.combat import Action, Observation
|
||||
from pf1e_simulator.nl_policy import KindValue
|
||||
from pf1e_simulator.nl_policy import get_args as _get_args
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import (
|
||||
PromptBuilder,
|
||||
_render_examples,
|
||||
_supported_kinds,
|
||||
action_schema,
|
||||
render_candidates,
|
||||
)
|
||||
|
||||
_FIXTURE_PACK = Path(__file__).resolve().parent / "fixtures" / "prompt_pack"
|
||||
_FIXTURE_PACK_FS = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
_FIXTURE_VERSION = "0.1.0-test"
|
||||
_FIXTURE_PROMPT_NAME = "nl_default"
|
||||
|
||||
|
||||
def _obs() -> Observation:
|
||||
"""Minimal valid observation for deterministic tests."""
|
||||
return Observation(
|
||||
round=1,
|
||||
acting_combatant=0,
|
||||
combatants=(
|
||||
{"id": "goblin", "side": "players", "active": True, "pos": (1, 1)},
|
||||
),
|
||||
pairs=(),
|
||||
terrain=((None,),),
|
||||
elevation=((0,),),
|
||||
)
|
||||
|
||||
|
||||
def _pack() -> PromptPack:
|
||||
return PromptPack.load(_FIXTURE_PACK)
|
||||
|
||||
|
||||
def _builder() -> PromptBuilder:
|
||||
return PromptBuilder(pack=_pack())
|
||||
|
||||
|
||||
def test_supported_kinds_filters_unsupported() -> None:
|
||||
kinds = _supported_kinds()
|
||||
assert "swift" not in kinds
|
||||
assert "free" not in kinds
|
||||
assert "full_round" not in kinds
|
||||
# 27 kinds remain
|
||||
assert len(kinds) == 27
|
||||
# Order preserved from source Literal
|
||||
assert kinds == (
|
||||
"attack", "move", "wait", "immediate", "full_attack", "charge",
|
||||
"withdraw", "5foot_step", "cast_spell", "maneuver", "channel_energy",
|
||||
"hex", "deed", "spell_combat", "inspire_courage", "retrieve_weapon",
|
||||
"repair_weapon", "activate_power_attack", "delay", "ready",
|
||||
"total_defense", "fight_defensively", "stand_up", "drop_prone",
|
||||
"aid_another", "run", "hide",
|
||||
)
|
||||
|
||||
|
||||
def test_action_schema_structure() -> None:
|
||||
schema = action_schema()
|
||||
assert schema["type"] == "object"
|
||||
assert schema["required"] == ["kind"]
|
||||
assert schema["additionalProperties"] is False
|
||||
|
||||
props = schema["properties"]
|
||||
# Properties match Action fields exactly
|
||||
action_field_names = {f.name for f in fields(Action)}
|
||||
assert set(props.keys()) == action_field_names
|
||||
|
||||
# Kind enum is the filtered 27
|
||||
kind_prop = props["kind"]
|
||||
assert kind_prop["type"] == "string"
|
||||
assert kind_prop["enum"] == list(_supported_kinds())
|
||||
|
||||
|
||||
def test_action_schema_spot_check_fragments() -> None:
|
||||
schema = action_schema()
|
||||
props = schema["properties"]
|
||||
|
||||
# target_id accepts either a string or null
|
||||
assert props["target_id"] == {"type": ["string", "null"]}
|
||||
|
||||
# destination is a nullable two-int pair (prefixItems + min/maxItems)
|
||||
dest = props["destination"]
|
||||
assert dest["type"] == ["array", "null"]
|
||||
assert dest["prefixItems"] == [{"type": "integer"}, {"type": "integer"}]
|
||||
assert dest["minItems"] == 2
|
||||
assert dest["maxItems"] == 2
|
||||
|
||||
# channel_exclude: string[] (array of strings)
|
||||
ch_ex = props["channel_exclude"]
|
||||
assert ch_ex["type"] == "array"
|
||||
assert ch_ex["items"] == {"type": "string"}
|
||||
|
||||
# cast_defensively is a boolean field
|
||||
assert props["cast_defensively"] == {"type": "boolean"}
|
||||
|
||||
# tumble is a boolean field
|
||||
assert props["tumble"] == {"type": "boolean"}
|
||||
|
||||
# channel_mode: nullable enum
|
||||
ch_mode = props["channel_mode"]
|
||||
assert ch_mode["type"] == ["string", "null"]
|
||||
assert ch_mode["enum"] == ["heal", "harm"]
|
||||
|
||||
# aid_mode: nullable enum
|
||||
aid = props["aid_mode"]
|
||||
assert aid["type"] == ["string", "null"]
|
||||
assert aid["enum"] == ["attack", "ac"]
|
||||
|
||||
# trick_condition: nullable enum (6 values)
|
||||
tc = props["trick_condition"]
|
||||
assert tc["type"] == ["string", "null"]
|
||||
assert tc["enum"] == ["dazzled", "blinded", "deafened", "entangled", "shaken", "sickened"]
|
||||
|
||||
# ready_trigger: nullable enum (4 values)
|
||||
rt = props["ready_trigger"]
|
||||
assert rt["type"] == ["string", "null"]
|
||||
assert rt["enum"] == ["enemy_enters_reach", "enemy_casts", "enemy_attacks", "enemy_hits_ally"]
|
||||
|
||||
# metamagic: string[]
|
||||
meta = props["metamagic"]
|
||||
assert meta["type"] == "array"
|
||||
assert meta["items"] == {"type": "string"}
|
||||
|
||||
# class_ability is a nullable string field
|
||||
assert props["class_ability"] == {"type": ["string", "null"]}
|
||||
|
||||
|
||||
def test_render_candidates() -> None:
|
||||
# Single action with payload
|
||||
rendered = render_candidates((Action(kind="move", destination=(2, 3)),))
|
||||
assert rendered == ('0. move {"destination":[2,3]}',)
|
||||
|
||||
# Action with no payload (all defaults/None)
|
||||
rendered = render_candidates((Action(kind="total_defense"),))
|
||||
assert rendered == ("0. total_defense",)
|
||||
|
||||
# Action with target_id
|
||||
rendered = render_candidates((Action(kind="attack", target_id="orc"),))
|
||||
assert rendered == ('0. attack {"target_id":"orc"}',)
|
||||
|
||||
# Empty menu
|
||||
assert render_candidates(()) == ()
|
||||
|
||||
# Multiple actions
|
||||
rendered = render_candidates((
|
||||
Action(kind="attack", target_id="a"),
|
||||
Action(kind="move", destination=(1, 1)),
|
||||
))
|
||||
assert rendered == (
|
||||
'0. attack {"target_id":"a"}',
|
||||
'1. move {"destination":[1,1]}',
|
||||
)
|
||||
|
||||
|
||||
def test_prompt_builder_build_deterministic() -> None:
|
||||
builder = _builder()
|
||||
obs = _obs()
|
||||
prompt1 = builder.build(obs)
|
||||
prompt2 = builder.build(obs)
|
||||
assert prompt1 == prompt2
|
||||
|
||||
|
||||
def test_prompt_builder_sections_present() -> None:
|
||||
builder = _builder()
|
||||
obs = _obs()
|
||||
prompt = builder.build(obs)
|
||||
|
||||
# Template content
|
||||
assert "tactical director" in prompt
|
||||
assert "battle state" in prompt
|
||||
|
||||
# Sections
|
||||
assert "## observation\n" in prompt
|
||||
assert "## plausible actions\n" in prompt
|
||||
assert "## action schema\n" in prompt
|
||||
|
||||
# Schema is valid JSON and contains kind enum
|
||||
schema_part = prompt.split("## action schema\n")[1]
|
||||
parsed = json.loads(schema_part)
|
||||
assert parsed["properties"]["kind"]["enum"] == list(_supported_kinds())
|
||||
|
||||
|
||||
def test_prompt_version_format() -> None:
|
||||
builder = _builder()
|
||||
pv = builder.prompt_version(_FIXTURE_PROMPT_NAME)
|
||||
assert pv.startswith(f"{_FIXTURE_VERSION}:{_FIXTURE_PROMPT_NAME}:")
|
||||
digest = pv.split(":")[-1]
|
||||
assert len(digest) == 12
|
||||
assert all(c in "0123456789abcdef" for c in digest)
|
||||
# Cross-check against actual pack content
|
||||
pack = _pack()
|
||||
content = pack.get(_FIXTURE_PROMPT_NAME)
|
||||
expected_digest = hashlib.sha256(content.encode("utf-8")).hexdigest()[:12]
|
||||
assert digest == expected_digest
|
||||
|
||||
|
||||
def test_prompt_builder_custom_prompt_name() -> None:
|
||||
builder = _builder()
|
||||
obs = _obs()
|
||||
# nl_default is the only fixture, but the API accepts any name
|
||||
prompt = builder.build(obs, prompt_name=_FIXTURE_PROMPT_NAME)
|
||||
assert "tactical director" in prompt
|
||||
|
||||
|
||||
def test_prompt_version_delegates_to_pack() -> None:
|
||||
builder = _builder()
|
||||
pack = _pack()
|
||||
assert builder.prompt_version(_FIXTURE_PROMPT_NAME) == pack.prompt_version(_FIXTURE_PROMPT_NAME)
|
||||
|
||||
|
||||
def test_action_schema_cross_pin_with_nl_policy_kindvalue() -> None:
|
||||
"""The kind enum in the schema must match nl_policy.KindValue exactly."""
|
||||
schema = action_schema()
|
||||
assert schema["properties"]["kind"]["enum"] == list(_get_args(KindValue))
|
||||
|
||||
|
||||
# ---- few-shot examples (P5.2) ----------------------------------------------
|
||||
|
||||
_FS_PACK = PromptPack.load(_FIXTURE_PACK_FS)
|
||||
|
||||
|
||||
def _fs_builder() -> PromptBuilder:
|
||||
return PromptBuilder(pack=_FS_PACK)
|
||||
|
||||
|
||||
def test_render_examples_block_format() -> None:
|
||||
"""Few-shot pairs render as numbered observation-to-response blocks."""
|
||||
rendered = _render_examples((_FS_PACK.examples(_FIXTURE_PROMPT_NAME)[0],))
|
||||
assert rendered == (
|
||||
"## few-shot examples\n"
|
||||
"### example 1\n"
|
||||
"observation:\n"
|
||||
'{"combatant": "goblin", "distance_ft": 10, "enemy": "orc", "round": 1}\n'
|
||||
"\n"
|
||||
"response:\n"
|
||||
'[{"kind": "attack", "target_id": "orc"}]'
|
||||
)
|
||||
|
||||
|
||||
def test_build_without_examples_unchanged() -> None:
|
||||
"""Packs without examples keep the pre-few-shot bit-for-bit section order."""
|
||||
plain = _builder().build(_obs())
|
||||
assert "## few-shot examples" not in plain
|
||||
# Template, then observation, then candidates, then schema — in order.
|
||||
sections = [
|
||||
"## observation\n",
|
||||
"## plausible actions\n",
|
||||
"## action schema\n",
|
||||
]
|
||||
positions = [plain.index(section) for section in sections]
|
||||
assert positions == sorted(positions)
|
||||
# The template comes first, unmodified.
|
||||
assert plain.startswith("You are the tactical director")
|
||||
|
||||
|
||||
def test_build_with_examples_inserts_between_template_and_observation() -> None:
|
||||
"""The few-shot section sits between the template and the observation."""
|
||||
prompt = _fs_builder().build(_obs())
|
||||
assert "## few-shot examples\n" in prompt
|
||||
assert prompt.index("## few-shot examples\n") < prompt.index("## observation\n")
|
||||
# Template still leads the prompt.
|
||||
assert prompt.startswith("You are the tactical director")
|
||||
# Observation/candidates/schema keep their relative order.
|
||||
assert prompt.index("## observation\n") < prompt.index("## plausible actions\n")
|
||||
assert prompt.index("## plausible actions\n") < prompt.index("## action schema\n")
|
||||
|
||||
|
||||
def test_build_with_examples_includes_example_content() -> None:
|
||||
"""The rendered example content is embedded verbatim (sorted keys)."""
|
||||
prompt = _fs_builder().build(_obs())
|
||||
assert '{"combatant": "goblin", "distance_ft": 10, "enemy": "orc", "round": 1}' in prompt
|
||||
assert '[{"kind": "attack", "target_id": "orc"}]' in prompt
|
||||
assert "### example 1" in prompt
|
||||
|
||||
|
||||
def test_empty_examples_array_no_section(tmp_path: Path) -> None:
|
||||
"""An empty examples array yields no few-shot section (not an error)."""
|
||||
root = tmp_path / "emptyfs"
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
(root / "version.txt").write_text("1.0.0", encoding="utf-8")
|
||||
(root / "nl_default.md").write_text("T", encoding="utf-8")
|
||||
(root / "nl_default.examples.json").write_text("[]", encoding="utf-8")
|
||||
pack = PromptPack.load(root)
|
||||
assert pack.examples("nl_default") == ()
|
||||
prompt = PromptBuilder(pack=pack).build(_obs())
|
||||
assert "## few-shot examples" not in prompt
|
||||
@@ -0,0 +1,336 @@
|
||||
"""Phase-2 real provider client (docs/phase-2 P5.1, sections 15/16).
|
||||
|
||||
``OpenAILLMClient`` (``provider.py``) speaks the OpenAI chat-completions
|
||||
wire format over ``urllib.request`` with zero HTTP dependency. These tests
|
||||
pin the wire format, the error taxonomy (transient 429/5xx retried, 4xx
|
||||
rejected, invalid payload rejected), the env-key resolution, the token
|
||||
ledger and the cache behaviour - all through an injectable fake transport,
|
||||
never the network. Every key used below is a fake test credential.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Self
|
||||
from urllib.error import HTTPError
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.client import (
|
||||
RetryExhaustedError,
|
||||
estimate_tokens,
|
||||
)
|
||||
from pf1e_simulator.config import (
|
||||
ENV_LLM_API_KEY,
|
||||
ENV_LLM_BASE_URL,
|
||||
ENV_LLM_MODEL,
|
||||
)
|
||||
from pf1e_simulator.provider import (
|
||||
OpenAILLMClient,
|
||||
ProviderRejectedError,
|
||||
ProviderResponseError,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from urllib.request import Request
|
||||
|
||||
_KEY = "sk-test"
|
||||
_ENV_KEY = "sk-env-key"
|
||||
_PROMPT_VERSION = "fixture:provider:0"
|
||||
_BASE_URL = "https://api.openai.com/v1"
|
||||
_MODEL = "gpt-4o-mini"
|
||||
|
||||
|
||||
class _FakeResponse:
|
||||
"""Minimal ``urllib.response`` double: status + body, context manager."""
|
||||
|
||||
def __init__(self, status: int, body: bytes) -> None:
|
||||
self.status = status
|
||||
self._body = body
|
||||
|
||||
def read(self) -> bytes:
|
||||
return self._body
|
||||
|
||||
def __enter__(self) -> Self:
|
||||
return self
|
||||
|
||||
def __exit__(self, _exc_type: object, _exc_value: object, _traceback: object) -> None:
|
||||
return None
|
||||
|
||||
|
||||
class _FakeTransport:
|
||||
"""Injectable ``urlopen``: replays a script, keeping the last entry.
|
||||
|
||||
An ``Exception`` entry is raised; provider code sees it exactly as it
|
||||
would see a real transport failure.
|
||||
"""
|
||||
|
||||
def __init__(self, script: Sequence[_FakeResponse | Exception]) -> None:
|
||||
self._script = tuple(script)
|
||||
self._calls: list[tuple[Request, float]] = []
|
||||
|
||||
def __call__(self, request: Request, *, timeout: float) -> _FakeResponse:
|
||||
self._calls.append((request, timeout))
|
||||
index = min(len(self._calls) - 1, len(self._script) - 1)
|
||||
entry = self._script[index]
|
||||
if isinstance(entry, Exception):
|
||||
raise entry
|
||||
return entry
|
||||
|
||||
@property
|
||||
def calls(self) -> tuple[tuple[Request, float], ...]:
|
||||
return tuple(self._calls)
|
||||
|
||||
|
||||
def _http_error(status: int) -> HTTPError:
|
||||
return HTTPError(
|
||||
f"{_BASE_URL}/chat/completions",
|
||||
status,
|
||||
"simulated HTTP error",
|
||||
hdrs={},
|
||||
fp=io.BytesIO(b"{}"),
|
||||
)
|
||||
|
||||
|
||||
def _ok_response(*, content: str = "action json", include_usage: bool = True) -> _FakeResponse:
|
||||
body: dict[str, object] = {
|
||||
"choices": [{"message": {"content": content}}],
|
||||
}
|
||||
if include_usage:
|
||||
body["usage"] = {"prompt_tokens": 12, "completion_tokens": 7}
|
||||
return _FakeResponse(200, json.dumps(body).encode("utf-8"))
|
||||
|
||||
|
||||
def _fake_transport(script: Sequence[_FakeResponse | Exception]) -> _FakeTransport:
|
||||
return _FakeTransport(script)
|
||||
|
||||
|
||||
def _client(
|
||||
transport: _FakeTransport,
|
||||
*,
|
||||
api_key: str = _KEY,
|
||||
retries: int = 2,
|
||||
timeout_s: float = 30.0,
|
||||
base_url: str = _BASE_URL,
|
||||
model: str = _MODEL,
|
||||
) -> OpenAILLMClient:
|
||||
return OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
model=model,
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
timeout_s=timeout_s,
|
||||
urlopen=transport,
|
||||
retries=retries,
|
||||
sleep=lambda _: None,
|
||||
)
|
||||
|
||||
|
||||
# ── wire format ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_wire_format_matches_openai_chat_completions() -> None:
|
||||
"""The request is a POST to /v1/chat/completions with the raw body."""
|
||||
transport = _fake_transport((_ok_response(),))
|
||||
client = _client(transport)
|
||||
assert client.complete("hello") == "action json"
|
||||
assert len(transport.calls) == 1
|
||||
request, timeout = transport.calls[0]
|
||||
assert timeout == 30.0
|
||||
assert request.full_url == f"{_BASE_URL}/chat/completions"
|
||||
assert request.method == "POST"
|
||||
assert request.get_header("Authorization") == f"Bearer {_KEY}"
|
||||
# urllib capitalizes header keys via str.capitalize(): "Content-Type" is
|
||||
# stored as "Content-type".
|
||||
assert request.get_header("Content-type") == "application/json"
|
||||
assert request.data is not None
|
||||
body = json.loads(request.data)
|
||||
assert body["model"] == _MODEL
|
||||
assert body["messages"] == [{"role": "user", "content": "hello"}]
|
||||
assert body["temperature"] == 0.0
|
||||
assert body["max_tokens"] == 1024
|
||||
assert body["top_p"] == 1.0
|
||||
assert body["response_format"] == {"type": "json_object"}
|
||||
|
||||
|
||||
def test_ledger_records_provider_usage() -> None:
|
||||
"""Provider-reported token counts land in the client ledger."""
|
||||
transport = _fake_transport((_ok_response(),))
|
||||
client = _client(transport)
|
||||
client.complete("hello")
|
||||
assert client.stats.calls == 1
|
||||
assert client.stats.prompt_tokens == 12
|
||||
assert client.stats.completion_tokens == 7
|
||||
assert client.stats.cost_usd == 0.0
|
||||
assert client.stats.cache_hits == 0
|
||||
assert client.stats.cache_misses == 1
|
||||
|
||||
|
||||
def test_usage_estimated_when_provider_omits_it() -> None:
|
||||
"""Without a usage block, tokens are estimated from character counts."""
|
||||
transport = _fake_transport((_ok_response(include_usage=False),))
|
||||
client = _client(transport)
|
||||
client.complete("hello")
|
||||
assert client.stats.prompt_tokens == estimate_tokens("hello")
|
||||
assert client.stats.completion_tokens == estimate_tokens("action json")
|
||||
|
||||
|
||||
# ── key resolution ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_missing_or_empty_api_key_raises(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A missing or empty key is rejected with the env-var name in the error."""
|
||||
monkeypatch.delenv(ENV_LLM_API_KEY, raising=False)
|
||||
with pytest.raises(ValueError, match="PF1E_LLM_API_KEY"):
|
||||
OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
api_key=None,
|
||||
base_url=_BASE_URL,
|
||||
urlopen=_fake_transport((_ok_response(),)),
|
||||
)
|
||||
with pytest.raises(ValueError, match="PF1E_LLM_API_KEY"):
|
||||
OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
api_key="",
|
||||
base_url=_BASE_URL,
|
||||
urlopen=_fake_transport((_ok_response(),)),
|
||||
)
|
||||
|
||||
|
||||
def test_api_key_read_from_environment(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""With no explicit key, PF1E_LLM_API_KEY is used for the bearer token."""
|
||||
monkeypatch.setenv(ENV_LLM_API_KEY, _ENV_KEY)
|
||||
transport = _fake_transport((_ok_response(),))
|
||||
client = OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
api_key=None,
|
||||
base_url=_BASE_URL,
|
||||
urlopen=transport,
|
||||
sleep=lambda _: None,
|
||||
)
|
||||
client.complete("hello")
|
||||
request, _timeout = transport.calls[0]
|
||||
assert request.get_header("Authorization") == f"Bearer {_ENV_KEY}"
|
||||
|
||||
|
||||
# ── error taxonomy and retries ──────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_rate_limit_retries_then_succeeds() -> None:
|
||||
"""HTTP 429 is transient: with retries=2 the third attempt succeeds."""
|
||||
transport = _fake_transport(
|
||||
(_http_error(429), _http_error(429), _ok_response()),
|
||||
)
|
||||
client = _client(transport, retries=2)
|
||||
assert client.complete("hello") == "action json"
|
||||
assert client.calls == 3
|
||||
|
||||
|
||||
def test_transient_oserror_retries_then_exhausts() -> None:
|
||||
"""A transport error is retried, then RetryExhaustedError names it."""
|
||||
transport = _fake_transport((TimeoutError("simulated timeout"),))
|
||||
client = _client(transport, retries=1)
|
||||
with pytest.raises(RetryExhaustedError, match="unreachable"):
|
||||
client.complete("hello")
|
||||
assert client.calls == 2
|
||||
|
||||
|
||||
def test_permanent_4xx_is_not_retried() -> None:
|
||||
"""HTTP 401 is a permanent rejection: one attempt, then raise."""
|
||||
transport = _fake_transport((_http_error(401),))
|
||||
client = _client(transport, retries=3)
|
||||
with pytest.raises(ProviderRejectedError, match="HTTP 401"):
|
||||
client.complete("hello")
|
||||
assert client.calls == 1
|
||||
|
||||
|
||||
def test_server_error_retries_then_exhausts() -> None:
|
||||
"""HTTP 500 is transient: retries exhausted raises with the status."""
|
||||
transport = _fake_transport((_http_error(500),))
|
||||
client = _client(transport, retries=1)
|
||||
with pytest.raises(RetryExhaustedError, match="HTTP 500"):
|
||||
client.complete("hello")
|
||||
assert client.calls == 2
|
||||
|
||||
|
||||
# ── payload validation ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_invalid_json_body_raises_provider_response_error() -> None:
|
||||
"""A 2xx body that is not JSON is a permanent response error."""
|
||||
transport = _fake_transport((_FakeResponse(200, b"<html>oops</html>"),))
|
||||
client = _client(transport)
|
||||
with pytest.raises(ProviderResponseError, match="not valid JSON"):
|
||||
client.complete("hello")
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "message"),
|
||||
[
|
||||
({"choices": []}, "no choices"),
|
||||
({"choices": [{}]}, "message"),
|
||||
({"choices": [{"message": {}}]}, "content"),
|
||||
([], "not a JSON object"),
|
||||
],
|
||||
)
|
||||
def test_malformed_payload_raises_provider_response_error(
|
||||
payload: object,
|
||||
message: str,
|
||||
) -> None:
|
||||
"""Non-chat-completions payloads are rejected with a stable message."""
|
||||
transport = _fake_transport(
|
||||
(_FakeResponse(200, json.dumps(payload).encode("utf-8")),),
|
||||
)
|
||||
client = _client(transport)
|
||||
with pytest.raises(ProviderResponseError, match=message):
|
||||
client.complete("hello")
|
||||
|
||||
|
||||
# ── caching and environment defaults ────────────────────────────────────────
|
||||
|
||||
|
||||
def test_cache_serves_second_call_without_provider() -> None:
|
||||
"""An identical second call is served from the cache, no wire traffic."""
|
||||
transport = _fake_transport((_ok_response(),))
|
||||
client = _client(transport)
|
||||
assert client.complete("hello") == "action json"
|
||||
assert client.complete("hello") == "action json"
|
||||
assert len(transport.calls) == 1
|
||||
assert client.calls == 1
|
||||
assert client.cache_hits == 1
|
||||
assert client.cache_misses == 1
|
||||
|
||||
|
||||
def test_base_url_and_model_read_from_environment(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""PF1E_LLM_BASE_URL/PF1E_LLM_MODEL drive the endpoint and the body."""
|
||||
monkeypatch.setenv(ENV_LLM_BASE_URL, "http://localhost:8080/v1/")
|
||||
monkeypatch.setenv(ENV_LLM_MODEL, "local-model")
|
||||
transport = _fake_transport((_ok_response(),))
|
||||
client = OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
api_key=_KEY,
|
||||
base_url=None,
|
||||
model=None,
|
||||
urlopen=transport,
|
||||
sleep=lambda _: None,
|
||||
)
|
||||
client.complete("hello")
|
||||
request, _timeout = transport.calls[0]
|
||||
assert request.full_url == "http://localhost:8080/v1/chat/completions"
|
||||
assert request.data is not None
|
||||
assert json.loads(request.data)["model"] == "local-model"
|
||||
|
||||
|
||||
def test_invalid_timeout_rejected() -> None:
|
||||
"""A non-positive timeout_s is a construction error, not a wire call."""
|
||||
with pytest.raises(ValueError, match="timeout_s"):
|
||||
OpenAILLMClient(
|
||||
prompt_version=_PROMPT_VERSION,
|
||||
api_key=_KEY,
|
||||
base_url=_BASE_URL,
|
||||
timeout_s=0,
|
||||
)
|
||||
@@ -0,0 +1,234 @@
|
||||
"""Phase-2 replay log (docs/phase-2-llm-architecture.md P0.5).
|
||||
|
||||
``replay.py`` records every ``(observation, action)`` pair a policy produced,
|
||||
keyed by the execution-order ``call_seq``, and serializes the whole battle
|
||||
(spec + seed + round cap + calls) to JSONL. ``replay()`` turns the log back
|
||||
into an observation-only policy returning the recorded actions.
|
||||
|
||||
These tests pin the contract: JSONL round-trips to canonically-equal records
|
||||
(``observation_key`` ignores JSON's tuple/list collapse), a replayed battle is
|
||||
bit-identical to the recorded one (including from a serialized log), a
|
||||
duplicate ``call_seq`` is rejected, and playback fails loudly on a mismatched
|
||||
observation and once the log is exhausted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.candidates import candidates
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.replay import (
|
||||
ReplayError,
|
||||
ReplayExhaustedError,
|
||||
ReplayLog,
|
||||
ReplayMismatchError,
|
||||
ReplayRecord,
|
||||
observation_key,
|
||||
replay,
|
||||
)
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.combat import Action, CombatantState, Observation
|
||||
from pf1e_simulator.models import Combatant
|
||||
from pf1e_simulator.runner import EncounterReport
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
SEED = 7
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _duel_spec() -> EncounterSpec:
|
||||
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class _Recorder:
|
||||
"""Legacy two-argument policy: log every call, then pick ``candidates[0]``."""
|
||||
|
||||
def __init__(self, spec: EncounterSpec, seed: int, round_cap: int) -> None:
|
||||
self.log = ReplayLog(spec=spec, seed=seed, round_cap=round_cap)
|
||||
|
||||
def policy(
|
||||
self, observation: Observation, _state: CombatantState
|
||||
) -> tuple[Action, ...]:
|
||||
action = candidates(observation)[0]
|
||||
self.log.add(
|
||||
ReplayRecord(
|
||||
call_seq=len(self.log),
|
||||
observation=observation,
|
||||
action=action,
|
||||
)
|
||||
)
|
||||
return (action,)
|
||||
|
||||
|
||||
def _record(spec: EncounterSpec, *, seed: int = SEED) -> tuple[_Recorder, EncounterReport]:
|
||||
"""Record one battle and return the recorder plus its captured report."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
recorder = _Recorder(spec, seed=seed, round_cap=spec.round_cap)
|
||||
report = run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=seed,
|
||||
llm_policy=recorder.policy,
|
||||
spell_registry=registry,
|
||||
capture_first_transcript=True,
|
||||
)
|
||||
return recorder, report
|
||||
|
||||
|
||||
def test_jsonl_round_trip_preserves_records() -> None:
|
||||
"""A log serializes to JSONL and loads back to structurally-equal records."""
|
||||
spec = _duel_spec()
|
||||
recorder, _report = _record(spec)
|
||||
log = recorder.log
|
||||
|
||||
assert len(log) > 0
|
||||
assert [record.call_seq for record in log.records] == list(range(len(log)))
|
||||
|
||||
text = log.to_jsonl()
|
||||
restored = ReplayLog.from_jsonl(text)
|
||||
|
||||
assert len(restored) == len(log)
|
||||
assert [record.call_seq for record in restored.records] == [
|
||||
record.call_seq for record in log.records
|
||||
]
|
||||
assert [
|
||||
observation_key(record.observation) for record in restored.records
|
||||
] == [observation_key(record.observation) for record in log.records]
|
||||
assert [record.action for record in restored.records] == [
|
||||
record.action for record in log.records
|
||||
]
|
||||
assert restored.spec == log.spec
|
||||
assert restored.seed == log.seed
|
||||
assert restored.round_cap == log.round_cap
|
||||
assert restored.to_jsonl() == text
|
||||
|
||||
|
||||
def test_replay_reproduces_the_recorded_battle() -> None:
|
||||
"""Replaying the in-memory log matches the recorded battle exactly."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
recorder, recorded = _record(spec)
|
||||
|
||||
replayed = run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=SEED,
|
||||
llm_policy=replay(recorder.log),
|
||||
spell_registry=registry,
|
||||
capture_first_transcript=True,
|
||||
)
|
||||
|
||||
assert replayed.wins == recorded.wins
|
||||
assert replayed.draws == recorded.draws
|
||||
assert replayed.avg_rounds == recorded.avg_rounds
|
||||
assert replayed.transcript == recorded.transcript
|
||||
|
||||
|
||||
def test_replay_from_serialized_log_reproduces_the_battle() -> None:
|
||||
"""A log thawed from JSONL replays the battle bit-for-bit."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
recorder, recorded = _record(spec)
|
||||
|
||||
restored = ReplayLog.from_jsonl(recorder.log.to_jsonl())
|
||||
replayed = run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=SEED,
|
||||
llm_policy=replay(restored),
|
||||
spell_registry=registry,
|
||||
capture_first_transcript=True,
|
||||
)
|
||||
|
||||
assert replayed.wins == recorded.wins
|
||||
assert replayed.draws == recorded.draws
|
||||
assert replayed.avg_rounds == recorded.avg_rounds
|
||||
assert replayed.transcript == recorded.transcript
|
||||
|
||||
|
||||
def test_replay_rejects_a_mismatched_observation() -> None:
|
||||
"""An observation that differs from the recorded one raises a mismatch error."""
|
||||
spec = _duel_spec()
|
||||
recorder, _report = _record(spec)
|
||||
log = recorder.log
|
||||
policy = replay(log)
|
||||
first = log.records[0].observation
|
||||
tampered = replace(first, round=first.round + 1)
|
||||
|
||||
with pytest.raises(ReplayMismatchError):
|
||||
policy(tampered)
|
||||
|
||||
|
||||
def test_replay_returns_each_recorded_action_then_exhausts() -> None:
|
||||
"""Playback walks every record in order, then raises once past the end."""
|
||||
spec = _duel_spec()
|
||||
recorder, _report = _record(spec)
|
||||
log = recorder.log
|
||||
policy = replay(log)
|
||||
|
||||
for record in log.records:
|
||||
assert policy(record.observation) == (record.action,)
|
||||
|
||||
with pytest.raises(ReplayExhaustedError):
|
||||
policy(log.records[-1].observation)
|
||||
|
||||
|
||||
def test_replay_log_rejects_a_duplicate_call_seq() -> None:
|
||||
"""Adding two records with the same ``call_seq`` is an ambiguity error."""
|
||||
spec = _duel_spec()
|
||||
recorder, _report = _record(spec)
|
||||
record = recorder.log.records[-1]
|
||||
|
||||
with pytest.raises(ReplayError, match="duplicate call_seq"):
|
||||
recorder.log.add(
|
||||
ReplayRecord(
|
||||
call_seq=record.call_seq,
|
||||
observation=record.observation,
|
||||
action=record.action,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_replay_log_rejects_an_out_of_order_call_seq() -> None:
|
||||
"""A ``call_seq`` below the last one is an ambiguity error (not a duplicate)."""
|
||||
spec = _duel_spec()
|
||||
recorder, _report = _record(spec)
|
||||
record = recorder.log.records[0]
|
||||
|
||||
with pytest.raises(ReplayError, match="is not after"):
|
||||
recorder.log.add(
|
||||
ReplayRecord(
|
||||
call_seq=record.call_seq,
|
||||
observation=record.observation,
|
||||
action=record.action,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_from_jsonl_rejects_garbage() -> None:
|
||||
"""An empty or unparseable payload raises ``ReplayError``, not a raw error."""
|
||||
with pytest.raises(ReplayError):
|
||||
ReplayLog.from_jsonl("")
|
||||
with pytest.raises(ReplayError):
|
||||
ReplayLog.from_jsonl("{not json}")
|
||||
@@ -0,0 +1,599 @@
|
||||
"""Tests for the robustness layer (P4.2/P4.3, docs section 10)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import itertools
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator import robustness
|
||||
from pf1e_simulator.adversary import AdversarySpec
|
||||
from pf1e_simulator.robustness import (
|
||||
DangerCell,
|
||||
MatchupBenchmark,
|
||||
RobustnessEntry,
|
||||
SensitivityEntry,
|
||||
adversarial_analysis,
|
||||
danger_map,
|
||||
format_danger_map,
|
||||
format_frontier,
|
||||
format_matchup,
|
||||
format_robustness,
|
||||
format_sensitivity,
|
||||
frontier,
|
||||
matchup_benchmark,
|
||||
sensitivity,
|
||||
)
|
||||
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec
|
||||
from pf1e_simulator.strategy import StrategySpec
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
RUNS = 5
|
||||
SEED = 7
|
||||
ROUND_CAP = 20
|
||||
PARTY = "players"
|
||||
|
||||
ARENA = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
|
||||
ORC = ROOT / "data" / "monsters" / "orc.json"
|
||||
|
||||
|
||||
def _scenario(
|
||||
*, name: str = "goblins-vs-orcs", sides: tuple[SideSpec, ...] | None = None
|
||||
) -> ScenarioSpec:
|
||||
return ScenarioSpec(
|
||||
name=name,
|
||||
map_file=ARENA,
|
||||
sides=sides
|
||||
or (
|
||||
SideSpec(PARTY, "A", (GOBLIN, GOBLIN, GOBLIN)),
|
||||
SideSpec("monsters", "B", (ORC, ORC)),
|
||||
),
|
||||
round_cap=ROUND_CAP,
|
||||
)
|
||||
|
||||
|
||||
def _party(*, name: str = "default") -> StrategySpec:
|
||||
return StrategySpec(name=name)
|
||||
|
||||
|
||||
def _three_side_scenario() -> ScenarioSpec:
|
||||
return ScenarioSpec(
|
||||
name="three-sides",
|
||||
map_file=ARENA,
|
||||
sides=(
|
||||
SideSpec(PARTY, "A", (GOBLIN,)),
|
||||
SideSpec("monsters", "B", (ORC,)),
|
||||
SideSpec("spectators", "C", (GOBLIN,)),
|
||||
),
|
||||
round_cap=ROUND_CAP,
|
||||
)
|
||||
|
||||
|
||||
def test_matchup_requires_exactly_two_sides() -> None:
|
||||
with pytest.raises(ValueError, match="exactly 2 sides"):
|
||||
matchup_benchmark(
|
||||
_three_side_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED
|
||||
)
|
||||
|
||||
|
||||
def test_matchup_requires_valid_party_side() -> None:
|
||||
scenario = _scenario()
|
||||
with pytest.raises(ValueError, match="party_side"):
|
||||
matchup_benchmark(
|
||||
scenario, _party(), AdversarySpec(), party_side="", runs=RUNS, seed=SEED
|
||||
)
|
||||
with pytest.raises(ValueError, match="is not a side"):
|
||||
matchup_benchmark(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
party_side="spectators",
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
|
||||
def test_matchup_reports_both_dangers_and_is_reproducible() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
adversary = AdversarySpec()
|
||||
first = matchup_benchmark(scenario, party, adversary, runs=RUNS, seed=SEED)
|
||||
assert isinstance(first, MatchupBenchmark)
|
||||
assert first.scenario == "goblins-vs-orcs"
|
||||
assert first.party == "default"
|
||||
assert first.adversary == "default"
|
||||
assert first.runs == RUNS
|
||||
assert first.seed == SEED
|
||||
assert first.party_side == PARTY
|
||||
assert first.report.runs == RUNS
|
||||
for danger in (first.party_danger, first.adversary_danger):
|
||||
assert 0.0 <= danger.p_survives <= 1.0
|
||||
assert 0.0 <= danger.p_tpk <= 1.0
|
||||
assert 0.0 <= danger.p_any_death <= 1.0
|
||||
assert danger.expected_hp_lost >= 0.0
|
||||
second = matchup_benchmark(scenario, party, adversary, runs=RUNS, seed=SEED)
|
||||
assert first == second
|
||||
assert second.party_danger == first.party_danger
|
||||
|
||||
|
||||
def test_adversarial_analysis_alternates_and_converges() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
specs = (
|
||||
AdversarySpec(name="retreat", retreat_hp_fraction=0.5),
|
||||
AdversarySpec(name="plain"),
|
||||
)
|
||||
entries = adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[party],
|
||||
adversary_candidates=specs,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
iterations=2,
|
||||
)
|
||||
assert len(entries) == 2
|
||||
assert all(isinstance(entry, RobustnessEntry) for entry in entries)
|
||||
assert [entry.iteration for entry in entries] == [1, 2]
|
||||
# the adversary best response is the candidate with the worst party outlook
|
||||
benchmarks = {
|
||||
spec.name: matchup_benchmark(
|
||||
scenario, party, spec, runs=RUNS, seed=SEED
|
||||
)
|
||||
for spec in specs
|
||||
}
|
||||
expected = min(
|
||||
benchmarks.values(),
|
||||
key=lambda b: (
|
||||
b.party_danger.p_survives,
|
||||
-b.party_danger.expected_hp_lost,
|
||||
b.adversary,
|
||||
),
|
||||
)
|
||||
assert all(entry.benchmark == expected for entry in entries)
|
||||
assert entries[0].adversary.name == expected.adversary
|
||||
|
||||
|
||||
def test_adversarial_party_best_response() -> None:
|
||||
scenario = _scenario()
|
||||
parties = (
|
||||
StrategySpec(),
|
||||
StrategySpec(name="bully", target_priority="lowest_hp"),
|
||||
)
|
||||
adversary = AdversarySpec()
|
||||
entries = adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=parties,
|
||||
adversary_candidates=[adversary],
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
iterations=1,
|
||||
)
|
||||
benchmarks = {
|
||||
spec.name: matchup_benchmark(
|
||||
scenario, spec, adversary, runs=RUNS, seed=SEED
|
||||
)
|
||||
for spec in parties
|
||||
}
|
||||
expected = max(
|
||||
benchmarks.values(),
|
||||
key=lambda b: (
|
||||
b.party_danger.p_survives,
|
||||
-b.party_danger.expected_hp_lost,
|
||||
b.party,
|
||||
),
|
||||
)
|
||||
assert entries[0].benchmark == expected
|
||||
|
||||
|
||||
def test_adversarial_candidates_required() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
with pytest.raises(ValueError, match="party_candidates must be non-empty"):
|
||||
adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[],
|
||||
adversary_candidates=[AdversarySpec()],
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="adversary_candidates must be non-empty"):
|
||||
adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[party],
|
||||
adversary_candidates=[],
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
|
||||
def test_adversarial_budget_is_enforced_before_any_battle() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
specs = (
|
||||
AdversarySpec(name="retreat", retreat_hp_fraction=0.5),
|
||||
AdversarySpec(name="plain"),
|
||||
)
|
||||
cost = 2 * (1 + len(specs)) * RUNS
|
||||
with pytest.raises(ValueError, match="robustness budget exceeded"):
|
||||
adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[party],
|
||||
adversary_candidates=specs,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
iterations=2,
|
||||
max_total_runs=cost - 1,
|
||||
)
|
||||
entries = adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[party],
|
||||
adversary_candidates=specs,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
iterations=2,
|
||||
max_total_runs=cost,
|
||||
)
|
||||
assert len(entries) == 2
|
||||
|
||||
|
||||
def test_danger_map_is_3x3_party_major_with_variant_names() -> None:
|
||||
scenario = _scenario()
|
||||
cells = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
|
||||
assert len(cells) == 9
|
||||
assert all(isinstance(cell, DangerCell) for cell in cells)
|
||||
order = [(cell.party_hp_fraction, cell.adversary_hp_fraction) for cell in cells]
|
||||
assert order == [(p, a) for p in (0.5, 0.7, 1.0) for a in (0.5, 0.7, 1.0)]
|
||||
assert cells[0].benchmark.scenario == "goblins-vs-orcs party_hp=0.5 adv_hp=0.5"
|
||||
assert cells[-1].benchmark.scenario == "goblins-vs-orcs party_hp=1.0 adv_hp=1.0"
|
||||
for cell in cells:
|
||||
assert 0.0 <= cell.benchmark.party_danger.p_survives <= 1.0
|
||||
|
||||
|
||||
def test_danger_map_rejects_out_of_range_fractions() -> None:
|
||||
scenario = _scenario()
|
||||
with pytest.raises(ValueError, match="fraction must be in"):
|
||||
danger_map(
|
||||
scenario, _party(), AdversarySpec(), fractions=(0.0,), runs=RUNS, seed=SEED
|
||||
)
|
||||
with pytest.raises(ValueError, match="fraction must be in"):
|
||||
danger_map(
|
||||
scenario, _party(), AdversarySpec(), fractions=(1.5,), runs=RUNS, seed=SEED
|
||||
)
|
||||
with pytest.raises(ValueError, match="fractions must be non-empty"):
|
||||
danger_map(
|
||||
scenario, _party(), AdversarySpec(), fractions=(), runs=RUNS, seed=SEED
|
||||
)
|
||||
|
||||
|
||||
def test_danger_map_is_reproducible() -> None:
|
||||
scenario = _scenario()
|
||||
first = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
|
||||
second = danger_map(scenario, _party(), AdversarySpec(), runs=RUNS, seed=SEED)
|
||||
assert first == second
|
||||
|
||||
|
||||
def test_frontier_finds_reached_threshold() -> None:
|
||||
scenario = _scenario()
|
||||
result = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
min_fraction=0.25,
|
||||
max_fraction=1.0,
|
||||
precision=0.05,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
# expected hp lost is always >= 0, so the threshold is crossed everywhere:
|
||||
# bisection converges to the smallest probed fraction, rounded to precision
|
||||
assert result == pytest.approx(0.25)
|
||||
|
||||
|
||||
def test_frontier_returns_none_when_threshold_unreachable() -> None:
|
||||
scenario = _scenario()
|
||||
result = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=1e9,
|
||||
min_fraction=0.25,
|
||||
max_fraction=1.0,
|
||||
precision=0.05,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
assert result is None
|
||||
|
||||
|
||||
def test_frontier_is_reproducible() -> None:
|
||||
scenario = _scenario()
|
||||
first = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
second = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
assert first == second
|
||||
|
||||
|
||||
def test_frontier_returns_precision_multiple() -> None:
|
||||
scenario = _scenario()
|
||||
result = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
min_fraction=0.3,
|
||||
max_fraction=1.0,
|
||||
precision=0.1,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
assert result is not None
|
||||
# bisection stops within precision of min_fraction; the result is the
|
||||
# nearest precision multiple in [min_fraction, min_fraction + precision]
|
||||
assert 0.3 <= result <= 0.4
|
||||
assert result == pytest.approx(round(result / 0.1) * 0.1)
|
||||
|
||||
|
||||
def test_frontier_p_survives_uses_inverse_threshold() -> None:
|
||||
scenario = _scenario()
|
||||
result = frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="p_survives",
|
||||
target=0.9999,
|
||||
min_fraction=0.5,
|
||||
max_fraction=1.0,
|
||||
precision=0.25,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
# convention: p_survives crosses when value <= target; a present result
|
||||
# is a precision multiple within the probed range
|
||||
if result is not None:
|
||||
assert 0.5 <= result <= 0.75
|
||||
assert result == pytest.approx(round(result / 0.25) * 0.25)
|
||||
|
||||
|
||||
def test_frontier_validates_arguments() -> None:
|
||||
scenario = _scenario()
|
||||
with pytest.raises(ValueError, match="criterion must be one of"):
|
||||
frontier(
|
||||
scenario, _party(), AdversarySpec(), criterion="bogus", runs=RUNS, seed=SEED
|
||||
)
|
||||
with pytest.raises(ValueError, match="target must be in"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="p_tpk",
|
||||
target=0.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="target must be in"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="p_survives",
|
||||
target=1.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="target must be >= 0"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=-1.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="min_fraction must be > 0"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
min_fraction=0.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="max_fraction must be <= 1"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
max_fraction=1.5,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="min_fraction must be <= max_fraction"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
min_fraction=0.9,
|
||||
max_fraction=0.5,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
with pytest.raises(ValueError, match="precision must be > 0"):
|
||||
frontier(
|
||||
scenario,
|
||||
_party(),
|
||||
AdversarySpec(),
|
||||
criterion="expected_hp_lost",
|
||||
target=0.0,
|
||||
precision=0.0,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
|
||||
def test_sensitivity_ranks_levers_and_reports_signed_deltas() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
entries = sensitivity(scenario, party, runs=RUNS, seed=SEED)
|
||||
assert len(entries) == 7
|
||||
assert all(isinstance(entry, SensitivityEntry) for entry in entries)
|
||||
assert {entry.levers for entry in entries} == {
|
||||
"weakest_ac",
|
||||
"lowest_hp",
|
||||
"healer",
|
||||
"spell_policy=none",
|
||||
"retreat=0.25",
|
||||
"retreat=0.5",
|
||||
"use_charges=false",
|
||||
}
|
||||
for previous, current in itertools.pairwise(entries):
|
||||
assert abs(previous.delta) >= abs(current.delta)
|
||||
if abs(previous.delta) == abs(current.delta):
|
||||
assert previous.levers <= current.levers
|
||||
# delta is the signed change vs the all-default baseline
|
||||
baseline = matchup_benchmark(
|
||||
scenario, party, AdversarySpec(), runs=RUNS, seed=SEED
|
||||
)
|
||||
base = baseline.party_danger.expected_hp_lost
|
||||
for entry in entries:
|
||||
assert entry.delta == pytest.approx(
|
||||
entry.benchmark.party_danger.expected_hp_lost - base
|
||||
)
|
||||
|
||||
|
||||
def test_sensitivity_rejects_unknown_criterion() -> None:
|
||||
scenario = _scenario()
|
||||
with pytest.raises(ValueError, match="criterion must be one of"):
|
||||
sensitivity(scenario, _party(), criterion="bogus", runs=RUNS, seed=SEED)
|
||||
|
||||
|
||||
def test_sensitivity_budget_is_enforced() -> None:
|
||||
scenario = _scenario()
|
||||
party = _party()
|
||||
with pytest.raises(ValueError, match="robustness budget exceeded"):
|
||||
sensitivity(
|
||||
scenario, party, runs=RUNS, seed=SEED, max_total_runs=8 * RUNS - 1
|
||||
)
|
||||
entries = sensitivity(
|
||||
scenario, party, runs=RUNS, seed=SEED, max_total_runs=8 * RUNS
|
||||
)
|
||||
assert len(entries) == 7
|
||||
|
||||
|
||||
def test_format_matchup_ascii_aligned() -> None:
|
||||
benchmark_ = matchup_benchmark(
|
||||
_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED
|
||||
)
|
||||
text = format_matchup(benchmark_)
|
||||
lines = text.splitlines()
|
||||
assert lines[0].split() == [
|
||||
"Scenario",
|
||||
"Runs",
|
||||
"Side",
|
||||
"Win%",
|
||||
"Survive",
|
||||
"TPK",
|
||||
">=1-death",
|
||||
"HP-lost",
|
||||
"Rounds",
|
||||
]
|
||||
assert lines[1] == "-" * len(lines[0])
|
||||
assert "party" in lines[2]
|
||||
assert "adversary" in lines[3]
|
||||
assert all(len(line) == len(lines[0]) for line in lines)
|
||||
assert text.isascii()
|
||||
|
||||
|
||||
def test_format_robustness_ascii_aligned() -> None:
|
||||
scenario = _scenario()
|
||||
entries = adversarial_analysis(
|
||||
scenario,
|
||||
party_candidates=[_party()],
|
||||
adversary_candidates=[AdversarySpec()],
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
iterations=2,
|
||||
)
|
||||
text = format_robustness(entries)
|
||||
lines = text.splitlines()
|
||||
assert lines[0].split() == [
|
||||
"Iter",
|
||||
"Party",
|
||||
"Adversary",
|
||||
"P(party-survive)",
|
||||
"P(party-TPK)",
|
||||
"Party-HP",
|
||||
"P(adv-survive)",
|
||||
"P(adv-TPK)",
|
||||
"Adv-HP",
|
||||
]
|
||||
assert len(lines) == 2 + 2
|
||||
assert all(len(line) == len(lines[0]) for line in lines)
|
||||
assert text.isascii()
|
||||
|
||||
|
||||
def test_format_danger_map_ascii_aligned() -> None:
|
||||
cells = danger_map(_scenario(), _party(), AdversarySpec(), runs=RUNS, seed=SEED)
|
||||
text = format_danger_map(cells)
|
||||
lines = text.splitlines()
|
||||
assert lines[0].startswith("Party\\Adv")
|
||||
assert "adv=0.5" in lines[0]
|
||||
assert "party=0.5" in lines[2]
|
||||
assert "party=1.0" in lines[4]
|
||||
assert "/" in lines[2]
|
||||
assert all(len(line) == len(lines[0]) for line in lines)
|
||||
assert text.isascii()
|
||||
|
||||
|
||||
def test_format_frontier_messages() -> None:
|
||||
assert (
|
||||
format_frontier("p_tpk", 0.2, None, min_fraction=0.25, max_fraction=1.0)
|
||||
== "no frontier for p_tpk 0.2 in [0.25, 1.00]"
|
||||
)
|
||||
assert (
|
||||
format_frontier("p_tpk", 0.2, 0.35, min_fraction=0.25, max_fraction=1.0)
|
||||
== "frontier p_tpk 0.2: party hp_fraction = 0.35"
|
||||
)
|
||||
|
||||
|
||||
def test_format_sensitivity_ascii_aligned() -> None:
|
||||
entries = sensitivity(_scenario(), _party(), runs=RUNS, seed=SEED)
|
||||
text = format_sensitivity(entries, "expected_hp_lost")
|
||||
lines = text.splitlines()
|
||||
assert lines[0].split() == ["Rank", "Levers", "Value", "Delta"]
|
||||
assert len(lines) == 2 + 7
|
||||
assert lines[1].split()[-1].startswith(("+", "-"))
|
||||
assert all(len(line) == len(lines[0]) for line in lines)
|
||||
assert text.isascii()
|
||||
|
||||
|
||||
def test_robustness_module_does_not_import_search() -> None:
|
||||
source = Path(robustness.__file__).resolve().read_text(encoding="utf-8")
|
||||
assert "search" not in source
|
||||
@@ -0,0 +1,153 @@
|
||||
"""Phase-2 runner policy entry points (docs/phase-2-llm-architecture.md P0.4).
|
||||
|
||||
``run_encounter`` grows two Phase-2 keyword arguments: ``llm_policy`` (the
|
||||
observation-only/legacy policy every ``CombatEngine`` gets) and
|
||||
``policy_factory`` (a per-run ``Policy`` builder called once per battle with
|
||||
the run index). These tests pin the contract: the factory is invoked exactly
|
||||
once per run in index order, its result is the run's policy (``None`` falls
|
||||
back to ``default_policy``), the three entry points are mutually exclusive,
|
||||
and an ``llm_policy`` really drives a full battle through the runner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.combat import Action, default_policy
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.combat import Observation, Policy
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _duel_spec() -> EncounterSpec:
|
||||
"""A fast 1v1 (goblin archer vs orc) on the sample arena."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _wait_policy(_observation: Observation) -> tuple[Action, ...]:
|
||||
"""An observation-only policy that never attacks (forces a draw)."""
|
||||
return (Action(kind="wait"),)
|
||||
|
||||
|
||||
def test_policy_factory_called_once_per_run_in_order() -> None:
|
||||
"""The factory sees each run index once, and its policy drives the run."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
calls: list[int] = []
|
||||
|
||||
def factory(index: int) -> Policy | None:
|
||||
calls.append(index)
|
||||
return default_policy
|
||||
|
||||
report = run_encounter(
|
||||
spec, runs=4, seed=3, policy_factory=factory, spell_registry=registry
|
||||
)
|
||||
direct = run_encounter(
|
||||
spec, runs=4, seed=3, policy=default_policy, spell_registry=registry
|
||||
)
|
||||
|
||||
assert calls == [0, 1, 2, 3]
|
||||
# A per-run factory returning default_policy is bit-identical to passing
|
||||
# default_policy directly (same seed stream, same policy each run).
|
||||
assert report.wins == direct.wins
|
||||
assert report.draws == direct.draws
|
||||
assert report.avg_rounds == direct.avg_rounds
|
||||
assert report.attrition == direct.attrition
|
||||
|
||||
|
||||
def test_policy_factory_none_falls_back_to_default_policy() -> None:
|
||||
"""A factory returning None leaves the engine on default_policy."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
|
||||
report = run_encounter(
|
||||
spec,
|
||||
runs=2,
|
||||
seed=5,
|
||||
policy_factory=lambda _index: None,
|
||||
spell_registry=registry,
|
||||
)
|
||||
direct = run_encounter(spec, runs=2, seed=5, spell_registry=registry)
|
||||
|
||||
assert report.wins == direct.wins
|
||||
assert report.avg_rounds == direct.avg_rounds
|
||||
|
||||
|
||||
def test_policy_entry_points_mutually_exclusive() -> None:
|
||||
"""policy / llm_policy / policy_factory reject every conflicting pairing."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
|
||||
def factory(_index: int) -> Policy | None:
|
||||
return default_policy
|
||||
|
||||
with pytest.raises(ValueError, match="policy_factory and policy"):
|
||||
run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=1,
|
||||
policy=default_policy,
|
||||
policy_factory=factory,
|
||||
spell_registry=registry,
|
||||
)
|
||||
with pytest.raises(ValueError, match="policy_factory and llm_policy"):
|
||||
run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=1,
|
||||
llm_policy=_wait_policy,
|
||||
policy_factory=factory,
|
||||
spell_registry=registry,
|
||||
)
|
||||
with pytest.raises(ValueError, match="policy and llm_policy"):
|
||||
run_encounter(
|
||||
spec,
|
||||
runs=1,
|
||||
seed=1,
|
||||
policy=default_policy,
|
||||
llm_policy=_wait_policy,
|
||||
spell_registry=registry,
|
||||
)
|
||||
|
||||
|
||||
def test_llm_policy_drives_a_full_battle_through_the_runner() -> None:
|
||||
"""The observation-only policy is invoked each turn and settles the battle."""
|
||||
spec = _duel_spec()
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
seen: list[int] = []
|
||||
|
||||
def wait(observation: Observation) -> tuple[Action, ...]:
|
||||
seen.append(observation.round)
|
||||
return (Action(kind="wait"),)
|
||||
|
||||
report = run_encounter(
|
||||
spec, runs=1, seed=1, llm_policy=wait, spell_registry=registry
|
||||
)
|
||||
|
||||
# Both combatants wait every round: the policy is called for every turn
|
||||
# and nobody can win, so the battle draws at the round cap.
|
||||
assert seen
|
||||
assert report.draws == 1
|
||||
assert report.wins == {"players": 0, "monsters": 0}
|
||||
@@ -0,0 +1,206 @@
|
||||
"""Phase-2 scenario spec tests (docs/phase-2-llm-architecture.md 3.1, 6, 10 P3.1).
|
||||
|
||||
``ScenarioSpec``/``SideSpec`` are the reusable scenario search space of
|
||||
section 6; ``build_encounter`` compiles one deterministically into a runner
|
||||
``EncounterSpec`` (map + sides + round cap), and ``scenario_variants``
|
||||
implements the deterministic per-side HP-fraction sampling of section 3.1.
|
||||
These tests pin the round-trip (side order, ids, hp scaling, round cap), the
|
||||
validation errors of both layers, the variant naming/scaling, determinism,
|
||||
and one cheap integration run proving the compiled encounter is playable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.runner import run_encounter
|
||||
from pf1e_simulator.scenarios import (
|
||||
ScenarioSpec,
|
||||
SideSpec,
|
||||
build_encounter,
|
||||
scenario_variants,
|
||||
)
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
|
||||
ORC = ROOT / "data" / "monsters" / "orc.json"
|
||||
|
||||
|
||||
def _scenario(goblins: int = 3, orcs: int = 2) -> ScenarioSpec:
|
||||
return ScenarioSpec(
|
||||
name="goblins-vs-orcs",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(
|
||||
name="players",
|
||||
zone="A",
|
||||
combatant_files=tuple(GOBLIN for _ in range(goblins)),
|
||||
),
|
||||
SideSpec(
|
||||
name="monsters",
|
||||
zone="B",
|
||||
combatant_files=tuple(ORC for _ in range(orcs)),
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_build_encounter_round_trip() -> None:
|
||||
"""Side order/names/zones, combatant counts and ids, round cap pass through."""
|
||||
spec = build_encounter(_scenario(goblins=3, orcs=2))
|
||||
|
||||
assert [side.name for side in spec.sides] == ["players", "monsters"]
|
||||
assert [side.zone for side in spec.sides] == ["A", "B"]
|
||||
assert [len(side.combatants) for side in spec.sides] == [3, 2]
|
||||
assert [c.id for c in spec.sides[0].combatants] == ["goblin", "goblin", "goblin"]
|
||||
assert [c.id for c in spec.sides[1].combatants] == ["orc", "orc"]
|
||||
assert spec.round_cap == 100
|
||||
assert spec.map.deployment == {"players": "A", "monsters": "B"}
|
||||
|
||||
|
||||
def test_build_encounter_hp_fraction() -> None:
|
||||
"""hp_fraction 0.5 halves hp_max (rounded); 1.0 leaves it identical."""
|
||||
half = build_encounter(
|
||||
ScenarioSpec(
|
||||
name="half-hp",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(
|
||||
name="players",
|
||||
zone="A",
|
||||
combatant_files=(GOBLIN,),
|
||||
hp_fraction=0.5,
|
||||
),
|
||||
SideSpec(
|
||||
name="monsters",
|
||||
zone="B",
|
||||
combatant_files=(ORC,),
|
||||
hp_fraction=0.5,
|
||||
),
|
||||
),
|
||||
)
|
||||
)
|
||||
assert half.sides[0].combatants[0].hp_max == 3
|
||||
assert half.sides[1].combatants[0].hp_max == 3
|
||||
|
||||
full = build_encounter(_scenario(goblins=1, orcs=1))
|
||||
assert full.sides[0].combatants[0].hp_max == 6
|
||||
assert full.sides[1].combatants[0].hp_max == 6
|
||||
|
||||
|
||||
def test_scenario_spec_validation() -> None:
|
||||
"""Empty name, empty sides, bad hp_fraction, and round_cap 0 all raise."""
|
||||
with pytest.raises(ValueError, match="name"):
|
||||
ScenarioSpec(name="", map_file=MAP, sides=(_scenario().sides[0],))
|
||||
with pytest.raises(ValueError, match="sides"):
|
||||
ScenarioSpec(name="x", map_file=MAP, sides=())
|
||||
for fraction in (0.0, 1.1, -0.5):
|
||||
with pytest.raises(ValueError, match="hp_fraction"):
|
||||
ScenarioSpec(
|
||||
name="x",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(
|
||||
name="players",
|
||||
zone="A",
|
||||
combatant_files=(GOBLIN,),
|
||||
hp_fraction=fraction,
|
||||
),
|
||||
),
|
||||
)
|
||||
with pytest.raises(ValueError, match="round_cap"):
|
||||
ScenarioSpec(
|
||||
name="x",
|
||||
map_file=MAP,
|
||||
sides=(_scenario().sides[0],),
|
||||
round_cap=0,
|
||||
)
|
||||
|
||||
|
||||
def test_build_encounter_validation() -> None:
|
||||
"""Shared zone letters, undeclared zones, and empty combatant lists raise."""
|
||||
with pytest.raises(ValueError, match="share zone letters"):
|
||||
build_encounter(
|
||||
ScenarioSpec(
|
||||
name="x",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(name="a", zone="A", combatant_files=(GOBLIN,)),
|
||||
SideSpec(name="b", zone="A", combatant_files=(ORC,)),
|
||||
),
|
||||
)
|
||||
)
|
||||
with pytest.raises(ValueError, match="not declared in map deployment"):
|
||||
build_encounter(
|
||||
ScenarioSpec(
|
||||
name="x",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(name="a", zone="A", combatant_files=(GOBLIN,)),
|
||||
SideSpec(name="b", zone="C", combatant_files=(ORC,)),
|
||||
),
|
||||
)
|
||||
)
|
||||
with pytest.raises(ValueError, match="combatant file"):
|
||||
build_encounter(
|
||||
ScenarioSpec(
|
||||
name="x",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(name="a", zone="A", combatant_files=()),
|
||||
SideSpec(name="b", zone="B", combatant_files=(ORC,)),
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_variants_default_returns_base() -> None:
|
||||
"""The default hp_fractions=(1.0,) yields exactly one variant equal to base."""
|
||||
base = _scenario()
|
||||
assert scenario_variants(base) == (base,)
|
||||
|
||||
|
||||
def test_variants_fractions_names_and_scaling() -> None:
|
||||
"""(0.5, 1.0) yields two variants with suffixed names and scaled HP."""
|
||||
base = _scenario()
|
||||
variants = scenario_variants(base, hp_fractions=(0.5, 1.0))
|
||||
|
||||
assert len(variants) == 2
|
||||
assert variants[0].name == "goblins-vs-orcs hp=0.5"
|
||||
assert all(side.hp_fraction == 0.5 for side in variants[0].sides)
|
||||
assert variants[1] == base
|
||||
|
||||
half = build_encounter(variants[0])
|
||||
assert half.sides[0].combatants[0].hp_max == 3
|
||||
assert half.sides[1].combatants[0].hp_max == 3
|
||||
|
||||
|
||||
def test_variants_empty_fractions_raise() -> None:
|
||||
"""An empty hp_fractions tuple is a ValueError."""
|
||||
with pytest.raises(ValueError, match="hp_fractions"):
|
||||
scenario_variants(_scenario(), hp_fractions=())
|
||||
|
||||
|
||||
def test_variants_invalid_fraction_raises() -> None:
|
||||
"""A fraction outside (0, 1] is a ValueError."""
|
||||
with pytest.raises(ValueError, match="hp_fraction"):
|
||||
scenario_variants(_scenario(), hp_fractions=(1.5,))
|
||||
|
||||
|
||||
def test_variants_deterministic() -> None:
|
||||
"""Two calls with the same inputs produce equal variant tuples."""
|
||||
base = _scenario()
|
||||
fractions = (0.25, 0.5, 0.75, 1.0)
|
||||
assert scenario_variants(base, hp_fractions=fractions) == scenario_variants(
|
||||
base, hp_fractions=fractions
|
||||
)
|
||||
|
||||
|
||||
def test_build_encounter_playable() -> None:
|
||||
"""A compiled encounter runs end-to-end (20 seeded battles)."""
|
||||
spec = build_encounter(_scenario(goblins=3, orcs=2))
|
||||
report = run_encounter(spec, runs=20, seed=1)
|
||||
assert report.runs == 20
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Phase-2 strategy search tests (docs/phase-2-llm-architecture.md 10 P3.2/P3.3).
|
||||
|
||||
``evaluate`` measures one strategy on one scenario; ``optimize`` searches the
|
||||
strategy space (coarse grid, retreat refinement, LLM candidates) and returns a
|
||||
ranked playbook; ``strategy_name`` renders deterministic labels; and
|
||||
``format_playbook`` renders the playbook as aligned ASCII columns. These tests
|
||||
pin the contract: benchmark fields, label rendering, rank ordering, candidate
|
||||
deduplication, determinism, the ``max_total_runs`` cost bound, and the text
|
||||
renderer. Small n (runs <= 5) keeps the file fast and deterministic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.scenarios import ScenarioSpec, SideSpec
|
||||
from pf1e_simulator.search import evaluate, format_playbook, optimize, strategy_name
|
||||
from pf1e_simulator.strategy import StrategySpec
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.search import PlaybookEntry
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
GOBLIN = ROOT / "data" / "monsters" / "goblin.json"
|
||||
ORC = ROOT / "data" / "monsters" / "orc.json"
|
||||
|
||||
RUNS = 5
|
||||
SEED = 7
|
||||
ROUND_CAP = 20
|
||||
PARTY = "players"
|
||||
|
||||
|
||||
def _scenario() -> ScenarioSpec:
|
||||
return ScenarioSpec(
|
||||
name="goblins-vs-orcs",
|
||||
map_file=MAP,
|
||||
sides=(
|
||||
SideSpec(
|
||||
name="players",
|
||||
zone="A",
|
||||
combatant_files=(GOBLIN, GOBLIN, GOBLIN),
|
||||
),
|
||||
SideSpec(
|
||||
name="monsters",
|
||||
zone="B",
|
||||
combatant_files=(ORC, ORC),
|
||||
),
|
||||
),
|
||||
round_cap=ROUND_CAP,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def playbook() -> tuple[PlaybookEntry, ...]:
|
||||
return optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
|
||||
|
||||
|
||||
def test_evaluate_default_strategy() -> None:
|
||||
"""evaluate returns a benchmark with sane danger metrics and win band."""
|
||||
benchmark = evaluate(
|
||||
StrategySpec(),
|
||||
_scenario(),
|
||||
party=PARTY,
|
||||
runs=100,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
assert benchmark.scenario_name == "goblins-vs-orcs"
|
||||
assert benchmark.strategy == StrategySpec()
|
||||
assert benchmark.danger.runs == 100
|
||||
assert 0.0 <= benchmark.danger.p_survives <= 1.0
|
||||
assert 0.0 <= benchmark.danger.p_tpk <= 1.0
|
||||
assert benchmark.danger.expected_rounds > 0.0
|
||||
assert set(benchmark.wins) == {"players", "monsters"}
|
||||
assert benchmark.draws >= 0
|
||||
low, high = benchmark.win_rate_band
|
||||
assert 0.0 <= low <= high <= 1.0
|
||||
|
||||
|
||||
def test_strategy_name() -> None:
|
||||
"""Labels render non-default levers in field order; the name field is excluded."""
|
||||
assert strategy_name(StrategySpec()) == "default"
|
||||
assert strategy_name(StrategySpec(target_priority="weakest_ac")) == "weakest_ac"
|
||||
assert (
|
||||
strategy_name(StrategySpec(target_priority="weakest_ac", spell_policy="none"))
|
||||
== "weakest_ac, spell_policy=none"
|
||||
)
|
||||
assert (
|
||||
strategy_name(
|
||||
StrategySpec(target_priority="lowest_hp", retreat_hp_fraction=0.5)
|
||||
)
|
||||
== "lowest_hp, retreat=0.5"
|
||||
)
|
||||
assert (
|
||||
strategy_name(StrategySpec(name="custom", use_charges=False))
|
||||
== "use_charges=false"
|
||||
)
|
||||
|
||||
|
||||
def test_optimize_small_ranks_and_default(playbook: tuple[PlaybookEntry, ...]) -> None:
|
||||
"""A small optimize returns contiguous ranks ordered by the rank key."""
|
||||
assert len(playbook) >= 24
|
||||
assert [entry.rank for entry in playbook] == list(range(1, len(playbook) + 1))
|
||||
assert any(entry.strategy == StrategySpec() for entry in playbook)
|
||||
|
||||
keys = [
|
||||
(
|
||||
-entry.benchmark.danger.p_survives,
|
||||
entry.benchmark.danger.expected_hp_lost,
|
||||
-entry.benchmark.danger.worst_decile_party_hp,
|
||||
entry.benchmark.danger.expected_rounds,
|
||||
strategy_name(entry.strategy),
|
||||
)
|
||||
for entry in playbook
|
||||
]
|
||||
assert keys == sorted(keys)
|
||||
|
||||
|
||||
def test_optimize_deterministic() -> None:
|
||||
"""Two optimize calls with the same inputs return identical playbooks."""
|
||||
first = optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
|
||||
second = optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED)
|
||||
assert first == second
|
||||
|
||||
|
||||
def test_optimize_candidates_deduped() -> None:
|
||||
"""LLM candidates run once each; duplicates and grid duplicates collapse."""
|
||||
in_grid = StrategySpec(target_priority="weakest_ac", spell_policy="none")
|
||||
custom = StrategySpec(
|
||||
target_priority="lowest_hp",
|
||||
spell_policy="none",
|
||||
use_charges=False,
|
||||
use_class_abilities=False,
|
||||
retreat_hp_fraction=0.75,
|
||||
)
|
||||
playbook = optimize(
|
||||
_scenario(),
|
||||
party=PARTY,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
candidates=(custom, custom, in_grid),
|
||||
)
|
||||
|
||||
custom_matches = [entry for entry in playbook if entry.strategy == custom]
|
||||
assert len(custom_matches) == 1
|
||||
in_grid_matches = [entry for entry in playbook if entry.strategy == in_grid]
|
||||
assert len(in_grid_matches) == 1
|
||||
|
||||
direct = evaluate(custom, _scenario(), party=PARTY, runs=RUNS, seed=SEED)
|
||||
assert custom_matches[0].benchmark.danger == direct.danger
|
||||
|
||||
|
||||
def test_budget_raises_below_required_total() -> None:
|
||||
"""A max_total_runs below the required total raises before running."""
|
||||
with pytest.raises(ValueError, match="budget") as excinfo:
|
||||
optimize(_scenario(), party=PARTY, runs=RUNS, seed=SEED, max_total_runs=1)
|
||||
assert "max_total_runs" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_budget_generous_cap_raises_nothing() -> None:
|
||||
"""A generous max_total_runs cap does not raise."""
|
||||
playbook = optimize(
|
||||
_scenario(),
|
||||
party=PARTY,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
max_total_runs=10_000,
|
||||
)
|
||||
assert len(playbook) >= 24
|
||||
|
||||
|
||||
def test_format_playbook_columns_and_ascii(
|
||||
playbook: tuple[PlaybookEntry, ...],
|
||||
) -> None:
|
||||
"""The renderer shows header tokens, a dash separator, aligned rows, ASCII."""
|
||||
text = format_playbook(playbook, PARTY)
|
||||
|
||||
for token in (
|
||||
"Rank",
|
||||
"Strategy",
|
||||
"Runs",
|
||||
"P(survive)",
|
||||
"P(TPK)",
|
||||
"P(>=1 death)",
|
||||
"HP lost",
|
||||
"Worst-decile HP",
|
||||
"Rounds",
|
||||
"Win% (3σ)",
|
||||
):
|
||||
assert token in text
|
||||
|
||||
lines = text.splitlines()
|
||||
assert len(lines) == 2 + len(playbook)
|
||||
separator = lines[1]
|
||||
assert separator
|
||||
assert set(separator) <= {"-", " "}
|
||||
assert len({len(line) for line in lines}) == 1
|
||||
|
||||
non_ascii = {char for char in text if ord(char) > 127}
|
||||
assert non_ascii <= {"σ"}
|
||||
|
||||
top = playbook[0]
|
||||
rate = top.benchmark.wins.get(PARTY, 0) / top.benchmark.danger.runs
|
||||
low, high = top.benchmark.win_rate_band
|
||||
expected = f"{rate * 100:.1f} [{low * 100:.1f}, {high * 100:.1f}]"
|
||||
assert expected in text
|
||||
@@ -0,0 +1,231 @@
|
||||
"""Phase-2 Etage A strategy compiler tests (docs/phase-2-llm-architecture.md 10 P2.1).
|
||||
|
||||
Deterministic, offline: no network, no LLM. Pins the ``StrategySpec`` /
|
||||
``compile_strategy`` contract: the all-default spec compiles to
|
||||
``default_policy`` by identity, ``select_target`` honours each priority with
|
||||
deterministic tie-breaks, non-default specs drive real battles through
|
||||
``run_encounter`` bit-identically under a fixed seed, and invalid specs are
|
||||
rejected at construction.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.combat import (
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
default_policy,
|
||||
)
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
from pf1e_simulator.strategy import StrategySpec, compile_strategy, select_target
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
_MAP_SPEC = load_map(MAP)
|
||||
_SPELLS = load_spell_registry(SPELLS_DIR)
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _spec() -> EncounterSpec:
|
||||
"""A fast 2v2 (goblin archer + orc vs orc + goblin) on the sample arena."""
|
||||
return EncounterSpec(
|
||||
map=_MAP_SPEC,
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(
|
||||
name="players",
|
||||
combatants=(_c("monsters/goblin.json"), _c("monsters/orc.json")),
|
||||
zone="A",
|
||||
),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=(_c("monsters/orc.json"), _c("monsters/goblin.json")),
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _state(
|
||||
cid: str, side: str, pos: tuple[int, int], hp: int = 10, path: str = "monsters/orc.json"
|
||||
) -> CombatantState:
|
||||
"""A bare runtime state over a loaded combatant (no battle run)."""
|
||||
combatant = _c(path).model_copy(update={"id": cid})
|
||||
return CombatantState(combatant=combatant, side=side, pos=pos, hp=hp, resources_left={})
|
||||
|
||||
|
||||
def _mini_engine(states: list[CombatantState]) -> CombatEngine:
|
||||
"""A live engine over the 2v2 map with the given states (no battle run)."""
|
||||
return CombatEngine(SeededRng(1), Grid.from_spec(_MAP_SPEC), states, spell_registry=_SPELLS)
|
||||
|
||||
|
||||
def test_compile_strategy_default_is_default_policy() -> None:
|
||||
"""The all-default spec compiles to ``default_policy`` by identity."""
|
||||
assert compile_strategy(StrategySpec()) is default_policy
|
||||
|
||||
|
||||
def test_select_target_nearest() -> None:
|
||||
"""nearest picks the closest active enemy (ties keep list order)."""
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
far_orc = _state("orc2", "players", (1, 16), hp=6)
|
||||
engine = _mini_engine([orc, goblin, far_orc])
|
||||
assert select_target(engine, orc, "nearest") is goblin
|
||||
|
||||
|
||||
def test_select_target_weakest_ac() -> None:
|
||||
"""weakest_ac picks the lowest total AC (goblin 16 > orc 13)."""
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
far_orc = _state("orc2", "players", (1, 16), hp=6)
|
||||
engine = _mini_engine([orc, goblin, far_orc])
|
||||
assert select_target(engine, orc, "weakest_ac") is far_orc
|
||||
|
||||
|
||||
def test_select_target_lowest_hp() -> None:
|
||||
"""lowest_hp picks the most wounded active enemy."""
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
healthy = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
wounded = _state("goblin2", "players", (1, 16), hp=1, path="monsters/goblin.json")
|
||||
engine = _mini_engine([orc, healthy, wounded])
|
||||
assert select_target(engine, orc, "lowest_hp") is wounded
|
||||
|
||||
|
||||
def test_select_target_no_active_enemy() -> None:
|
||||
"""No active enemy -> None for every priority."""
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
dying = _state("goblin1", "players", (1, 2), hp=-5, path="monsters/goblin.json")
|
||||
engine = _mini_engine([orc, dying])
|
||||
assert select_target(engine, orc, "nearest") is None
|
||||
assert select_target(engine, orc, "weakest_ac") is None
|
||||
assert select_target(engine, orc, "lowest_hp") is None
|
||||
|
||||
|
||||
def test_compiled_policy_melee_ladder() -> None:
|
||||
"""A non-default spec still resolves the melee ladder on a live engine."""
|
||||
spec = StrategySpec(name="no-charge", use_charges=False)
|
||||
policy = compile_strategy(spec)
|
||||
assert policy is not default_policy
|
||||
orc = _state("orc1", "monsters", (1, 1))
|
||||
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
engine = _mini_engine([orc, goblin])
|
||||
assert policy(engine, orc) == (Action(kind="full_attack", target_id="goblin1"),)
|
||||
|
||||
|
||||
def test_compiled_policy_charge_disabled() -> None:
|
||||
"""use_charges=False skips the charge and falls to step + full-attack."""
|
||||
spec = StrategySpec(name="no-charge", use_charges=False)
|
||||
policy = compile_strategy(spec)
|
||||
# The celestial wolf is melee-only (Bite, reach 5 ft): at 2 cells no
|
||||
# weapon_for short-circuit, so the ladder reaches the charge branch. The
|
||||
# orc sits DIAGONALLY at (2, 3): straight ahead at (1, 3) the only
|
||||
# attackable charge end is 1 cell away (below the 2-cell minimum charge
|
||||
# move), while from (1, 1) the diagonal end (1, 3) is a legal 2-cell
|
||||
# straight line AND a single 5-foot step to (1, 2) still enters reach.
|
||||
wolf = _state("wolf1", "players", (1, 1), hp=13, path="monsters/celestial_wolf.json")
|
||||
orc = _state("orc1", "monsters", (2, 3))
|
||||
engine = _mini_engine([wolf, orc])
|
||||
assert default_policy(engine, wolf) == (Action(kind="charge", target_id="orc1"),)
|
||||
assert policy(engine, wolf) == (
|
||||
Action(kind="5foot_step", target_id="orc1"),
|
||||
Action(kind="full_attack", target_id="orc1"),
|
||||
)
|
||||
|
||||
|
||||
def test_compiled_policy_retreats_below_threshold() -> None:
|
||||
"""retreat_hp_fraction withdraws below the threshold, not at/above it."""
|
||||
spec = StrategySpec(name="coward", retreat_hp_fraction=0.5)
|
||||
policy = compile_strategy(spec)
|
||||
orc = _state("orc1", "monsters", (1, 1), hp=2) # 2 < 0.5 * 6 = 3
|
||||
goblin = _state("goblin1", "players", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
engine = _mini_engine([orc, goblin])
|
||||
assert policy(engine, orc) == (Action(kind="withdraw", target_id="goblin1"),)
|
||||
orc_ok = _state("orc2", "monsters", (1, 1), hp=3) # 3 is not below 3
|
||||
engine_ok = _mini_engine([orc_ok, goblin])
|
||||
assert all(a.kind != "withdraw" for a in policy(engine_ok, orc_ok))
|
||||
|
||||
|
||||
def test_compiled_policy_spell_policy_none_skips_casting() -> None:
|
||||
"""spell_policy="none" never proposes a cast even for a caster with slots."""
|
||||
spec = StrategySpec(name="no-spells", spell_policy="none")
|
||||
policy = compile_strategy(spec)
|
||||
oni = _c("pcs/oni.json")
|
||||
oni_state = CombatantState(
|
||||
combatant=oni,
|
||||
side="players",
|
||||
pos=(1, 1),
|
||||
hp=oni.hp_max,
|
||||
resources_left={},
|
||||
spell_slots_left={1: 4, 2: 3},
|
||||
)
|
||||
goblin = _state("goblin1", "monsters", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
engine = _mini_engine([oni_state, goblin])
|
||||
# With the default spell policy the same setup casts (Shield of Faith,
|
||||
# slot 1): no ally is below 50% HP and Sound Burst is an area spell, so
|
||||
# _find_offensive_spell (top-level DamageEffect only) skips it.
|
||||
assert default_policy(engine, oni_state) == (
|
||||
Action(kind="cast_spell", target_id="oni", spell_name="Shield of Faith"),
|
||||
)
|
||||
# With spell_policy="none" the melee ladder runs instead.
|
||||
actions = policy(engine, oni_state)
|
||||
assert all(a.kind != "cast_spell" for a in actions)
|
||||
|
||||
|
||||
def test_compiled_policy_class_abilities_disabled() -> None:
|
||||
"""use_class_abilities=False skips channel/hex even when they would fire."""
|
||||
spec = StrategySpec(name="no-class", use_class_abilities=False)
|
||||
policy = compile_strategy(spec)
|
||||
oni = _c("pcs/oni.json")
|
||||
oni_state = CombatantState(
|
||||
combatant=oni,
|
||||
side="players",
|
||||
pos=(1, 1),
|
||||
hp=oni.hp_max,
|
||||
resources_left={"channel_energy": 6},
|
||||
spell_slots_left={1: 4, 2: 3},
|
||||
)
|
||||
hurt = _state("hurt", "players", (1, 3), hp=1, path="monsters/goblin.json")
|
||||
goblin = _state("goblin1", "monsters", (1, 2), hp=6, path="monsters/goblin.json")
|
||||
engine = _mini_engine([oni_state, hurt, goblin])
|
||||
# With class abilities enabled the channel fires (ally below 50% hp).
|
||||
assert default_policy(engine, oni_state) == (
|
||||
Action(kind="channel_energy", target_id="goblin1", channel_mode="heal"),
|
||||
)
|
||||
# Disabled: the channel is skipped entirely.
|
||||
actions = policy(engine, oni_state)
|
||||
assert all(a.kind != "channel_energy" for a in actions)
|
||||
|
||||
|
||||
def test_compiled_policy_deterministic_runs() -> None:
|
||||
"""A compiled policy drives bit-identical battles under a fixed seed."""
|
||||
spec = StrategySpec(name="focus", target_priority="weakest_ac")
|
||||
policy = compile_strategy(spec)
|
||||
_, runs_a = run_encounter(_spec(), runs=3, seed=7, policy=policy, collect_runs=True)
|
||||
_, runs_b = run_encounter(_spec(), runs=3, seed=7, policy=policy, collect_runs=True)
|
||||
assert runs_a == runs_b
|
||||
|
||||
|
||||
def test_strategy_spec_validation() -> None:
|
||||
"""Invalid specs are rejected at construction."""
|
||||
with pytest.raises(ValueError, match="non-empty"):
|
||||
StrategySpec(name="")
|
||||
with pytest.raises(ValueError, match="within"):
|
||||
StrategySpec(retreat_hp_fraction=1.5)
|
||||
with pytest.raises(ValueError, match="within"):
|
||||
StrategySpec(retreat_hp_fraction=-0.1)
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Phase-2 sweep runner tests (docs/phase-2-llm-architecture.md 10 P2.3, D9/D10).
|
||||
|
||||
The ``SweepRunner`` executes a grid of (scenario x strategy) cells in
|
||||
parallel, each cell running ``run_encounter(..., collect_runs=True)`` under
|
||||
one policy and reporting the aggregate report plus the party's danger
|
||||
metrics. These tests pin the contract: scenario-major grid order, bit-for-bit
|
||||
determinism, thread-pool equivalence (workers=1 vs workers=4), the
|
||||
``max_total_runs`` cost bound, constructor/cell validation, the equivalence
|
||||
of a ``None`` policy cell with a direct ``run_encounter`` call, and the
|
||||
aligned text renderer. All cells use small n (runs <= 5) so the file stays
|
||||
fast and deterministic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.objectives import danger_metrics
|
||||
from pf1e_simulator.phase2_spike import focus_fire_policy
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.sweep import SweepCell, SweepRunner, format_sweep
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
RUNS = 5
|
||||
SEED = 7
|
||||
ROUND_CAP = 20
|
||||
PARTY = "players"
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _spec(goblins: int, orcs: int) -> EncounterSpec:
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=ROUND_CAP,
|
||||
sides=(
|
||||
Side(
|
||||
name="players",
|
||||
combatants=tuple(_c("monsters/goblin.json") for _ in range(goblins)),
|
||||
zone="A",
|
||||
),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=tuple(_c("monsters/orc.json") for _ in range(orcs)),
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
VARIANT_A = _spec(goblins=3, orcs=2)
|
||||
VARIANT_B = _spec(goblins=2, orcs=3)
|
||||
|
||||
SCENARIOS = (("variant_a", VARIANT_A), ("variant_b", VARIANT_B))
|
||||
STRATEGIES = (("default", None), ("focus_fire_monsters", focus_fire_policy("monsters")))
|
||||
|
||||
|
||||
def _cells() -> tuple[SweepCell, ...]:
|
||||
return (
|
||||
SweepCell(
|
||||
scenario="variant_a",
|
||||
strategy="default",
|
||||
spec=VARIANT_A,
|
||||
policy=None,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
),
|
||||
SweepCell(
|
||||
scenario="variant_b",
|
||||
strategy="focus_fire_monsters",
|
||||
spec=VARIANT_B,
|
||||
policy=focus_fire_policy("monsters"),
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_run_grid_2x2_scenario_major() -> None:
|
||||
"""A 2x2 grid returns 4 results in scenario-major order with sane metrics."""
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
results = runner.run_grid(SCENARIOS, STRATEGIES, runs=RUNS, seed=SEED)
|
||||
|
||||
assert len(results) == 4
|
||||
assert [r.cell.scenario for r in results] == [
|
||||
"variant_a",
|
||||
"variant_a",
|
||||
"variant_b",
|
||||
"variant_b",
|
||||
]
|
||||
assert [r.cell.strategy for r in results] == [
|
||||
"default",
|
||||
"focus_fire_monsters",
|
||||
"default",
|
||||
"focus_fire_monsters",
|
||||
]
|
||||
for result in results:
|
||||
assert result.report.runs == RUNS
|
||||
assert result.danger.runs == RUNS
|
||||
assert 0.0 <= result.danger.p_survives <= 1.0
|
||||
assert 0.0 <= result.danger.p_tpk <= 1.0
|
||||
|
||||
|
||||
def test_run_is_deterministic() -> None:
|
||||
"""Two runs over identical cells return identical frozen results."""
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
cells = _cells()
|
||||
|
||||
first = runner.run(cells)
|
||||
second = runner.run(cells)
|
||||
|
||||
assert first == second
|
||||
|
||||
|
||||
def test_parallel_equivalence_workers_1_vs_4() -> None:
|
||||
"""workers=1 and workers=4 produce identical results (thread-safe registry)."""
|
||||
cells = _cells()
|
||||
|
||||
serial = SweepRunner(party=PARTY, workers=1).run(cells)
|
||||
parallel = SweepRunner(party=PARTY, workers=4).run(cells)
|
||||
|
||||
assert serial == parallel
|
||||
|
||||
|
||||
def test_cost_bound_raises_on_budget_exceeded() -> None:
|
||||
"""Cells summing past max_total_runs raise a ValueError naming runs/budget."""
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
cells = (
|
||||
SweepCell(
|
||||
scenario="a", strategy="s", spec=VARIANT_A, policy=None, runs=3, seed=1
|
||||
),
|
||||
SweepCell(
|
||||
scenario="b", strategy="s", spec=VARIANT_B, policy=None, runs=3, seed=1
|
||||
),
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="runs") as excinfo:
|
||||
runner.run(cells, max_total_runs=5)
|
||||
|
||||
assert "budget" in str(excinfo.value)
|
||||
|
||||
|
||||
def test_validation_workers_party_runs() -> None:
|
||||
"""workers=0, an empty party, and a runs=0 cell all raise ValueError."""
|
||||
with pytest.raises(ValueError, match="workers"):
|
||||
SweepRunner(party=PARTY, workers=0)
|
||||
with pytest.raises(ValueError, match="party"):
|
||||
SweepRunner(party="", workers=2)
|
||||
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
bad = SweepCell(
|
||||
scenario="a", strategy="s", spec=VARIANT_A, policy=None, runs=0, seed=1
|
||||
)
|
||||
with pytest.raises(ValueError, match="runs"):
|
||||
runner.run((bad,))
|
||||
|
||||
|
||||
def test_policy_none_matches_direct_run_encounter() -> None:
|
||||
"""A None-policy cell equals a direct run_encounter on the same scenario."""
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
cell = SweepCell(
|
||||
scenario="variant_a",
|
||||
strategy="default",
|
||||
spec=VARIANT_A,
|
||||
policy=None,
|
||||
runs=RUNS,
|
||||
seed=SEED,
|
||||
)
|
||||
|
||||
(result,) = runner.run((cell,))
|
||||
report, outcomes = run_encounter(VARIANT_A, RUNS, SEED, collect_runs=True)
|
||||
|
||||
assert result.report == report
|
||||
assert result.danger == danger_metrics(outcomes, PARTY)
|
||||
|
||||
|
||||
def test_format_sweep_columns_and_win_pct() -> None:
|
||||
"""The renderer shows header tokens, one row per result, and the party win%."""
|
||||
runner = SweepRunner(party=PARTY, workers=2)
|
||||
results = runner.run_grid(SCENARIOS, STRATEGIES, runs=RUNS, seed=SEED)
|
||||
|
||||
text = format_sweep(results, PARTY)
|
||||
for token in ("Scenario", "Win%", "P(TPK)"):
|
||||
assert token in text
|
||||
|
||||
lines = text.splitlines()
|
||||
assert len(lines) == 2 + len(results)
|
||||
|
||||
first = results[0]
|
||||
expected_win = f"{first.report.wins.get(PARTY, 0) / first.report.runs * 100:.1f}"
|
||||
assert lines[2].split()[3] == expected_win
|
||||
@@ -0,0 +1,279 @@
|
||||
"""Phase-2 compact observation view (docs/phase-2-llm-architecture.md P0.2).
|
||||
|
||||
``llm_view.build_view`` + ``render`` turn the raw ``Observation`` snapshot
|
||||
into a compact, deterministic text view for the LLM prompt. This module
|
||||
measures the compaction on a representative caster-heavy encounter (same
|
||||
shape as the measurement spike, but with its OWN spec builder - the spike
|
||||
file is out of scope), prints a summary block for the orchestrator, and
|
||||
pins LOOSE upper bounds so a regression (a dropped key re-added, a sparse
|
||||
grid turned dense, ...) fails loudly. It also pins the structural contract:
|
||||
sparse grids only carry non-default cells, dropped keys stay dropped, the
|
||||
nested attack/spell expansions stay compact, and the render is
|
||||
deterministic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pf1e_simulator.combat import (
|
||||
OBSERVATION_SCHEMA_VERSION,
|
||||
Action,
|
||||
CombatantState,
|
||||
CombatEngine,
|
||||
Observation,
|
||||
default_policy,
|
||||
)
|
||||
from pf1e_simulator.grid import Grid
|
||||
from pf1e_simulator.llm_view import build_view, render
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.rng import SeededRng
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, build_states
|
||||
from pf1e_simulator.spells import SPELLS_DIR, SpellSpec, load_spell_registry
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.models import Combatant
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
# Loose ceilings (regression guards, not exact pins): the docs target a
|
||||
# compact view of ~2-4k tokens against a raw snapshot of ~16k tokens, i.e.
|
||||
# 12.5-25% of the raw size. 25% of raw bytes and a 4k-token byte ceiling
|
||||
# (4 bytes/token) are the outer bounds.
|
||||
MAX_COMPACT_RATIO = 0.25
|
||||
MAX_COMPACT_BYTES = 16_000
|
||||
|
||||
# Keys the compact view deliberately drops from each combatant snapshot.
|
||||
_DROPPED_COMBATANT_KEYS = frozenset(
|
||||
{
|
||||
"occupied_cells",
|
||||
"abilities",
|
||||
"skills",
|
||||
"initiative_mod",
|
||||
"initiative_total",
|
||||
"temp_hp_rounds_left",
|
||||
"features",
|
||||
"feature_details",
|
||||
"carried_items",
|
||||
"weapon_damage",
|
||||
"aware",
|
||||
"alignment",
|
||||
"armor_spell_failure",
|
||||
"arcane_caster",
|
||||
"run_rounds",
|
||||
}
|
||||
)
|
||||
|
||||
# Every pair key must survive (the pairs are the tactical core); the raw
|
||||
# ``target_index`` is replaced by the stable combatant ``target`` id.
|
||||
_PAIR_KEYS = frozenset(
|
||||
{
|
||||
"target",
|
||||
"distance_ft",
|
||||
"weapon_for",
|
||||
"weapon_reason",
|
||||
"range_penalty",
|
||||
"can_charge",
|
||||
"charge_reason",
|
||||
"can_step5_to_attack",
|
||||
"has_line_of_effect",
|
||||
"flanking",
|
||||
"movement_cost_ft",
|
||||
"target_threatens_me",
|
||||
"enemies_threatening_me",
|
||||
"cover_bonus",
|
||||
}
|
||||
)
|
||||
|
||||
# Keys dropped from the nested attack/spell expansions.
|
||||
_ATTACK_DROPPED_KEYS = frozenset({"splash", "hardness", "hit_points"})
|
||||
_SPELL_DROPPED_KEYS = frozenset(
|
||||
{"summary", "school", "half_on_save", "save_negates", "harmless", "components"}
|
||||
)
|
||||
|
||||
|
||||
class _Capture:
|
||||
"""Policy wrapper that snapshots the first observation, then delegates."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.observation: Observation | None = None
|
||||
|
||||
def policy(self, engine: CombatEngine, state: CombatantState) -> tuple[Action, ...]:
|
||||
if self.observation is None:
|
||||
self.observation = engine._build_observation(state)
|
||||
return default_policy(engine, state)
|
||||
|
||||
|
||||
def _c(path: str) -> Combatant:
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _representative_spec() -> EncounterSpec:
|
||||
"""Caster-heavy 3v4 on the sample arena (walls, rubble, water, pillars)."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(
|
||||
name="players",
|
||||
combatants=(_c("pcs/oni.json"), _c("pcs/misty.json"), _c("pcs/nairda.json")),
|
||||
zone="A",
|
||||
),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=(
|
||||
_c("monsters/lich.json"),
|
||||
_c("monsters/thrune_agent.json"),
|
||||
_c("monsters/orc.json"),
|
||||
_c("monsters/goblin.json"),
|
||||
),
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _first_observation(
|
||||
spec: EncounterSpec, seed: int, registry: dict[str, SpellSpec]
|
||||
) -> Observation:
|
||||
grid = Grid.from_spec(spec.map)
|
||||
states = build_states(spec)
|
||||
cap = _Capture()
|
||||
CombatEngine(
|
||||
SeededRng(seed),
|
||||
grid,
|
||||
states,
|
||||
round_cap=spec.round_cap,
|
||||
policy=cap.policy,
|
||||
spell_registry=registry,
|
||||
).run()
|
||||
assert cap.observation is not None
|
||||
return cap.observation
|
||||
|
||||
|
||||
def _emit(line: str) -> None:
|
||||
sys.stdout.write(line + "\n")
|
||||
|
||||
|
||||
def _is_default_terrain(semantics: dict[str, object]) -> bool:
|
||||
"""Mirror of ``llm_view._is_default_terrain`` for the structural test."""
|
||||
return (
|
||||
semantics.get("move_cost") == 1
|
||||
and not semantics.get("blocks_los")
|
||||
and not semantics.get("cover")
|
||||
and semantics.get("cover_bonus") == 0
|
||||
)
|
||||
|
||||
|
||||
def test_compact_view_size(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
"""Measure raw-vs-compact on the representative battle; pin loose bounds."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
|
||||
raw = json.dumps(obs.to_dict())
|
||||
view = build_view(obs)
|
||||
compact = render(view)
|
||||
|
||||
raw_bytes = len(raw)
|
||||
compact_bytes = len(compact)
|
||||
|
||||
# Determinism: rebuilding the view from the same observation renders
|
||||
# byte-identically.
|
||||
assert render(build_view(obs)) == compact
|
||||
|
||||
# Schema version is pinned to the engine's contract.
|
||||
assert view.schema_version == OBSERVATION_SCHEMA_VERSION
|
||||
|
||||
# Loose bounds (regression guards, not exact pins).
|
||||
assert compact_bytes < raw_bytes * MAX_COMPACT_RATIO
|
||||
assert compact_bytes < MAX_COMPACT_BYTES
|
||||
|
||||
_emit("=== PHASE2 SUMMARY (compact observation view) ===")
|
||||
_emit("encounter: 7 combatants, round_cap=20, seed=1")
|
||||
_emit(f"raw observation bytes: {raw_bytes} (~{raw_bytes // 4} tokens)")
|
||||
_emit(f"compact view bytes: {compact_bytes} (~{compact_bytes // 4} tokens)")
|
||||
_emit(f"compaction ratio: {compact_bytes / raw_bytes:.1%} of raw")
|
||||
_emit(f"combatants in view: {len(view.combatants)}")
|
||||
_emit(f"pairs in view: {len(view.pairs)}")
|
||||
_emit(f"sparse terrain cells: {len(view.terrain)}")
|
||||
_emit(f"sparse elevation cells: {len(view.elevation)}")
|
||||
_emit(f"threatened cells: {len(view.threatened)}")
|
||||
_emit(f"ranged_threatened cells: {len(view.ranged_threatened)}")
|
||||
out = capsys.readouterr().out
|
||||
assert "PHASE2 SUMMARY" in out
|
||||
|
||||
|
||||
def test_view_sparse_grids() -> None:
|
||||
"""Pin the sparse-grid contract: only non-default/non-zero/True cells."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
|
||||
raw = obs.to_dict()
|
||||
view = build_view(obs)
|
||||
semantics_by_type = {str(sem["type"]): sem for sem in raw["terrain_semantics"]}
|
||||
|
||||
# Dimensions match the raw terrain grid.
|
||||
assert view.dimensions == (len(raw["terrain"]), len(raw["terrain"][0]))
|
||||
|
||||
# Sparse terrain only carries non-default cells, with the raw char.
|
||||
for r, c, terrain_char in view.terrain:
|
||||
assert not _is_default_terrain(semantics_by_type[terrain_char])
|
||||
assert raw["terrain"][r][c] == terrain_char
|
||||
|
||||
# Sparse elevation only carries non-zero cells.
|
||||
for r, c, height in view.elevation:
|
||||
assert height != 0
|
||||
assert raw["elevation"][r][c] == height
|
||||
|
||||
# Sparse threatened grids only carry True cells.
|
||||
for r, c in view.threatened:
|
||||
assert raw["threatened"][r][c] is True
|
||||
for r, c in view.ranged_threatened:
|
||||
assert raw["ranged_threatened"][r][c] is True
|
||||
|
||||
|
||||
def test_view_dropped_keys() -> None:
|
||||
"""Pin the key contract: dropped keys stay dropped, pair keys survive."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
|
||||
view = build_view(obs)
|
||||
|
||||
for combatant in view.combatants:
|
||||
assert _DROPPED_COMBATANT_KEYS.isdisjoint(combatant.keys())
|
||||
for pair in view.pairs:
|
||||
assert _PAIR_KEYS.issubset(pair.keys())
|
||||
|
||||
# Nested expansions stay compact: no dropped attack/spell keys, and the
|
||||
# attack damage is rendered as formula+type strings.
|
||||
for combatant in view.combatants:
|
||||
for attack in combatant["attacks"]:
|
||||
assert _ATTACK_DROPPED_KEYS.isdisjoint(attack.keys())
|
||||
assert all(isinstance(d, str) for d in attack["damage"])
|
||||
for spell in combatant["spells"]:
|
||||
assert _SPELL_DROPPED_KEYS.isdisjoint(spell.keys())
|
||||
|
||||
|
||||
def test_view_ids_and_render() -> None:
|
||||
"""Pin the id contract and the deterministic line-based render."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
obs = _first_observation(_representative_spec(), seed=1, registry=registry)
|
||||
raw = obs.to_dict()
|
||||
view = build_view(obs)
|
||||
|
||||
# The acting combatant and pair targets are rendered as ids.
|
||||
assert view.acting_combatant in raw["initiative_order"]
|
||||
ids = {combatant["id"] for combatant in view.combatants}
|
||||
for pair in view.pairs:
|
||||
assert pair["target"] in ids
|
||||
|
||||
# The render is line-based and self-describing.
|
||||
text = render(view)
|
||||
assert text.startswith("schema_version=1\n")
|
||||
assert "combatants:" in text
|
||||
assert "pairs:" in text
|
||||
assert "condition_definitions" not in text
|
||||
@@ -0,0 +1,162 @@
|
||||
"""Tests for ReportBudget and budget exceeded handling (P5.3 T6)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from pf1e_simulator.client import CostBudget, FakeLLMClient
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.prompt_builder import PromptBuilder
|
||||
from pf1e_simulator.report import (
|
||||
LLMReportPolicy,
|
||||
ReportBudget,
|
||||
build_report_context,
|
||||
)
|
||||
from pf1e_simulator.runner import EncounterReport
|
||||
|
||||
FIXTURE_DIR = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
|
||||
|
||||
def _make_policy(
|
||||
script: list[str],
|
||||
*,
|
||||
max_retries: int = 1,
|
||||
budget: ReportBudget | None = None,
|
||||
) -> LLMReportPolicy:
|
||||
pack = PromptPack.load(FIXTURE_DIR)
|
||||
builder = PromptBuilder(pack)
|
||||
client = FakeLLMClient(script=script, prompt_version=pack.prompt_version("report")) # type: ignore[arg-type]
|
||||
return LLMReportPolicy(
|
||||
client=client,
|
||||
builder=builder,
|
||||
prompt_name="report",
|
||||
max_retries=max_retries,
|
||||
budget=budget or ReportBudget(),
|
||||
)
|
||||
|
||||
|
||||
def test_report_budget_default_allows_one_call() -> None:
|
||||
"""Default budget allows exactly 1 successful call."""
|
||||
policy = _make_policy(script=["Rapport de test."])
|
||||
ctx = build_report_context(
|
||||
encounter=EncounterReport(
|
||||
runs=10, seed=1, wins={"players": 5, "monsters": 5}, draws=0, avg_rounds=5.0,
|
||||
attrition={}
|
||||
),
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == "Rapport de test."
|
||||
assert policy.stats.calls == 1
|
||||
|
||||
|
||||
def test_report_budget_exceeded_on_second_call() -> None:
|
||||
"""Budget exceeded when attempting a second call."""
|
||||
policy = _make_policy(
|
||||
script=["First call.", "Second call (should not happen)."],
|
||||
max_retries=0, # No retries
|
||||
)
|
||||
ctx = build_report_context(
|
||||
encounter=EncounterReport(
|
||||
runs=10, seed=1, wins={"players": 5, "monsters": 5}, draws=0, avg_rounds=5.0,
|
||||
attrition={}
|
||||
),
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
# First call succeeds
|
||||
result1 = policy(ctx)
|
||||
assert result1 == "First call."
|
||||
assert policy.stats.calls == 1
|
||||
|
||||
# Second call should exceed budget and return empty string (fallback)
|
||||
result2 = policy(ctx)
|
||||
assert result2 == ""
|
||||
assert policy.stats.calls == 2
|
||||
assert policy.stats.fallbacks == 1
|
||||
|
||||
|
||||
def test_report_budget_for_retries_allows_retry() -> None:
|
||||
"""Budget adjusted for retries allows the retry to succeed."""
|
||||
policy = _make_policy(
|
||||
script=[Exception("boom"), "Rapport après retry."], # type: ignore[list-item]
|
||||
max_retries=1,
|
||||
)
|
||||
ctx = build_report_context(
|
||||
encounter=EncounterReport(
|
||||
runs=10, seed=1, wins={"players": 5, "monsters": 5}, draws=0, avg_rounds=5.0,
|
||||
attrition={}
|
||||
),
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == "Rapport après retry."
|
||||
assert policy.stats.calls == 2
|
||||
assert policy.stats.retries == 1
|
||||
assert policy.stats.fallbacks == 0
|
||||
|
||||
|
||||
def test_report_budget_for_retries_exceeded() -> None:
|
||||
"""Budget still exceeded after max retries exhausted."""
|
||||
policy = _make_policy(
|
||||
script=[Exception("boom1"), Exception("boom2"), "Should not reach here."], # type: ignore[list-item]
|
||||
max_retries=1,
|
||||
)
|
||||
ctx = build_report_context(
|
||||
encounter=EncounterReport(
|
||||
runs=10, seed=1, wins={"players": 5, "monsters": 5}, draws=0, avg_rounds=5.0,
|
||||
attrition={}
|
||||
),
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
result = policy(ctx)
|
||||
|
||||
assert result == ""
|
||||
assert policy.stats.calls == 2 # initial + 1 retry (budget allows max_retries + 1 = 2)
|
||||
assert policy.stats.retries == 1
|
||||
assert policy.stats.fallbacks == 1
|
||||
|
||||
|
||||
def test_report_budget_to_cost_budget() -> None:
|
||||
"""ReportBudget converts to CostBudget correctly."""
|
||||
budget = ReportBudget(max_calls=2, max_total_tokens=1000, max_cost_usd=0.01)
|
||||
cost_budget = budget.to_cost_budget()
|
||||
|
||||
assert isinstance(cost_budget, CostBudget)
|
||||
assert cost_budget.max_calls == 2
|
||||
assert cost_budget.max_total_tokens == 1000
|
||||
assert cost_budget.max_cost_usd == 0.01
|
||||
|
||||
|
||||
def test_report_budget_for_retries_adjusts_max_calls() -> None:
|
||||
"""for_retries adjusts max_calls to max_retries + 1."""
|
||||
budget = ReportBudget(max_calls=1)
|
||||
adjusted = budget.for_retries(2)
|
||||
|
||||
assert adjusted.max_calls == 3 # 2 retries + 1 initial
|
||||
assert adjusted.max_total_tokens is None
|
||||
assert adjusted.max_cost_usd is None
|
||||
|
||||
|
||||
def test_report_budget_for_retries_preserves_other_limits() -> None:
|
||||
"""for_retries preserves token and cost limits."""
|
||||
budget = ReportBudget(max_calls=1, max_total_tokens=500, max_cost_usd=0.005)
|
||||
adjusted = budget.for_retries(3)
|
||||
|
||||
assert adjusted.max_calls == 4
|
||||
assert adjusted.max_total_tokens == 500
|
||||
assert adjusted.max_cost_usd == 0.005
|
||||
|
||||
|
||||
def test_report_budget_for_retries_preserves_non_default_max_calls() -> None:
|
||||
"""for_retries preserves non-default max_calls."""
|
||||
budget = ReportBudget(max_calls=5)
|
||||
adjusted = budget.for_retries(2)
|
||||
|
||||
assert adjusted.max_calls == 5 # unchanged because it wasn't the default 1
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Tests for ReportContext and build_report_context (P5.3 T1)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import FrozenInstanceError
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from pf1e_simulator.client import ClientStats
|
||||
from pf1e_simulator.loaders import load_combatant
|
||||
from pf1e_simulator.map import load_map
|
||||
from pf1e_simulator.nl_policy import NLPolicyStats
|
||||
from pf1e_simulator.objectives import DangerMetrics
|
||||
from pf1e_simulator.report import ReportContext, build_report_context
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, run_encounter
|
||||
from pf1e_simulator.spells import SPELLS_DIR, load_spell_registry
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MAP = ROOT / "data" / "maps" / "sample_arena.yaml"
|
||||
|
||||
|
||||
def _c(path: str):
|
||||
return load_combatant(ROOT / "data" / path)
|
||||
|
||||
|
||||
def _spec() -> EncounterSpec:
|
||||
"""Fast 1v1 (goblin vs orc) on the sample arena."""
|
||||
return EncounterSpec(
|
||||
map=load_map(MAP),
|
||||
round_cap=20,
|
||||
sides=(
|
||||
Side(name="players", combatants=(_c("monsters/goblin.json"),), zone="A"),
|
||||
Side(name="monsters", combatants=(_c("monsters/orc.json"),), zone="B"),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def test_build_report_context_minimal() -> None:
|
||||
"""Minimal context: only EncounterReport, no RunOutcome."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
spec = _spec()
|
||||
report = run_encounter(spec, runs=10, seed=1, spell_registry=registry)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=report,
|
||||
spec_name="goblin-vs-orc",
|
||||
)
|
||||
|
||||
assert isinstance(ctx, ReportContext)
|
||||
assert ctx.encounter is report
|
||||
assert ctx.spec_name == "goblin-vs-orc"
|
||||
assert ctx.danger is None
|
||||
assert ctx.nl_stats is None
|
||||
assert ctx.client_stats is None
|
||||
assert ctx.matchup is None
|
||||
assert ctx.danger_map is None
|
||||
assert ctx.frontier_value is None
|
||||
assert ctx.sensitivity is None
|
||||
|
||||
|
||||
def test_build_report_context_with_outcomes() -> None:
|
||||
"""Context with RunOutcome -> DangerMetrics computed for the party."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
spec = _spec()
|
||||
report, outcomes = run_encounter(
|
||||
spec, runs=100, seed=1, spell_registry=registry, collect_runs=True
|
||||
)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=report,
|
||||
spec_name="goblin-vs-orc",
|
||||
outcomes=outcomes,
|
||||
party="players",
|
||||
)
|
||||
|
||||
assert ctx.danger is not None
|
||||
assert isinstance(ctx.danger, DangerMetrics)
|
||||
assert ctx.danger.runs == 100
|
||||
# p_survives, p_tpk, p_any_death are probabilities in [0, 1]
|
||||
assert 0.0 <= ctx.danger.p_survives <= 1.0
|
||||
assert 0.0 <= ctx.danger.p_tpk <= 1.0
|
||||
assert 0.0 <= ctx.danger.p_any_death <= 1.0
|
||||
assert ctx.danger.expected_hp_lost >= 0.0
|
||||
assert ctx.danger.expected_rounds > 0.0
|
||||
assert ctx.danger.worst_decile_party_hp >= 0.0
|
||||
|
||||
|
||||
def test_build_report_context_with_all_optional_fields() -> None:
|
||||
"""Context with all optional fields populated."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
spec = _spec()
|
||||
report, outcomes = run_encounter(
|
||||
spec, runs=100, seed=1, spell_registry=registry, collect_runs=True
|
||||
)
|
||||
|
||||
nl_stats = NLPolicyStats(calls=5, repairs_attempted=1, repairs_succeeded=1, fallbacks=0)
|
||||
client_stats = ClientStats(
|
||||
calls=5,
|
||||
prompt_tokens=1000,
|
||||
completion_tokens=200,
|
||||
cost_usd=0.001,
|
||||
cache_hits=0,
|
||||
cache_misses=5,
|
||||
)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=report,
|
||||
spec_name="goblin-vs-orc",
|
||||
outcomes=outcomes,
|
||||
party="players",
|
||||
nl_stats=nl_stats,
|
||||
client_stats=client_stats,
|
||||
)
|
||||
|
||||
assert ctx.nl_stats is nl_stats
|
||||
assert ctx.client_stats is client_stats
|
||||
assert ctx.nl_stats.calls == 5 # type: ignore[reportOptionalMemberAccess]
|
||||
assert ctx.nl_stats.repairs_attempted == 1 # type: ignore[reportOptionalMemberAccess]
|
||||
assert ctx.nl_stats.repairs_succeeded == 1 # type: ignore[reportOptionalMemberAccess]
|
||||
assert ctx.nl_stats.fallbacks == 0 # type: ignore[reportOptionalMemberAccess]
|
||||
assert ctx.client_stats.prompt_tokens == 1000 # type: ignore[reportOptionalMemberAccess]
|
||||
assert ctx.client_stats.cost_usd == 0.001 # type: ignore[reportOptionalMemberAccess]
|
||||
|
||||
|
||||
def test_report_context_immutable() -> None:
|
||||
"""ReportContext is a frozen dataclass (immutable)."""
|
||||
registry = load_spell_registry(SPELLS_DIR)
|
||||
spec = _spec()
|
||||
report = run_encounter(spec, runs=5, seed=1, spell_registry=registry)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=report,
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
# Frozen dataclass -> assignment raises FrozenInstanceError
|
||||
with pytest.raises(FrozenInstanceError):
|
||||
ctx.encounter = None # type: ignore[misc]
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Tests for report prompt pack (P5.3 T2)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from pf1e_simulator.prompt import PromptPack
|
||||
from pf1e_simulator.report import build_report_context, build_report_prompt
|
||||
from pf1e_simulator.runner import CombatantAttrition, EncounterReport
|
||||
|
||||
|
||||
def test_report_prompt_pack_version_stable() -> None:
|
||||
"""Prompt pack version hash is stable for the fixture."""
|
||||
pack_dir = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
pack = PromptPack.load(pack_dir)
|
||||
|
||||
report_version = pack.prompt_version("report")
|
||||
assert report_version.startswith("0.1.0-test:report:")
|
||||
assert len(report_version.split(":")[2]) == 12 # hash12
|
||||
|
||||
|
||||
def test_report_prompt_renders() -> None:
|
||||
"""build_report_prompt renders the context correctly."""
|
||||
pack_dir = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
pack = PromptPack.load(pack_dir)
|
||||
template = pack.get("report")
|
||||
|
||||
encounter = EncounterReport(
|
||||
runs=100,
|
||||
seed=1,
|
||||
wins={"players": 65, "monsters": 33},
|
||||
draws=2,
|
||||
avg_rounds=7.2,
|
||||
attrition={
|
||||
"goblin": CombatantAttrition(2.45, 0.12, 18.30, 4.20, 0.00),
|
||||
"orc": CombatantAttrition(3.20, 0.25, 28.40, 15.60, 0.00),
|
||||
},
|
||||
)
|
||||
|
||||
ctx = build_report_context(
|
||||
encounter=encounter,
|
||||
spec_name="test",
|
||||
)
|
||||
|
||||
prompt = build_report_prompt(ctx, template)
|
||||
|
||||
# Should contain the system prompt text
|
||||
assert "Vous êtes le narrateur" in prompt
|
||||
assert "Bilan d'équilibrage" in prompt
|
||||
assert "Danger et attrition" in prompt
|
||||
# Should contain the rendered encounter data
|
||||
assert "Victoires players : 65.0%" in prompt or "Victoires players : 65" in prompt
|
||||
assert "goblin" in prompt
|
||||
assert "orc" in prompt
|
||||
# Should contain attrition
|
||||
assert "Attrition moyenne par combatant" in prompt
|
||||
assert "touches" in prompt
|
||||
|
||||
|
||||
def test_report_prompt_version_different_from_nl_default() -> None:
|
||||
"""report prompt_version differs from nl_default."""
|
||||
pack_dir = Path(__file__).resolve().parent / "fixtures" / "prompt_pack_fs"
|
||||
pack = PromptPack.load(pack_dir)
|
||||
|
||||
report_version = pack.prompt_version("report")
|
||||
nl_version = pack.prompt_version("nl_default")
|
||||
assert report_version != nl_version
|
||||
+78
-1
@@ -22,7 +22,13 @@ from pf1e_simulator.models import (
|
||||
DamageComponent,
|
||||
Saves,
|
||||
)
|
||||
from pf1e_simulator.runner import EncounterSpec, Side, build_states, run_encounter
|
||||
from pf1e_simulator.runner import (
|
||||
EncounterReport,
|
||||
EncounterSpec,
|
||||
Side,
|
||||
build_states,
|
||||
run_encounter,
|
||||
)
|
||||
|
||||
|
||||
def make_combatant(
|
||||
@@ -391,3 +397,74 @@ def test_run_encounter_reports_healing_done() -> None:
|
||||
report = run_encounter(spec, runs=5, seed=11, policy=_self_healing_policy)
|
||||
assert report.attrition["healer"].healing_done > 0.0
|
||||
assert report.attrition["healer"].damage_taken > 0.0
|
||||
|
||||
|
||||
def _duel_spec() -> EncounterSpec:
|
||||
return EncounterSpec(
|
||||
map=make_map(),
|
||||
sides=(
|
||||
Side(name="players", combatants=(make_combatant("p1", hp=20),), zone="A"),
|
||||
Side(name="monsters", combatants=(make_combatant("m1", hp=20),), zone="B"),
|
||||
),
|
||||
round_cap=20,
|
||||
)
|
||||
|
||||
|
||||
def test_run_encounter_default_unchanged_without_collect_runs() -> None:
|
||||
"""Without collect_runs the call still returns the bare EncounterReport."""
|
||||
result = run_encounter(_duel_spec(), runs=5, seed=3)
|
||||
assert isinstance(result, EncounterReport)
|
||||
|
||||
|
||||
def test_run_encounter_collect_runs_returns_report_and_summaries() -> None:
|
||||
"""collect_runs=True returns (report, per-run summaries) with consistent aggregates."""
|
||||
spec = _duel_spec()
|
||||
report, outcomes = run_encounter(spec, runs=10, seed=3, collect_runs=True)
|
||||
assert isinstance(report, EncounterReport)
|
||||
assert len(outcomes) == 10
|
||||
wins = sum(1 for o in outcomes if o.winner == "players")
|
||||
draws = sum(1 for o in outcomes if o.winner is None)
|
||||
assert wins == report.wins["players"]
|
||||
assert draws == report.draws
|
||||
assert sum(o.rounds for o in outcomes) / 10 == report.avg_rounds
|
||||
for o in outcomes:
|
||||
assert {c.id for c in o.combatants} == {"p1", "m1"}
|
||||
by_id = {c.id: c for c in o.combatants}
|
||||
assert by_id["p1"].side == "players"
|
||||
assert by_id["m1"].side == "monsters"
|
||||
assert by_id["p1"].hp_max == 20
|
||||
# dead is the engine death condition: hp <= -CON (con 12 here)
|
||||
for c in o.combatants:
|
||||
assert c.dead == (c.hp <= -12)
|
||||
|
||||
|
||||
def test_run_encounter_collect_runs_deterministic() -> None:
|
||||
"""Same seed -> identical per-run summaries."""
|
||||
spec = _duel_spec()
|
||||
_, a = run_encounter(spec, runs=8, seed=9, collect_runs=True)
|
||||
_, b = run_encounter(spec, runs=8, seed=9, collect_runs=True)
|
||||
assert a == b
|
||||
|
||||
|
||||
def test_run_encounter_collect_runs_dead_matches_winner() -> None:
|
||||
"""A side that never wins still ends runs with its member dead in the summaries."""
|
||||
spec = EncounterSpec(
|
||||
map=make_map(),
|
||||
sides=(
|
||||
Side(name="players", combatants=(make_combatant("weak", hp=4),), zone="A"),
|
||||
Side(
|
||||
name="monsters",
|
||||
combatants=(make_combatant("strong", hp=40, attack_bonus=10, damage="2d6"),) * 2,
|
||||
zone="B",
|
||||
),
|
||||
),
|
||||
round_cap=20,
|
||||
)
|
||||
report, outcomes = run_encounter(spec, runs=5, seed=5, collect_runs=True)
|
||||
assert report.wins["monsters"] == 5
|
||||
for o in outcomes:
|
||||
weak = next(c for c in o.combatants if c.id == "weak")
|
||||
# PF1e: the battle ends when weak is INACTIVE (hp < 0), not necessarily
|
||||
# dead - dead must mirror the engine condition hp <= -CON, not the loss.
|
||||
assert weak.hp < 0
|
||||
assert weak.dead == (weak.hp <= -12)
|
||||
|
||||
Reference in New Issue
Block a user