feat: establish simulation scaling baseline

This commit is contained in:
Rijad Zuzo
2026-07-16 13:07:01 +02:00
parent 4f88d15e12
commit 4be72639d8
14 changed files with 1004 additions and 15 deletions
+8
View File
@@ -57,6 +57,10 @@ would otherwise obscure that lifecycle:
- `simulation/definitions/` owns stable IDs and immutable action/profession
definitions.
Outside the runtime lifecycle, `simulation/benchmark/` owns reusable,
schema-valid workload fixtures. CLI tools and headless scenarios consume those
fixtures; production simulation does not depend on benchmark code.
The manager deliberately remains a façade instead of being split into a
collection of scene-tree manager nodes. A new collaborator is justified when
one cohesive rule set has several real consumers or makes the tick lifecycle
@@ -76,6 +80,7 @@ hard to read.
| `simulation/state/` | Versioned, serializable mutable records |
| `simulation/definitions/` | Stable IDs and immutable gameplay definitions |
| `simulation/persistence/` | Validated local save-file storage |
| `simulation/benchmark/` | Reproducible full-fidelity headless workloads and metrics |
| `world/` | Loaded-world interaction geometry and presentation adapters |
| `world/resource_nodes/` | Finite resource presentation bound by stable ID |
| `world/storage/` | Storage interaction geometry, never stored quantities |
@@ -110,6 +115,9 @@ improving ownership.
- New mutable features define serialization and deterministic continuation at
the same time as their first gameplay use. Cross-record causes use stable
event IDs rather than object references or prose.
- Benchmark fixtures use ordinary simulation records and must round-trip
through the current state schema. They may control workload setup, but must
not add benchmark-only fields or branches to production saves and ticks.
- Opportunity records reference stable NPC, storage, resource, trigger-event,
and resolution-event IDs. Their generator may observe authoritative state
and history, but it does not mutate the economy or command NPC behavior. The
+10 -5
View File
@@ -750,14 +750,19 @@ Completed:
derives one concise ordinary harvest route from current player-usable finite
resources and the real pantry or woodpile. It clears when help emerges or
the need closes, and adds no quest, waypoint, tracker, or saved UI state.
32. Deterministic scaling baseline: one schema-valid full-fidelity fixture now
drives a repeatable CLI ledger and fast headless regression across rising
NPC and event counts. The first reviewed capture sets a local 600-NPC target,
records identical checksums across samples, and exposes repeated scarce-food
population scans plus objective-event growth as the next measured work.
Next:
1. Begin Milestone 8 with a deterministic headless scaling baseline over
increasing NPC and event counts. Measure current tick throughput,
serialized-state growth, and retained history before choosing a target
population or adding spatial partitions and simulation LOD, as sequenced in
`LEARNING_ROADMAP.md`.
1. Reuse one stable per-tick population view in the existing
trusted-starving-subject query, preserve exact action/checksum behavior, and
rerun simulation scaling baseline 01. Let the resulting profile determine
whether the following slice needs another bounded query optimization, a
spatial index, or the first active/abstract LOD contract.
Do not start with GIS data, a full city, a large asset pack, or more NPC
mechanics. The next proof is a beautiful stage for the systems that already
+24 -6
View File
@@ -467,7 +467,10 @@ a smaller active population remains fully represented. Moving an NPC between
fidelity levels preserves identity, inventory, task, relationships, and
important history.
Set the target population only after collecting baseline measurements.
[Simulation scaling baseline 01](benchmarks/SIMULATION_SCALING_BASELINE_01.md)
sets the first local target at 600 data-only full-fidelity NPCs while preserving
deterministic state. Revisit that target when world presentation or LOD enters
the measured workload.
## Milestone 9 — Player interaction and social agency
@@ -849,14 +852,29 @@ cared-about shortage now produces an inspectable opportunity whose NPC and
player responses both use the originating resource, relationship, knowledge,
and event systems rather than quest-only duplicates.
The immediate next slice should begin Milestone 8 with a deterministic headless
scaling baseline: measure tick throughput, serialized-state growth, and retained
history across increasing NPC/event counts using the current full-fidelity
simulation. Set an evidence-backed target before adding spatial partitions,
batching, or active/abstract LOD transitions.
The first Milestone 8 measurement slice is complete. A reusable schema-valid
headless fixture and CLI runner now measure three fresh deterministic samples
across 6, 60, and 600 NPCs plus 600 and 6,000 seeded event histories. The
reviewed Apple M1 Max baseline reaches 65.84 ticks per second at 600 NPCs, but
10x population from 60 to 600 costs about 24.4x per tick and creates 20,950
objective events over 200 measured ticks. The machine-readable samples,
workload exclusions, checksums, and local 50-ticks-per-second reference target
live in
[Simulation scaling baseline 01](benchmarks/SIMULATION_SCALING_BASELINE_01.md).
The immediate next slice should build one bounded per-tick population view for
the existing trusted-starving-subject query. That scarce-food path currently
rebuilds an all-NPC lookup for each applicable idle decision, making it the
first code-level candidate consistent with the measured superlinear growth.
Preserve exact selection/checksum behavior and rerun the same ledger before
choosing a spatial index or active/abstract LOD design.
Recently completed:
- Deterministic simulation scaling baseline: schema-valid full-fidelity
fixtures, a repeatable CLI runner, and a fast headless regression now record
population throughput, phase timing, serialized-state growth, history
retention, event growth, and deterministic checksums before optimization.
- Simulation responsibility cleanup: storage/inventory transactions now live
in `VillageEconomy`, ordered history and rate queries live in
`SimulationEventLog`, and `SimulationManager` exposes a shorter tick
+15 -4
View File
@@ -943,10 +943,21 @@ the need closes. The result and HUD add no quest acceptance, waypoint, tracker,
save field, RNG draw, or simulation mutation. This completes the Milestone 7
simulation-garden exit proof.
The next slice should start Milestone 8 with a deterministic headless scaling
baseline over increasing NPC and event counts. Measure tick throughput,
serialized-state growth, and retained history before selecting a target
population or implementing spatial partitions and active/abstract LOD.
The first Milestone 8 measurement slice is complete. A reusable schema-valid
headless fixture and CLI runner now measure full-fidelity simulation throughput,
phase timing, serialized-state growth, retained knowledge, objective event
growth, and deterministic checksums across 6, 60, and 600 NPCs plus 600 and
6,000 seeded histories. The reviewed Apple M1 Max baseline reaches 65.84 ticks
per second for 600 data-only NPCs, above the initial local 50-ticks-per-second
reference target, while exposing superlinear population cost and rapid
objective-log growth. Workload details and raw samples live in
[`docs/benchmarks/`](benchmarks/SIMULATION_SCALING_BASELINE_01.md).
The next slice should build one stable per-tick population view for the
existing trusted-starving-subject query, which currently rebuilds an all-NPC
map for each applicable scarce-food decision. Preserve exact selection and
continuation checksums, rerun the same benchmark, and use the remaining measured
cost before committing to spatial partitions or active/abstract LOD.
The remaining simulation-garden target still aims for:
+3
View File
@@ -25,6 +25,9 @@ sources of truth.
current data contracts.
7. [`decisions/`](decisions/) contains durable architectural decisions,
including consequences and revisit conditions.
8. [`benchmarks/`](benchmarks/) contains reviewed, workload-specific
performance ledgers and their machine-readable samples. Measurements are
local evidence, not portable CI limits.
When documents disagree:
+23
View File
@@ -0,0 +1,23 @@
# Simulation benchmark ledger
This directory stores reviewed benchmark snapshots. Each snapshot must name the
workload, seed, Godot version, hardware context, sample count, simulated tick
count, and important exclusions so later comparisons remain honest.
Run the current scaling harness from the project root with Godot 4.7:
```bash
/Applications/Godot.app/Contents/MacOS/Godot \
--headless --path "$PWD" \
--script res://tools/benchmark_simulation_scaling.gd
```
The default report is written under `user://`. Pass
`-- --host-label="<hardware>" --output=res://docs/benchmarks/<name>.json` only
when intentionally capturing a reviewed project baseline.
Reviewed captures:
- [Simulation scaling baseline 01](SIMULATION_SCALING_BASELINE_01.md) records
the first full-fidelity population/history measurements and the bounded next
optimization.
@@ -0,0 +1,99 @@
# Simulation scaling baseline 01
This is the first reviewed Milestone 8 measurement of the current
full-fidelity, data-only simulation. It establishes a reproducible reference
before introducing batching, spatial partitions, or simulation LOD.
The machine-readable samples and checksums are stored beside this note in
[`simulation_scaling_baseline_01.json`](simulation_scaling_baseline_01.json).
## Capture context
- captured: 2026-07-16;
- engine: Godot 4.7 stable;
- host: Apple M1 Max with 64 GB memory on macOS, with 10 processors reported
by Godot;
- benchmark seed: `8088`;
- workload: `full_fidelity_headless_arrival`;
- samples: three fresh managers per case, using the median;
- cadence: 10 warmup ticks, then 200 measured ticks at 1.2 simulated seconds
per tick.
Run the reviewed capture from the project root:
```bash
/Applications/Godot.app/Contents/MacOS/Godot \
--headless --path "$PWD" \
--script res://tools/benchmark_simulation_scaling.gd -- \
--host-label="Apple M1 Max, 64 GB" \
--output=res://docs/benchmarks/simulation_scaling_baseline_01.json
```
## Workload contract
Every NPC remains a named `SimNPC` and receives the normal needs, schedule,
task, opportunity, and action-selection work each tick. The pantry starts
empty, so ordinary food-supply decisions exercise the current population-wide
queries. Travel completes through the same deterministic immediate-arrival
convention used by existing headless continuation scenarios. Seeded history is
made of valid immutable storage-deposit events plus at most three known-event
references per NPC, and every prepared fixture must parse through the normal
`SimulationStateRecord` schema.
The timed section excludes manager construction, fixture preparation,
serialization, world scenes, rendering, navigation, and `NpcVisual`. This is a
simulation-throughput baseline, not a complete frame-time or memory profile.
## Results
| Case | NPCs | Seeded history | Median us/tick | Ticks/s | Simulated realtime | Arrival share | End JSON | Events added |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Population 6 | 6 | 0 | 55.74 | 17,942.05 | 21,530.5x | 5.2% | 0.06 MiB | 209 |
| Population 60 | 60 | 0 | 621.87 | 1,608.07 | 1,929.7x | 6.4% | 0.56 MiB | 2,090 |
| Population 600 | 600 | 0 | 15,187.74 | 65.84 | 79.0x | 14.0% | 5.67 MiB | 20,950 |
| History 600 | 60 | 600 | 677.12 | 1,476.85 | 1,772.2x | 5.6% | 0.71 MiB | 2,090 |
| History 6,000 | 60 | 6,000 | 1,421.19 | 703.64 | 844.4x | 2.7% | 2.04 MiB | 2,090 |
All three samples in each case produced the same deterministic checksum and
state counters. The benchmark runner rejects the report if they diverge.
## Findings
Population scaling is the first measured knee. Raising the fixture from 60 to
600 NPCs increases median tick cost by about 24.4x for 10x the population.
Immediate-arrival completion accounts for only 14.0% of the 600-NPC timed
section, so most of that growth remains inside the ordinary simulation tick.
The benchmark is not a function profiler, but code inspection identifies a
bounded first candidate: while food is scarce, each applicable idle decision
calls `RelationshipSystem.get_trusted_starving_subject()`, which rebuilds a map
by scanning every NPC. Building the same stable-ID population view once per
tick should remove repeated work without changing action semantics or tie
breaks. The benchmark and checksum give that change a concrete comparison.
History also has a visible but less urgent cost. At a fixed 60 NPCs, increasing
seeded objective events from 600 to 6,000 raises tick cost about 2.1x and final
serialized state from 0.71 MiB to 2.04 MiB. The 180 seeded known-event
references age out during the measured window, while the objective event log
remains complete. Event-log indexing or archiving therefore remains a later,
separate decision.
Organic event growth matters independently of CPU time. The 600-NPC case adds
20,950 objective events and grows serialized state by 5,235,086 bytes over only
200 measured ticks. Future long-session work should measure event retention and
save cost explicitly rather than treating tick throughput as the whole scale
problem.
## Reference target and next slice
The first local reference target is at least **50 measured ticks per second for
600 data-only full-fidelity NPCs** on this Apple M1 Max workload, with identical
deterministic state and checksum. Baseline 01 reaches 65.84 ticks per second.
This is a local comparison target, not a cross-machine CI timing assertion or a
claim about 600 rendered and navigating characters.
The next slice should build one reusable per-tick population view for the
existing trusted-starving-subject consumer, preserve exact selection and
continuation results, and rerun this ledger. Spatial partitioning and
active/abstract LOD should wait until that bounded change shows what cost
remains.
@@ -0,0 +1,308 @@
{
"benchmark_seed": 8088,
"captured_utc": "2026-07-16T11:05:19Z",
"cases": [
{
"arrival_share_percent": 5.23010675518077,
"arrival_usec_median": 583,
"arrival_usec_samples": [
578,
583,
587
],
"arrivals_processed": 209,
"case_id": "population_006",
"elapsed_usec_max": 11309,
"elapsed_usec_median": 11147,
"elapsed_usec_min": 10991,
"elapsed_usec_samples": [
10991,
11147,
11309
],
"end_event_count": 221,
"end_known_reference_count": 0,
"end_state_bytes": 58576,
"end_tick": 210,
"events_recorded": 209,
"final_checksum": "100dd5dc1e0c22ea23f35dd876d6f56fd5c33c58b1586b6fa9aec47ee1536511",
"history_seed_events": 0,
"measured_ticks": 200,
"npc_updates": 1200,
"population": 6,
"realtime_factor_median": 21530.4566251009,
"sample_count": 3,
"schema_version": 1,
"seed": 8088,
"setup_usec_median": 772,
"setup_usec_samples": [
535,
772,
3251
],
"simulation_usec_median": 10544,
"simulation_usec_samples": [
10387,
10544,
10700
],
"start_event_count": 12,
"start_known_reference_count": 0,
"start_state_bytes": 7632,
"start_tick": 10,
"state_growth_bytes": 50944,
"tick_interval": 1.2,
"ticks_per_second_median": 17942.0471875841,
"usec_per_tick_median": 55.735,
"warmup_arrivals": 12,
"warmup_ticks": 10,
"workload_id": "full_fidelity_headless_arrival"
},
{
"arrival_share_percent": 6.39769081713877,
"arrival_usec_median": 7957,
"arrival_usec_samples": [
7747,
7957,
8499
],
"arrivals_processed": 2090,
"case_id": "population_060",
"elapsed_usec_max": 142845,
"elapsed_usec_median": 124373,
"elapsed_usec_min": 123769,
"elapsed_usec_samples": [
123769,
124373,
142845
],
"end_event_count": 2210,
"end_known_reference_count": 0,
"end_state_bytes": 587324,
"end_tick": 210,
"events_recorded": 2090,
"final_checksum": "58c7f96fc5df7f5399fd82d1a9c212b3c53067e807059e88ffa484ba0eb41e16",
"history_seed_events": 0,
"measured_ticks": 200,
"npc_updates": 12000,
"population": 60,
"realtime_factor_median": 1929.67927122446,
"sample_count": 3,
"schema_version": 1,
"seed": 8088,
"setup_usec_median": 3734,
"setup_usec_samples": [
3603,
3734,
5045
],
"simulation_usec_median": 116595,
"simulation_usec_samples": [
115787,
116595,
134317
],
"start_event_count": 120,
"start_known_reference_count": 0,
"start_state_bytes": 71278,
"start_tick": 10,
"state_growth_bytes": 516046,
"tick_interval": 1.2,
"ticks_per_second_median": 1608.06605935372,
"usec_per_tick_median": 621.865,
"warmup_arrivals": 120,
"warmup_ticks": 10,
"workload_id": "full_fidelity_headless_arrival"
},
{
"arrival_share_percent": 14.0226590657991,
"arrival_usec_median": 425945,
"arrival_usec_samples": [
425585,
425945,
427652
],
"arrivals_processed": 20950,
"case_id": "population_600",
"elapsed_usec_max": 3040453,
"elapsed_usec_median": 3037548,
"elapsed_usec_min": 3028975,
"elapsed_usec_samples": [
3028975,
3037548,
3040453
],
"end_event_count": 22150,
"end_known_reference_count": 0,
"end_state_bytes": 5948447,
"end_tick": 210,
"events_recorded": 20950,
"final_checksum": "fa4efac63fde63f92eb28141bebd1922f665e85ffe7b7ac2b9702dae6e4c24b7",
"history_seed_events": 0,
"measured_ticks": 200,
"npc_updates": 120000,
"population": 600,
"realtime_factor_median": 79.0110971085889,
"sample_count": 3,
"schema_version": 1,
"seed": 8088,
"setup_usec_median": 35480,
"setup_usec_samples": [
35329,
35480,
35664
],
"simulation_usec_median": 2609839,
"simulation_usec_samples": [
2603329,
2609839,
2614455
],
"start_event_count": 1200,
"start_known_reference_count": 0,
"start_state_bytes": 713361,
"start_tick": 10,
"state_growth_bytes": 5235086,
"tick_interval": 1.2,
"ticks_per_second_median": 65.8425809238241,
"usec_per_tick_median": 15187.74,
"warmup_arrivals": 1200,
"warmup_ticks": 10,
"workload_id": "full_fidelity_headless_arrival"
},
{
"arrival_share_percent": 5.63567488535921,
"arrival_usec_median": 7632,
"arrival_usec_samples": [
7621,
7632,
7830
],
"arrivals_processed": 2090,
"case_id": "history_000600",
"elapsed_usec_max": 136012,
"elapsed_usec_median": 135423,
"elapsed_usec_min": 134887,
"elapsed_usec_samples": [
134887,
135423,
136012
],
"end_event_count": 2810,
"end_known_reference_count": 0,
"end_state_bytes": 743004,
"end_tick": 810,
"events_recorded": 2090,
"final_checksum": "7ae86044f7683359d6bfbc8753e2656f3af90f23d040039154c6869dca48cf65",
"history_seed_events": 600,
"measured_ticks": 200,
"npc_updates": 12000,
"population": 60,
"realtime_factor_median": 1772.22480671673,
"sample_count": 3,
"schema_version": 1,
"seed": 8088,
"setup_usec_median": 22795,
"setup_usec_samples": [
22782,
22795,
22881
],
"simulation_usec_median": 127766,
"simulation_usec_samples": [
127030,
127766,
128361
],
"start_event_count": 720,
"start_known_reference_count": 180,
"start_state_bytes": 253098,
"start_tick": 610,
"state_growth_bytes": 489906,
"tick_interval": 1.2,
"ticks_per_second_median": 1476.85400559728,
"usec_per_tick_median": 677.115,
"warmup_arrivals": 120,
"warmup_ticks": 10,
"workload_id": "full_fidelity_headless_arrival"
},
{
"arrival_share_percent": 2.72202422626189,
"arrival_usec_median": 7737,
"arrival_usec_samples": [
7650,
7737,
7809
],
"arrivals_processed": 2090,
"case_id": "history_006000",
"elapsed_usec_max": 287265,
"elapsed_usec_median": 284237,
"elapsed_usec_min": 280605,
"elapsed_usec_samples": [
280605,
284237,
287265
],
"end_event_count": 8210,
"end_known_reference_count": 0,
"end_state_bytes": 2141615,
"end_tick": 6210,
"events_recorded": 2090,
"final_checksum": "b4df2bac21ed5ce6dcd92736e210db5f3114d959dcb15ae7796493362911b6d7",
"history_seed_events": 6000,
"measured_ticks": 200,
"npc_updates": 12000,
"population": 60,
"realtime_factor_median": 844.365793334436,
"sample_count": 3,
"schema_version": 1,
"seed": 8088,
"setup_usec_median": 154433,
"setup_usec_samples": [
153698,
154433,
155153
],
"simulation_usec_median": 276401,
"simulation_usec_samples": [
272849,
276401,
279590
],
"start_event_count": 6120,
"start_known_reference_count": 180,
"start_state_bytes": 1649700,
"start_tick": 6010,
"state_growth_bytes": 491915,
"tick_interval": 1.2,
"ticks_per_second_median": 703.63816111203,
"usec_per_tick_median": 1421.185,
"warmup_arrivals": 120,
"warmup_ticks": 10,
"workload_id": "full_fidelity_headless_arrival"
}
],
"engine_version": "4.7-stable (official)",
"exclusions": [
"manager_and_fixture_setup",
"state_serialization",
"world_scene",
"rendering",
"navigation",
"npc_visual"
],
"host_label": "Apple M1 Max, 64 GB",
"measured_ticks": 200,
"platform": "macOS",
"processor_count": 10,
"sample_count": 3,
"schema_version": 1,
"timed_phases": [
"simulation_tick",
"headless_arrival_completion"
],
"warmup_ticks": 10,
"workload": "All NPCs receive full per-tick needs/task updates and ordinary action decisions; travel resolves through the deterministic immediate-arrival headless convention.",
"workload_id": "full_fidelity_headless_arrival"
}