Definition-of-Done audit
This document scores the repository against the specification’s §76 Definition of Done at each milestone. It is maintained honestly: an item is marked done only when the corresponding capability is real, registered, and verified — never when it is merely designed. The audit is a living document; re-run the checks below before updating a row.
Last audited: real simulator wiring (RealSimulator over the crucible-simulator flow engine, run_real_simulator / run_integration on the harness).
Scorecard
| # | DoD item | Status | Evidence |
|---|---|---|---|
| 1 | Scenarios can be registered | ✅ | scenario-registry, 48 scenarios registered across ten packs — 43 judged by the in-memory double, 5 (conformance-real) by the real crucible-simulator engine (crucible-scenarios list) |
| 2 | Scenarios can be discovered | ✅ | registry + list with filters |
| 3 | Scenarios can be filtered | ✅ | list/run --category/--tag/--pack |
| 4 | Scenarios execute deterministically | ✅ | fixed-clock harness; byte-stable outcomes; parallel ≡ serial (tested) |
| 5 | Simulator integration works | ✅ | RealSimulator drives the real crucible-simulator flow engine ([RealSimulator], pinned revision): operations translated onto the engine’s lifecycle API, real state root/event codes/error codes reported, closed public observation vocabulary; exercised via MockHarness::run_real_simulator and end-to-end with the prover via MockHarness::run_integration |
| 6 | Prover integration works | ✅ | prover adapter wires the real crucible-prover service ([RealProver], pinned revision): ABI preflight, envelopes, mandatory round-trip on the hermetic mock backend; exercised via MockHarness::run_real |
| 7 | Proof verification can be asserted | ✅ | proof-valid/invalid assertions; verification classification in packs |
| 8 | State transitions can be asserted | ✅ | balance/ownership/commitment/state assertions + invariants |
| 9 | Happy paths exist for supported operations | ✅ | flows catalog: register, deposit, merge, transfer, withdraw, lifecycle (CT-HAPPY-*) |
| 10 | Negative paths exist | ✅ | CT-NEG-001..011 |
| 11 | Adversarial paths exist | ✅ | CT-ADV-001..004 |
| 12 | Replay scenarios exist | ✅ | CT-ADV-003, CT-NEG-008, CT-REG-003 |
| 13 | Stale-state scenarios exist | ✅ | CT-ADV-003, CT-NEG-002 family |
| 14 | Public-input mutation scenarios exist | ✅ | CT-ADV-002, CT-ADV-004 |
| 15 | Proof-tampering scenarios exist | ✅ | CT-ADV-001, negative invalid/tampered/malformed |
| 16 | Privacy tests exist | ✅ | CT-PRIV-001..004 + definition-time guard + privacy invariant |
| 17 | Invariants exist | ✅ | seven built-in checks, attached across packs |
| 18 | Conformance tests exist | ✅ | CT-CONF-001..005 (against doubles; see note) |
| 19 | Regression tests exist | ✅ | CT-REG-001..003 from real bugs; never deleted |
| 20 | Fuzz targets exist | ✅ | four targets, seeded, finding→reduction→CT-REG pipeline |
| 21 | Concurrency scenarios exist | ✅ | CT-CONC-001..004 (sequential composition) + --parallel runner |
| 22 | Test vectors exist | ✅ | CT-VEC-001..024 corpus, validated for coherence |
| 23 | Results are machine-readable | ✅ | JSON, JUnit XML, Markdown; CI-usable exit codes |
| 24 | Failures are reproducible | ✅ | seed + environment on every outcome; deterministic replay |
| 25 | Private data is not leaked | ✅ | redaction by construction; privacy pack + invariant + reporter audit |
| 26 | Mock tests clearly distinguished from real crypto tests | ✅ | doubles labeled everywhere; docs/PR template enforce |
| 27 | Soroban integration isolated behind adapters | ✅ | adapters/soroban (contract surface, translation, events; client not wired) |
| 28 | Testnet integration is optional | ✅ | adapters/testnet opt-in; never required by CI |
| 29 | CI covers appropriate scenario classes | ✅ | Level-1/2 + conformance/adversarial/regression/performance/security/testnet/release |
| 30 | Documentation explains the architecture | ✅ | 27 docs + README status maps |
| 31 | Contributors have clear issue surfaces | ✅ | 8 issue templates + PR template + chooser |
| 32 | No simulator implementation duplicated | ✅ | boundary enforced; no state engine in scenarios |
| 33 | No prover implementation duplicated | ✅ | boundary enforced; no proving in scenarios |
| 34 | No unsupported protocol semantics invented | ✅ | agent scenarios deferred (no agent protocol); replay/etc. per declared model |
Done: 34 of 34. DoD 5 and 6 are both closed at the integration level:
the real crucible-simulator flow engine and the real crucible-prover
service machinery each run behind their adapter, together in one run via
MockHarness::run_integration, and CT-CONF-R01..R05 state conformance
contracts the engine itself judges. The one caveat that survives is the
proving backend: it is crucible-prover’s deterministic mock, so no scenario
here is yet a cryptographic conformance check, and real UltraHonk/bb
proving remains
an opt-in exercised in crucible-prover’s own dedicated CI. Everything else
marked done is verifiable with the commands below.
How to verify the done claims
scripts/test-all.sh # fmt, clippy, tests, benches, validate, report
cargo test --workspace # 376 tests
target/debug/crucible-scenarios validate # 48 scenarios + 24 vectors + 2 declarative docs
target/debug/crucible-scenarios report # 29 pass, 19 expected failures, 0 failed
target/debug/crucible-scenarios run --tag real-engine --real-engine # 5 contracts judged by the real engine
target/debug/crucible-scenarios report --parallel # byte-identical to serial
target/debug/crucible-scenarios list --category conformance
target/debug/crucible-scenarios fuzz --seed 42 --iterations 100
cargo bench # six benches
cargo test -p scenario-format committed_example_documents_parse_and_validate
Conformance caveat (DoD 18)
The conformance category holds two packs, and a report says which system judged each contract:
conformance(CT-CONF-001..005) — stated against the deterministic in-memory double, asserting the double’s vocabulary (balance.<actor>.<token>, fixture commitment ids,ct_*event codes). These exercise orchestration and binding semantics.conformance-real(CT-CONF-R01..R05) — stated against the realcrucible-simulatorflow engine, asserting only facts the engine publishes (op.<id>.accepted, its own event codes, and its public state counts). These are judged by the engine itself, viaMockHarness::run_real_simulator, andcrucible-scenarios reportnow includes them: 5 contracts judged by crucible-simulator, 43 by the double.
A test asserts that the real-engine contracts are not all satisfiable by the double, so the distinction they draw is real rather than nominal.
What is still not done — and the reason this caveat stays — is the proving
axis. Contracts still run against crucible-prover’s hermetic mock backend;
real UltraHonk/bb proving is opt-in and heavyweight in the prover
repository, so no scenario here is yet a cryptographic conformance check.
That is the remaining work, and it is gated on the same external audit the
prover repository needs.
A real run is never a silent substitution: RealSimulator reports its own
adapter name, run_real_simulator/run_integration are separate entry
points from run, the CLI refuses to judge a double-backed contract with the
real engine, and announces any real-engine contract it skips.
Change log
- Privacy/concurrency/test-vectors batch — closed DoD 16 (privacy tests), 21 (concurrency scenarios), 22 (test vectors).
- Regression/fuzz batch — closed DoD 19 (regression tests), 20 (fuzz targets).
- Adapters/packs/reporting/declarative/parallel/CI batch — closed DoD 27 (Soroban adapter isolation) and 28 (opt-in testnet), extended DoD 29 (dedicated CI levels), added the reporting, declarative-format, and parallel-execution surfaces, and completed DoD 31 (contributor issue surfaces).
- Benches/scripts/docs batch — completed DoD 30 (full documentation set and accurate status maps), added the six benches, the script set, and the worked-examples map, and produced this audit document.
DoD 5–6 were open at every milestone and blocked on the upstream
repositories. DoD 6 was closed by wiring the real crucible-prover service
behind the prover adapter (pinned revision, hermetic mock backend) and
raising the workspace MSRV to 1.98 to compile the edition-2024 prover
crates. DoD 5 was closed by adding RealSimulator over the real
crucible-simulator flow engine (pinned revision) and exposing
run_real_simulator / run_integration on the harness, so the SIMULATE
and PROVE seams are both real and can be driven together in one run. The
conformance pack itself is still evaluated against the double; see the
conformance caveat above for the remaining work and why it is deliberate.