Chapter at a Glance

Evidence is useful only when we know what was executed, what it establishes, and its limits.

Learn / Chapter 06

Verification and Engineering Evidence

The checks are green. Has the engineering requirement been satisfied?

Learning objective: identify which claim an execution supports, distinguish test success from requirement satisfaction, and explain the limits of its evidence.

Prerequisite: Experimentation and Trade-offs.

Start with the claim, not the badge

A reviewer sees successful software checks alongside a failing reference coverage result. Another engineer demonstrates a passing browser configuration. These observations look inconsistent only if their subjects are left unstated. Before accepting a conclusion, identify the criterion, evaluated configuration, method and activity that produced each observation.

The claim here is narrow: whether the reference configuration satisfies a geometric coverage threshold under V0 assumptions. The operator’s broader monitoring need remains separate. A test badge cannot supply a missing operational basis for that threshold, and a passing idealized configuration cannot establish physical sensing performance. Evidence is useful when its scope is explicit.

Verification in this lesson means checking specified behavior or a criterion against defined expectations. We make no claim of conformity with a verification standard. Follow the actual requirement and registered test to see what this example checks, rather than inferring authority from their identifiers.

Ask what each check establishes

Source validation checks supported fields, parameter ranges and resolvable relationships. Independent model tests compare known geometries and exercise boundary and invalid-input behavior. The evaluator compares an estimated percentage with a criterion. These activities answer related but different questions; succeeding at one does not automatically establish the others.

The registered reference test deliberately expects the canonical configuration to fail the coverage criterion. Its assertions check that the estimate lies in the expected neighborhood, the threshold remains unchanged and the evaluation reports failure with a negative margin. A successful test run means those assertions held. It preserves the engineering finding instead of disguising FAIL as PASS.

A changed browser radius supplies different inputs to the same model. Its passing outcome does not contradict the reference result, because it is a different evaluated configuration. Nor does the browser execute Vitest. The source scenario and registered test remain unchanged when you manipulate temporary controls. To establish that a test executed, inspect its actual command and logs.

Experiment: reconcile the observations

  1. Reset proposal. Identify the canonical criterion, inputs and geometric outcome.
  2. Try the radius example while keeping the criterion fixed. Explain which definition changed and what the new outcome establishes.
  3. Follow TEST-COV-001’s source. Did your browser edit change its canonical inputs or execute its assertions?
  4. Reset and explain how a successful software suite, a failing canonical criterion and a passing candidate can all be correct.
  5. Suppose an obstruction challenges the ideal footprint. Which conclusion needs review even if the calculation repeats?

A definition change, not an approval

Temporary, unapproved browser proposal. This workspace changes a copy of the definitions. It does not edit YAML, execute TEST-COV-001, retain evidence or approve a requirement. Reset restores the baseline; leaving or reloading discards edits.

Canonical baseline from this site build

Requirement REQ-COV-001: 50%. Scenario SCN-001: radius 35, center (50, 50), area 100 × 100 model units.

Under reference scenario SCN-001 and the V0 model assumptions, the monitoring configuration shall cover at least 50 percent of the defined area geometrically.

FAIL — 38.52% estimated coverage. Margin: -11.48 percentage points.

This is a build-time calculation, not a displayed CI verification artifact. The baseline stays unchanged while you edit.

Temporary proposal controls

Threshold: 0–100%. Radius: 0–200 model units. Choosing a different change restores that field’s baseline value.

This edit applies only to the temporary copy. Its source_need must resolve to the existing need.

Enable JavaScript to validate and evaluate a temporary proposal.

Exact field differences

The requirement statement and basis change with the threshold so the proposed definition stays consistent. These differences describe the current trial, not a committed revision.

Canonical value → temporary proposed value
FieldBaselineProposal
Inspect the validated proposed definition
No proposal evaluated yet.

Both calculations use MODEL-COV-001 v0.1.0: a 100 × 100 cell-center area estimate. No physical sensor calibration, terrain, occlusion, directional effects, time, uncertainty, faults or communications are modeled. A PASS establishes neither operational suitability nor an optimal design.

Compare your review

The canonical evaluation and the radius proposal use different scenario inputs under the same criterion. The reference test expects the unchanged canonical result to fail; a successful test therefore agrees with that finding. The browser proposal executes calculation/evaluation, not the reference assertions, and creates no retained CI evidence. An obstruction challenges model applicability; none of these outputs calculates its effect or establishes physical suitability.

Inspect the activity behind the record

Follow the repository evidence review from an identified checkout. Run the focused reference test and inspect its assertions. Then generate a reference record. Keep these steps separate: npm run evidence loads validated sources and calls the calculation and evaluator, but does not itself invoke Vitest or execute the assertions.

The generated record’s test ID associates the reference context. That field alone is not proof that the named test ran. In CI, inspect the workflow steps and logs for test execution, then the separate evidence-generation step. Locally, retain the command output needed to explain what you ran. Never turn an association into an execution claim merely because the IDs match.

Inspect the record’s definition fields: requirement, scenario and model identities, model version, exact inputs, threshold and assumptions. Its calculation carries covered cells, total cells, method, grid, percentage, margin and outcome. It does not embed every YAML statement or title. Inspect source definitions at the recorded revision for that surrounding meaning.

Next inspect source commit, working-tree status, timestamp, context and runtime. A dirty checkout cannot be reconstructed from its commit alone. Preserve relevant source differences and the evaluated inputs. Repeated numerical fields may agree while execution identifiers and timestamps differ. Neither repeatability nor a plausible timestamp establishes that every required verification activity occurred.

No retained execution record is loaded on this page. Its baseline is calculated during the site build; interactive results are temporary. The reference artifact instructions explain how to inspect a real chosen-revision record. Artifacts expire after 30 days and can require sign-in. Reproducing an expired artifact creates a new local execution, with its own context.

The proposal analysis record has a different meaning. It includes exact baseline and proposed definitions and explicitly states that the reference test was not executed against the proposal. A passing proposal analysis therefore cannot replace canonical reference evidence or a test log. Read each record according to its generation path.

State what remains unjustified

The shared calculation keeps browser and tests from drifting into competing implementations. Independent geometry checks can detect numerical mistakes in selected cases; invalid-input and equality tests check other contracts. These checks strengthen confidence within their scope. They do not establish a universal discretization bound or prove that an ideal circle represents a physical sensor.

An obstruction, directional sensitivity or calibration issue can undermine interpretation without changing the software’s deterministic output. The model does not include those effects, and the teaching threshold has no derived operational basis. Additional model or physical evidence would be needed for those questions. Avoid inventing a correction factor or treating a geometric margin as protection against every omitted effect.

A useful review conclusion states the checked claim, evaluated definitions, activity and outcome, then names its limits. “The canonical geometric criterion fails as expected by the reference assertions” is different from “the monitoring system is unsuitable in every setting.” Neither a green workflow nor a red criterion answers the broader operational question by itself.

Check your reasoning

Answer before revealing each explanation.

1. How can the reference test pass while the criterion fails?

The assertions expect the canonical geometric failure. A successful assertion confirms that the implementation reports that finding correctly.

2. Does the test ID in reference-evidence.json prove Vitest ran?

No. The generator calculates and evaluates canonical inputs without invoking Vitest. Inspect the separate test command and logs to establish assertion execution.

3. Does a passing radius proposal replace the reference evidence?

No. It evaluates changed inputs in an unapproved proposal context and does not execute the registered reference test against them.

4. What does a repeated number say about physical model validity?

It supports numerical repeatability under the same rules. It does not establish calibration, occlusion behavior or operational suitability.

Next: bound the assistance

These distinctions also constrain future engineering assistance. A tool may help inspect sources, propose a change or repeat a calculation, but its output must retain the activity’s actual meaning. Before delegating work, define the allowed changes, independent checks and review responsibility. Chapter 7: Agentic Engineering explores that boundary through a manual task-contract and proposal-review exercise.

← Chapter 5 · Learning path · Inspect engineering sources →