Chapter at a Glance

Controlled experiments compare alternatives without confusing feasibility with preference.

Learn / Chapter 05

Experimentation and Trade-offs

Several configurations can pass. What would make one preferable?

Learning objective: design a controlled comparison, distinguish feasibility from preference, and identify the evidence missing from a recommendation.

Prerequisite: Executable Engineering Models.

Ask a decision question before running trials

The monitoring team can change where it places a sensor or what footprint it assumes. Some modeled configurations satisfy the coverage criterion. The next question is not simply how to obtain another PASS, but what a comparison can establish about choosing a configuration. A larger percentage may be useful without being sufficient to recommend a design.

Begin by stating the decision and the available evidence. Here we can compare geometric coverage under one ideal circular footprint. We cannot compare equipment prices, installation effort or detection reliability because those properties have no model or data in this demonstrator. An honest experiment can reveal this gap rather than conceal it behind a ranking.

A reference configuration gives us an identified starting point. A candidate changes specified input values. A different architecture would change how the system is organized, such as introducing several sensors. This lesson compares configurations within the existing single-sensor architecture. Each recorded row is an independent run; recording four rows never creates a four-sensor system.

Control what the experiment means

A factor is an input deliberately varied to answer a question. Controlled inputs stay fixed so that differences in response can be attributed clearly. Our measured responses are estimated coverage and its margin against the criterion. The area, threshold, grid and model remain the same throughout the browser comparison.

First ask about placement. Hold radius and y fixed, and compare the center with a location on the left boundary. Predict how much of the footprint contributes inside the rectangle. Then ask about assumed radius: keep the center fixed and compare two footprints. Separate these questions so that a changed position does not obscure the effect of a changed radius.

Changing two factors together can still describe a candidate, but it weakens attribution. If the result improves, which change caused it? A useful protocol makes that ambiguity visible. Record exact inputs, not just a memorable label such as “better sensor,” and state what each trial is meant to test before looking at its outcome.

Keep the criterion fixed when comparing configurations against the same requirement. The threshold proposal exercise shows why a lower threshold may produce PASS without improving coverage. That is a different engineering question. It must not be mixed into the table as if the design had improved under an unchanged requirement.

Experiment: four separate configurations

Predict each result before recording it. All coordinates and radii below use model units.

  1. Clear comparison, then reset to SCN-001. Record trial 1: x=50, y=50, radius=35.
  2. Change only x to 0. Record trial 2: x=0, y=50, radius=35. Compare these two rows to investigate boundary clipping.
  3. Reset, then change radius to 45. Record trial 3: x=50, y=50, radius=45.
  4. Change only radius to 50. Record trial 4: x=50, y=50, radius=50. Compare trials 3 and 4 under the same criterion.
  5. Identify which rows pass. Write a conditional recommendation: what did you observe, what assumptions support it, and what missing evidence prevents a final choice?

Reset restores current inputs while keeping recorded rows. Clear comparison removes the rows; reloading discards the temporary table. The fifth record is blocked until you clear it.

One sensor. One explicit criterion.

Initialized from SCN-001. Change position or radius, then record separate trials below to compare them against the same criterion.

Interactive browser what-if run. These changes are temporary. They do not edit the source scenario or create saved CI verification evidence.
Ideal single-sensor coverageA rectangular monitoring area with one circular footprint, clipped to the area.

Area: 100 × 100 model units. Origin is the upper-left corner; x increases right and y increases down. Filled disk: ideal footprint. Dark dot: sensor position.

Sensor configuration (model units)

Use arrow keys or type whole model units. x: 0–100; y: 0–100; radius: 0–200. Position controls stay within the area; the engineering model also accepts outside positions.

FAIL — 38.52% estimated coverage; threshold 50.00%.

3852 / 10000 cell centers covered. Margin: -11.48 percentage points.

Compare separate single-sensor trials

Record up to four configurations. Each row is a separate one-sensor what-if run with the same area, threshold and model; rows do not combine sensors. Reset keeps your rows. Reloading or leaving this page discards them.

No configurations recorded.

Temporary browser comparisons — not retained CI evidence
TrialxyRadiusCoverage estimateMargin (pp)Criterion

Positions and radii use model units; margins use percentage points. A larger margin does not establish optimality: cost, installation constraints and real sensing performance are not evaluated.

Definition and interpretation

REQ-COV-001 defines the 50% criterion. This threshold is a teaching assumption, not an operationally derived target. TEST-COV-001 evaluates the reference scenario using MODEL-COV-001 v0.1.0.

Method: cell-center-grid, 100 × 100 equal-area cells. A cell is covered when its center’s distance is ≤ radius; zero radius means zero coverage. This is a sampled estimate, with discretization error. The smooth circle depicts the ideal footprint, not the sampled cells. PASS uses the unrounded estimate and includes equality.

  • One fixed ideal circular footprint in a uniformly weighted rectangular area.
  • No terrain, occlusion, uncertainty, time, faults, or communication effects.

No directional sensitivity or real sensor calibration is modeled. Area coverage is not detection probability, tracking performance or mission effectiveness. A PASS only satisfies this idealized geometric criterion.

Compare your interpretation

The placement pair isolates clipping at a fixed radius. The centered footprint pair isolates assumed radius. Read the calculated outcomes in your table rather than substituting an expected PASS. A larger assumed footprint can increase geometric coverage, but the model does not show that such a sensor exists, is affordable or can be installed. These observations support a bounded geometric comparison, not a preferred operational design.

Feasible under a criterion is not preferred overall

A passing row satisfies this geometric criterion under the model assumptions. It does not establish every condition required for a feasible real installation. The margin reports distance from one threshold in percentage points. It is not a safety margin against every omitted effect, and a higher margin is not automatically a better engineering decision.

Preference requires an objective and justified constraints. If installation access matters, obtain evidence about acceptable locations. If cost matters, obtain relevant cost data. If a larger footprint is proposed, establish the physical performance and conditions supporting that assumption. Do not turn model units into prices or treat assumed radius as a calibrated product specification.

A trade-off concerns competing considerations in a decision. This model supplies only geometric observations; it cannot manufacture the other side of a numerical trade-off. Assigning arbitrary weights to unknown cost or reliability would make a score look precise while leaving its basis unjustified. It is legitimate to conclude that the available evidence is insufficient to select between passing candidates.

Likewise, two tested radii do not establish a globally optimal footprint or robustness across all scenarios. A placement with the same modeled coverage may have different practical constraints. Uniformly weighted area also ignores whether particular locations are more important. Keep these limitations attached to the comparison instead of promoting a small set of trials into an architecture recommendation.

Make the comparison reproducible

Follow the repository configuration experiment from an identified checkout. After installation and checks, its executable example calls the same pure coverage function and evaluator for the four stated inputs. Compare its numerical outputs with your browser rows. It prints exploratory calculations; it does not create a retained verification record or execute the registered reference test against the candidates.

Canonical YAML remains the source of the area, criterion, reference configuration and model registration. Candidate input copies do not overwrite it. The existing radius proposal fixture reproduces the radius-45 trial, but does not reproduce the boundary-placement trial or every browser edit. Preserve those distinctions when explaining what was executed.

For retained reference evidence, use npm run evidence and inspect the actual revision, working-tree status, exact inputs, method and execution context. The artifact instructions explain reproduction and retention limits. Browser rows and terminal output are useful observations, but neither becomes CI evidence merely because it resembles a saved result.

Check your reasoning

Answer before revealing each explanation.

1. Why keep the threshold fixed across the four trials?

To compare configurations against the same criterion. Lowering the threshold changes the question and can change PASS without improving geometric coverage.

2. Why separate placement and radius changes?

It makes attribution clearer. Changing both together describes another candidate, but does not isolate which factor explains the response.

3. Does a passing radius-50 trial demonstrate an available sensor?

No. Radius is an assumed input in model units. Physical range, calibration and installation suitability require separate evidence.

4. Can the best recommendation be to defer selection?

Yes. If relevant objectives, constraints or evidence are missing, report the bounded comparison and identify the information needed before choosing.

Next: judge the evidence

We can now frame a comparison without confusing a passing candidate with an accepted design. The next question is how strongly each observation supports its intended conclusion. Chapter 6: Verification and Engineering Evidence examines the checks, execution context and model limits behind that conclusion.

← Chapter 4 · Learning path · Inspect engineering sources →