Break-to-Verify

Break-to-Verify is a check that a generated test can actually fail. Before merge, the test runs against the normal application, then against the same application with a controlled fault switched on, then against the normal application again. A test that stays green while the application is broken is flagged, and merging it requires an explicit, logged human acceptance.

Why a green test is not enough (Demonstrated)

A new test that passes tells you very little on its own. It may check the right behaviour, or it may check something that is always true: an element that is always visible, an assertion on the wrong field, or a step that never reaches the logic under test.

Generated tests make this question sharper. Reviewers read the code and the mapping, but a test can look correct and still be insensitive to the fault it is meant to catch.

Break-to-Verify answers the question with a run instead of a review. The idea is similar to mutation testing: change the application in a known way and see whether the test notices. The difference is scope. Mutation testing generates many automatic changes to the code. Break-to-Verify uses a small number of deliberate, meaningful faults that are prepared in the test environment.

Break-to-Verify is deterministic and uses no model, so it does not need a model evaluation. Its current label is on the Capabilities page.

How a run works

The engineer starts Break-to-Verify from the script page of a generated test. The platform then runs three separate test runs, each recorded on its own:

1. Green: the test runs against the normal application and must pass.
2. Red: the test runs again with a controlled fault switched on for this run only. The expected outcome is a failure on an assertion.
3. Green again: the fault is switched off and the test must pass once more.

The fault is scoped to the run. In the reference setup, it applies only to the test user of that run, so other users, other runs and the code generation agent are not affected. The switch is reachable only from the application’s API and the test runner with a dedicated token, not from the agent’s browser.

The portal shows the three phases as a timeline with the result of each run. Runs created by Break-to-Verify are tagged as such and are not counted in Control Tower KPIs such as pass rate.

Reading the result

A Break-to-Verify run ends in one of three states.

Result	Meaning	Effect on merge
Passed	Green, then red on an assertion, then green: the test catches the fault	No extra step
Weak	The test stayed green while the fault was on	Merge requires an explicit acceptance note
Failed	A run could not complete, for example because the environment was unhealthy	Merge requires an explicit acceptance note; rerun is recommended

A weak result is a signal, not an automatic block. It is often expected: a fault in the cancellation window rule should not affect a test that only checks the order list. The platform does not hide the result; it asks a human to say so explicitly.

The acceptance note is stored as the reason for the merge decision and recorded in the audit log with the person and time. While Break-to-Verify is running, merge is refused.

Example: STORY-142

In the fictional Acme Retail web shop, the test environment provides a fault switch for the cancellation rule. When it is on, the application lets a customer cancel an order that has already shipped.

• The test “Shipped order shows Request return instead of Cancel” runs green on the normal app. With the fault on, the Cancel button appears for a shipped order and the assertion fails. With the fault off, it is green again. The result is passed.
• The test “Order list shows the latest order first” also runs green on the normal app. With the fault on, it is still green, because the order list does not depend on the cancellation rule. The result is weak. The engineer reads the result, writes “Not related to the cancellation rule; list ordering is covered elsewhere” and merges.

Both decisions are visible later. The first pull request records a passed Break-to-Verify. The second records a weak result together with the engineer’s note.

What it proves and what it does not

Break-to-Verify gives a specific kind of evidence. It is important to read it precisely.

It proves that:

• The test is sensitive to the specific fault that was switched on.
• The failure comes from an assertion, not from a crash or a timeout.
• The test returns to green when the fault is removed, so the red run was not caused by a broken environment.

It does not prove that:

• The test catches every fault in the feature. It covers the faults that were prepared, nothing more.
• The test suite as a whole has good coverage. Break-to-Verify works test by test.
• The application is free of defects.

Treat a passed result as “this test can fail for the right reason”. It complements code review and the step mapping; it does not replace them.

Requirements in your test environment

Break-to-Verify needs a way to switch a meaningful fault on and off in the test environment. The platform does not modify your application code to create faults.

In practice this means a small test hook or feature flag in the test environment that:

• Changes one business rule in a known way, such as accepting a cancellation that should be refused.
• Can be scoped to a single test user or a single run, so parallel work is not affected.
• Is available only in test environments and protected by a token that only the test runner holds.

Which faults to prepare, and where, is agreed during the pilot together with the engineering team. Tests for parts of the application without a prepared fault will report a weak result. That is expected, and the human acceptance step records why the test was merged anyway.

Related checks after merge

Break-to-Verify is one of several deterministic checks around a generated test.

• Before the pull request: strict TypeScript compilation, linting, the AST guard and repeated runs of the new test. Inconsistent results mark the test as flaky.
• At the merge gate: required checks on the protected branch and a human approval. Merge is also blocked while the branch is behind main.
• After merge: a push run validates main with the full suite. PR, push, deploy and nightly triggers all run the same approved suite without AI.

If a merged test later fails because a test ID changed while the element’s role and name stayed the same, triage proposes a repair PR that changes only the locator value. A failure where the expected result differs from the actual result is never repaired automatically; it is reported as a possible defect.