Concepts

AI Testing Probe is built around a few ideas that appear on every page of this documentation. Approval Gates put a human decision after every AI step, evidence-backed findings make AI claims checkable, and the audit log records who decided what. This page defines each term once.

Approval Gates

An Approval Gate is a point in the flow where an AI draft waits for a human decision. Nothing moves to the next stage, and nothing is written to Jira, Xray or the main branch, until the gate is passed.

The gates in the flow are:

• Finding decisions: the analyst decides for each pre-review finding whether the document or the code is wrong.
• Product owner approval: the product owner approves the corrected story in Jira.
• Test case review: the analyst approves, edits or rejects each AI-drafted case before publishing.
• Pull request review and merge: the engineer reviews generated code and merges it through a protected branch.
• Weak-test acceptance: if Break-to-Verify cannot show that a test catches a fault, merging requires an explicit, logged acceptance.
• Bug draft decision: the tester files a new bug, links to an existing one, edits or rejects the draft.
• Repair PR merge: a human merges every locator repair.

Approvals are bound to a content hash. If the content changes after approval, the approval no longer applies and the item must be reviewed again. API tokens can submit drafts, but they cannot approve.

AI, human and system steps

Every step in the flow is performed by one of three actors, and the interface labels each step with an icon and text, not colour alone.

• AI steps produce drafts: pre-review findings, consistency findings, test cases, step-to-page-object mappings, failure classifications that rules could not decide, bug drafts and duplicate rankings.
• Human steps are decisions: approving, editing, rejecting, merging and filing.
• System steps are deterministic: reading Jira, re-verifying evidence, running compilers and linters, the AST guard, test runs, Break-to-Verify, rule-based triage and locator repair proposals.

A useful way to read any screen is to ask which of the three produced each item. AI output is always marked as a draft until a human step changes its state. When a deterministic rule can make a decision, the decision is not left to a model. Triage, for example, applies rules first and calls the model only when no rule matches.

User roles

The platform has five portal roles, enforced by role-based access control.

Role	Main responsibilities
Analyst	Pulls requirements, decides on findings, generates and reviews test cases, publishes them, creates API tokens for the MCP channel
Product owner	Approves the corrected requirement. For Jira stories, approval is given in Jira; for free-text requirements, in the portal
Engineer	Generates code, runs Break-to-Verify, reviews and merges pull requests, opens and merges repair PRs
Tester	Decides on bug drafts: file new, link to existing, edit or reject
Manager	Triggers deploy and nightly runs, watches the Control Tower and reports

Each role sees only the actions it is allowed to take. A person can hold more than one role, but every decision is recorded under the person and role that made it.

Evidence-backed findings

An evidence-backed finding is an AI claim that comes with proof the platform can check without a model. During the consistency check, for example, a finding that the code disagrees with the story must point to a file and line in the repository, or to the result of a read-only database query.

Before a finding is shown, the platform re-verifies its evidence deterministically. If the cited line does not contain what the finding says, or the query does not return the stated value, the finding is dropped and never reaches the analyst.

This changes how people review AI output. Instead of asking “is the model right?”, the analyst looks at the evidence and decides what it means: is the document wrong, or is the code wrong? The same principle applies to bug drafts, whose expected and actual values must appear in the run’s own artefacts, and to duplicate rankings, which may only name issues that the Jira search actually returned.

REPLAY and live mode

The model gateway can run in different modes. In live mode, AI steps call the configured model provider. In REPLAY mode, the output of each AI step comes from a curated record whose input has been verified against a declaration.

REPLAY exists for reliable demonstrations and testing. It follows the same code path as live mode: logging, schema validation, evidence re-verification and human gates are identical. The difference is only where the model output comes from.

The platform never presents REPLAY as live. Every REPLAY output carries a REPLAY badge. If an input does not match the record, the output is marked REPLAY-MISS and nothing is invented in its place. In the code generation agent, the browser steps run for real in the application even in REPLAY; only the model’s decisions come from the record.

Before a prompt bundle is used live, it must pass its evaluation set against a live model. Passing that evaluation does not turn REPLAY output into live output.

Maturity labels

Every capability on this site carries one of three maturity labels, written as text so the meaning does not depend on colour or icons.

• Live-verified: evaluated against a live model and run on real systems (Jira Cloud or Xray, GitHub, GitHub Actions). The evidence and its date are listed on the Capabilities page.
• Demonstrated: works end to end and is shown either with REPLAY outputs or on a live platform. The live model evaluation is not complete.
• Roadmap: not available yet, or only designed. Each item states a quarter or a pilot condition, or says that the date is set by pilot demand.

In this documentation, the label appears in headings where it matters, for example “Selenium migration (Roadmap)”. Roadmap features are described so you know they are planned, but they are never written up as instructions. The Capabilities page is the single source for the current label of each capability.

Audit log

The audit log records every AI call, tool call, approval and rejection. Records are append-only: the application’s database role cannot update or delete audit events.

Each audit event is chained to the previous one with a SHA-256 hash of the previous hash and the event’s canonical JSON. Changing or removing an old event breaks the chain, and the chain can be verified at any time from the portal.

The head of the chain is also pinned outside the platform. When approved cases are published, the Jira comment includes the chain hash; generated pull requests and repair PRs include it in their descriptions. This lets a reviewer match a Jira issue or a pull request to a specific state of the log.

The log can be exported as JSON Lines with a SHA-256 checksum file. A full environment reset exports the log first and stops without deleting anything if the export fails.

Zero-AI CI and Repair PRs

Two more signature terms describe what happens after merge.

Zero-AI CI means the tests that run in your pipeline are approved, deterministic Playwright code. The CI and test images contain no model key, no agent and no model SDK, and an automated security check confirms this. A flaky model cannot turn a build red, because no model runs there.

Repair PRs are how maintenance works. When a test fails because an element’s test ID changed while its role and accessible name stayed the same, a deterministic rule classifies it as test maintenance. The platform proposes a pull request that changes only the test ID value in the page object locators, checked by the AST guard and a full local run. A human reviews and merges it. There is no silent self-healing: every change to test code is a reviewed pull request.

A failure where the expected result differs from the actual result is never repaired. It becomes a defect candidate and goes to a human.