AI Probe family · Governed AI test engineering

AI drafts the tests. Nothing merges without your experts.

AI drafts test cases and Playwright code from your Jira stories, in your own repo. Every AI step is approved, every test is checked against a deliberately broken app, and no AI runs in CI.

Illustrative flow for an example story, STORY-142 Order cancellation
Illustrative
  1. AI reviewed the requirement and raised 3 findings, each with evidence. Done.
  2. The analyst and product owner accepted 2 findings and rejected 1. Done.
  3. AI drafted 8 test cases; experts approved 7, which were sent to Xray. Done.
  4. AI drafted Playwright code from your templates and opened pull request 27. Done.
  5. The system broke the app on purpose and the new test failed as expected. Done.
  6. An engineer reviewed and approved the merge. Done.
  7. Nightly CI ran 42 tests, all passed, with no AI calls. Done.
  8. AI triaged a failure as a changed locator and drafted a repair pull request, which awaits approval. In progress.
AI Human System 6 stages · 8 steps 1 awaiting approval · audit chain

What holds on every run

  • A test that misses a seeded fault can't merge without explicit sign‑off
  • Plain Playwright, your repo
  • 0 AI calls in CI
  • Deploys in your environment. You control what reaches the model.

Model calls go to Anthropic's API today; an in-tenant model path is on the roadmap.

Works with:
  • Jira
  • Xray
  • GitHub
  • GitHub Actions
  • Playwright
  • MCP clients

The problem

Test automation is under pressure from both sides

Stories arrive faster than tests. AI can help, but only if its output never skips an expert.

  • AI output shouldn't become unreviewed tests

    Generated test cases and scripts look plausible. Without an expert gate, wrong assumptions quietly become part of your regression suite.

  • The automation backlog can't keep pace

    Every sprint adds stories that need tests. Writing cases and Playwright code by hand leaves automation trailing behind delivery.

  • Locator upkeep drains the team

    UI changes break selectors all the time. Silent self-healing hides real regressions instead of fixing tests in review.

How it works

From story to stable test, in six stages

Every stage shows who acts: AI drafts, your experts decide, the system runs and records.

  1. Requirement

    Pull the Jira story, or paste free text. Your analyst picks what goes in.

    SystemHuman
  2. Requirement review

    AI flags gaps and code/data inconsistencies, each with evidence. Analyst and PO decide.

    AIHuman
  3. Test cases

    AI drafts test cases. Only approved cases go to Xray or Jira.

    AIHuman
  4. Code

    AI fills your templates and page objects. A test that misses a seeded fault can't merge without explicit sign-off.

    AISystemHuman
  5. CI

    Plain Playwright runs in your pipeline. Deterministic, with no AI calls.

    System
  6. Triage

    Rules first, then AI. Failures become a repair PR or a bug draft for review.

    AIHuman

See the full flow

How we start

Bring one Jira story. See reviewed test cases and a draft Playwright PR.

Prerequisites: a page object library and test IDs. If they are missing, setting them up is the first pilot step.

  1. 30-min fit call

    We look at your stack, your stories and your constraints, and say plainly whether a pilot makes sense.

  2. Pilot on one application

    One application and a few real stories, in your environment, with your analysts and engineers in the loop.

  3. Decide on measured results

    The same stories are handled with and without AI, and your analysts time both. You decide on that.

Pilot details

Signature features

Proof and maintenance you can review

Break-to-VerifyDemonstrated

A test that misses a seeded fault can't merge without explicit sign‑off

Before a generated test can merge, the platform breaks the app on purpose and runs the test against both builds. A test that stays green on the broken build is flagged, and merging it takes explicit, logged sign-off. Similar to mutation testing, applied to every pull request.

  • The model does not write TypeScript; it fills your templates and page objects.
  • The check runs in the pull request, next to your usual review.
  • An engineer approves the merge. Always.
PR #27 · checksIllustrative
  1. System Build with the change broken on purposeTest failed, as expected
  2. System Build with the real changeTest passed
  3. Human MergeWaiting for engineer review

Repair PRs + triageDemonstrated

Maintenance through reviewed pull requests, not silent healing

When a nightly run fails, rules classify the failure first, then AI. A changed locator becomes a repair PR; a real defect becomes a bug draft that is checked for duplicates first.

  • A repair PR changes only the testId. Nothing else.
  • Bug drafts are searched against existing issues before anyone sees them.
  • Your team reviews both. Nothing is applied silently.
Nightly CI · 1 failureIllustrative
SystemRule match: locator changed
AIRepair PR #31 · 1 file · testId only
- page.getByTestId('cancel-btn')
+ page.getByTestId('order-cancel')
HumanAwaiting review
AIOther failure → duplicate search → bug draft for review

Approach

Built for control, not for a demo

How the approach differs from common ways of adding AI to test automation.

How Testing Probe differs from common approaches
Topic Testing Probe Common approach
Test maintenance Testing ProbeRepair PR that your team reviews Common approachSilent self-healing at run time
Test format Testing ProbePlain Playwright in your repo Common approachClosed, tool-specific format
Ownership Testing ProbeCode and test cases are yours Common approachScripts stay with the vendor
Deployment Testing ProbeRuns in your environment; you decide what reaches the model Common approachSaaS only
CI Testing ProbeZero-AI CI: deterministic runs Common approachAI in the CI loop

Trust is part of the product

  • Approval Gates

    Every AI step waits for a named expert. Rejected drafts never become test assets.

  • Evidence-backed findings

    Each requirement finding points to the evidence behind it, so reviewers can check instead of trust.

  • Hash-chained audit trail

    Decisions are recorded in a tamper-evident chain: who approved what, and when.

  • Control Tower

    One view of stories, approvals, pull requests, runs and repairs across the flow.

Security details

Start with one story.

In a 30-minute call we look at your stack and a few of your stories, and tell you honestly whether a pilot fits.