Architecture

AI Testing Probe runs as a set of containers in your environment. AI calls happen in a single service, generated code runs in an isolated runner with no internet access and no secrets, and every external system sits behind an adapter. This page describes the services, the network boundaries, the model gateway and the data flow. A shorter summary for security reviewers is on the Security page.

Services

The platform is made of a small number of services with separate responsibilities.

Service	Responsibility
Core API	REST API, sessions, live events to the portal, approval records, and the MCP endpoint for IDE clients
Core worker	State machine and job queue. The only service that calls the model, writes to Jira, Xray and Git, and holds model keys and write credentials
Toolchain worker	TypeScript tooling: symbol table of the page object library, tsc, eslint, the AST guard and locator probes. Sole writer of merge gate results. It runs no generated code and has no database credentials
Test runner	Runs generated Playwright tests against the test environment. No internet access and no secrets
Agent browser	The code generation agent’s browser, through Playwright MCP. It can reach only the application under test
Review portal	Web interface where people review, approve and merge
Database	PostgreSQL, the single source of truth for domain data and the audit log

For evaluation and development, simulators can replace Jira and Git, and a local runner can replace CI. They use the same client code as the real adapters.

Network isolation

Services are split across separate networks, and most of them are internal networks with no route to the internet.

• Test runner: can reach the application under test and the toolchain, nothing else. It cannot reach the database, the core API, the agent browser, the merge gate results, any secret or the internet.
• Agent browser: can reach only the application’s web front end. Test-only endpoints of the application are closed to it, and only the core worker can connect to it.
• Merge gate results: stored on a separate volume. Only the toolchain worker writes there. The test runner, which executes generated code, cannot see it and carries no token for it.
• Ports: every port published to the host binds only to 127.0.0.1.

These boundaries are not just configuration. An automated security check verifies them: ports, runner isolation, audit log permissions, secret scanning of the repository, the read-only data role, where model keys and write credentials are, agent browser isolation, merge gate isolation, and the absence of agents in CI images. The check is run after every configuration change.

Model gateway

All model calls go through the model gateway inside the core worker. The core API does not know the model key or the model mode.

The gateway provides a provider-independent contract. Provider-specific code lives only in adapters. Today the working adapter is Anthropic Claude. Azure OpenAI and other providers are on the roadmap.

Every call follows the same rules:

• Structured output. Each call must produce output that matches a JSON schema. Invalid output is rejected and retried with the error a limited number of times, then handed to a human. Free text is never the sole basis for a decision.
• Untrusted input. Jira content, page snapshots, code, data and CI logs are treated as untrusted. They are passed inside labelled data blocks, and instructions inside them are not followed.
• Full record. Each call records the model, version, prompt bundle hash, masked input, output, tokens, cost, channel and person.
• Evaluation before use. A prompt or rule change is not merged until its evaluation set has run against a live model.

REPLAY and live mode

The gateway can run in several modes.

Mode	Behaviour
replay	AI step outputs come from curated records; any configured key is ignored
live	Calls the provider; fails clearly if no key is configured
auto	Live when a key is present, otherwise replay
record	Live, and stores successful outputs as candidate records for human review

In REPLAY, each record declares the input it belongs to, and the gateway verifies the input before returning the output. Every REPLAY output carries a REPLAY badge in the portal. If the input does not match, the result is REPLAY-MISS: no output from another record is substituted, and the item goes to a human. A strict option turns a miss into an error, which is useful for rehearsals.

In code generation, the agent’s browser tools run for real in REPLAY; only the model’s decisions come from the record. If a locator or page differs from the recording, the step is marked as a replay deviation and is not treated as verified.

What data reaches the model

The model receives only what a step needs.

• Requirement review: the story text, acceptance criteria and the product owner’s answers.
• Consistency check: relevant code from the application repository, read with a read-only token, and the results of restricted database queries. Queries run through a dedicated read-only role on allowed tables and columns, as single SELECT statements with row and time limits, with masked results.
• Test case generation: the approved story, acceptance criteria, product owner answers and the application’s test data.
• Code generation: case steps and the agent’s view of the page through its allowlisted browser tools. Screenshots are taken outside the model loop and are not sent to the model.
• Triage and bug drafts: failure details and CI log excerpts, only when no deterministic rule could classify the failure.

Personal-data masking in the model gateway (national ID, IBAN, phone, e-mail) is available and must be switched on whenever customer data is processed; pilots can run on synthetic or non-production data. Customer data is not used for model training or fine-tuning by the platform. For the provider’s own data retention terms, see the provider’s commercial terms.

Audit log and data model

PostgreSQL holds all domain data: requirements, findings, cases, scripts, runs, failures, bug drafts and decisions.

The AI call, tool call and audit event tables are append-only. The application’s database role can insert into them but cannot update or delete audit events. Each audit event stores the hash of the previous event, and its own hash is computed with SHA-256 over the previous hash and the event’s canonical JSON. The chain can be verified from the portal.

The chain head is pinned outside the platform in Jira comments and pull request descriptions. A full reset of an environment first exports the log as JSON Lines with a SHA-256 checksum file, and stops without deleting anything if the export fails.

Run artefacts, such as JUnit XML, Allure and Playwright reports, traces and screenshots, are stored on volumes with retention limits. Reports are served to the portal through short-lived signed links, inside a sandbox.

Deployment today

Today the platform is installed with Docker Compose on a single Linux server inside your environment. Container images are built for both arm64 and amd64.

• No inbound ports are opened beyond the local host; access to the portal is arranged in your network.
• Images are pinned by digest, CI actions by commit SHA, and lockfiles are required. Static analysis runs on every pull request.
• One command brings the environment up and waits until every service is healthy. Another verifies the security boundaries described above.
• A reset exports the audit log before clearing data, and a separate demo reset restores demo data without touching the audit log.

Not available today: deployment on Kubernetes or OpenShift, a model running in your own cloud account or tenant, and single sign-on for the portal. If any of these is a requirement for you, raise it early so that it can be scoped as a pilot condition.