Black-box testing · over HTTP
Black-box testing for AI support agents, from the outside in.
Novera sends a versioned scenario suite to your agent's HTTP endpoint, checks every reply with rules, your own system of record and two independent models, and seals the result into a report a person can read and a pipeline can act on.
No card. About ten minutes to a first report.
- 49
- scenarios in every run of eu-support v7
- 3 stages
- rules, read-back, then models
- 5 exit codes
- pass, fail, incomplete, config, infra
- SHA-256
- seals every report to its evidence
Connect an endpoint, not a codebase
No SDK, no agent framework required. If it answers HTTP, it can be tested.
A request template
The endpoint URL, a JSON body with
{{input}}where the scenario goes, and the path to the reply. An auth header's value is encrypted and only ever decrypted on the server.A receipt before the run
One harmless request goes first and its reply is saved, so you can confirm you are testing the thing you think you are testing.
Limits on every call
30 seconds and 256 KB per reply, the whole body included. Only public addresses are called; a redirect is reported, never followed.
Each scenario on its own channel
Multi-turn scenarios need a
{{history}}slot; injected data needs{{context}}. Without one, the scenario is recorded as not run — never flattened into a different test.
Cheapest sufficient evidence first
Each reply goes through three stages in order. The first that settles it decides, and the report names which one did.
- 01
Rules
Text that must or must not appear, tools allowed, forbidden, required or in order, approval before an action, no retry after a failure, latency.
A rule can fail a scenario and can never pass one. A rule-settled case names no model.
- 02
Read-back
A read-only GET to your own system checks whether the action the agent claimed actually happened. The response body is never stored.
Confirmed, contradicted or unavailable. A contradiction fails the case with no model asked.
- 03
Models
Whatever is still open goes to models at temperature 0, each reading the scenario's expected behaviour and assertions against the reply.
On Novera's keys, two vendors must agree; a third model settles a split.
Two vendors, and a third to settle
A single model re-grading identical replies drifts. A verdict here is the finding of two models from different vendors.
Independence is derived, not assumed. It is read from the stored votes, so a second opinion from the same vendor is reported as exactly that.
Your own key grades on your key alone. Its route is built from the models you named and Novera proved; the report says corroborated within one vendor, or not corroborated.
A deadlock is not a guess. If two models split and a third cannot settle it, the scenario is recorded as disputed and given no verdict.
Every report names its graders. And counts how its verdicts were reached: agreed, settled by a third model, or not corroborated.
Missing evidence is never a pass
A broken integration that turns into a plausible verdict is the failure this design exists to prevent.
Counted separately
A scenario that errored, never ran, or had an action nothing could evidence is its own state — listed, never counted as a pass.
Three coverage figures
Execution, resolution and evidence. Each is left empty, not shown as 0 or 100, when the run carries nothing to compute it from.
INCOMPLETE or WITHHELD
INCOMPLETE: scenarios never ran — run the suite again. WITHHELD: everything ran but the evidence does not support a letter — fix what made it unusable.
Sealed, then read by your pipeline
A report is sealed when its run completes. A run that was stopped or failed is never sealed.
A SHA-256 digest covers the sealed content and chains to the previous report for the same agent.
Four downloads. Markdown, CSV, JSON and JUnit XML. In JUnit, a scenario with no result is an error — never a failure and never a pass.
Start runs from CI. A REST call with an Idempotency-Key starts a run; retrying it returns the same run instead of a second one.
A gate that reads the evidence. Missing evidence exits 2 and stops a release just as a failure does.
Exit codes, read from the counts sealed in the report
- 0
- pass
- 1
- fail
- 2
- incomplete
- 3
- config
- 4
- infra
What Novera is not
A document that overstates itself is worth less — especially to the person you hand it to.
Testing evidence, not certification
This report records Novera's testing of the declared agent, scenarios and configuration at the time shown. It is not a legal opinion, a conformity assessment, a certification or an assurance that any law is met, and Novera is not a notified body.
Not a certification. Novera is evidence of testing.
Model grading is not infallible. The report says how many verdicts agreed, how many a third model settled, and how many could not be corroborated.
One dated run. One suite version against one policy version.
Only the suite. Not your agent's behaviour on every conversation.
Every report carries the same notice and its own scope and limitations. Read the limits in full.
Point it at your agent's endpoint
Three free runs, graded on Novera's keys or your own. No card, and about ten minutes to a first report.
- No card
- Email, Google or GitHub
- Database in Frankfurt (EU)