VALIDATION / ACCEPTANCE CRITERIA

Start with measurement.
Finish with evidence.

If it cannot be measured, it does not pass. We establish a baseline in the customer environment and compute results with criteria agreed in advance.

01 / SUCCESS CRITERIA

Success criteria for automated action

An automated action is recorded as successful only when it meets the criteria below; otherwise it is reverted automatically.

≥ 40% reductionBacklog cleared automatically

Backlog falls by 40% or more within 10 minutes of the action

Within 5 minReplication anomaly resolved

Replication mismatch confirmed cleared within 5 minutes of a node restart

Within 15 minRisk state stabilized

Sustained descent from critical to warning or below within 15 minutes

≥ 85%Anomaly detection precision

Share of detection alerts that correspond to a real anomaly

Threshold values are tuned to the customer environment at adoption. The figures shown illustrate a validation design and are not guaranteed outcomes.

02 / SCENARIOS

Validation requires fixed scenarios too

Normal: The intraday curve from average to peak is reproduced continuously
Surge: A 5–10× ingress spike is injected for 10 to 30 minutes
Throughput drop: Consumer throughput is cut by 40–50% and held for 30 minutes
Chaos injection: Fault-injection scenarios run, then the resilience score is measured

03 / SIX GATES

Six gates, from scope to evidence

G0SCOPE

Lock the environment and measurement scope

Define clusters, topics, consumer groups, intervals and metrics first. Synthetic and live data are separated, while tenant boundaries and access controls are verified.

Environment label · source · period · access boundary

G1BASELINE

Establish an observe-mode baseline

Measure lag, backlog time, rates, incident frequency and recovery without automated action. Comparable workload cycles and peak periods are required.

Sample size · normal/peak windows · missing data · outliers

G2INTELLIGENCE

Validate prediction and detection quality

Time-ordered holdout data measures forecast error and detection quality. MAE, RMSE, MAPE, precision, recall, F1, lead time and model drift are reviewed together.

Forecast error · classification · lead time · drift

G3POLICY

Validate policy and approval safety

Allow, deny and approval conditions, role permissions, duplicate blocking, cooldowns and hourly attempt limits are tested—including fail-closed behavior when policy is unavailable.

Fail-closed · RBAC · HITL · duplicate blocking

G4ACTION

Validate action effect and rollback

Before-and-after metrics use the same measurement rule. Success rate, lag reduction, recovery time, retries, blast radius and rollback must satisfy the agreed criteria.

Before/after · retry · rollback · blast radius

G5EVIDENCE

Validate evidence completeness and reproducibility

Every transition and decision must form a chronological record. SHA-256 values, audit events, report figures and raw measurements are cross-checked.

Audit timeline · hash · raw data · report consistency

04 / COVERAGE

A pass must be reproducible

Unit and contract

Check calculations, API contracts, model-to-engine mappings and policy input/output consistency.

Integration and E2E

Verify the live transition from collection through approval, action, verification and reporting.

Performance and concurrency

Measure p50, p95 and p99 latency, throughput, error rate and service health under concurrency.

Security and isolation

Verify authentication, role permissions, tenant isolation, sensitive-data removal and security headers.

Long-run and recovery

Verify sustained operation, restart recovery, data loss, retry, rollback and terminal state.

Evidence reconciliation

Cross-check verdicts, raw output, screenshots and hashes to block unsupported passes.

05 / VERDICT MODEL

Only three verdicts

PASS: All mandatory criteria met with measured evidence
CONDITIONAL: Approved with bounded conditions and explicit follow-up
FAIL: Blocked on a missed target, missing evidence or environment mismatch

06 / IMPLEMENTATION

The capabilities these gates apply to

The six gates apply to the capabilities below. Each is implemented as an independently deployed service, and fixed layer interfaces allow any one of them to be replaced.

Capability	Scope
Incident management	Approve · reject · assign owner · severity classification
Autonomous operations loop	MAPE-K · automated backlog action · rebalance detection
12 AI/ML endpoints	Forecasting · anomaly detection · causal inference · reinforcement learning
AI chatbot with RAG	SSE streaming · multi-LLM · hybrid retrieval
Complete security set	MFA · RBAC · audit hash chain · SIEM · UEBA
Observability stack	Prometheus · OpenTelemetry · Loki · Grafana
Operations management	SLA · on-call · post-mortem · compliance
Chaos engineering	Fault injection · resilience score · scheduled runs
SOAR workflows	DAG automation · YAML-defined workflows
Marketplace	Plugin catalog · one-click install · version management
Autonomous agents	Triage · post-mortem · on-call · weekly review
Deployment and infrastructure	Helm · Docker Compose · CI/CD pipeline

The Kafka driver is one target system implemented on this structure; others are added through the same four methods.

OPERATING LOOP

The operating loop behind these criteria.

View how it works Request an assessment

Start with measurement.Finish with evidence.