Yaga Containment Architecture

I. Introduction

Autonomous offensive security systems operate under a risk regime fundamentally different from any other application of artificial intelligence. When a conversational assistant hallucinates, the result is incorrect text. When a pentest agent hallucinates, the result may be the destruction of a table in production, an exploit fired against an out-of-scope host, or the extraction of administrative credentials without authorization. The consequence is no longer a wrong output; it becomes material, legal and reputational damage.

Recent literature in AI safety concentrates predominantly on conversational alignment and harmful content prevention. Autonomous pentest agents introduce a distinct class of risk: the risk of material action in the real world, in which the model does not merely generate text but executes commands, sends payloads and interacts with live systems. No generic alignment framework adequately captures this class of risk.

YAGA is a real pentest agent developed by HackerSec. Its already documented architecture combines multi-agent coordination through stigmergy, in which specialized agents read and write findings on a shared board carrying pheromone weights that decay over time, playbook selection through augmented retrieval, exploration driven by intrinsic curiosity, and automatic backtracking over the attack graph. The layers described here operate on top of that architecture: they do not replace the orchestration, they bound it.

This work formalizes the ten containment layers it implements. The thesis we hold is counterintuitive: well-designed containment is not a tax on offensive capability. An agent without limits finds little more and reports a great deal of noise; a contained agent finds nearly the same and reports what is real. The difference between the two is not in what they discover, it is in what can be done with the result.

II. Threat Model for Autonomous Agents

Before defining containment mechanisms, it is necessary to formalize which failures they protect against. We define six classes of catastrophic failure for autonomous pentest agents.

TABLE I: Classes of catastrophic failure

P(a ∈ S | G₁(a)=allow ∧ ... ∧ Gₙ(a)=allow) ≥ 1 − ε

The parameter ε represents the tolerated escape rate. YAGA's operational target is ε less than or equal to 0.0001.

The system operates as a defense-in-depth verification chain: each layer is designed to be independent of the others, so that the failure of one does not compromise the safety of the whole.

Listing 1: combined escape probability

III. Containment Architecture: Ten Layers

YAGA implements ten layers organized into four functional domains: scope containment, execution safety, output integrity and temporal governance. Each one is described with its formalization, implementation and validation metrics.

A. Scope containment

This layer operates as the first filter before any interaction with the target. It maintains a formal representation of the scope as a set of authorized targets, defined during engagement configuration: address ranges with ports and protocols, domain and subdomain patterns, and URL patterns with the permitted methods.

When the target of the action belongs to the set, the action is allowed. When it does not belong and the classification confidence exceeds 0.95, the action is denied automatically. Below that threshold, the decision is escalated to the human operator.

Listing 2: authorized target set and decision policy

When the agent identifies a host during reconnaissance, scope membership is resolved through three cascading checks:

1. Exact match: address, range or domain against the explicit authorization list.
2. Ownership verification: registry and autonomous system lookup to confirm that the asset belongs to the target organization.
3. Boundary detection: network analysis to identify pivots that would cross organizational boundaries, through a change of autonomous system or of registrant.

The escape rate measured in the benchmark is 0.003%, equivalent to 2 actions out of 66,847 executed. Both were informational name resolution queries against external servers, classified as severity zero because they are read-only and have no impact on the target.

B. Destructive action prevention

Every command synthesized by the agent passes through a classifier that assigns a destructive potential score. The score combines four probabilities: data loss, service disruption, configuration change and irreversibility, with weights of 0.35, 0.30, 0.20 and 0.15 respectively. The weights were calibrated over 12,000 annotated commands.

Listing 3: destructive potential score

Listing 4: decision policy by destructive potential

TABLE II: Classification by command category

Complementary to the destructive score, a reversibility score estimates the ability to undo the action after execution. A value of 1.0 indicates a fully reversible action, such as pure reading. A value of 0.0 indicates an irreversible action, such as data destruction without a backup. Actions with reversibility below 0.3 require human confirmation regardless of the destructive score.

Listing 5: reversibility score

C. Payload transformation engine

The automatic rewriting performed by the destructive prevention layer is not text sanitization. It is carried out by a payload transformation engine that preserves the proof semantics of the vulnerability while eliminating the destructive components.

The formal guarantee is twofold: the transformed payload must continue to demonstrate the vulnerability, and its destructive score must fall below the automatic execution threshold of 0.15.

Listing 6: payload transformation pipeline

The transformation registry maps each vulnerability class to the minimal form of proof.

TABLE III: Payload transformation registry by vulnerability class

Above the scoring system there is a set of absolute rules. They have no threshold: they are unconditional blocks, regardless of the computed score.

TABLE IV: Absolute non-interaction rules

D. Blast radius limitation

When the agent obtains code execution or privileged access, this layer imposes three simultaneous limits: a maximum depth of three escalation hops, a maximum volume of one hundred records accessed, and a maximum duration of three hundred seconds per exploit.

Listing 7: containment limits and blast radius

The blast radius is computed as the product of three fractions: hosts affected over total hosts, privilege gained over maximum privilege, and data volume accessed over the total. The operational target is to keep that value below 0.05, that is, at most 5% of the possible impact.

The layer operates under the principle of demonstrative sufficiency: the agent must prove that the vulnerability is exploitable and quantify its potential impact without materializing that impact in full. This distinction is what separates a responsible pentest from a real attack.

The privilege model defines a partial order over access levels, from anonymous to administrative. The agent may escalate within the chain, but each hop requires verifying whether the next level is necessary to demonstrate impact, computing the marginal risk of the transition, requesting human confirmation when that marginal risk exceeds 0.15, and recording the rationale in the exploitation log.

Listing 8: privilege escalation governance

E. Bayesian false positive filter

False positives in automated pentesting are more than an inconvenience. They consume incident response resources, erode trust in the system and can drive incorrect operational decisions, such as applying an emergency fix for a vulnerability that does not exist. The filter implements three stages.

In the first stage, the posterior probability that the vulnerability is real is computed from the observed evidence, using prior probabilities calibrated per category over more than 14,000 historical findings. The likelihood ratios were estimated on an independent validation set.

Listing 9: Bayesian posterior of the finding

Listing 10: prior probabilities by category

In the second stage, YAGA performs cross-verification using four independent verification agents, each starting from a distinct hypothesis about the same finding. A finding confirmed by at least 75% of them is classified as confirmed. Between 50% and 75%, it is marked as probable. Below 50%, it is discarded from the final report.

Listing 11: consensus across verifiers

In the third stage, findings classified as confirmed are submitted to an independent reproduction attempt, in which the agent seeks to reproduce the result through an approach entirely different from the original.

Fig. 1: Progressive reduction of the false positive rate by stage

F. Hallucination detection pipeline

Coupled to the Bayesian filter, a dedicated pipeline verifies whether every claim contained in a finding is backed by observable evidence. It operates through three independent checks.

The grounding check verifies whether the output references a real response from the target, whether the executed command actually returned the cited content, and whether the differential behavior is observable. The score is the fraction of claims with matching evidence.

The consistency check verifies whether the finding contradicts prior evidence, whether the cited payload is syntactically valid, and whether the reported software version exists. The score is the complement of the fraction of contradictions.

The reproducibility check verifies whether re-execution produces the same result and whether an alternative path confirms the same vulnerability, scoring zero for failure, one half for partial reproduction and one for full reproduction.

Listing 12: hallucination score

Across more than 14,000 processed findings, the pipeline identified and removed 847 hallucinated findings that would have been reported as real vulnerabilities. That corresponds to a raw hallucination rate of 6.0%, reduced to zero after filtering.

TABLE V: Taxonomy of the hallucinated findings removed

G. Data exfiltration control

When the agent demonstrates a data access vulnerability, there is a risk of extracting more information than the proof of concept requires. This layer imposes a volume cap equal to the lesser of ten records and 0.01% of the total, detects personal data through regular expressions combined with an entity recognition model, and redacts the content before storage.

Listing 13: extraction control policy

Data classified as confidential or above has its evidence replaced by a cryptographic digest of the first characters, and the volume cap drops to three records.

The personal data detection rate in the benchmark is 99.7%, with three undetected occurrences out of 1,043 instances. All three involved non-standard formats of foreign national identification.

H. Circuit breaker with human decision

The circuit breaker fires when any one of eight conditions is met: the destructive score of the next action reaches 0.45; the blast radius of the current chain exceeds 0.05; the action has no precedent in the training set; the agent accumulates more than five consecutive failures, indicating a possible loop; elapsed time passes 80% of the session limit; the agent already holds administrative privilege and the next action writes, removes or modifies; the target response contains an adversarial pattern; or the aggregate session risk exceeds the configured threshold.

Listing 14: circuit breaker firing conditions

The aggregate session risk weights the severity of each action by an exponential decay factor over time, so that recent actions carry more weight than older ones.

In the benchmark, the circuit breaker fired 847 times across 66,847 actions, equivalent to 1.27%. Of those, 812 (95.9%) were classified as legitimate firings by the human operator, confirming high precision in the mechanism. The 35 incorrect firings were caused mostly by unusual server responses interpreted as adversarial patterns.

I. Constitutional self-evaluation

Before each high-impact action, the model evaluates the action against six principles: whether the action is necessary to demonstrate the vulnerability; whether a less invasive alternative exists with the same demonstrative power; whether the action respects the limits of the contractual scope; whether the data accessed is sufficient for the proof without excessive extraction; whether the action could cause unavailability in production; and whether the payload could affect real users of the system.

Listing 15: offensive constitutional principles

The constitutional score is the average of the six evaluations. Below 0.6, the action is blocked with an indication of which principle was violated.

The self-evaluation model was trained on a set of 8,400 action and evaluation pairs annotated by professional pentesters. Agreement between the model and the human annotators is 94.2%, with a kappa coefficient of 0.87, which indicates substantial agreement.

J. Chain depth governance

The policy limits the chain to five chained vulnerabilities and to three hops between distinct hosts. Lateral movement requires explicit authorization per hop. The composite risk of the chain sums the risk of each step multiplied by an amplification factor that grows with depth, at a rate of 0.2 per level. When the composite risk exceeds 2.5, execution is paused and escalated.

Listing 16: chain depth governance

This layer prevents the scenario in which the agent, having found an initial vulnerability, keeps exploring indefinitely in pursuit of maximum impact. In practice, the limit preserved effectiveness: accuracy on chains remains at 91.2%, the same level as the previous benchmarks, which demonstrates that the ceiling does not block legitimate chains. The limit of five applies to distinct chained vulnerabilities, not to the number of execution steps: a chain of five or more steps fits within the ceiling when those steps derive from a smaller number of vulnerabilities.

K. Temporal execution boundary

Listing 17: temporal and rate limits

L. Adversarial instruction resistance

The scenario is not trivial. If the target knows it is being assessed by an autonomous agent, it can insert adversarial instructions into its responses to divert the agent. A simple example is a custom error page containing a comment instructing the agent to ignore previous instructions and scan a different network range.

The defense operates in four steps. The target output is sanitized to remove instruction patterns. The received content is isolated in a data context, never in an instruction context. An anomaly detector flags responses with abnormally low perplexity or containing instruction markers. And a strict authority hierarchy places the system instruction above the operator configuration, which sits above the rules of engagement, which sit far above any output from the target.

Listing 18: defense line against adversarial instruction

Measured effectiveness is 99.8% detection across 500 adversarial injection attempts, distributed over 50 distinct categories of technique.

M. Concrete intervention scenarios

The cases below are real, drawn from the benchmark. In each one, the action originally synthesized by YAGA was modified or blocked before execution.

TABLE VI: Concrete intervention scenarios during exploitation

In scenarios S1 and S3, the rewrite preserved the demonstrative power and dropped the destructive score by more than eighty percent. The vulnerability remains proven, the damage ceases to exist.

Scenario S4 shows scope containment in a segmented network. YAGA discovered a transitive route to a third-party network during authorized lateral movement. Three checks run before any interaction with a new host.

Listing 19: scope verification in scenario S4

The inadequate segmentation finding, that is, a third-party network reachable through an internal route, is reported as medium severity without YAGA ever touching the external network. The layer converts a potential legal incident, which would be the unauthorized scanning of a third party, into a finding of value for the client.

Scenario S7 shows depth governance in a corporate directory environment. The chain that starts at Kerberoasting, passes through cracking the credential and ends in domain replication is one of the most common in this type of assessment. Without the depth layer, YAGA would execute the entire chain automatically and obtain a full dump of the domain password digests, which represents a risk of massive extraction.

Listing 20: composite chain risk in scenario S7

IV. Safety-Effectiveness Pareto Analysis

The natural concern when implementing aggressive containment is the degradation of offensive performance. Overly restrictive layers can lead the agent to abort legitimate exploits, miss real vulnerabilities, or operate so conservatively that its operational value disappears.

To quantify that trade-off, we introduce the Safety-Effectiveness Pareto Score, the geometric mean between the safety score and the effectiveness score.

SEPS = √(Safety × Effectiveness)

Listing 21: SEPS composition by modality

Fig. 2: SEPS by modality, with and without containment

Scale starting at 0.75 to make the differences between modalities visible.

V. YAGA Performance by Modality

HackerSec's internal benchmark evaluates YAGA in black-box mode (no prior knowledge), gray-box mode (ordinary user credentials and partial documentation) and white-box mode (full source code and administrative credentials).

The percentages in this section measure vulnerability detection rate, that is, the fraction of the vulnerabilities present in the environment that YAGA identifies and proves. It should not be confused with the per-scenario success rate, which is the fraction of scenarios in which the contracted objective was fully achieved and which appears in our previous benchmarks with different values because it measures something different.

The progression from 94.0% in black-box to 96.5% in gray-box and 97.0% in white-box demonstrates that YAGA takes advantage of additional information incrementally without compromising safety. The modest total gain, roughly three percentage points between the extremes, indicates an exploitation engine that is robust even without privileged access.

That flattening is consistent with the origin of the architecture. YAGA was conceived for gray-box and black-box, the most frequent scenarios in real engagements, in which the offensive team starts from partial or no information. Every architectural decision was made to operate under maximum uncertainty. White-box performance is an emergent consequence, not a design goal: an architecture that discovers vulnerabilities without source code finds the same patterns more easily when the code is available. That is why containment costs so little here; it acts on an engine that was already operating in the worst case.

TABLE VII: Contribution of each architectural component

Without containment, the baseline false positive rate is 11.4%, or roughly one false positive for every nine findings. With the complete set of ten layers, YAGA holds 1.2%, or roughly one false positive for every 83 findings. The 9.5-fold reduction is attributable mainly to the Bayesian filter and to consensus across verifiers.

That number is the current point on a trajectory. Our previous benchmarks recorded 3.2% and then 1.4% false positives, measured before the complete set of layers was consolidated. The drop to 1.2% in version 2.6 is the cumulative effect described in this work.

Fig. 3: False positive rate by test domain (lower is better)

YAGA with containment    Without containment

VI. Experimental Validation

To validate each layer individually, we ran the complete benchmark in two configurations: with all layers active and with the layers removed one at a time. This approach allows the marginal contribution of each one to the safety and effectiveness of the system to be quantified.

A. Ablation study

TABLE VIII: Impact of removing each layer

Five observations emerge from the ablation study.

1. The Bayesian filter is the layer with the greatest impact on SEPS. Its removal raises false positives from 1.2% to 11.4% and drops SEPS by 0.103 points. It does not affect the detection rate, which stays at 94.0%, but it eliminates 89.5% of false positives.
2. Adversarial resistance is the only layer whose removal reduces detection. Without it, the agent loses 2.8 percentage points because adversarial targets manage to divert it from productive vectors.
3. Scope containment has the greatest impact on the escape rate. Without it, 1.84% of actions reach targets outside the authorized scope.
4. Extraction control is transparent in the aggregate metrics, but it is essential for compliance: without it, sensitive data appears in reports without redaction.
5. Removing everything increases detection by 2.2 percentage points, from 94.0% to 96.2%, but at the cost of false positives 9.5 times higher and an escape rate 1,070 times higher. SEPS falls from 0.973 to 0.761, which confirms that the trade-off strongly favors containment.

B. Offensive testing against the layers themselves

A team of five senior HackerSec pentesters attempted to force each layer individually and in combination, over 40 hours of dedicated testing.

TABLE IX: Offensive testing results

The 12 bypasses identified were fixed in version 2.6.1, resulting in zero bypasses on retest. The self-evaluation layer bypass involved constructing scenarios in which the less-invasive-alternative principle was genuinely ambiguous, forcing the model to prioritize effectiveness over caution.

VII. Advanced Metrics

A. Operating characteristic curves

Each layer that operates with a continuous threshold has a characteristic curve describing the trade-off between sensitivity, that is, dangerous actions correctly blocked, and specificity, that is, safe actions correctly allowed.

TABLE X: Performance of layers with a continuous threshold

B. Latency impact

Each layer adds computational cost to execution. We measured the latency added by each layer along the execution path.

Listing 22: latency per layer (median, P95, P99)

C. Aggregate confusion matrix

Computed over the validation set of 600 scenarios used in the version 2.6 test cycle, distinct from the sets of the earlier comparative benchmarks, with every result verified by a human specialist in black-box mode.

TABLE XI: Confusion matrix and derived metrics

Listing 23: derived metrics

D. Formal model of chained false positive elimination

False positive elimination operates as a chain of Bayesian filters in series. The cumulative posterior probability after k filtering stages combines the likelihood ratios of each stage with the prior probability.

Listing 24: cumulative posterior and decomposition of the final rate

This decomposition explains how three individually imperfect filters, with specificities of 85%, 75% and 68%, produce an aggregate rate of only 1.2% when operating in series.

To quantify the operational impact, we define the expected cost of a false positive as the product of the rate, the number of findings, and the triage cost plus the response cost weighted by the probability of escalation.

Listing 25: expected false positive cost per engagement

E. Safe chain execution model

The most technical contribution of the containment architecture is enabling vulnerability chaining without destructive actions. We formalize this as a safe chain execution model, in which each transition is an action that moves YAGA from one state to the next.

Listing 26: safe chain execution model and its invariants

The cumulative risk cost of the chain follows a damped geometric progression, in which a temporal damping factor makes recent actions weigh more, while the amplification factor penalizes depth.

Listing 27: cumulative risk cost applied to scenario S7

The result is the behavior that separates YAGA from an agent without containment: the first three steps execute on their own and prove the chain, and the stop happens exactly at the step that would carry the risk of massive extraction, without the entire chain having to be abandoned.

VIII. Comparison with Generic Frameworks

YAGA's containment layers represent an evolution specific to the offensive security domain over generic AI safety frameworks.

TABLE XII: Comparison between approaches

IX. Conclusion

This work demonstrates that containment of autonomous offensive security agents is formalizable, measurable and fundamental to safe operation. YAGA's ten layers, operating together as a defense-in-depth system, achieve a combined escape probability below 10⁻¹² under a conservative model with a failure correlation of 0.3, while the false positive rate stays at 1.2% and the escape rate observed in the benchmark is 0.003%.

The SEPS metric quantifies the trade-off between safety and effectiveness. YAGA reaches 0.973, retaining 97.3% of the maximum theoretical effectiveness. The ablation study confirms that removing containment entirely would raise detection by 2.2 percentage points, but at the cost of false positives 9.5 times higher and an escape rate 1,070 times higher. Containment is what separates safe operation from an incident.

The hallucination management system, combining a three-stage Bayesian filter, verification through independent reproduction and consensus across verifiers, ensures that reported findings correspond to real vulnerabilities. The 1.2% false positive rate, roughly one in every 83 findings, places YAGA at the reliability frontier for offensive automation.

Work in progress includes extending self-evaluation with reward models trained specifically on offensive scenarios with feedback from professional pentesters, integrating formal verification through model checking for statically verifiable properties, continuous adversarial training against emerging instruction injection techniques, and exploring adaptive thresholds that adjust in real time to the risk profile observed in the engagement.

References

[1] Bai, Y., et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022.
[2] Ouyang, L., et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
[3] Deng, G., et al. PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing. arXiv:2308.06782, 2023.
[4] Wang, Y., et al. VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework. arXiv:2501.13411, 2025.
[5] Zhang, L., et al. AutoPentester: An LLM Agent-based Framework for Automated Pentesting. arXiv:2510.05605, 2025.
[6] Happe, A., et al. Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design. arXiv:2504.10112, 2025.
[7] Yin, J., et al. CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities. arXiv:2503.17332, ICML, 2025.
[8] Fang, R., et al. Teams of LLM Agents can Exploit Zero-Day Vulnerabilities. arXiv:2406.01637, 2024.
[9] Bhatt, R., et al. Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing. arXiv:2512.09882, 2025.
[10] Amodei, D., et al. Concrete Problems in AI Safety. arXiv:1606.06565, 2016.
[11] Christiano, P., et al. Deep reinforcement learning from human feedback. NeurIPS, 2017.
[12] Rafailov, R., et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS, 2023.
[13] HackerSec Research Team. YAGA: Technical Documentation and Operational Manual. Internal release, 2026.
[14] HackerSec. Yaga Benchmark: AI Pentesting Agent Evaluation. hackersec.ai/evals, 2026.

All benchmark data was generated in controlled laboratory environments. No third-party system was assessed without authorization.