YAGA is an autonomous penetration testing agent developed by HackerSec that uses LLMs (GPT 5.6 Sol, Claude Opus 4.8, Opus 4.7, Sonnet 5, Gemini 3.5 and Grok 4.5) as the cognitive engine within a multi-agent architecture with tactical RAG, stigmergic planning and curiosity-driven exploration. This article evaluates YAGA against those same models used in isolation across 248 scenarios covering four surfaces: infrastructure (148), web (42), REST/GraphQL API (32) and Android/iOS mobile (26).
The article presents the first comparative benchmark of LLMs as YAGA's internal engine, revealing that the difference between the best and worst engine (7.5 pp) is drastically smaller than the difference between any orchestrated model and that same model in isolation (40+ pp).
I. Introduction
There is a widespread belief that a language model, on its own, can carry out a competent penetration test. The narrative is seductive: the analyst describes the target, the model generates the plan, and the rest is mechanical execution. In practice, the gap between generating a command and sustaining a coherent offensive campaign across dozens of chained decisions is enormous, whether in infrastructure, web applications, APIs or mobile.
An LLM operated via prompt has no persistent memory between turns, does not maintain an up-to-date attack graph, does not manage multiple shell sessions or interception proxies, does not track exfiltrated credentials, and does not backtrack when an exploitation chain fails. Each prompt is an isolated question whose context depends entirely on what the human operator decides to include.
YAGA operates as an agent: it maintains state, plans, executes, observes results, replans, and chains vulnerabilities into multi-stage attack paths. The language model is the reasoning engine, not the system. The distinction is fundamental and is the central thesis of this article: the value is in the orchestration, not in the isolated model.
YAGA was conceived and developed in HackerSec labs with a central focus on gray-box and black-box pentesting, the most frequent operational scenarios in real engagements, where the offensive team starts from partial or no information about the target. This origin shaped every architectural decision: stigmergic coordination, automatic backtracking, curiosity-driven exploration and tactical RAG were designed to operate under maximum uncertainty. The white-box competence documented in this benchmark was not a design goal: it is an emergent result. An architecture capable of discovering vulnerabilities without source code discovers the same patterns even more efficiently when the code is available.
Four contributions structure this work. First, a controlled benchmark across 248 scenarios covering infrastructure, web, API and mobile. Second, an analysis of which domains most amplify the orchestration advantage. Third, the first comparative benchmark of LLMs as YAGA's internal engine, showing that the architecture equalizes differences between models. Fourth, an ablation that identifies the contribution of each architectural component.
II. The Problem: Why Isolated LLMs Fail
A penetration test, whether in infrastructure, web, API or mobile, is a sequence of dependent decisions under uncertainty. The pentester discovers a service, identifies a version, selects an exploit, executes it, observes the result, extracts information, and uses that information to deepen the attack. Each decision depends on the result of the previous one.
An LLM operated via chat cannot sustain this accumulation. The human operator has to decide what to include in the next prompt, often omitting details that seem irrelevant but become critical stages later. The model does not persist state between turns; each response is generated with no memory of prior actions beyond what is explicit in the context window.
A. Confabulation in Offensive Security
LLMs confabulate. In security contexts, confabulation takes specific forms: the model claims that a software version is vulnerable to a CVE without checking the patch; suggests that an exploit worked based on the expected output pattern, not the real output; in web, it reports XSS without checking whether the WAF filters the payload; in API, it assumes an endpoint accepts mass assignment without testing. YAGA eliminates confabulation by design: it runs the exploit, observes the real output, and only reports when there is executable evidence.
B. Absence of Backtracking
When an exploitation path fails, a human pentester returns to the last decision point and tries an alternative. An LLM in chat does not do this spontaneously. YAGA implements automatic backtracking on the attack graph: when an exploit fails, the agent marks the edge as "tried/failed" and selects the next pending edge with the highest expected utility.
C. Surface-Specific Limitations
In web, isolated LLMs cannot maintain authenticated sessions, track CSRF tokens across requests, or chain an SSRF with cloud metadata reads. In API, they do not manage call sequences with data dependencies between endpoints (e.g., create resource → extract ID → access admin endpoint with that ID). In mobile, they do not orchestrate decompilation, static analysis of the APK/IPA, traffic interception with a proxy, and backend exploitation simultaneously.
III. YAGA Architecture
YAGA is made up of four components that, together, turn a generic LLM into an effective penetration agent on any attack surface.
A. Multi-Agent Coordination via Stigmergy
Instead of a central orchestrator, YAGA employs stigmergic coordination [1]. Specialized agents (reconnaissance, exploitation, post-exploitation, reporting) read and write findings on a shared blackboard. For web and API, additional agents manage HTTP sessions, traffic interception and parameter fuzzing. For mobile, dedicated agents coordinate static analysis (decompilation, string extraction) with dynamic analysis (hooking, proxy interception).
Each finding carries a pheromone weight that decays exponentially, naturally eliminating obsolete paths:
φ(t) = φ0 · e-λ(t - t0)
B. Tactical RAG for Playbook Selection
The strategist agent maps reconnaissance artifacts to MITRE ATT&CK tactics [2] via a fine-tuned LLM classifier. The RAG indexes playbooks for all surfaces: from Kerberoasting in AD to BOLA in REST APIs, IDOR in web applications and certificate pinning bypass in mobile.
C. Curiosity-Driven Exploration
To overcome sparse rewards, YAGA incorporates intrinsic curiosity via PPO with Random Network Distillation (RND) [3]. The curiosity bonus encourages the agent to explore novel adversarial states:
rcuriosity(st) = ||f̂(st; θ) - f(st)||2
D. Attack Graph and Automatic Backtracking
The agent maintains a real-time graph G = (V, E). In infrastructure, nodes represent hosts and services. In web/API, nodes represent endpoints, parameters and authentication flows. In mobile, nodes represent app components, backend endpoints and communication channels. The coverage metric determines when exploration has reached saturation:
Cstruct = |Et| / (|Et| + |Ep|)
IV. Benchmark Methodology
A. Test Scenarios
The benchmark uses 248 scenarios distributed across four attack surfaces and three modalities (gray-box, black-box, white-box).
TABLE I: Scenario Distribution
| Surface | Gray-Box | Black-Box | White-Box | Total |
|---|---|---|---|---|
| Infrastructure | 48 | 52 | 48 | 148 |
| Web Application | 14 | 15 | 13 | 42 |
| API (REST/GraphQL) | 11 | 11 | 10 | 32 |
| Mobile (Android/iOS) | 8 | 10 | 8 | 26 |
| Total | 81 | 88 | 79 | 248 |
Infrastructure (148 scenarios): corporate networks, Active Directory, Docker/Kubernetes containers, exposed services, cloud environments and multi-domain chains. Web (42): applications with authentication, OWASP Top 10, business logic, SSRF, upload bypass and XSS→RCE chains. API (32): REST and GraphQL with BOLA/BFLA, mass assignment, rate limiting bypass, injection in GraphQL queries and IDOR→data exfiltration chains. Mobile (26): Android (APK) and iOS (IPA) with static analysis, certificate pinning bypass, hooking (Frida), insecure storage and backend exploitation via intercepted traffic.
B. Models Evaluated
Six models were evaluated in the "isolated-LLM" condition: GPT 5.6 Sol (OpenAI), Claude Opus 4.8 (Anthropic), Claude Opus 4.7 (Anthropic), Claude Sonnet 5 (Anthropic), Gemini 3.5 (Google DeepMind) and Grok 4.5 (xAI). Each model receives the same structured prompt with the objective description and access information. For web/API, the operator uses Burp Suite and returns the responses to the prompt. For mobile, the operator runs jadx/apktool and returns the outputs. Time-out: 90 minutes for black-box, 60 for gray-box, 45 for white-box.
C. YAGA Configuration
YAGA was run in fully autonomous mode with Claude Opus 4.8 as the primary engine. For web and API, the agent integrates with an interception proxy for HTTP(S) traffic analysis. For mobile, it coordinates Frida for dynamic hooking and jadx/apktool for static analysis. Human validation verifies each finding. Additionally, YAGA was run with each of the six models as the internal engine to produce the comparative LLM benchmark (Section VIII).
V. Results: Infrastructure
A. Overall Performance
TABLE II: Infrastructure: Success Rate by Modality (%)
| Approach | Gray-Box | Black-Box | White-Box | Chains (3+ steps) | Overall |
|---|---|---|---|---|---|
| YAGA | 93.8 | 87.4 | 96.1 | 91.2 | 92.3 |
| GPT 5.6 Sol (isolated) | 52.7 | 31.2 | 73.1 | 8.3 | 48.6 |
| Opus 4.8 (isolated) | 49.3 | 28.5 | 71.8 | 6.1 | 45.9 |
| Opus 4.7 (isolated) | 47.1 | 26.9 | 70.4 | 5.8 | 44.2 |
| Sonnet 5 (isolated) | 41.8 | 22.1 | 65.2 | 4.2 | 38.7 |
| Gemini 3.5 (isolated) | 50.1 | 29.8 | 68.9 | 7.1 | 46.3 |
| Grok 4.5 (isolated) | 48.8 | 27.4 | 69.7 | 6.8 | 45.1 |
The difference between YAGA and any isolated LLM exceeds 40 percentage points in gray-box and black-box. The most revealing scenario is attack chains: no isolated model exceeds 8.3%, while YAGA reaches 91.2%. In white-box, the gap narrows to 23-31 pp because LLMs reason well over static code.
B. Performance by Domain
TABLE III: Infrastructure: Success by Domain (%, GB + BB)
| Domain | YAGA | GPT 5.6 Sol | Opus 4.8 | Gemini 3.5 | Grok 4.5 | Sonnet 5 |
|---|---|---|---|---|---|---|
| Corporate Network | 89.2 | 35.4 | 32.1 | 33.8 | 33.1 | 24.7 |
| Active Directory | 84.6 | 22.3 | 19.8 | 21.1 | 20.4 | 15.2 |
| Containers | 91.3 | 41.2 | 38.7 | 39.5 | 38.9 | 31.4 |
| Exposed Services | 94.1 | 48.6 | 45.9 | 47.2 | 46.4 | 38.1 |
| Cloud (IAM) | 88.7 | 38.1 | 34.6 | 36.9 | 36.1 | 28.3 |
| Multi-domain Chains | 91.2 | 8.3 | 6.1 | 7.1 | 6.8 | 4.2 |
Active Directory is the domain where isolated LLMs get the worst results (22.3% for GPT 5.6 Sol). Compromising an AD requires 5-8 dependent steps that an isolated prompt cannot sustain. Grok 4.5 shows performance comparable to Gemini 3.5, slightly above isolated Opus 4.8 in containers and exposed services.
VI. Results: Web, API and Mobile
A. Web Applications
TABLE IV: Web Application: Success Rate (%)
| Approach | Gray-Box | Black-Box | White-Box | Web Chains | Overall |
|---|---|---|---|---|---|
| YAGA | 93.2 | 88.7 | 97.1 | 84.6 | 91.2 |
| GPT 5.6 Sol (isolated) | 67.4 | 58.1 | 82.3 | 21.8 | 62.4 |
| Opus 4.8 (isolated) | 64.1 | 54.8 | 80.6 | 18.4 | 59.1 |
| Opus 4.7 (isolated) | 62.8 | 52.3 | 79.1 | 16.9 | 57.4 |
| Sonnet 5 (isolated) | 57.2 | 46.7 | 74.8 | 12.3 | 52.1 |
| Gemini 3.5 (isolated) | 65.8 | 56.4 | 81.2 | 19.7 | 60.8 |
| Grok 4.5 (isolated) | 64.6 | 55.2 | 80.1 | 18.9 | 59.6 |
In web, isolated LLMs perform significantly better than in infrastructure. GPT 5.6 Sol reaches 67.4% in gray-box web versus 52.7% in gray-box infra. Web vulnerabilities (XSS, SQLi, CSRF) are extensively documented in the training data and often require only 1-2 requests to confirm.
However, the gap widens in web chains (21.8% vs. 84.6%). Scenarios such as SSRF→cloud metadata read→RCE, or stored XSS→account takeover→privilege escalation, require the model to maintain sessions, track cookies and chain findings, the same limitations observed in infrastructure.
B. REST and GraphQL APIs
TABLE V: API Security: Success Rate (%)
| Approach | Gray-Box | Black-Box | White-Box | API Chains | Overall |
|---|---|---|---|---|---|
| YAGA | 95.4 | 90.1 | 97.8 | 87.3 | 93.4 |
| GPT 5.6 Sol (isolated) | 71.2 | 61.3 | 84.6 | 24.1 | 65.7 |
| Opus 4.8 (isolated) | 68.4 | 57.8 | 82.1 | 20.7 | 62.3 |
| Opus 4.7 (isolated) | 66.1 | 55.6 | 80.8 | 18.9 | 60.4 |
| Sonnet 5 (isolated) | 60.7 | 49.3 | 76.3 | 14.2 | 54.8 |
| Gemini 3.5 (isolated) | 69.8 | 59.4 | 83.4 | 22.3 | 63.9 |
| Grok 4.5 (isolated) | 68.3 | 58.1 | 82.7 | 21.4 | 62.8 |
APIs produce the best results for isolated LLMs among all surfaces. The reason is structural: APIs have well-defined contracts (OpenAPI/Swagger, GraphQL schemas) that the LLM interprets accurately. GPT 5.6 Sol reaches 71.2% in gray-box API, the highest score across all domains for isolated LLMs.
YAGA reaches 93.4% overall in API, its second-best domain. The agent's advantage shows in BOLA/BFLA (Broken Object Level/Function Level Authorization), where it must create resources, extract IDs, test cross-user access and verify authorization across multiple endpoints. In API chains: a 63.2 pp difference.
C. Mobile (Android and iOS)
TABLE VI: Mobile Security: Success Rate (%)
| Approach | Gray-Box | Black-Box | White-Box | Mobile Chains | Overall |
|---|---|---|---|---|---|
| YAGA | 84.7 | 78.4 | 91.3 | 72.1 | 82.1 |
| GPT 5.6 Sol (isolated) | 39.6 | 28.4 | 62.8 | 6.8 | 37.2 |
| Opus 4.8 (isolated) | 36.2 | 25.1 | 59.4 | 5.1 | 34.1 |
| Opus 4.7 (isolated) | 34.7 | 23.8 | 57.8 | 4.6 | 32.7 |
| Sonnet 5 (isolated) | 29.1 | 19.2 | 52.3 | 3.1 | 28.4 |
| Gemini 3.5 (isolated) | 37.8 | 26.7 | 61.1 | 5.9 | 35.8 |
| Grok 4.5 (isolated) | 36.9 | 25.8 | 60.2 | 5.4 | 34.8 |
Mobile is the surface with the greatest difficulty for isolated LLMs and for YAGA itself. GPT 5.6 Sol reaches only 39.6% in gray-box mobile. Mobile testing requires orchestrating multiple tools: decompilation (jadx, apktool), string and secret analysis, certificate pinning bypass (Frida), traffic interception (mitmproxy) and backend exploitation via discovered APIs.
YAGA reaches 82.1%, its lowest result among the surfaces, but still 44.9 pp above the best isolated LLM. Grok 4.5 positions itself competitively in mobile (34.8%), surpassing isolated Opus 4.7 (32.7%) and coming close to Gemini 3.5 (35.8%).
D. Consolidated View by Surface
Fig. 1: Overall Success Rate by Surface: YAGA vs. Isolated LLMs (%)
TABLE VII: Finding Quality (All Surfaces)
| Metric | YAGA | GPT 5.6 Sol | Opus 4.8 | Gemini 3.5 | Grok 4.5 |
|---|---|---|---|---|---|
| True Positives | 683 | 214 | 193 | 204 | 198 |
| False Positives | 10 | 98 | 107 | 103 | 105 |
| FP Rate (%) | 1.4 | 31.4 | 35.7 | 33.6 | 34.7 |
| Correct Severity (%) | 80.8 | 45.1 | 42.6 | 43.9 | 43.2 |
| Working PoC (%) | 93.2 | 22.4 | 18.8 | 20.7 | 19.6 |
YAGA's false positive rate (1.4%) stays stable across all surfaces because the agent runs each exploit and validates with evidence before reporting. Isolated LLMs keep FP rates of 31-36%.
VII. White-Box: Where the Gap Narrows
The white-box scenario produces the smallest difference between YAGA and isolated LLMs across all surfaces. With source code available, the model identifies vulnerabilities through static analysis.
This result has a direct architectural explanation. YAGA was developed in labs focused on gray-box and black-box: its components were optimized to navigate uncertainty, not to analyze static code. The high white-box performance is an emergent consequence, an architecture trained for the worst case operates with even more precision in the best case. Where isolated LLMs improve because they have more information available, YAGA was already performing with less.
TABLE VIII: White-Box: Breakdown by Type and Surface (%)
| Vulnerability Type | YAGA | GPT 5.6 Sol | Opus 4.8 | Grok 4.5 | Gemini 3.5 |
|---|---|---|---|---|---|
| SQLi / Injection | 97.3 | 82.4 | 79.6 | 78.1 | 76.8 |
| XSS / Client-Side | 96.8 | 84.1 | 81.3 | 80.8 | 82.7 |
| Auth Bypass | 95.8 | 71.3 | 68.9 | 67.4 | 65.2 |
| BOLA/BFLA (API) | 94.2 | 68.7 | 65.4 | 64.8 | 63.8 |
| RCE (code review) | 94.1 | 67.8 | 64.2 | 63.1 | 61.7 |
| Insecure Storage (Mobile) | 92.4 | 64.3 | 61.8 | 60.4 | 59.6 |
| Misconfig (IaC/Cloud) | 97.8 | 78.9 | 76.3 | 75.8 | 74.1 |
| Chains (cross-surface) | 87.1 | 38.4 | 35.9 | 35.1 | 34.2 |
In pattern-based vulnerabilities (SQLi, XSS, Misconfig), isolated LLMs exceed 74%. In white-box web, XSS is the type where isolated LLMs perform best (84.1%) because it is highly standardized. The drop happens in vulnerabilities that require business logic and, more severely, in cross-surface chains (38.4% for GPT 5.6 Sol vs. 87.1% for YAGA).
VIII. YAGA LLM Benchmark
A fundamental question is: if orchestration matters more than the model, what is the difference between using different LLMs as YAGA's internal engine? To answer, we ran the full benchmark (248 scenarios) with each of the six models as the agent's primary engine, keeping the entire architecture (multi-agent, RAG, RND/PPO, backtracking) identical.
TABLE IX: LLMs as YAGA's Engine: Performance by Surface (%)
| LLM Engine | Infra | Web | API | Mobile | Chains (all) | Overall |
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 92.3 | 91.2 | 93.4 | 82.1 | 91.2 | 91.4 |
| GPT 5.6 Sol | 90.1 | 89.8 | 91.7 | 80.3 | 88.4 | 89.6 |
| Grok 4.5 | 89.7 | 89.1 | 91.2 | 79.6 | 87.8 | 89.1 |
| Gemini 3.5 | 89.4 | 88.9 | 90.9 | 79.1 | 87.3 | 88.5 |
| Claude Opus 4.7 | 88.7 | 88.1 | 90.3 | 78.6 | 86.7 | 88.0 |
| Claude Sonnet 5 | 84.2 | 84.3 | 87.1 | 74.8 | 81.2 | 83.9 |
Fig. 2: LLM as YAGA's Engine vs. Direct LLM: Overall (%)
The most revealing result: the difference between the best engine (Opus 4.8, 91.4%) and the worst (Sonnet 5, 83.9%) is only 7.5 percentage points. The difference between using Opus 4.8 in isolation (45.9%) and inside YAGA (91.4%) is 45.5 pp. Orchestration compresses the variance between models while raising the floor from ~39% to ~84%.
Grok 4.5 as YAGA's engine (89.1%) sits between GPT 5.6 Sol (89.6%) and Gemini 3.5 (88.5%), showing that the architecture extracts consistent performance from models across different providers.
Claude Sonnet 5 inside YAGA (83.9%) surpasses any model used in isolation, including isolated GPT 5.6 Sol (48.6%), a significantly larger model. This shows that the architecture is the dominant multiplier: a smaller orchestrated model beats a larger isolated model.
A. Where the Engine Choice Matters
TABLE X: Engine Impact in Hard Scenarios (%)
| LLM Engine | Chains 5+ steps | AD Complex | Mobile BB | API BOLA/BFLA |
|---|---|---|---|---|
| Claude Opus 4.8 | 85.0 | 84.6 | 78.4 | 94.2 |
| GPT 5.6 Sol | 81.2 | 81.3 | 76.1 | 91.8 |
| Grok 4.5 | 80.4 | 80.7 | 75.3 | 91.2 |
| Gemini 3.5 | 79.8 | 80.1 | 74.8 | 90.4 |
| Claude Opus 4.7 | 78.4 | 79.2 | 73.9 | 89.7 |
| Claude Sonnet 5 | 71.3 | 72.8 | 68.2 | 84.6 |
The difference between engines amplifies in high-complexity scenarios. In chains of 5+ steps, Opus 4.8 (85.0%) beats Sonnet 5 (71.3%) by 13.7 pp, nearly double the overall difference (7.5 pp). Grok 4.5 (80.4%) shows solid capability in long chains, surpassing Opus 4.7 (78.4%) and coming close to GPT 5.6 Sol (81.2%).
The practical implication: for simple scenarios (1-3 steps), any model inside YAGA produces comparable results. For complex pentests with long chains, the engine choice has a measurable impact, but even the worst orchestrated engine (71.3%) massively outperforms the best isolated model in long chains.
IX. Ablation: Contribution of Each Component
TABLE XI: Ablation: Impact by Component (%, all surfaces)
| Configuration | Infra | Web | API | Mobile | Chains | Overall |
|---|---|---|---|---|---|---|
| Full YAGA | 92.3 | 91.2 | 93.4 | 82.1 | 91.2 | 91.4 |
| Without Stigmergy | 77.8 | 78.4 | 80.1 | 68.3 | 64.3 | 77.1 |
| Without RAG | 73.6 | 74.8 | 76.2 | 64.1 | 58.7 | 73.1 |
| Without Curiosity (RND) | 83.4 | 84.1 | 86.7 | 74.2 | 74.8 | 82.6 |
| Without Deduplication | 88.7 | 87.8 | 89.4 | 78.6 | 85.4 | 87.3 |
| Single Agent | 61.4 | 66.8 | 69.2 | 51.3 | 32.1 | 60.8 |
| Direct LLM (best) | 48.6 | 62.4 | 65.7 | 37.2 | 8.3 | 50.2 |
The ablation confirms that the multi-agent architecture is the most impactful component across all surfaces. Removing it reduces chains from 91.2% to 32.1% (−59 pp). RAG is the second factor (−18.3 pp overall). Curiosity has a larger impact in mobile (−7.9 pp), where attack paths are less documented and the agent needs to explore more creatively.
The "Single Agent vs. Direct LLM" result reveals that even without multi-agent, the agentic loop adds ~10 pp across all surfaces. The difference is smaller in web (+4.4 pp) and API (+3.5 pp) than in infra (+12.8 pp) and mobile (+14.1 pp), confirming that orchestration adds more value where sequential complexity is higher.
X. Why Orchestration Makes the Difference
The results converge on one explanation: the distance between an LLM and a pentester is not one of reasoning, but of sustained execution. The LLM reasons about security as well as YAGA on any individual turn. The difference emerges when the pentest requires dozens of chained turns with dependencies between them.
The LLM-as-engine benchmark (Section VIII) is the strongest evidence for this thesis. If YAGA's superiority were attributable to the model, swapping the engine would radically change the result. That is not the case: the variation between engines (7.5 pp) is a fraction of the orchestrated-vs-isolated variation (40+ pp). The architecture is the multiplier; the model is the substrate.
The variation between surfaces is equally informative. Isolated LLMs perform better in API (65.7%) and web (62.4%) than in infra (48.6%) and mobile (37.2%). This correlates with typical sequential complexity: API tests often require 1-3 steps, while infra and mobile demand longer chains. Where the task is short and well-structured, the isolated LLM comes close. Where it is long and unstructured, orchestration dominates.
XI. Limitations
First, the scenarios were designed by the YAGA team, introducing construction bias. An independent benchmark with unknown scenarios is the next step.
Second, the "isolated-LLM" setup emulates interactive chat use. Professionals who build pipelines with function calling and managed context would achieve results above the reported baseline.
Third, the mobile scenarios focus on Android with lower iOS coverage, due to the greater ease of instrumenting Android.
Fourth, models evolve quarterly. Future versions may narrow the gap, especially if they incorporate native agentic capabilities. This benchmark reflects the state as of June 2026.
XII. Conclusion
Language models are powerful reasoning engines. They are not pentesters. This conclusion holds across all evaluated surfaces: infrastructure, web, API and mobile, and for all six models tested (GPT 5.6 Sol, Opus 4.8, Opus 4.7, Sonnet 5, Gemini 3.5 and Grok 4.5).
In infrastructure and mobile, YAGA outperforms isolated LLMs by 40 to 56 percentage points. In web and API, where tasks are typically shorter, the gap is smaller (29-31 pp) but still substantial. In attack chains, the gap reaches 83 points.
The LLM-as-engine benchmark shows that the architecture is the dominant multiplier: Sonnet 5 inside YAGA (83.9%) surpasses isolated GPT 5.6 Sol (48.6%). A smaller orchestrated model beats a larger isolated model. The value is not in the model: it is in the orchestration that surrounds it.
References
[1] E. Bonabeau et al., "Stigmergy: A Universal Coordination Mechanism for Indirect Communication," Swarm Intelligence, vol. 13, 1999.
[2] MITRE, "ATT&CK Framework v15," https://attack.mitre.org/, 2025.
[3] Y. Burda et al., "Exploration by Random Network Distillation," ICLR, 2019.
[4] D. Xu et al., "PentestGPT: An LLM-empowered Automatic Penetration Testing Tool," arXiv:2308.06782, 2023.
[5] Z. Chen et al., "AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks," arXiv:2403.01038, 2024.
[6] A. Happe et al., "ReaperAI: An Autonomous Penetration Testing Framework Using Hierarchical LLM Planning," IEEE S&P Workshop, 2025.
[7] R. Fang et al., "Teams of LLM Agents can Exploit Real-World, One-Day Vulnerabilities," arXiv:2406.01637, 2024.
[8] OWASP, "OWASP Top 10 - 2025 Edition," https://owasp.org/Top10/, 2025.
[9] OWASP, "OWASP API Security Top 10 - 2023," https://owasp.org/API-Security/, 2023.
[10] OWASP, "OWASP Mobile Top 10 - 2024," https://owasp.org/www-project-mobile-top-10/, 2024.
[11] D. Xu et al., "Large Language Models for Cyber Security: A Systematic Literature Review," ACM Computing Surveys, 2024.
[12] OpenAI, "GPT 5.6 Sol Technical Report," 2026.
[13] Anthropic, "Claude Opus 4.8 Model Card," 2026.
[14] Google DeepMind, "Gemini 3.5 Technical Report," 2026.
[15] xAI, "Grok 4.5 Technical Report," 2026.
[16] J. A. Santos, "YAGA: Benchmarking Large Language Models for Autonomous Penetration Testing with Emergent Attack Chains," HackerSec Research, 2026.