Martin's Blog

Five AI Pentest Frameworks, and Where the Walls Actually Are

Autonomous AI pentest frameworks have stopped being a research curiosity. ARTEX forks are showing up in incident reports, Cairn panels are exposed on the internet, and CyberStrikeAI made the news for running 221 tool calls without anyone approving them.

I wanted to know what these things actually are underneath the README claims. So I read the source of five of them: HexStrike AI, PentAGI, Cairn, ARTEX-ymh and CyberStrikeAI. I looked at three questions. Who plans the work? What executes it? And when a framework says it limits targets or actions, where does that limit live?

The answer to the last question is the one that matters. Where scope exists at all, it is mostly prompt text or opt-in configuration. Only two of the five ship any default target restriction in code, and both are narrow. None of them requires a human to approve anything out of the box.

A note on method and scope: This is a source reading of local clones taken in November 2026. The file:line references point into those clones and will drift. The ARTEX findings come from the ymh fork, not upstream ARTEX, and that difference matters for scope enforcement (more on that below). I describe architecture and controls for defensive purposes and leave out anything that would make these tools easier to point at someone.

Five very different things

All five call themselves AI offensive-security platforms. They are not the same kind of software.

HexStrike AIPentAGICairnARTEX-ymhCyberStrikeAI
What it isMCP tool server with no agent of its ownMulti-agent pentest platformGeneral state-space search engine, validated on pentestingPentest platform aimed at external breach through to intranet depthAI security workspace on a Go web platform
StackPython (Flask, FastMCP)Go backend, React UIPython (FastAPI server, dispatcher)Go monolith, Next.js UI, PostgreSQLGo, embedded SPA, Node.js Pi runtime
Who plansThe external LLM; in-repo “decision engine” is hand-coded heuristicsGenerator and Refiner agentsNobody, explicitly: a blackboard graph and equal workersA Planner agent woken by graph changesOrchestrator modes: single, deep, plan-execute, supervisor
What executessubprocess.Popen(shell=True) on the server hostA Docker sandbox per flowClaude Code, Codex or Pi CLIs in per-project Kali containersAgent sessions running shell tools in per-intent work directoriesMCP workers with cgroup process containment
StateNone; in-memory cachePostgreSQL with pgvector, optional Neo4jSQLite fact-intent graphPostgreSQL, two graphsSQLite, 49 tables

HexStrike AI: a tool box for someone else’s model

HexStrike is two large scripts. hexstrike_mcp.py (about 5,500 lines) is a FastMCP bridge that proxies to hexstrike_server.py (about 17,300 lines), a Flask server on port 8888 with 156 routes. That server shells out to nmap, nuclei, sqlmap and the rest.

MCP client (Claude, GPT, Cursor) --stdio--> hexstrike_mcp.py
        --HTTP--> hexstrike_server.py (Flask, :8888)
                --subprocess shell=True--> nmap / nuclei / sqlmap / ...

The README talks about an “Intelligent Decision Engine” and “12+ AI agents”. In the code these are deterministic rule tables: static tool scores per target type, regex technology signatures and parameter templates (IntelligentDecisionEngine, hexstrike_server.py:572). BugBountyWorkflowManager returns workflow dictionaries and never executes them. Nothing in the repository calls a model. All of the thinking happens in whatever MCP client the operator connects.

There is no database and no isolation. /api/command is an open shell pipe running as the server user.

PentAGI: the conventional multi-agent pipeline

PentAGI is the most familiar design of the five. Work is organised as Flow, Task and Subtask. A Generator agent splits a task into at most 15 subtasks, and a Refiner agent rewrites the remaining plan after every one (subtasks.go). There are specialist agents for pentesting, coding, installing, searching, memory and advice, each with its own prompt template.

Execution is properly separated. Every flow gets its own Docker container, pentagi-terminal-<flowID>, and commands run through docker exec. Capabilities are dropped and added back explicitly. An optional remote worker node runs Docker over TLS behind OPA authorisation.

Interestingly, security tools are not registered anywhere. An Image Chooser agent picks an image, falling back to vxcontrol/kali-linux, an Installer agent adds what is missing, and everything runs through a generic terminal tool.

Cairn: no planner at all

Cairn is the most opinionated design, and the opinion is to have as little structure as possible. The server stores three things: Facts (confirmed, append-only), Intents (a hyper-edge from a set of facts to a proposed next step) and Hints (human input at any time). Workers coordinate only through that shared board. There are no roles and no direct communication between agents.

The dispatcher generates three kinds of task from the graph state. Bootstrap tries to solve the goal directly. Reason runs when the frontier is empty and something has changed. Explore claims one intent, executes it and writes back one fact.

The workers are literally the claude, codex and pi command-line tools. The first two are started with --dangerously-skip-permissions and --dangerously-bypass-approvals-and-sandbox; pi runs with a limited toolset instead. Cairn exposes no MCP tools and no custom tools. The agents use their own bash, read and write tools inside a well-stocked Kali container. In local mode there is no container, and the agents run as the host user.

ARTEX-ymh: a planner, workers and a guard chain

ARTEX is a Go monolith with a Next.js front end, PostgreSQL and an embedded go-mitmproxy recording proxy. The ymh fork pushes it towards full-chain operations: tunnels (suo5, chisel, ligolo), webshell sessions and honeypot detection.

It keeps two graphs. A global asset graph holds companies, assets, services and endpoints across tasks. A per-task exploration graph holds goals, intents, facts, findings, hints and digests. The frontier is just the open intents.

A Planner (agent/planner.go) wakes on graph changes, debounced to 800 ms, and emits up to four intents at a time. It can only close the goal through prove_goal, which demands confirmed findings and numeric coverage against scope. At least three workers per task claim one intent each and write back through a strict contract: insert_assets, record_fact with a confidence tag, and report_finding, which requires a proof of concept.

Every tool call passes through a guard chain before it runs, which I cover below. All worker HTTP traffic goes through the recording proxy, so every action leaves evidence on disk that a reporter agent can bind to findings.

CyberStrikeAI: governance as a product

CyberStrikeAI is a Go server with about 140 handlers, a static SPA, SQLite and a Node.js worker hosting the Pi agent SDK. Go owns state and approvals; the SDK owns the model loop.

It offers four orchestration modes. Deep mode delegates to sub-agents whose tool catalogue can only narrow the parent’s. Plan-Execute records a plan, hands one step at a time to an executor and replans on evidence, capped at ten iterations. Supervisor transfers between specialists. The agents themselves are Markdown files with YAML front matter: three orchestrators and thirteen kill-chain sub-agents from recon to reporting.

Tools are 90 YAML recipes compiled into command lines and exposed as MCP tools, plus around 100 built-in MCP tools. Those include an exec escape hatch, eight WebShell tools and eight C2 tools. External MCP servers can be federated in, each behind a circuit breaker.

On paper this is the most governed of the five: human-in-the-loop approval with an optional audit-agent reviewer, a regex tool guard, RBAC and four audit streams. I come back to how much of that is on by default.

How the work gets planned

The planning designs fall into four camps.

HexStrike outsources planning entirely. The model in the operator’s client improvises across about 150 tools and replans after each result.

PentAGI and CyberStrikeAI run explicit plan-and-revise loops. PentAGI’s Refiner rewrites the remaining plan after every subtask. CyberStrikeAI’s Plan-Execute mode does the same after every step and keeps the plan on a mutable board.

ARTEX makes planning an event-driven daemon. The Planner is the only agent allowed to create intents, and it wakes whenever the graph changes: a worker finishes, a finding lands, the goal is edited, or a watchdog fires.

Cairn dissolves the planner. Nobody plans ahead. The plan is the growing fact-intent graph, and any worker can reason over it.

Three of the five (Cairn, ARTEX and CyberStrikeAI, with its project-facts board) coordinate through a shared store of facts and next steps. PentAGI has a similar graph as an option. That is a pattern, not a law: the outside telemetry below includes at least one framework that plans from a task list.

How the work gets done

Isolation is where the designs diverge most, and it decides what a compromise of the framework itself would mean.

ExecutorIsolationFailure handling
HexStrikesubprocess with shell=True on the hostNoneRegex error patterns, retry with backoff, fall back to another tool
PentAGIdocker exec into the flow’s sandboxContainer per flow, capability allowlist, optional remote nodeUp to three retries, repeat-call detection, a Reflector agent, a tool-call fixer
CairnCLI agent inside a per-project Kali containerContainer, with the CLI’s own sandbox switched off; none in local modeFallback on timeout or parse failure; losing a heartbeat lease kills the worker
ARTEX-ymhAgent sessions running Bash in per-intent directoriesNo per-call container; recording proxy, egress checks and the guard chainGuard deny or ask, stuck detection, blocked intents fed back to the planner
CyberStrikeAIMCP workers running compiled recipescgroup v2 or Job Object process containment, cancellable workersCircuit breakers, cancellation, hard timeouts, a blocked status on guard hits

PentAGI is the only one that gives every flow a fresh container. Cairn uses one container per project, and none at all in local mode. ARTEX and CyberStrikeAI rely on controls around the call rather than a box around it. HexStrike relies on nothing.

Where the walls actually are

I split limits into two kinds: what may be attacked (targets) and what may be done (actions). For each, the question is whether the limit lives in a prompt or in code that runs before the tool does.

ProjectTarget limitsAction limitsWhere it lives
HexStrike AINoneNoneThe only lever is alwaysAllow: [] in the operator’s MCP client config, outside the codebase
CairnNoneNoneThe container, if you use one. Humans can steer with Hints. Worker prompts tell agents never to refuse
PentAGINone, by designYesPer-agent tool sets and call caps in executor code. Targets are left to the network
ARTEX-ymhDefault .gov/.edu blocklist at graph level; per-call scope check is opt-inYesPre-tool hooks in Go: scope matcher, intercept rules, LLM judge, human decision
CyberStrikeAIPrompt scope, plus a code-enforced .gov blockYesA regex guard on every call, optional human approval, narrowing tool catalogues

The pattern is simple. Prompts state the rules. Where anything is actually enforced, it is in code, and the authors who built the strongest controls clearly assumed the model would not reliably follow the prompt.

ARTEX-ymh: a guard chain on every call

In the ymh fork, every tool call runs this chain before execution (guard/guard.go, intercept/intercept.go):

egress fingerprint check -> audit -> RoE scope check -> tarpit
  -> intercept rules (allow / deny / ask) -> LLM judge -> human decision

The rules-of-engagement check (guard/roe.go) matches targets pulled from the tool call against the task’s registered scope: domains, IPs, CIDRs and ICP records. Keyword rules exist in the schema but are never matched. The verdict has three states. If the task has no registered scope, the call passes. If no target can be extracted from the call, it passes. Loopback counts as in scope, but RFC1918 deliberately does not. A comment notes that otherwise “RoE is pointless for intranet work”.

Enforcement has three modes. off disables the check. warn allows the call and logs the out-of-scope hosts. strict routes the call to a human, and blocks it outright if no human approval path is configured. An RoE approval only exempts the RoE gate. Deny rules are still evaluated, and the source comment says so explicitly.

Intercept rules are priority-ordered regex or string matches, and the default for the dangerous surface (Bash, WebFetch, file writes, session tools) is fail-closed. When no rule matches, a second model, separate from the working agent, judges the call. Timed blanket approvals of two, four or eight hours exist for intranet phases, but they never override a deny.

One detail shows how much thought went into this. When a rule blocks a call, the message tells the agent it hit 【ARTEX 平台管控·非目标防御】: platform governance, not the target’s defences. That framing is meant to stop the agent treating the block as a WAF to work around.

The egress check denies any tool output containing platform secrets such as LLM keys, the JWT key or the database password. The audit log records only which kind of secret matched.

There are two important caveats. Elastic describes upstream ARTEX as having removed the RoE mechanism. The ymh fork kept and extended it. If you are looking at a different fork, check before assuming any of this is there.

And the scope allow-list is operator-written and empty by default. With no scope registered, every call passes the RoE gate.

ARTEX-ymh: a default blocklist that only covers the graph

ARTEX also seeds built-in asset-intercept rules for .gov, .gov.cn, .edu and .edu.cn, marked as built-in and enabled (db/db.go:333-336). A match rejects the intent or asset.

The catch is where it is checked. It runs when add_intent or insert_assets puts a target into the graph, not on every shell command. As Elastic puts it, a shell command can still reach any host.

CyberStrikeAI: a guard that approval cannot bypass

CyberStrikeAI’s tool guard (internal/toolguard/, internal/mcp/tool_call_guard.go) checks every call immediately before execution. That covers internal MCP tools, external MCP servers, the HTTP MCP endpoint and the standalone stdio process. It inspects the tool name, the full serialised JSON arguments including nested keys, and up to three rounds of URL percent-decoding. That last part is aimed squarely at a model that can shape its own arguments.

A block gets its own blocked status, does not trip circuit breakers, and cannot be overridden by a human approval or by editing the arguments. Operators can add up to 100 RE2 rules through the admin API.

The one built-in rule is hard-coded into DefaultConfig, so it applies even if a deployment omits the tool_guard section. Its message translates roughly as “Recognised {match}: attacking government websites is prohibited”. The test suite shows it blocking agency.gov, www.agency.gov.cn:443, EXAMPLE.GOV.UK, wildcard forms and encoded variants such as agency%252Egov buried inside shell commands. It deliberately ignores governance, .govt, agency.gov-example.com and similar near-misses. There is no .edu or .mil rule.

Target scope is a different story. A project’s scope_json (targets, excludes, notes) is formatted into a Chinese-language block injected into the agent’s context. It reads roughly as “this is the authorised boundary, you must comply: test only the listed targets, avoid excludes, do not expand scope on your own”. It is advisory. Nothing blocks a call to a target outside it.

Human approval is real code. Pending decisions live in a table and the Go server interrupts execution until someone decides. A reviewer can be a person or an audit agent with a long judge prompt: a rule ladder of must-reject, then must-approve, then default-allow, with mandatory rejection of password changes, destructive actions, service stoppers and DoS. In review_edit mode it can only narrow arguments.

The shipped default is default_mode: off.

PentAGI: action limits, no target limits

PentAGI is upfront about it. Its documentation says it “does not restrict targets on its own, so enforce a hard boundary at the network level”, meaning a firewall on the worker node.

Its action limits are in code. Each agent’s tool set is fixed in a Go executor config (backend/pkg/tools/tools.go) and cannot be widened by prompting. General agents are capped at 100 tool calls and limited agents at 20. Repeat detection stops loops after three identical calls. The terminal tool times out after 1,200 seconds by default, with a hard cap of three hours.

The layering they share

Put together, the frameworks use the same four layers, filled in to very different degrees:

  1. Prompts. Scope blocks, planner discipline, mentor reviews. Always present and always bypassable by a misbehaving model.
  2. Per-call code gates. ARTEX’s pre-tool hooks and CyberStrikeAI’s tool guard. These are the only layers that stop a call whatever the model decides.
  3. Human decisions. ARTEX’s ask and CyberStrikeAI’s approval interrupts, both with a code floor that approval cannot lower.
  4. The environment. PentAGI’s firewall doctrine and Cairn’s containers.

HexStrike has none of them. Cairn has only the fourth, and only in container mode.

Shipped defaults

Defaults are what attackers actually run. Here is what each framework does out of the box.

Human approvalPrompts claim authorisationDefault target limits
ARTEX-ymhOff (PermissionMode: ModeBypass)Yes.gov, .gov.cn, .edu, .edu.cn at graph level; empty allow-list
CyberStrikeAIOff (hitl.default_mode: off)Yes.gov and .gov.cn on every call
CairnBypass flags on two of three worker typesWorkers told never to refuseNone
PentAGIOff (ASK_USER=false)Yes; prompts open with an AUTHORIZATION FRAMEWORK blockNone
HexStrike AILeft to the MCP clientNoAn out_of_scope list is accepted and stored, then never read

That last row deserves a second look. HexStrike accepts an out_of_scope list in API payloads and stores it on workflow objects (hexstrike_server.py:2442, 10108, 10117). No execution path ever consults it.

The defaults also say something about whose targets the authors cared about. Both built-in blocklists cover government domains: .gov, .gov.cn and, in CyberStrikeAI’s case, forms such as .gov.uk. ARTEX adds .edu and .edu.cn. Nothing else is protected by default.

CyberStrikeAI shows how little a feature list says about a deployment. It has the richest approval system of the five. In the Huntback investigation reported in November, the approval gate had been switched off and 221 tool calls ran without a human decision. The scope prompt was still there. It made no difference.

How this lines up with Elastic and Cloudflare

Two industry write-ups appeared on 13 November 2026 covering the same ground from different angles.

Elastic Security Labs’ Inside the loop: Five AI attack frameworks that never ask permission covers the same five frameworks. I checked its two comparison figures against the source, and they hold up.

The process-lineage figure, showing what starts nmap, is correct for all five. ARTEX and CyberStrikeAI start tools directly under the framework process. HexStrike’s server never calls a model; the operator’s MCP client does. Cairn and PentAGI go through the Docker socket into a container. There are three simplifications. The figure’s dashed “tool container” box for ARTEX is really just the Bash tool, since the fork does no per-call containerisation. CyberStrikeAI’s model calls run in a Node.js child process, not the Go binary itself. And Cairn’s local mode, a supported option, has no container at all.

The design-and-safeguards figure matched every cell I could verify. Its central claim is that none of the five enforces authorisation in code, and that the only default target limits are ARTEX’s .gov/.edu blocklist and CyberStrikeAI’s .gov block. That is what the code shows. One difference is not a contradiction: Elastic analysed upstream ARTEX, which removed RoE, while I read the ymh fork, which kept it.

Cloudflare’s Adapting to AI penetration testing orchestrators is telemetry and incident response rather than source analysis. Its figures come from Cloudflare’s network, and the product recommendations at the end read as marketing. The rest is useful.

According to Cloudflare, ARTEX and its forks make up more than 60% of known AI pentest orchestrators, with over 12,000 repositories. It counts 204 forks with their own code changes; the artex.sh tracker uses stricter criteria and lists 57. Request volume from this tooling grew more than thirtyfold since 1 October, and more than tenfold since the original ARTEX repository was taken down on 8 October. Call it a hydra effect. In an ecosystem held together by AGPL forks, a takedown spreads the code rather than stopping it.

Its table adds three families I did not read. LuaN1aoAgent plans over three graphs and has built-in SSH and Chisel tunnelling that tries to reconnect after disruption. Decepticon dispatches specialist sub-agents and ships dedicated Active Directory and cloud-pivoting skills. PentestGPT-Agent plans from a task list with dependencies, runs on the host, and has no built-in post-exploitation.

The incident-response section is the most practical part. In one engagement, agents used a compromised account-level API token to create Cloudflare Worker scripts with names such as artex-pgrecon-* and artex-do-*. Worker names starting with artex- are a hunt you can run in Cloudflare audit logs today. Cloudflare saw more than 69 requests a minute and 400 database dump requests in an hour, “lightly guided by the human operator”. That matches the ARTEX design: once the goal is set, the planner largely runs itself.

The recurring root cause across Cloudflare’s engagements was static, non-expiring CI/CD tokens with no IP restrictions or rate limiting, used to pivot from a hyperscaler environment into customer environments. An agent inherits everything an identity allows, and it chains credentials at machine speed.

What I take from this

These frameworks are not exotic. They are an LLM, a planning loop, a tool runner and a database. The differences are in planning style and isolation, and the market seems to be converging on graph-based planning, which is mostly an engineering choice and says little about danger.

What matters to a defender is the shipped defaults. Approval is off everywhere. Scope is a paragraph in a prompt. The only hard target limits protect government domains, plus .edu and .edu.cn in ARTEX, and in ARTEX’s case only at graph level. If your organisation does not sit under one of those suffixes, nothing in any of these frameworks stops one from being pointed at you.

So the controls that matter are the ones you own:

The authors who built real guards, ARTEX-ymh and CyberStrikeAI, put them in code and made sure a prompt or an approval could not lower them. That was the right call. But the guards that ship enabled only cover a handful of domain suffixes. Everything else, the scope allow-list and the human approval, is empty or off until an operator decides otherwise, and the people running these tools against you have no reason to.

Sources

#AI Agents #Offensive Security #ARTEX #PentAGI #CyberStrikeAI #Threat-Intelligence