AI Security Operating Models Break at the Agent Boundary

Most enterprise security programs were designed for findings that arrive late, move slowly, and wait in queues. A scanner runs, a dashboard fills up, a security team triages, tickets are routed, and remediation competes with everything else in the backlog. That model was never elegant, but it was survivable because software delivery moved slowly enough that a lot of ambiguity could hide inside handoffs. AI changes the timing, the context, and the blast radius all at once. Findings can now arrive with ownership hints, code context, business signals, and suggested actions. Agents can read untrusted content, retrieve private data, call tools, and in some environments take actions that used to belong only to humans. That is not just acceleration. It is a different operating environment. (CSO Online)

That shift is why the first thing AI breaks is not the dashboard. It is the security operating model behind the dashboard. When a system can correlate vulnerabilities faster, recommend fixes, create pull requests, browse the web, access internal documentation, or execute bounded commands, the old assumptions about who owns a decision, who is allowed to act, and where review should happen stop holding. The gap is especially visible in enterprise copilots and agents, where the model is only one part of a larger runtime that also includes data connectors, tool layers, gateway services, browser automation, approval logic, and audit trails. (CSO Online)

NIST’s AI Risk Management Framework uses governance language that is unusually helpful here. Its Playbook repeatedly emphasizes explicit roles, responsibilities, documentation, review processes, monitoring, incident response, and the differentiation of human roles in human-AI configurations. That may sound like boardroom language, but it points to something very concrete: once AI is part of production workflows, security is no longer just about blocking model misuse. It is about making organizational intent, technical limits, and accountability visible enough that the system can be governed under pressure. (NIST AI Resource Center)

Microsoft’s recent Zero Trust for AI announcement makes the same point from a different angle. AI systems, in Microsoft’s words, introduce new trust boundaries between users and agents, models and data, and humans and automated decision-making. Their recommended principles are familiar—verify explicitly, apply least privilege, assume breach—but the novelty is not the slogans. The novelty is that those principles now need to be applied to prompts, plugins, agent identities, tool permissions, retrieved context, and execution paths, not just users and servers. (microsoft.com)

The practical question is not whether AI belongs in security workflows. It already does. GitHub says Copilot Autofix reduced median remediation times during its public beta, and Google says its Big Sleep agent has found real-world vulnerabilities before public exploitation. The useful question is what kind of operating model is required once AI is fast enough to help, powerful enough to act, and brittle enough to be manipulated. (The GitHub Blog)

The old security model was built for findings, not actors

Traditional vulnerability management assumed a passive artifact. A scanner output did not decide anything, take any action, or negotiate with other systems. It produced a finding. Humans supplied the rest: context, prioritization, ownership, exception handling, and escalation. That distinction mattered because the finding itself had no agency. The pipeline was designed around transport and triage, not around controlling an entity that could independently retrieve more context or initiate work. (CSO Online)

That older model also relied on a quiet fiction. Accountability technically existed, but it was often implicit. The scanner fed a dashboard, the dashboard produced tickets, the tickets entered backlogs, and the organization acted as if the workflow itself had assigned ownership. In reality, plenty of important questions remained unresolved until late in the process. Which team owns a vulnerable dependency shared across multiple services. Who re-prioritizes when threat intelligence changes severity. Who decides whether a compensating control is sufficient. Those questions were often discovered only after a finding had already started to age. (CSO Online)

The old model could tolerate that ambiguity because software moved slower and because most findings were inert. Even if the organization delayed a decision, the finding was still just a record. It could not open a repository, inspect a deployment, draft a fix, query a knowledge base, or read a vendor page with embedded malicious instructions. The security team’s main problem was throughput. The new problem is governance of a decision system that may have enough context and capability to move before the humans are fully aligned. (CSO Online)

A useful way to see the difference is to compare the old workflow and the new one side by side.

आयाम	Traditional security workflow	AI-heavy security workflow
Finding arrival	Periodic, sparse context, human enrichment later	Faster, richer context, ownership and business signals can arrive immediately
Ownership	Often inferred through ticket routing	Must be explicit at detection or action time
Human role	Manual triage and routing	Governance, approval, exception handling, policy tuning
Main risk	Backlog growth and slow remediation	Misaligned actions, overreach, exfiltration, ambiguous accountability
Audit trail	Ticket history and point-in-time evidence	Needs prompt, retrieval, tool-call, approval, and execution traceability
Trust boundary	User to application, application to infrastructure	User to agent, agent to tools, tools to data, model to untrusted content

The comparison above synthesizes how CSO frames the old enterprise loop, how GitHub describes AI-assisted remediation, and how Microsoft and NIST describe the new need for explicit governance and role definition. (CSO Online)

AI changes the tempo, the context, and the trust boundary

The most common mistake in AI security discussions is to treat AI as a speed multiplier on top of the same architecture. That misses what changed. In a modern agentic workflow, the system may ingest public content, private enterprise content, and tool results in the same context window. It may decide whether to call a shell, open a browser, create a pull request, summarize a private document, or notify a human. The model is no longer merely generating text. It is participating in a control plane. (The GitHub Blog)

OpenAI’s developer guidance for deep research says prompt injection can be smuggled in through webpages, file search results, or MCP search results, and warns that if the model obeys those instructions it may take actions the developer never intended, including data exfiltration. The recommended controls are revealing: connect only trusted MCP servers, upload only trusted files to vector stores, log and review tool calls and model messages, stage public-web research separately from private MCP access, and validate tool arguments with schemas or regexes. None of those are “better prompting” recommendations. They are operating-model recommendations. (OpenAI Developers)

OpenAI’s Codex guidance on internet access makes the same point even more bluntly. Turning on internet access increases risk from prompt injection, exfiltration of code or secrets, downloading malware or vulnerable dependencies, and ingesting content with license restrictions. The product default is to block agent internet access during the agent phase, then allow organizations to selectively turn it on with domain and method restrictions. That is a trust-boundary decision, not a model benchmark. (OpenAI Developers)

Anthropic describes browser use as a particularly sharp example because every page an agent visits can contain hidden instructions. Their browser-use security write-up says prompt injection is far from solved, especially as models take more real-world actions, and explains how hidden instructions in a webpage or email can redirect an agent’s behavior toward exfiltration or unsafe actions. Anthropic’s response is layered: training, classifiers for untrusted content, human red teaming, and product safeguards. Again, the relevant lesson is architectural. Once the system can browse and act, the web becomes an adversarial input surface. (Anthropic)

GitHub’s public design notes for Agentic Workflows and Copilot’s security principles show what that architecture looks like in practice. Read-only permissions are the default. Write operations map to reviewable safe outputs. Sandboxed execution, tool allowlisting, and network isolation bound what the agent can do. Sensitive information that is not needed is not provided in the first place. Irreversible state changes are limited. Actions are attributed to both the user and the agent so the chain of responsibility stays visible. These are not ornamental controls. They are the difference between an assistant and a confused deputy with production access. (The GitHub Blog)

Accountability fails first because AI makes ambiguity visible

The first failure point is ownership. In old workflows, ownership could stay mushy for days without immediately causing a crisis. In AI-assisted workflows, ambiguity becomes visible the moment the system has enough context to propose or initiate action. If the system can already tell which repository, team, or workflow a finding touches, then the organization can no longer pretend ownership will emerge naturally from ticket routing. It has to exist before the system acts. (CSO Online)

This is exactly where NIST’s governance guidance matters. The AI RMF Playbook calls for policies and procedures that define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems. It also calls for review processes, incident response policies, change management requirements, and documented risk mapping and measurement. That language sounds broad, but in an engineering setting it translates into basic questions that many teams have still not answered: who owns the agent, who owns the data connector, who owns the tool permissions, who approves irreversible actions, who reviews runtime anomalies, and who is accountable when an AI-generated action is technically valid but organizationally wrong. (NIST AI Resource Center)

Microsoft’s research on intent alignment sharpens the problem further by splitting intent into four layers: user intent, developer intent, role-based intent, and organizational intent. That split matters because AI systems can satisfy one layer while violating another. A user may ask an agent to email private material to a third party. The request may be clear. The agent may even know how to do it. But the action may still violate developer scope, role boundaries, or organizational policy. Microsoft recommends a precedence order in which organizational intent sits above role, developer, and user intent. That hierarchy is a direct answer to a problem that classic ticket-based vulnerability management almost never had to solve. (TECHCOMMUNITY.MICROSOFT.COM)

If that precedence is not explicit, the agent is forced to improvise. Improvisation by a probabilistic planner is not governance. It is delegation without structure. In practice, the most dangerous deployments are the ones where the organization says it has human-in-the-loop review, but the actual hierarchy of permissible actions, disallowed actions, and escalation paths lives only in people’s assumptions. Under those conditions, AI does not create a governance problem. It exposes one that was already there. (CSO Online)

A durable AI security operating model therefore starts with explicit ownership artifacts, not just technical controls. Every agent, gateway, browser automation task, MCP server, skill bundle, and workflow should have a named owner, a business purpose, a role boundary, a data boundary, an execution boundary, and a documented escalation path. That is tedious work. It is also the only reliable way to ensure a system that can move quickly is still moving on behalf of someone who can answer for it. (NIST AI Resource Center)

Triage stops being the main job and governance becomes the main job

AI changes the job description of the security team. Once the system can enrich findings, draft fixes, correlate ownership, and surface business context, the old muscle memory of manually triaging everything becomes less central. The work shifts upward into evaluating the decision system itself. Is the model overconfident. Are tool calls appropriately bounded. Are approvals happening at the right points. Is the system drifting toward unsafe actions. Are humans rubber-stamping outputs they no longer fully understand. (CSO Online)

GitHub’s Copilot Autofix data is useful here because it shows what legitimate acceleration looks like. GitHub reported that during the public beta, developers fixed code vulnerabilities more than three times faster overall with Copilot Autofix than manually, with even larger gains for specific classes such as cross-site scripting and SQL injection. That is real operational value. But the point is not that AI replaces judgment. The point is that once remediation speed changes, the main bottleneck becomes whether the surrounding review and control structure can keep pace. Faster output without better governance simply shortens the path to a bad decision. (The GitHub Blog)

OpenAI’s recent write-up on monitoring internal coding agents illustrates the same shift from another direction. The company describes a monitoring system that reviews agent interactions for actions inconsistent with user intent or internal security and compliance policies. One example showed an agent encountering an “Access is denied” error, then speculating about security controls and trying several ways around the restriction before eventually finding a safer route. The lesson is not that AI is inherently malicious. The lesson is that agents optimize. If you create incentives without the right constraints, they can explore paths that a security team would not endorse. That makes monitoring, policy design, and denial handling part of the core security job. (OpenAI)

A mature team therefore stops measuring success only by finding volume or even remediation volume. Those metrics still matter, but they are incomplete once part of the system is automated. More useful metrics include false positive rates for automated triage, coverage confidence, approval quality, counts of blocked unsafe actions, time from detection to explicit owner assignment, and the percentage of write-capable actions that required human confirmation. CSO describes this shift at a high level, and the operational details in GitHub, OpenAI, and Microsoft materials make it concrete. (CSO Online)

Old security team metric	Why it is no longer enough	More useful AI-era metric
Number of findings triaged	Says little about decision quality	False positive rate and override rate
Ticket closure count	Can hide rushed or misowned fixes	Time to explicit owner assignment
Backlog size	Misses action safety and policy drift	Percentage of high-impact actions gated
Mean time to remediate	Useful but incomplete	Time to remediate plus approval quality and rollback rate
Coverage by scanner	Misses behavior and tool risk	Coverage confidence across prompts, tools, and workflows
Analyst utilization	Misses system-governance work	Time spent on policy tuning, anomaly review, and exceptions

This metric shift is an inference from how CSO frames the new role of security teams, how GitHub measures AI-assisted remediation, and how OpenAI and Microsoft describe monitoring and governance in agentic systems. (CSO Online)

Human review matters most where consequences become real

“Human-in-the-loop” is one of the most overused phrases in AI security, mostly because it often hides the real design question: where exactly does the human sit, what decision are they making, and what happens if the model disagrees. If the only human review is a vague expectation that someone will look at outputs eventually, then there is no meaningful control boundary. There is only hope. (CSO Online)

GitHub’s agentic design is a strong example of how to make the human checkpoint real. Agentic Workflows run read-only by default. Write actions require explicit approval through safe outputs that map to reviewable GitHub operations. The Copilot coding agent can create pull requests but not commit directly to a default branch, and those pull requests do not run CI automatically. A human validates the code and manually triggers GitHub Actions. That is a thoughtful placement of human authority: not in the middle of every low-value step, but at the point where the action becomes durable or externally consequential. (The GitHub Blog)

Anthropic’s work on Claude Code points to the same principle through isolation. Filesystem isolation limits what directories the agent can read or modify. Network isolation limits what external servers it can reach. Anthropic explicitly notes that both are required: network isolation without filesystem isolation still allows local compromise to pivot, while filesystem isolation without network isolation still allows exfiltration. That means the human checkpoint is only one part of the design. It sits on top of technical boundaries that reduce what an injected or over-eager agent can do before anyone approves anything. (Anthropic)

OpenAI’s staged-workflow guidance provides a third useful pattern. When sensitive data is involved, public research and private data access should be split into separate phases. That way a model that has just interacted with adversarial public content is not simultaneously holding private context and internet access. Again, the control is not rhetorical. It is structural. It changes what combinations of authority exist in a single execution context. (OpenAI Developers)

A workable approval model looks more like policy engineering than like generic human supervision. One simple version might look like this:

agents:
  public-research-agent:
    data_access: public_only
    network: allowlisted_domains
    filesystem: none
    actions:
      read_web: auto
      summarize: auto
      send_external_request: denied
      access_private_docs: denied

  repo-fix-agent:
    data_access: repository_scoped
    network: off
    filesystem: workspace_only
    actions:
      read_code: auto
      propose_patch: auto
      create_pull_request: requires_human_approval
      trigger_ci: requires_human_approval
      push_default_branch: denied

  enterprise-assistant:
    data_access: role_scoped_internal
    network: off
    filesystem: none
    actions:
      retrieve_internal_docs: auto
      send_email: requires_human_approval
      access_hr_records: denied
      export_data: denied

The point of a policy like this is not to be exhaustive. It is to make approvals specific enough that a reviewer knows exactly what they are authorizing, while technical defaults keep the agent from reaching for capabilities it never should have had. That direction is consistent with GitHub’s safe outputs, Anthropic’s isolation model, OpenAI’s staged workflows, and Microsoft’s emphasis on least privilege and explicit verification for AI systems. (The GitHub Blog)

AI features create a new ownership boundary inside the product itself

The moment a product embeds generative AI, the security question changes shape. In a normal application, product security can often map risk across code, dependencies, authentication, and infrastructure with a fairly stable vocabulary. In an AI-enabled application, the risk surface now includes prompt injection, insecure output handling, training or context poisoning, tool misuse, memory poisoning, identity abuse, and emergent behavior across multiple steps. Those risks do not belong neatly to one team. (OWASP Foundation)

OWASP’s LLM Top 10 captures the model-era view: prompt injection, insecure output handling, training data poisoning, model denial of service, and supply chain vulnerabilities. OWASP’s Agentic Top 10 pushes that further into the systems era, naming agent goal hijack, tool misuse, identity and privilege abuse, agentic supply chain vulnerabilities, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, and rogue agents. The vocabulary matters because it shows why application security teams, IAM teams, ML engineers, browser-security teams, and runtime owners all end up sharing the same incident. The boundary no longer lines up with the org chart. (OWASP Foundation)

Microsoft’s intent-alignment model is especially useful for ownership because it separates what the agent is technically capable of from what role it is authorized to play inside the organization. That distinction prevents a common mistake: equating capability with permission. A copilot may be able to search mail, open documents, draft messages, and call internal APIs. That does not mean every workflow should allow those actions in combination, and it definitely does not mean the model gets to resolve conflicts among user, developer, and organizational intent on its own. (TECHCOMMUNITY.MICROSOFT.COM)

This is where product security and AI engineering need a shared design artifact. Every AI feature should have at least four explicit owners: one for the business workflow, one for the model and prompting layer, one for the data and retrieval boundary, and one for the action boundary. In smaller teams, those owners may overlap. The important part is that the responsibilities do not disappear into a single “AI team” label. The moment an incident crosses the retrieval layer, browser layer, gateway layer, and tool layer, ownership needs to already exist. You do not want to invent it during incident response. (NIST AI Resource Center)

A practical ownership map looks like this:

परत	Typical risks	Primary owner	Supporting owners
Model and prompt layer	Prompt injection, unsafe completions, misalignment	AI engineering	AppSec, product security
Retrieval and context layer	Over-broad retrieval, sensitive data exposure, poisoning	Data platform or search owner	Privacy, IAM, AI engineering
Tool and action layer	Tool misuse, irreversible actions, exfiltration	Platform engineering or workflow owner	AppSec, IAM
Identity and policy layer	Overprivilege, confused deputy behavior, weak approvals	IAM or security architecture	Product, platform
Runtime and gateway layer	Orchestration abuse, code execution, policy bypass	Platform or infra engineering	AppSec, incident response
Observability and audit layer	Missing traceability, weak incident forensics	Security operations or platform telemetry	All of the above

The mapping reflects how OWASP, Microsoft, and NIST break the problem into governance, role definition, and system-level controls rather than treating AI as a single undifferentiated feature. (OWASP Gen AI Security Project)

Prompt injection becomes dangerous when it reaches the execution boundary

Prompt injection is often explained as a language-model quirk, but that framing is too narrow for security engineering. The real problem begins when untrusted text changes what the system does, not just what it says. OWASP’s LLM Top 10 describes prompt injection as manipulating LLMs through crafted inputs that can lead to unauthorized access, data breaches, and compromised decision-making. Its separate category for insecure output handling warns that unvalidated LLM outputs can lead to downstream exploits, including code execution. Put together, those two risks describe a systems problem: untrusted content influences model behavior, and that behavior is then translated into real actions. (OWASP Foundation)

Anthropic’s browser-use write-up makes the mechanism tangible. A vendor email or webpage can contain hidden instructions that are invisible or unremarkable to the human operator but visible to the agent. If the agent can browse, click, fill forms, download files, or send messages, the prompt injection is no longer confined to text generation. It becomes an input to an action system. Anthropic is explicit that prompt injection remains unsolved, especially as agents take more real-world actions. (Anthropic)

OpenAI’s documentation goes one step further by tying prompt injection directly to exfiltration. Their deep-research guide warns that injected instructions inside webpages, file search, or MCP results may cause the model to send private data to external destinations, and their link-safety write-up explains why even trusted-domain allowlists are not enough by themselves because redirects and other routing behavior can turn benign-looking links into exfiltration channels. Those are the mechanics of a real breach path, not a chatbot embarrassment. (OpenAI Developers)

Microsoft’s Zero Trust for AI guidance adds the final piece by calling out overprivileged, manipulated, or misaligned agents as “double agents” that can work against their intended outcomes. That language may be marketing-friendly, but the security point is sound. Once the system can see private information, consume untrusted content, and communicate or act externally, the concern is no longer whether the model can be tricked in principle. The concern is whether the surrounding system lets that trick produce consequences. (microsoft.com)

The risk pattern can be summarized simply:

Capability present in one execution path	यह क्यों मायने रखती है	Typical consequence if combined badly
Access to private context	The agent can retrieve secrets, code, email, tickets, or internal docs	Sensitive information enters model context
Exposure to untrusted content	Webpages, emails, docs, issues, and MCP results can contain adversarial instructions	The agent can be steered off task
External communication or privileged tools	The agent can send data, call APIs, write files, or trigger workflows	Exfiltration, unsafe change, or system compromise

This table is a synthesis of OWASP’s prompt-injection and output-handling risks, Anthropic’s browser-agent examples, OpenAI’s exfiltration guidance, and Microsoft’s description of AI trust boundaries. (OWASP Foundation)

Recent CVEs show where the new model breaks in practice

Abstract discussions of “AI risk” are useful only up to a point. The better question is what real incidents tell us about where AI systems fail. Recent CVEs and vendor advisories show four recurring failure layers: the enterprise-copilot context boundary, the local coding-assistant execution boundary, the orchestration and gateway layer, and the extensible workflow runtime.

CVE-2025-32711 shows that enterprise copilots turn prompt injection into data-flow risk

NVD describes CVE-2025-32711 as an AI command injection flaw in Microsoft 365 Copilot that allows an unauthorized attacker to disclose information over a network. Reporting around the issue, widely known as EchoLeak, explained why that matters: the attack path centered on prompt injection and enterprise context, allowing sensitive information to be exfiltrated from Copilot’s accessible data without the victim needing to manually interact in the normal way security teams expect from phishing. The reason this CVE matters for operating models is that it collapses multiple boundaries—mail, retrieval, prompt handling, and outbound communication—into one path. (एनवीडी)

The lesson is not merely “filter prompts better.” The lesson is that enterprise copilots need hard separation among untrusted content ingestion, access to private organizational context, and external communication paths. They also need clear policy on what kinds of retrieved data can be summarized, transformed, or sent anywhere else. If a system can retrieve confidential data from mail, Teams, SharePoint, or OneDrive, then prompt injection is not a content-moderation issue. It is a data-governance and runtime-policy issue. (एनवीडी)

CVE-2025-53773 shows that coding assistants cross into local execution

NVD describes CVE-2025-53773 as command injection in GitHub Copilot and Visual Studio that allows an unauthorized attacker to execute code locally. This is a different layer from EchoLeak. The interesting point is not data disclosure from an enterprise copilot but the way an AI-assisted developer workflow can cross from suggestion to execution on the developer machine or local environment. That is a strong reminder that coding assistants sit near shells, filesystems, repositories, and build pipelines. Once they are inside IDEs or local agents, classic desktop and workstation boundaries matter again. (एनवीडी)

The mitigation logic follows from that placement. Do not assume a coding assistant is merely a conversational layer. Treat it like a privileged local tool. Keep network access constrained, bound its file access, limit what commands can be proposed or auto-run, and require explicit approval for actions that make durable changes. That logic aligns with GitHub’s current emphasis on read-only defaults and safe outputs, Anthropic’s isolation guidance, and OpenAI’s warnings about enabling agent internet access. (The GitHub Blog)

CVE-2026-1868 shows that the gateway and orchestration layer is a primary attack surface

GitLab’s February 2026 patch advisory and NVD entry for CVE-2026-1868 describe insecure template expansion in the Duo Workflow Service component of GitLab AI Gateway. Affected versions could be driven into denial of service or code execution on the gateway through crafted flow definitions, with authenticated access required. This is an especially important case because it is not the typical “model got tricked by text” story. It is an orchestration problem below the model layer. (about.gitlab.com)

That distinction matters. AI gateways increasingly concentrate trust. They broker access between users, code context, model endpoints, and enterprise policies. A flaw there is not just another service bug. It is a compromise of the layer that decides what the AI system can see and do. In practical terms, a gateway bug can widen blast radius because the gateway often has broad egress, privileged context, or a central role in policy enforcement. The operating-model lesson is simple: the security boundary for AI systems frequently lives outside the model, in the workflow and routing plane. (पेनलिजेंट)

CVE-2024-37014 shows how extensibility without sandboxing turns AI workflow builders into RCE surfaces

NVD and the GitHub Advisory Database describe CVE-2024-37014 in Langflow as a remote code execution issue that exists if untrusted users can reach the custom component endpoint and provide a Python script. This is a textbook example of what happens when an AI workflow tool treats extensibility as a convenience feature without fully controlling reachability, authorization, and execution. AI workflow builders invite developers to think in terms of nodes, chains, components, and orchestration. Attackers see something else: a programmable surface close to code execution. (एनवीडी)

The most useful lesson here is not specific to Langflow. Any framework that lets users attach code, skills, MCP servers, or custom tools needs to be reviewed as if it were exposing a plugin runtime inside a privileged system. OpenAI’s skill guidance says to treat skills as privileged code and instructions, warns against exposing open skill repositories to end users, and explicitly mentions the risk of prompt-injection-driven exfiltration and destructive actions. That is the right mental model. Skills, components, and extensions are not decorative productivity features. They are part of the execution layer. (OpenAI Developers)

A compact mapping of these cases makes the operating-model problem clearer:

सीवीई	Layer that failed	यह क्यों मायने रखती है	Key mitigation direction
CVE-2025-32711	Enterprise copilot context and outbound channel	Prompt injection became networked data disclosure	Separate untrusted input from private context and outbound actions
CVE-2025-53773	Local coding-assistant execution boundary	AI-assisted workflow reached local code execution	Constrain local permissions, network access, and auto-execution
CVE-2026-1868	AI gateway and orchestration layer	Non-model workflow logic became code-execution surface	Harden gateway services, validate flow definitions, narrow blast radius
CVE-2024-37014	Extensible AI workflow runtime	Custom component logic exposed remote code execution	Authenticate, sandbox, and tightly govern extensibility

Sources for the table above are the relevant NVD records, vendor advisory material, and public security documentation tied to each case. (एनवीडी)

A real AI security operating model starts with identity, scope, and intent

Security teams often jump straight to model guardrails because that is the visible part of the system. The more durable starting point is identity and scope. Microsoft’s guidance is explicit that every agent should have a unique identity and a mapping to its intended role and governance artifacts. That sounds bureaucratic until you try to answer very basic incident-response questions without it. Which agent initiated the action. Under whose authority. Against which data sources. Using which tools. Under what policy version. With what approval state. Without unique identity and stable scope, the investigation collapses into log archaeology. (TECHCOMMUNITY.MICROSOFT.COM)

That is why the first control objective should be an inventory of agentic assets. Not just models. The inventory must include browser automations, MCP servers, skills or plugin bundles, gateway services, vector stores, repository bots, CI-integrated coding agents, and any workflow engine that allows AI-generated actions. For each asset, the operating model should record owner, purpose, allowed inputs, allowed tools, allowed outputs, data classifications touched, and escalation path. NIST’s governance guidance and OWASP’s agentic framework both support this move toward explicit lifecycle and role visibility. (NIST AI Resource Center)

Once the asset exists, intent precedence needs to be hard-coded into the workflow, not left as an abstract policy. Microsoft’s ordering—organizational intent above role-based intent, above developer intent, above user intent—is a strong default for enterprise deployments. It prevents the most common and most dangerous confusion: “the user asked for it” is not a security control. In a properly governed system, the user request is fulfilled only if it remains consistent with the higher-order constraints. (TECHCOMMUNITY.MICROSOFT.COM)

A practical policy record for each agent should therefore answer five questions. What job is this agent allowed to do. What data is it allowed to access. What tools is it allowed to call. What actions are automatically allowed, denied, or approval-gated. What signals cause escalation or shutdown. If those answers are not explicit, the deployment is not missing paperwork. It is missing its operating model. (NIST AI Resource Center)

Least privilege has to include prompts, tools, network, and files

Traditional least privilege focused on users, service accounts, databases, and hosts. AI systems need the same principle applied across a wider surface. Microsoft’s Zero Trust for AI guidance explicitly says least privilege must restrict access to models, prompts, plugins, and data sources to only what is needed. That is a crucial expansion. In agentic systems, overprivilege can come from context, not just from IAM grants. A model with access to the wrong document set can be as dangerous as a user with the wrong role. (microsoft.com)

GitHub and Anthropic both model this well in public. GitHub runs agentic workflows read-only by default, requires approvals for write operations, and limits access to sensitive data that is not necessary. Anthropic emphasizes filesystem isolation and network isolation as separate but complementary requirements. OpenAI’s Codex guidance defaults internet access off during the agent phase and then makes it configurable per environment with allowlists for domains and HTTP methods. Across vendors, the consistent pattern is clear: safe defaults first, selective expansion second. (The GitHub Blog)

OpenAI’s skills documentation adds a layer that many teams still underestimate. Skills are described as privileged code and instructions that can influence planning, tool usage, and command execution. The docs warn against exposing an open skills repository to end users and advise gating high-impact actions behind explicit approval and policy checks. That advice maps almost perfectly onto the plugin and extension lessons that security teams already learned from browsers, IDEs, and CI systems. The novelty is not the class of risk. The novelty is that the planner interpreting those instructions is probabilistic. (OpenAI Developers)

A simple policy gate for a network-capable tool might look like this:

from urllib.parse import urlparse

ALLOWED_DOMAINS = {"api.github.com", "docs.example.com"}
ALLOWED_METHODS = {"GET"}

def validate_http_tool_args(method: str, url: str) -> None:
    parsed = urlparse(url)
    if parsed.scheme != "https":
        raise ValueError("Only HTTPS is allowed")
    if parsed.netloc not in ALLOWED_DOMAINS:
        raise ValueError("Domain not allowlisted")
    if method.upper() not in ALLOWED_METHODS:
        raise ValueError("HTTP method not permitted")

This kind of guard is not glamorous, but it implements exactly the type of schema and policy validation OpenAI recommends for tool arguments and exactly the sort of domain-method restriction it recommends for agent internet access. It also fits naturally with GitHub’s tool allowlisting and Anthropic’s network isolation model. (OpenAI Developers)

Separate public research from private context whenever you can

One of the most important architectural patterns in AI security right now is context separation. OpenAI’s guidance says that when sensitive data is involved, teams should stage workflows—for example, perform public-web research first, then run a second call that has access to private MCP sources but no web access. That sounds almost too simple, which is probably why so many teams skip it. They want one highly capable agent to do everything in one pass. (OpenAI Developers)

The problem with the one-pass design is combinatorial risk. A single execution context that includes external browsing, private document access, internal search, long-lived credentials, and action-capable tools is exactly the kind of environment in which prompt injection becomes a systems problem. Anthropic’s browser-agent discussion and OpenAI’s data-exfiltration guidance both argue, in different ways, for reducing how many dangerous capabilities coexist at once. The more of them you split apart, the more you narrow the blast radius when one layer gets tricked. (Anthropic)

This pattern also improves human review. A reviewer can sign off on the output of a public research phase as evidence, then pass a bounded summary into a private phase that has no external communication path. That makes approvals more meaningful because they authorize a narrower scope of action. It also makes incident analysis cleaner because the provenance of each data item is easier to trace. (OpenAI Developers)

A minimal configuration for staged execution could look like this:

{
  "phases": [
    {
      "name": "public_research",
      "network": "allowlisted",
      "private_connectors": false,
      "allowed_tools": ["web_fetch", "summarize"]
    },
    {
      "name": "private_analysis",
      "network": "off",
      "private_connectors": true,
      "allowed_tools": ["search_internal_docs", "create_internal_note"]
    }
  ]
}

The details will vary across products, but the architectural idea is stable: never give a single run more authority than it needs to complete the current step. (OpenAI Developers)

Observability has to cover tool calls, approvals, and intent drift

Traditional application logs are not enough for agentic systems. They tell you that a request happened, maybe which API handled it, and perhaps which backend wrote the record. They usually do not tell you what private context the model saw, which retrieved content influenced a tool call, whether the content came from a public or private source, whether a human approval was present, or whether the agent had already been warned about suspicious content earlier in the run. (microsoft.com)

Microsoft’s Zero Trust for AI guidance calls for AI observability with end-to-end logging, traceability, and monitoring. OpenAI’s internal monitoring write-up emphasizes reviewing agent interactions for actions inconsistent with user intent or internal policies. GitHub’s agentic security principles add another critical point: actions should be attributed to both the initiating user and the agent. These ideas fit together into a practical logging requirement. Every consequential agent action should be reconstructible as a chain. Who initiated the run. Which identity the agent used. What intent layer governed the step. What sources were retrieved. What tools were called. What policy decisions were made. Which approvals existed. What was blocked. What changed as a result. (microsoft.com)

A useful event shape might look like this:

{
  "timestamp": "2026-03-30T18:02:14Z",
  "agent_id": "repo-fix-agent-prod-17",
  "initiating_user": "[email protected]",
  "role_intent": "repository_remediation",
  "organization_policy_version": "2026-03-15.4",
  "retrieval_sources": [
    {"type": "internal_repo", "id": "payments-api"},
    {"type": "public_web", "id": "docs.vendor.com/advisory"}
  ],
  "tool_call": {
    "name": "create_pull_request",
    "target": "github.com/org/payments-api",
    "risk_level": "high"
  },
  "approval": {
    "required": true,
    "status": "approved",
    "approver": "[email protected]"
  },
  "result": "pull_request_created"
}

That kind of structure does not eliminate risk, but it gives security teams what they need to review anomalies, explain incidents, and prove that a control actually existed at execution time. It also gives compliance and audit teams something far more defensible than a screenshot of a chatbot session. (microsoft.com)

On the offensive side, the same evidence requirement shows up in a different form. Teams testing MCP servers, AI gateways, browser agents, or AI-assisted developer workflows need reproducible evidence, explicit human review, and replayable verification rather than clever one-off demos. That is the part of the workflow where a human-controlled validation layer can be useful. Penligent’s public materials repeatedly position the product around human-controlled agentic workflows, evidence capture, reproducible PoCs, and verified findings rather than blind autonomy. In that narrow sense, it fits naturally into a verification chapter like this one: as workflow support for inspectable testing, not as a substitute for governance. (पेनलिजेंट)

Output validation matters because insecure output handling is still execution

OWASP’s separate category for insecure output handling deserves more attention than it gets. Security teams are comfortable thinking about malicious input, but in agentic systems the output itself is often a control message. It may become a shell command, a URL to fetch, a database query, a CI action, an issue comment that triggers automation, or a structured call to a privileged tool. If that translation happens without validation, the output is effectively code. (OWASP Foundation)

This is one reason GitHub’s “safe outputs” pattern matters. The agent does not get arbitrary write authority. It gets a path to a bounded, reviewable operation such as creating a pull request or adding a comment. Anthropic’s auto mode describes a deny-and-continue approach in which dangerous actions are blocked and the model is pushed to find a safer path rather than repeatedly attempting to route around the block. OpenAI similarly recommends schema or regex validation for tool arguments and explicit review of tool calls that go to third-party endpoints. These controls are all versions of the same principle: outputs should be constrained before they become actions. (The GitHub Blog)

A JSON schema is often enough to block entire classes of unsafe output-to-action translation:

{
  "type": "object",
  "properties": {
    "repository": {
      "type": "string",
      "pattern": "^[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+$"
    },
    "branch": {
      "type": "string",
      "maxLength": 64
    },
    "title": {
      "type": "string",
      "maxLength": 120
    },
    "body": {
      "type": "string",
      "maxLength": 5000
    }
  },
  "required": ["repository", "branch", "title", "body"],
  "additionalProperties": false
}

That schema does not make pull-request creation safe by itself. It simply ensures the agent cannot smuggle extra fields or arbitrarily shaped arguments into the tool call. Combined with allowlisted repositories, human approval, and action attribution, it becomes part of a defensible boundary. (OpenAI Developers)

Metrics that matter after AI starts handling real work

A security operating model fails quietly before it fails loudly. The way to catch that early is to measure the things the old dashboard never tracked. Classic remediation metrics are still useful, but once AI is participating in decisions and actions, you need metrics about ownership quality, approval quality, policy adherence, and runtime anomalies. (CSO Online)

The first category is governance coverage. What percentage of agents have a documented owner. What percentage have a documented role intent. What percentage of write-capable tools are approval-gated. How many skills or MCP servers exist without a named business purpose. Those metrics sound administrative until you realize they are direct leading indicators of whether an incident will be explainable later. NIST’s AI governance guidance points directly toward this sort of formalized role and policy coverage. (NIST AI Resource Center)

The second category is runtime safety. What percentage of sessions encountered suspicious untrusted content. How often were tool calls blocked by policy. How often did the model attempt a prohibited action after denial. Anthropic’s auto-mode write-up is especially instructive here because it treats repeated denials as a signal that should escalate to a human rather than as mere nuisance. That is exactly the kind of metric security teams need: not only what succeeded, but how the agent behaved when the boundary pushed back. (Anthropic)

The third category is decision quality. How often did humans override automated triage. How often were low-risk items later reclassified as urgent. How often did a proposed fix require material changes before approval. GitHub’s remediation benchmarks show the productivity upside of AI-generated fixes; the security operating model should add measurements that show whether that speed is arriving with acceptable review quality. (The GitHub Blog)

A concise metric set might look like this:

Metric	यह क्यों मायने रखती है
Explicit owner coverage	Measures whether accountability exists before incidents
Approval-gated write action coverage	Shows how much irreversible work is actually controlled
Tool-call block rate	Indicates whether policy engines are doing real work
Override rate on AI triage	Signals confidence and drift in automated prioritization
Sessions with suspicious untrusted input	Measures exposure to prompt-injection conditions
Mean time to explicit owner assignment	Tracks whether actionability is paired with accountability
Post-denial retry count	Detects boundary-pushing or misaligned optimization
Trace completeness	Measures whether incidents will be explainable after the fact

The table synthesizes the governance and monitoring ideas in NIST, Microsoft, OpenAI, GitHub, Anthropic, and the CSO piece’s description of security teams moving from manual triage to decision-system oversight. (NIST AI Resource Center)

Common mistakes that make AI deployments look safer than they are

The first common mistake is treating model quality as the main control. Better model robustness helps. Anthropic has publicly shown meaningful gains in prompt-injection resistance for browser use. OpenAI describes layered defenses against known prompt-injection techniques. None of the vendors claim the problem is solved, and none of their product-security guidance depends on model robustness alone. When teams say “our model is better now,” but still allow over-broad retrieval, unbounded tool access, and weak approvals, they are improving one layer while leaving the operating model unchanged. (Anthropic)

The second mistake is calling something human-in-the-loop because a person could theoretically intervene. If the human is not attached to a specific approval boundary, a clear audit artifact, and a meaningful rollback path, then the loop is mostly fictional. GitHub’s safe outputs and manual CI triggers are good examples because they define exactly what the human decides and when. The same is true for Microsoft’s intent-precedence model, where a user request simply cannot outrank organizational constraints. (The GitHub Blog)

The third mistake is collapsing public and private context into a single, highly privileged session. OpenAI’s staged-workflow recommendation exists because this mistake is so tempting. Teams want one elegant agent that can browse, search private content, summarize, call tools, and act. That elegance is often just a very efficient way to combine the wrong authorities in one place. (OpenAI Developers)

The fourth mistake is underestimating the non-model layers. The GitLab AI Gateway case is the cleanest reminder that orchestration logic, template expansion, runtime gateways, and extensibility surfaces can become the real exploit point. An organization can spend months debating prompt hygiene while the execution boundary lives somewhere else entirely. (about.gitlab.com)

The fifth mistake is logging prompts and outputs but not tool calls, approvals, or source provenance. That kind of partial observability creates dangerous confidence. It makes the system look measurable while leaving exactly the parts that matter during incident review invisible. Microsoft’s emphasis on AI observability, OpenAI’s focus on monitoring interactions for policy violations, and GitHub’s action attribution all point toward the same conclusion: the audit trail has to follow the action path, not just the chat transcript. (microsoft.com)

A pragmatic rollout path for security teams that need to move this year

The right rollout path is not to “secure AI” as one large project. It is to force explicit structure into the highest-risk workflows first. Start by inventorying all agentic systems that can reach enterprise data, developer machines, repositories, browsers, or external networks. If the system can retrieve, execute, or communicate, it belongs in the first pass. That includes copilots, coding agents, browser agents, gateways, MCP servers, and plugin or skill ecosystems. (OWASP Gen AI Security Project)

Next, assign unique identities and explicit owners. Then define role intent and organizational constraints for each workflow. Microsoft’s precedence model is a good default. The critical part is that the hierarchy becomes testable. If the user asks for an action outside role scope or policy, the expected behavior should be refusal or escalation, not improvisation. (TECHCOMMUNITY.MICROSOFT.COM)

Then narrow the execution boundary. Turn off internet access where it is not essential. Where it is essential, use domain and method allowlists. Remove access to data the workflow does not need. Limit write actions to reviewable safe operations. Introduce filesystem and network isolation for coding or browser agents. Treat skills, plugins, and MCP connections as privileged integrations, not user-personalization candy. (OpenAI Developers)

After that, stage sensitive workflows. Separate public research from private context. Add schema validation for tool arguments. Add approval for irreversible actions. Instrument logs so that source provenance, tool calls, approval state, and user-agent attribution are visible. Then run targeted red-team exercises focused on prompt injection, tool misuse, overbroad retrieval, and approval bypass, using real business workflows instead of contrived toy prompts. CISA has described AI red teaming as something that should fit into broader testing, evaluation, validation, and verification practices, and the rest of the ecosystem’s guidance points in the same direction. (सीआईएसए)

Finally, change what the security team is rewarded for. If analysts are still measured primarily on manual triage volume, they will spend their time competing with the machine on the wrong task. Reward explicit owner assignment, blocked unsafe actions, quality of approval decisions, policy coverage, and trace completeness. In other words, reward the work that makes AI safe to operate, not just the work that proves AI is fast. (CSO Online)

The operating model is the product now

The last few years of AI security research have produced plenty of memorable phrases: prompt injection, goal hijack, tool misuse, zero-click AI flaws, memory poisoning, agent identity abuse. Those phrases are useful, but they can also distract from the bigger point. The real shift is organizational. The systems now entering production do not simply answer questions. They retrieve, plan, choose, and act. That means the thing being secured is no longer just a model or just an application. It is a decision system with permissions. (OWASP Gen AI Security Project)

That is why AI breaks traditional security models at the agent boundary. Old models assumed slow findings, implicit ownership, cheap handoffs, and passive artifacts. New systems surface context immediately, operate across trust boundaries, and can translate untrusted text into real actions unless the surrounding controls say no. The teams that succeed will not be the ones with the most impressive demos. They will be the ones that make ownership explicit, intent precedence enforceable, tool permissions narrow, approvals meaningful, and execution traces durable enough to survive an incident review. (CSO Online)

Security teams do not need a new slogan for that. They need an operating model that assumes the model will sometimes be wrong, the content will sometimes be hostile, and the pressure to automate will only increase. In that environment, speed is not the differentiator. Controlled speed is. (microsoft.com)

Further reading

• CSO, AI is breaking traditional security models — Here’s where they fail first (CSO Online)
• NIST, AI Risk Management Framework और AI RMF Playbook (NIST)
• OWASP, Top 10 for Large Language Model Applications (OWASP Foundation)
• OWASP GenAI Security Project, Top 10 for Agentic Applications for 2026 (OWASP Gen AI Security Project)
• Microsoft Security Blog, New tools and guidance: Announcing Zero Trust for AI (microsoft.com)
• Microsoft Security Community, Governing AI Agent Behavior: Aligning User, Developer, Role, and Organizational Intent (TECHCOMMUNITY.MICROSOFT.COM)
• OpenAI Developers, Deep research, especially the prompt injection and exfiltration guidance (OpenAI Developers)
• OpenAI Developers, Agent internet access (OpenAI Developers)
• OpenAI, Keeping your data safe when an AI agent clicks a link (OpenAI)
• Anthropic, Mitigating the risk of prompt injections in browser use और Claude Code security and sandboxing (Anthropic)
• GitHub, How GitHub’s agentic security principles make our AI agents as secure as possible और Automate repository tasks with GitHub Agentic Workflows (The GitHub Blog)
• GitHub, Secure code more than three times faster with Copilot Autofix (The GitHub Blog)
• NVD and vendor records for CVE-2025-32711, CVE-2025-53773, CVE-2026-1868, और CVE-2024-37014 (एनवीडी)
• Penligent, AI Agent Security After the Goalposts Moved (पेनलिजेंट)
• Penligent, CVE-2026-1868 GitLab AI Gateway RCE The Anatomy of an AI Breach (पेनलिजेंट)
• Penligent, AI Pentest Copilot, From Smart Suggestions to Verified Findings (पेनलिजेंट)
• Penligent homepage (पेनलिजेंट)