SSH agent forwarding is convenient because it allows an administrator to authenticate through an intermediate system without copying a private key onto that system. That convenience, however, creates a trust relationship that is easy to underestimate.
CVE-2023-38408 exposed one of the most important examples of that problem.
The vulnerability affected ssh-agent in OpenSSH versions 5.5 through 9.3p1 inclusive. Under the right conditions, an attacker controlling a system to which a victim had forwarded an SSH agent could abuse OpenSSH’s PKCS#11 provider-loading functionality and ultimately achieve remote code execution on the machine where the victim’s ssh-agent was actually running. OpenSSH fixed the issue in OpenSSH 9.3p2, released on July 19, 2023. (OpenSSH)
The interesting part of CVE-2023-38408 is not simply that it is an OpenSSH vulnerability or that NVD assigns it a CVSS 3.1 score of 9.8. The vulnerability exposes a much deeper security lesson: a filesystem path that contains trusted operating-system packages is not automatically a safe code-loading boundary. NVD summarizes the weakness as an insufficiently trustworthy search path in the PKCS#11 functionality of ssh-agent, but the underlying exploitation research shows why that description matters. (NVD)
A shared object under /usr/lib may be legitimate, signed, packaged by the distribution, and completely non-malicious. It may still be unsafe to load into an arbitrary security-sensitive process.
That distinction is at the heart of CVE-2023-38408.
What Is CVE-2023-38408?
CVE-2023-38408 is a remote-code-execution vulnerability associated with OpenSSH’s forwarded ssh-agent and PKCS#11 provider loading.
OpenSSH’s own security advisory states that affected ssh-agent versions range from 5.5 through 9.3p1, and that exploitation requires two particularly important conditions:
- specific libraries must exist on the victim system;
- the victim’s SSH agent must have been forwarded to a system controlled by the attacker.
The issue was corrected in OpenSSH 9.3p2. (OpenSSH)
That immediately gives us an important distinction.
CVE-2023-38408 is not a conventional remotely exploitable vulnerability in sshd listening on TCP port 22.
An attacker cannot simply find an arbitrary Internet-facing OpenSSH server running a vulnerable version and send a packet that compromises it through this flaw.
The vulnerable component is the victim’s SSH authentication agent, and the interesting attack path appears after the victim has deliberately exposed access to that agent through agent forwarding.
This is why understanding ssh-agent forwarding is essential before discussing the exploit itself.
How ssh-agent Works
ssh-agent is an authentication agent that holds private keys for SSH public-key authentication.
Applications such as the OpenSSH client communicate with the agent through a Unix-domain socket. Instead of repeatedly reading a private key from disk, the SSH client can ask the agent to perform a cryptographic operation using an identity already loaded into it. The OpenSSH documentation describes ssh-agent as a program used to hold private keys for public-key authentication. (OpenBSD Manual Pages)
A simplified local authentication flow looks like this:
SSH Client
|
| request signature
v
ssh-agent
|
| uses loaded private key
v
Cryptographic Signature
|
v
SSH Client -> Remote Server
The private-key material does not need to be transmitted to the SSH server.
That is an important security property.
But administrators frequently need to make more complicated connections.
Suppose Alice’s workstation can reach a bastion server, and the bastion can reach an internal production server:
Alice's Workstation
|
| SSH
v
Bastion
|
| SSH
v
Production Server
Alice could copy her production private key onto the bastion.
That is usually undesirable.
Instead, SSH provides agent forwarding.
What SSH Agent Forwarding Actually Forwards
When Alice uses:
ssh -A bastion.example.com
or configures:
Host bastion.example.com
ForwardAgent yes
her private key is not copied to the bastion.
Instead, OpenSSH forwards access to her authentication agent.
OpenSSH’s documentation explicitly states that authentication passphrases and private keys themselves do not travel across the network in the normal agent-forwarding workflow. Requests reach the agent through the forwarded connection and results are sent back to the requester. (OpenBSD Manual Pages)
Conceptually:
Alice's Workstation Bastion
+-------------------+
| private SSH key |
| inside ssh-agent |
+---------+---------+
^
|
| forwarded agent protocol
|
+==============================+
|
+------v------+
| SSH_AUTH_SOCK|
| proxy/socket |
+-------------+
This allows a process running on the bastion to request authentication operations from Alice’s workstation.
It is convenient.
It is also why forwarding an SSH agent is fundamentally different from simply authenticating to a remote host.
You have extended part of your local authentication environment into that remote machine.
SSH Agent Forwarding Was Already a Security Risk
OpenSSH has long warned users to enable agent forwarding cautiously.
The current ssh(1) documentation explains that users capable of bypassing permissions protecting the forwarded agent socket on the remote machine can access the local agent through that forwarded connection. Although they cannot simply extract private-key material from the agent, they may be able to perform operations using identities loaded into it. OpenSSH explicitly suggests that a jump host using -J may be safer. (OpenBSD Manual Pages)
This is the traditional ssh-agent forwarding risk.
Imagine:
Developer Laptop
|
| ssh -A
v
Compromised Build Server
|
+----> forwarded ssh-agent
|
+----> authenticate as developer
to another reachable host
The attacker does not need the private key file.
As long as the forwarded agent remains reachable and an appropriate identity is available, the attacker may attempt to make authentication requests through it.
CVE-2023-38408 made this trust problem considerably more serious.
Instead of merely using an identity through the agent, researchers discovered that an attacker could under the right conditions make the local machine load code-related objects selected through the forwarded agent protocol.
That changes the security boundary substantially.
Where PKCS#11 Enters the Picture
PKCS#11 is an API commonly associated with cryptographic tokens, smart cards, Hardware Security Modules, and similar cryptographic providers.
OpenSSH supports adding keys exposed through PKCS#11 providers to ssh-agent.
A PKCS#11 provider is typically implemented as a shared library.
At a high level:
ssh-agent
|
v
ssh-pkcs11-helper
|
| dlopen()
v
PKCS#11 Provider Library
|
v
Smart Card / Token / HSM
Qualys explains that OpenSSH’s PKCS#11 support was introduced so the agent could add and remove PKCS#11-backed identities. Instead of loading provider code directly inside the process holding private keys, OpenSSH uses a dedicated ssh-pkcs11-helper process. That helper loads provider shared libraries using dlopen().
That process separation is itself a meaningful defensive design choice.
But it did not eliminate the danger of loading untrusted or unexpectedly behaving native libraries.
The Crucial Provider-Loading Primitive
Qualys discovered that, through a forwarded agent, a remote attacker could cause the victim’s PKCS#11 helper to attempt to load shared libraries from allowed locations on the victim machine.
This is where CVE-2023-38408 begins to depart from ordinary SSH agent abuse.
The attacker is located here:
Victim Workstation
+---------------------------------------+
| |
| ssh-agent |
| | |
| v |
| ssh-pkcs11-helper |
| | |
| v |
| local /usr/lib/... libraries |
| |
+-------------------^-------------------+
|
| forwarded agent
|
+-------------------+-------------------+
| Attacker-controlled remote machine |
+---------------------------------------+
The important inversion is that the attacker selects a provider path that is resolved on the victim’s workstation, rather than loading a library from the attacker’s own filesystem.
That initially appears relatively safe.
After all, the attacker is not simply uploading evil.so and asking ssh-agent to execute it.
There was already a path allowlist.
The problem was the assumption behind that allowlist.
Why Was There Already an Allowlist?
CVE-2023-38408 was not the first security problem involving OpenSSH PKCS#11 provider loading.
Both NVD and Qualys identify it as related to an incomplete fix for CVE-2016-10009. (NVD)
Before the earlier mitigation, provider paths were not sufficiently restricted. The earlier fix introduced an allowlist intended to constrain provider libraries to trusted system locations.
Qualys describes the historical default as matching locations such as /usr/lib* and /usr/local/lib*.
Modern ssh-agent documentation similarly exposes the -P allowed_providers option. The current default documented by OpenSSH is:
/usr/lib/*,/usr/local/lib/*
Libraries outside the configured pattern list are refused. (OpenBSD Manual Pages)
At first glance, this design seems reasonable.
An unprivileged attacker usually cannot write arbitrary shared libraries into /usr/lib.
Therefore:
Attacker-controlled /tmp/evil.so
|
X
path filter rejects it
But CVE-2023-38408 demonstrates the difference between:
trusted origin
and:
safe to execute in this context.
They are not the same thing.
Why /usr/lib Was Not a Safe Security Boundary
Qualys attempted several obvious ways to bypass the provider allowlist.
Their research says they investigated potential bugs in the path-matching logic, path traversal, and writable locations under the approved library paths. These straightforward approaches did not provide the needed bypass. OpenSSH canonicalized the requested path with realpath() before applying its filtering logic, which blocked simple traversal approaches.
So the researchers focused on something more subtle.
They asked:
What happens if we load legitimate libraries that are already present under /usr/lib?
This is where the vulnerability became much more interesting.
A shared library is not necessarily a passive collection of callable functions.
Loading it can itself cause code to run.
dlopen() Does More Than Map Bytes
Unix-like operating systems frequently use dlopen() to dynamically load a shared library into a running process.
A developer might imagine the operation conceptually as:
dlopen("provider.so")
|
v
map library
|
v
look up functions
In reality, native shared libraries can carry initialization behavior.
Constructors can execute automatically when the library is loaded.
Destructors may execute when the library is unloaded.
Libraries can register callbacks, initialize frameworks, manipulate signal handlers, allocate executable mappings, alter process state, or assume they are executing inside a very specific application environment.
Qualys identified exactly this class of behavior.
The researchers found that many normal distribution libraries have side effects when loaded or unloaded, including constructor and destructor functions automatically executed during dlopen() and dlclose().
That means:
Signed OS package
≠
Safe arbitrary plugin
and:
Located in /usr/lib
≠
Safe to dlopen() inside security-sensitive helper
This is the conceptual vulnerability behind CVE-2023-38408.
The Provider Does Not Even Need to Be a Real PKCS#11 Provider
Another critical detail comes from the OpenSSH provider-loading behavior described by Qualys.
ssh-pkcs11-helper loads the requested library and expects to find the PKCS#11 symbol C_GetFunctionList.
If that symbol is absent, the library is not a valid PKCS#11 provider and is unloaded.
The simplified operation is roughly:
request provider
|
v
dlopen(library)
|
v
find C_GetFunctionList?
|
+--+--+
| |
yes no
| |
use dlclose()
From an application’s perspective, a non-PKCS#11 library has been rejected.
From a security perspective, however, it may already be too late.
Its initialization code may already have executed.
And during unloading, additional code may execute again.
So even this apparently unsuccessful operation:
"That isn't a valid PKCS#11 provider"
can still produce meaningful changes to process state.
The Attack Surface Is Library Behavior, Not Just Library Contents
That distinction explains why simplistic mitigations failed conceptually.
If the security policy asks:
Is this library located inside an approved directory?
the answer can be yes.
If the real security question is:
Is this library safe to dynamically load and unload inside this exact process, under an attacker-selected sequence?
the answer may be very different.
Qualys systematically investigated library behavior and found several categories of side effects that could potentially be chained. Their published research discusses executable-stack behavior, signal handlers, callbacks, memory mappings and other effects of dynamically loading and unloading different libraries.
The key point is not one particular .so.
It is composition.
An attacker controlling the order of multiple library loads may gain significantly more expressive power than the security model assumed.
From a Weak Primitive to Code Execution
Initially, the primitive available to the attacker appeared extremely constrained.
The attacker did not have:
- arbitrary memory writes;
- arbitrary files on the victim;
- arbitrary library upload;
- direct shell execution;
- direct control over the helper’s environment;
- direct control over its memory layout.
Qualys describes the central primitive essentially as controlling the order in which selected system libraries were loaded and unloaded.
That sounds weak.
But native code has a huge amount of implicit state.
Consider a hypothetical sequence:
Library A loaded
|
v
changes process property
Library A unloaded
|
v
property remains
Library B loaded
|
v
registers callback
Library B unloaded
|
v
callback state becomes unsafe
Library C loaded
|
v
triggers condition
|
v
unexpected control flow
No individual library needs to contain malicious code.
The exploit emerges from interactions between valid components.
That makes CVE-2023-38408 particularly valuable as a security-design case study.
What Qualys Demonstrated
Qualys ultimately demonstrated practical remote code execution by combining side effects from libraries present in Ubuntu environments.
Their advisory states that researchers obtained reliable one-shot RCE despite mitigations including ASLR, PIE, and NX. Their strongest proof-of-concept work was performed against Ubuntu Desktop 22.04 and Ubuntu Desktop 21.10. For one Ubuntu 22.04 attack path, the research environment used the default desktop installation plus three additional packages from Ubuntu’s official Universe repository.
This detail matters because it prevents two opposite misconceptions.
The first misconception is:
Every vulnerable Linux workstation can be compromised with the exact same library sequence.
That is not established by the research.
Library availability and behavior matter.
The second misconception is:
Because special library combinations are required, the vulnerability is theoretical.
That is also incorrect.
Qualys demonstrated working exploitation against real Linux installations and OpenSSH treated the issue as a security vulnerability requiring a dedicated release. (OpenSSH)
The CVE-2023-38408 Attack Chain
The full trust chain can be modeled as follows:
1. Victim workstation
|
| runs vulnerable ssh-agent
v
2. Victim connects to remote host
|
| ssh -A / ForwardAgent yes
v
3. Agent socket becomes reachable remotely
|
v
4. Remote system is malicious or compromised
|
v
5. Attacker communicates with forwarded agent
|
v
6. Attacker requests PKCS#11 provider loading
|
v
7. Victim's ssh-pkcs11-helper loads local libraries
|
v
8. Library load/unload side effects accumulate
|
v
9. Suitable local library combination creates
exploitable process state
|
v
10. Arbitrary code execution on victim workstation
OpenSSH’s advisory confirms the two major environmental preconditions: the victim must have suitable libraries and the agent must have been forwarded to an attacker-controlled system. (OpenSSH)
Why This Is More Serious Than Traditional Agent Hijacking
Traditional ssh-agent forwarding risk usually revolves around credential use.
If a compromised server can reach your forwarded agent:
Compromised host
|
v
Victim ssh-agent
|
v
"Please sign this SSH authentication request"
The attacker may use your identity while the agent is available.
But your workstation itself remains a separate security boundary.
CVE-2023-38408 potentially changed the attack to:
Compromised host
|
v
Victim ssh-agent
|
v
Victim's local shared-library loader
|
v
Code execution on victim machine
The direction of compromise is the surprising part.
Administrators typically think:
I am connecting from my trusted laptop into an untrusted server.
CVE-2023-38408 shows how agent forwarding can create a path in the opposite direction:
The untrusted server can influence security-sensitive behavior back on my trusted laptop.
That is why SSH forwarding features should be understood as extensions of a local trust boundary rather than simple network conveniences.
CVE-2023-38408 Is Not an sshd Pre-Authentication RCE
This distinction deserves emphasis because vulnerability scanning reports often blur client and server components.
CVE-2023-38408 should not be interpreted as:
Internet attacker
|
| packet to TCP/22
v
OpenSSH sshd
|
v
instant RCE
The official OpenSSH advisory instead describes an attack involving ssh-agent, PKCS#11 support and a forwarded agent socket. (OpenSSH)
That means an organization assessing exposure must ask more than:
ssh -V
Version identification is only the beginning.
Operational exposure depends heavily on how SSH is actually used.
What Systems Were Affected?
OpenSSH states that the affected ssh-agent range was:
OpenSSH 5.5 through OpenSSH 9.3p1 inclusive.
The bug was fixed in:
OpenSSH 9.3p2. (OpenSSH)
However, enterprise Linux distributions frequently backport security patches without replacing the visible upstream version string with the latest OpenSSH release.
Therefore, this command:
ssh -V
is useful for inventory, but it is not by itself sufficient to decide whether a distribution package remains vulnerable.
For example, Ubuntu published security updates specifically addressing CVE-2023-38408 for supported releases, and Red Hat issued OpenSSH security advisories for affected enterprise products. (Ubuntu)
The correct assessment method is:
OpenSSH upstream version
+
distribution package version
+
vendor security advisory
+
local configuration
+
actual agent-forwarding usage
not merely:
version < 9.3p2 = definitely vulnerable
when dealing with vendor-backported packages.
Why CVSS 9.8 Needs Context
NVD currently lists CVE-2023-38408 with a CVSS 3.1 base score of 9.8 Critical, using the vector:
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
(NVD)
That number accurately communicates the potentially severe impact of successful remote exploitation.
But it should not replace architecture-level exposure analysis.
An enterprise should not conclude:
All machines running affected OpenSSH versions are immediately remotely exploitable.
The OpenSSH project explicitly documents additional conditions for exploitation, including agent forwarding to an attacker-controlled system and the presence of suitable libraries. (OpenSSH)
A more useful risk model is:
Risk =
vulnerable implementation
×
agent forwarding exposure
×
remote-host compromise probability
×
usable provider-loading path
×
suitable victim library state
×
resulting impact
This gives defenders a much more realistic remediation strategy.
Patch broadly, but prioritize environments where the trust chain actually exists.
Why Developer Workstations Are Especially Interesting
CVE-2023-38408 is conceptually dangerous in environments where engineers routinely move through infrastructure using SSH.
Consider:
Developer Laptop
|
v
Development Bastion
|
v
CI Worker
|
v
Production Host
If the developer globally enables:
ForwardAgent yes
then every host matching that configuration becomes part of the agent’s effective trust environment.
The most sensitive machine in the diagram may therefore not be the server.
It may be the laptop at the beginning of the chain.
The laptop could contain:
SSH credentials
Git credentials
cloud CLI sessions
source code
Kubernetes configuration
deployment tooling
browser sessions
developer secrets
CVE-2023-38408 is a useful reminder that infrastructure access controls must consider reverse trust paths back toward privileged administrator endpoints.
The Danger of Global ForwardAgent yes
A particularly poor SSH configuration pattern is globally enabling agent forwarding:
Host *
ForwardAgent yes
This makes agent forwarding the default rather than an exceptional privilege.
OpenSSH itself recommends caution around agent forwarding and notes that a jump host may provide a safer alternative for many use cases. (OpenBSD Manual Pages)
A better model is explicit per-host enablement:
Host trusted-internal-host
ForwardAgent yes
and:
Host *
ForwardAgent no
Even after patching CVE-2023-38408, minimizing forwarded-agent exposure remains valuable because the traditional agent-abuse problem still exists.
The patch fixes this specific provider-loading path.
It does not make unrestricted agent forwarding equivalent to not forwarding the agent at all.
How OpenSSH 9.3p2 Fixed CVE-2023-38408
OpenSSH 9.3p2 did more than change a filesystem pattern.
This is arguably the most important part of the fix.
The release changed the trust model.
OpenSSH’s 9.3p2 release notes state that the agent refuses requests from remote clients to load PKCS#11 modules by default. Administrators can restore the previous behavior using:
-Oallow-remote-pkcs11
(OpenSSH)
The corresponding OpenBSD source change describes the policy explicitly: remote addition of FIDO/PKCS#11 provider libraries is disabled by default, while allow-remote-pkcs11 restores the earlier behavior. (GitHub)
This is much stronger than saying:
Only load libraries from trusted paths.
The new default effectively says:
A remotely forwarded agent client does not need authority to dynamically load local provider code.
That is a better security boundary.
Why the Fix Is Architecturally Better
Before the change, the model looked roughly like this:
Remote agent client
|
v
provider request
|
v
path allowlist
|
v
local library loading
The security boundary depended heavily on determining whether a provider path was acceptable.
After the change:
Remote agent client
|
v
Is provider loading allowed remotely?
|
+---- NO ----> reject
|
+---- YES ----> provider-path controls
This introduces an authorization decision before native-code loading.
That is generally preferable.
The question changes from:
Which libraries may the remote side ask me to load?
to:
Why should the remote side be able to ask me to load native libraries at all?
For most users, the answer is that it should not.
How Does OpenSSH Know the Client Is Remote?
The fix contains another subtle detail worth understanding.
OpenSSH’s release notes explain that ssh-agent relies on the SSH client to identify that a request originates from a remote session. OpenSSH clients version 8.9 and newer support the required mechanism, but forwarding access to the agent socket using other tools can circumvent this distinction. (OpenSSH)
The current ssh-agent manual carries the same caveat: signalling that an agent client is remote is performed by ssh(1), and forwarding an agent socket with other mechanisms may bypass the restriction. (OpenBSD Manual Pages)
That means defenders should avoid interpreting the fix as a generic network-level ACL attached to an agent socket.
The security model depends partly on OpenSSH’s own session-binding behavior.
What Is allow-remote-pkcs11?
Current OpenSSH exposes the following ssh-agent option:
allow-remote-pkcs11
When enabled, clients communicating with a forwarded agent may once again load PKCS#11 or FIDO provider libraries.
By default, only local clients may perform this operation. (OpenBSD Manual Pages)
Organizations should therefore treat configurations equivalent to:
ssh-agent -O allow-remote-pkcs11
as deliberate security exceptions.
If the business does not have a concrete requirement for remote provider addition, it should generally remain disabled.
Provider Allowlists Still Have Value
OpenSSH continues to support:
ssh-agent -P allowed_providers
The current documentation describes this as a pattern list controlling acceptable PKCS#11 provider and FIDO middleware library paths, with a documented default of:
/usr/lib/*,/usr/local/lib/*
After CVE-2023-38408, the correct lesson is not that provider allowlists are useless.
The lesson is that they should not be the only security boundary.
A robust design uses both:
WHO may request provider loading?
+
WHICH providers may be loaded?
Authorization and path restriction solve different problems.
Emergency Mitigation for Vulnerable OpenSSH
OpenSSH’s original advisory provided a mitigation for environments that could not immediately patch.
Starting ssh-agent with an empty PKCS#11/FIDO provider allowlist prevents provider loading:
ssh-agent -P ''
OpenSSH also recommends that organizations that need providers configure an allowlist containing only the specific provider libraries actually required. (OpenSSH)
However, the preferred long-term action is to install a vendor-fixed OpenSSH package.
A mitigation should not become a substitute for patch management.
Practical Exposure Assessment
Organizations investigating CVE-2023-38408 should divide the assessment into several questions.
1. What OpenSSH package is installed?
Start with:
ssh -V
Then identify the distribution package:
dpkg -l | grep openssh
or on RPM-based systems:
rpm -qa | grep openssh
Do not decide patch status solely from the upstream-looking version number because distribution vendors may backport fixes.
Ubuntu and Red Hat both published distribution-specific fixes and advisories for CVE-2023-38408. (Ubuntu)
2. Is agent forwarding configured?
Search SSH client configuration:
grep -Rni "ForwardAgent" ~/.ssh /etc/ssh 2>/dev/null
Pay particular attention to:
Host *
ForwardAgent yes
A broad rule greatly expands the number of systems that receive agent access.
3. Do users routinely invoke ssh -A?
Configuration scanning will not catch command-line usage.
Operational documentation, shell history where appropriate and permitted, endpoint telemetry, and administrator interviews may be needed to understand real usage.
4. Are vulnerable developer or administrator endpoints involved?
CVE-2023-38408 is primarily interesting from the perspective of the host running the forwarded ssh-agent.
Do not focus only on the remote SSH server.
5. Has the organization explicitly restored remote PKCS#11 loading?
On modern OpenSSH installations, investigate whether agents are launched with:
-O allow-remote-pkcs11
because OpenSSH now disables this behavior for forwarded clients by default. (OpenBSD Manual Pages)
Do Not Test Only the SSH Server
This vulnerability also illustrates a common weakness in vulnerability-management programs.
A scanner may see:
Server A
OpenSSH package detected
and assign CVE-2023-38408 to that asset.
But the actual security architecture is more complicated.
The relevant scenario is:
Client Workstation A
vulnerable ssh-agent
|
| forwarded
v
Remote Server B
attacker-controlled
The system on which code may ultimately execute is Workstation A.
This is why CVE-to-asset mapping based entirely on listening services can produce misleading conclusions.
The vulnerable package, vulnerable process and attack entry point do not always reside on the same conceptual side of a connection.
Prefer ProxyJump When Agent Forwarding Is Unnecessary
OpenSSH explicitly identifies a jump host as a potentially safer alternative to agent forwarding. (OpenBSD Manual Pages)
Instead of:
Laptop
|
| ssh -A
v
Bastion
|
| uses forwarded agent
v
Target
a user can often establish a connection through a bastion:
ssh -J bastion.example.com target.internal
Conceptually:
Laptop SSH client
|
+------ encrypted transport ------+
|
Bastion
|
+------ encrypted transport ------+
|
v
Target
The bastion facilitates transport without necessarily receiving access to the user’s authentication agent in the same way.
This is an important architectural distinction.
Do Not Confuse ProxyJump With a Universal Solution
A jump host will not replace agent forwarding in every workflow.
Some interactive server-side workflows genuinely need the remote host to initiate additional SSH authentication using the user’s identity.
But many common administrative use cases do not.
For example:
"I need to SSH through Bastion A to Server B"
does not automatically imply:
"Bastion A needs access to my ssh-agent."
Those are different requirements.
CVE-2023-38408 is a good reason to audit where the latter has been enabled merely because it was convenient.
Restrict Agent Forwarding Per Destination
OpenSSH has continued improving controls around SSH keys and forwarded authentication.
Even without focusing specifically on CVE-2023-38408, organizations should adopt a principle of minimum agent exposure.
Instead of allowing an identity to be used everywhere the agent becomes reachable, modern OpenSSH workflows can make use of constraints on how identities may be used.
This does not replace upgrading OpenSSH.
It reduces the blast radius associated with future agent-related weaknesses.
The architectural goal is:
Key A
|
+--> permitted path A -> B -> C
|
X--> everything else
rather than:
Key A
|
+--> any reachable system while agent exists
Why Hardware-Backed Keys Do Not Automatically Eliminate This Class of Risk
It is tempting to conclude that using a smart card, security key, or HSM solves the problem because the private key itself cannot be exported.
CVE-2023-38408 demonstrates why that conclusion is incomplete.
The vulnerability was specifically connected to infrastructure designed to support PKCS#11-backed credentials.
The security failure was not:
private key extracted from HSM
It was:
remote client
|
v
provider-loading interface
|
v
native code loaded locally
Hardware-backed keys solve important key-protection problems.
They do not automatically make every middleware integration surrounding them safe.
The Broader PKCS#11 Path Risk
The phrase PKCS#11 path risk is useful beyond OpenSSH.
Any application that accepts a path to a cryptographic provider should ask several independent questions:
Who supplied this path?
Who controls the file?
Is the path canonicalized?
Is the file expected to be a provider?
Is its origin trusted?
Is it safe to load in this process?
Can initialization execute code?
Can unload behavior modify process state?
Can multiple providers be loaded sequentially?
What privileges does the helper possess?
CVE-2023-38408 demonstrates that checking only:
Does path start with /usr/lib?
is insufficient.
The trust relationship around native plugin loading is much larger than the pathname itself.
Trusted Software Can Become an Exploit Gadget
One of the most valuable lessons from the Qualys research is that exploit components do not necessarily need to be malicious.
A library can be:
- legitimate;
- installed from an official repository;
- useful for its intended application;
- free of a conventional vulnerability in normal use;
and still produce dangerous behavior when loaded into the wrong process.
Qualys’ investigation specifically focused on side effects from ordinary libraries and how their load/unload behavior could be composed into exploitation primitives.
This is conceptually similar to other forms of exploit composition.
The attacker’s question is not:
Is Library X malicious?
It is:
Does Library X give me a useful state transition?
That is a much more powerful way to think about system security.
Attackers Think in State Transitions
A defender may inspect several operations independently:
Load library A: allowed
Unload library A: allowed
Load library B: allowed
Unload library B: allowed
Load library C: allowed
Every individual operation passes policy.
But the attacker sees:
State 0
|
| library A
v
State 1
|
| library B
v
State 2
|
| library C
v
Exploit state
CVE-2023-38408 is therefore also a strong example of why security testing must evaluate sequences of valid operations, not just isolated invalid inputs.
Defensive Monitoring for CVE-2023-38408
Because the preferred fix has been available since 2023, the first defensive priority is patching rather than building a detection-only strategy.
Still, security teams may want to understand historical or residual exposure.
Useful telemetry includes:
OpenSSH package inventory
ssh-agent process inventory
ssh-agent command-line arguments
ForwardAgent configuration
use of ssh -A
unexpected ssh-pkcs11-helper execution
provider-library loading by ssh-pkcs11-helper
unusual child processes from SSH-related helpers
developer workstation EDR events
The presence of ssh-pkcs11-helper alone should not be treated as malicious.
It has legitimate functionality.
The detection problem is behavioral.
Security teams should look for combinations such as:
forwarded SSH session
+
unexpected PKCS#11 helper activity
+
unusual sequence of library loads
+
crashes / abnormal signals
+
unexpected execution behavior
rather than creating a naive rule that alerts every time a PKCS#11 helper starts.
Endpoint Visibility Matters More Than Server Logs Alone
A traditional SSH investigation may concentrate on:
/var/log/auth.log
sshd authentication events
failed logins
source IP addresses
Those remain useful.
But the interesting execution in CVE-2023-38408 occurs on the endpoint hosting the forwarded agent.
Therefore, endpoint telemetry from privileged administrator laptops and workstations becomes especially important.
A mature detection architecture might look like:
SSH Server Telemetry
|
|
v
+--------------------+
| Security Analytics |
+--------------------+
^
|
|
Endpoint Process Telemetry
|
+-- ssh
+-- ssh-agent
+-- ssh-pkcs11-helper
+-- library loads
+-- unexpected process activity
This correlation model is much more likely to detect cross-boundary behavior than either endpoint or server monitoring alone.
Patch Validation
The upstream fix landed in OpenSSH 9.3p2, but enterprise environments should validate using the distribution’s security metadata rather than blindly requiring that literal upstream version. OpenSSH announced 9.3p2 on July 19, 2023, while vendors such as Ubuntu and Red Hat subsequently shipped their own corrected packages. (OpenSSH)
A sensible validation process is:
Identify OS distribution
|
v
Identify installed OpenSSH package
|
v
Consult vendor CVE advisory
|
v
Confirm fixed package installed
|
v
Audit ForwardAgent usage
|
v
Verify modern ssh-agent policy
|
v
Review high-risk endpoints
This avoids both false positives and false confidence.
Security Teams Should Inventory Agent Forwarding as a Capability
Organizations often inventory:
open ports
installed packages
users
SSH keys
administrative accounts
but do not inventory credential delegation mechanisms.
Agent forwarding is one such mechanism.
A better SSH security inventory should answer:
Which users use ssh-agent?
Which keys are routinely loaded?
Which hosts receive forwarded agents?
Which bastions require forwarding?
Which workflows could use ProxyJump instead?
Which identities have destination restrictions?
How long do identities remain loaded?
Can remote PKCS#11 loading occur?
Which endpoints run vulnerable or unsupported SSH clients?
CVE-2023-38408 makes the value of this inventory obvious.
Treat Bastions as Security Boundaries
Organizations sometimes treat a bastion as safe merely because it is hardened.
But any machine receiving a forwarded SSH agent becomes part of the authentication trust chain.
If an attacker compromises that bastion while a privileged engineer connects using agent forwarding, the attacker may gain temporary interaction with the engineer’s agent.
OpenSSH’s own warning about agent-socket access makes this trust relationship explicit. (OpenBSD Manual Pages)
Therefore:
Bastion compromise
should be modeled not only as:
attacker controls bastion
but potentially as:
attacker may interact with
credential capabilities temporarily
delegated by connected administrators
CVE-2023-38408 showed that bugs in those delegated capabilities can even create attack paths back toward the administrator endpoint.
A Better SSH Administration Architecture
For many organizations, a more defensible design looks like:
+------------------+
| Production Host |
+---------^--------+
|
|
+----------------+ +--------+-------+
| Admin Endpoint |------->| Hardened Jump |
+-------+--------+ | Host |
| +----------------+
|
+-- local ssh-agent
|
+-- short-lived identities
|
+-- destination constraints
|
+-- ForwardAgent disabled by default
|
+-- patched OpenSSH
The important principles are:
Do not forward the agent merely to traverse a network boundary.
Do not grant remote hosts more agent capabilities than required.
Do not rely on filesystem location as the sole trust signal for native code loading.
Patch the endpoint, not just the SSH server.
CVE-2023-38408 vs. Private-Key Theft
Does CVE-2023-38408 allow an attacker to directly download your SSH private key from ssh-agent?
That is not the right way to describe the vulnerability.
OpenSSH’s normal agent-forwarding model deliberately avoids transmitting the private key itself, and its documentation states that an attacker accessing a forwarded agent cannot simply obtain the key material through the normal agent interface. (OpenBSD Manual Pages)
CVE-2023-38408 instead created a path toward code execution on the system running the agent.
Once arbitrary code execution on a workstation occurs, the overall security impact can obviously become much larger than ordinary agent misuse.
The distinction matters because defenders need to understand what control failed.
CVE-2023-38408 vs. SSH Key Forwarding
Another common misunderstanding is describing agent forwarding as “forwarding the SSH key.”
Technically, what is forwarded is access to the agent communication channel.
A better representation is:
WRONG mental model:
Laptop ---- private key ----> Server
versus:
BETTER mental model:
Server ---- request ----> Laptop's ssh-agent
Server <--- result ----- Laptop's ssh-agent
OpenSSH’s documentation explicitly explains that passphrases and private keys do not traverse the network in normal agent forwarding. (OpenBSD Manual Pages)
Understanding that difference makes the CVE much easier to understand.
The remote host could not simply take a private key.
But it could send requests into an API exposed by the forwarded agent.
One of those capabilities turned out to be far more powerful than expected.
Why CVE-2016-10009 Matters
The relationship to CVE-2016-10009 also contains an important lesson about incomplete fixes.
The earlier defense addressed the obvious scenario:
Attacker chooses arbitrary library
|
v
ssh-agent loads attacker-controlled code
The provider allowlist restricted where the library could reside.
CVE-2023-38408 revealed a more subtle variant:
Attacker chooses legitimate local libraries
|
v
ssh-agent helper loads them in attacker-controlled order
|
v
side effects become exploitation primitives
NVD explicitly describes CVE-2023-38408 as arising from an incomplete fix for CVE-2016-10009. (NVD)
This pattern appears repeatedly in software security.
The first patch blocks the known exploit representation.
The stronger fix removes or constrains the dangerous capability itself.
OpenSSH 9.3p2 moved much closer to the latter approach by preventing forwarded clients from remotely adding providers by default. (OpenSSH)
Security Design Lesson: Validate Authority Before Input
A weak design asks:
Remote request
|
v
Is pathname valid?
|
v
Perform privileged operation
A stronger design asks:
Remote request
|
v
Is this requester authorized
to perform this class of operation?
|
X if no
|
yes
v
Is pathname valid?
|
v
Perform operation
CVE-2023-38408 demonstrates why input validation cannot substitute for authorization.
The old provider path policy focused substantially on what could be loaded.
The 9.3p2 behavior also addresses who may initiate the load.
Security Design Lesson: Native Plugin Interfaces Are Dangerous Boundaries
Native plugin systems deserve security treatment similar to shell execution, scripting engines and other code-loading mechanisms.
Interfaces involving:
dlopen()
LoadLibrary()
shared objects
DLLs
PKCS#11 middleware
authentication plugins
PAM modules
browser native modules
database extensions
should not be treated as ordinary file-selection APIs.
A call such as:
dlopen(path, flags);
crosses a code-execution boundary.
The library’s location or vendor may reduce risk.
It does not turn loading attacker-selected native modules into a harmless action.
Security Design Lesson: Side Effects Are an API Too
Applications often reason only about documented return values.
For example:
load provider
|
v
provider invalid
|
v
return error
From the application’s logical perspective, nothing happened.
From the process’s perspective, however:
constructor executed
signal handler changed
memory mappings changed
destructor executed
callbacks registered
other global state modified
may already have occurred.
Qualys’ work on CVE-2023-38408 is a particularly clear example of security-relevant behavior existing outside the nominal API contract.
Recommended Remediation
For practical defensive work, remediation can be summarized as follows.
First, install the vendor-fixed OpenSSH package.
Upstream corrected the vulnerability in OpenSSH 9.3p2. Linux distributions including Ubuntu and Red Hat shipped corresponding security fixes through their normal package channels. (OpenSSH)
Second, disable SSH agent forwarding where it is unnecessary.
Do not use:
ForwardAgent yes
globally.
Third, prefer ProxyJump for network traversal when the intermediate host does not genuinely need agent access.
OpenSSH itself points to jump hosts as a potentially safer alternative. (OpenBSD Manual Pages)
Fourth, do not restore remote PKCS#11 provider loading unless there is a specific requirement.
Modern OpenSSH disables it for forwarded clients by default. (OpenBSD Manual Pages)
Fifth, restrict provider paths where PKCS#11 or FIDO middleware is required.
The -P option allows administrators to constrain provider libraries. (OpenBSD Manual Pages)
Sixth, treat developer and administrator workstations as part of the SSH attack surface.
CVE-2023-38408 is fundamentally a client-side agent problem, even though the attacker may operate from a remote server.
Frequently Asked Questions
Is CVE-2023-38408 remotely exploitable?
Yes, under specific conditions.
OpenSSH states that remote exploitation requires the victim’s agent to have been forwarded to an attacker-controlled system and requires suitable libraries to be present on the victim machine. (OpenSSH)
It is therefore not equivalent to an unauthenticated sshd vulnerability reachable directly from the Internet.
What OpenSSH versions are vulnerable to CVE-2023-38408?
OpenSSH lists versions 5.5 through 9.3p1 inclusive as affected.
OpenSSH 9.3p2 contains the upstream fix. (OpenSSH)
Distribution packages may contain backported patches, so consult the relevant operating-system security advisory.
Does the attacker need my private SSH key?
No.
The attack relies on access to a forwarded ssh-agent, not possession of the private key file.
The normal forwarding mechanism keeps private-key material on the system running the agent. (OpenBSD Manual Pages)
Does ssh -A send my private key to the server?
No.
ssh -A forwards access to the authentication agent. OpenSSH states that private keys and authentication passphrases themselves are not sent through the normal forwarding mechanism. (OpenBSD Manual Pages)
Is agent forwarding safe after patching?
The specific CVE-2023-38408 provider-loading vulnerability is addressed by the relevant OpenSSH fixes, but agent forwarding still has inherent security implications.
OpenSSH continues to warn that users capable of accessing the remote agent socket can perform operations using identities available in the local agent. (OpenBSD Manual Pages)
Therefore, agent forwarding should still be used selectively.
What does PKCS#11 have to do with SSH?
PKCS#11 allows applications to interact with cryptographic providers such as smart cards, tokens and HSMs.
OpenSSH can use PKCS#11 provider libraries to expose cryptographic identities through ssh-agent. Qualys’ research focused on how the helper process responsible for this functionality dynamically loaded provider libraries.
Why was /usr/lib unsafe?
The libraries were not necessarily malicious.
The problem was that normal shared libraries can execute constructors, destructors and other initialization or cleanup behavior when dynamically loaded and unloaded.
Qualys demonstrated that side effects from legitimate distribution libraries could be combined into exploitation chains.
Was this a path traversal vulnerability?
That is an oversimplification.
Qualys specifically investigated traversal as a potential allowlist bypass and found that OpenSSH’s use of realpath() prevented the straightforward traversal approach they tested. Their successful exploitation instead relied on side effects from libraries already inside allowed locations.
What is the easiest mitigation if patching is temporarily impossible?
OpenSSH’s advisory recommends starting ssh-agent with an empty provider allowlist:
ssh-agent -P ''
or configuring a narrowly scoped allowlist containing only required provider libraries. (OpenSSH)
This should be considered a mitigation while the underlying OpenSSH package is brought to a vendor-fixed state.
Should I replace ssh -A with ssh -J?
Where your requirement is simply to reach a destination through a bastion, often yes.
OpenSSH itself documents -J as a potentially safer alternative in its agent-forwarding warning. (OpenBSD Manual Pages)
If a process on the intermediate machine genuinely needs access to your authentication agent, the two features are not functionally identical.
Final Takeaway
CVE-2023-38408 is more than an old OpenSSH RCE with a high CVSS score.
It is a useful demonstration of what happens when SSH credential delegation, native library loading, filesystem trust and remote control intersect.
The vulnerability affected OpenSSH ssh-agent versions 5.5 through 9.3p1. A user first had to forward an agent to an attacker-controlled system, and suitable libraries needed to exist on the victim host. But once those conditions aligned, Qualys demonstrated that attacker-selected loading and unloading of legitimate local libraries could be transformed from an apparently narrow PKCS#11 primitive into remote code execution. (OpenSSH)
The most important technical lesson is simple:
A trusted library is not necessarily safe to load in an attacker-controlled context.
CVE-2016-10009 had already motivated OpenSSH to restrict provider paths. CVE-2023-38408 showed that filesystem provenance alone did not solve the deeper problem. OpenSSH 9.3p2 consequently changed the authorization model and disabled remote PKCS#11/FIDO provider loading through forwarded agents by default. (NVD)
For defenders, the remediation is equally clear: keep OpenSSH packages patched, minimize ForwardAgent usage, avoid global agent forwarding, prefer jump-host designs when the intermediate machine does not require agent access, and treat every forwarded authentication agent as an extension of the local endpoint’s security boundary.
For security engineers, CVE-2023-38408 offers an even broader lesson.
The most dangerous attack surface is not always an obviously malicious input.
Sometimes the attacker wins by taking perfectly legitimate operations, perfectly legitimate libraries and perfectly legitimate features—and arranging them in an order the original security model never expected.
English 
German 
French 
Spanish 
Portuguese 
Japanese 
Korean 
Hindi 
Arabic 
Turkish 
Hebrew