ペンリジェント・ヘッダー

SANSAR Linux Cryptomining: Ray Exposure and GPU Telemetry Checks

A GPU host can report little activity while an attacker uses its compute. Elastic's SANSAR investigation describes an intrusion in which an exposed Ray job agent provided command execution, a modified NVIDIA Management Library changed monitoring results, and a kernel component concealed selected host evidence. The operational problem is deciding which observations remain trustworthy after the workload host becomes untrusted.

について Elastic Security Labs investigation records probing and job execution between October 22 and 25, 2026. Its toolkit includes a Rust orchestrator named SANSAR, CPU and GPU miners, an NVML wrapper, a kernel rootkit and a separate backdoor. These are observations from that investigation, not evidence that every exposed Ray installation has the same tools or that NVIDIA firmware contains a newly disclosed vulnerability.

Inventory the job agent separately from the dashboard

In Elastic's case, the head-node dashboard listened on loopback while the job agent remained reachable from the Internet and accepted unauthenticated job submissions. A local dashboard binding therefore did not establish that the job submission surface was private. The first relevant inventory question is which services accept work, not whether the expected web interface appears inaccessible.

Ray's security documentation treats its developer services as interfaces for trusted parties and says security and isolation must be enforced outside the cluster. Apply that model to the actual deployment: head nodes, workers, job agents, gateways, security groups and service publication. A diagram showing one private endpoint is incomplete if a second route reaches the same execution authority.

Use the platform owner's deployment records before conducting any active checks. Record each job-related service's bind address, advertised address, ingress rule, workload identity and intended callers. Keep ports and authentication behavior deployment-specific; neither a service name nor an example from a different Ray version establishes your live configuration.

An outside caller can reach a published job agent while the dashboard is bound to loopback.
Reported Ray deployment relationship in Elastic’s investigation; port and policy details are deployment-specific.
ObservationEvidence to collectDecision it supports
Dashboard is bound locallyActive listener and service configurationScope of that dashboard listener
Job agent has a public routeService publication and network policyWhether untrusted callers can reach job acceptance
Gateway requires an identityDeployed policy and authorized test resultProtection on that gateway path
Worker accepts another routeWorker service inventory and access rulesWhether the gateway can be bypassed
Cluster runs submitted codeJob identity and process ownershipAuthority the accepted workload receives

These rows form a proposed review worksheet. They are not measured findings about a reader's cluster. A gateway's expected rejection and a legitimate user's normal job should be checked in an approved nonproduction environment, without submitting an attacker payload or contacting unrelated hosts.

Explain what an NVML reading actually measures

NVIDIA describes NVML as the management library underlying GPU monitoring tools, including nvidia-smi. Elastic's wrapper changed information returned through this layer. Two dashboards that both consume the affected library can agree because they share an altered source.

Agreement is useful only when the observation sources have the independence needed for the conclusion. A process list, an application metric and a collector running on the same compromised host may each be valuable, but none should automatically override evidence that the host's reporting stack was modified. Similarly, a monitoring gap is an investigation lead, not proof of malicious activity on its own.

Build an evidence-dependency map. Identify which collectors use the host library, which depend on the kernel, which records are retained outside the workload host, and which administrators can change each source. This reveals whether an apparent cross-check is just the same data presented twice.

Two NVML-based dashboards share a dependency; separately retained infrastructure records offer a different comparison.
Proposed evidence-dependency review informed by the reported NVML manipulation; separate logs also have coverage limits.
Evidence sourceUseful comparisonImportant limit
NVML-based GPU metricsExpected job and reported utilizationShared library integrity affects callers
Installed library identityTrusted package or image provenanceA host-side checksum alone cannot establish its own trust
Job scheduling recordsAuthorized workload, account and execution intervalCoverage depends on logging and retention
Kernel and boot artifactsApproved modules and recovery imageRoot-level compromise can affect local visibility
Independently retained infrastructure recordsHost timeline and external activityDifferent sources may cover different intervals

Do not prescribe one universal mismatch threshold. Normal workloads, sampling, batching and collection failures can all change GPU readings. The actionable result is an unexplained discrepancy tied to known source dependencies and a defined investigation owner.

Preserve the path from workload to host change

Elastic reports that the attacker first submitted reconnaissance and a reverse-shell job, later obtained root, and deployed components affecting libraries and the boot environment. The report does not make initial job submission synonymous with immediate root privileges. Preserve that distinction when reconstructing an incident: job identity, privilege changes and subsequent system modifications require their own evidence.

Retain approved job history, relevant service logs, available process ancestry, package or image references, and trusted copies of boot configuration. Record collection time and source integrity. An endpoint command executed after kernel tampering may still provide leads, but it should not be presented as an independent clean-state attestation.

Prioritize evidence already retained outside the host. Reconcile that history with local findings rather than treating missing local entries as a negative result. When a collector was absent or retention expired, label the interval unobserved and identify what decision remains open.

Recover both compute and reporting authority

Separate containment, restoration and acceptance. Network isolation can stop further untrusted job submission while an investigation continues. Replacing the workload image addresses only the components that image actually contains. A response involving altered kernel or early-boot components must account for the host and boot trust path, using the organization's recovery process.

Before returning a GPU node to service, identify the trusted operating-system source, driver and library provenance, Ray deployment configuration and workload identities. Review exposed credentials according to the authority the affected account could access. A restored filesystem does not revoke a credential previously copied elsewhere.

Run a small organization-owned workload with a known schedule and ordinary success criteria on the recovered environment. Check that authorized submission works, unauthorized access is rejected at the intended boundary, and monitoring records agree with the controlled job. This is an acceptance test for the restored path; it cannot prove the absence of all earlier attacker activity.

Keep an evidence-based closure record

One row per node should connect job-entry routes, intended callers, the trusted host image, GPU library provenance, the controlled workload and the source of each observation. Include unresolved history and the owner accepting it. Avoid a fleet-wide conclusion from a single node or one utilization screenshot.

For an authorized assessment of surrounding web and API entry points, 寡黙 can be considered within the defined application-testing scope. Host forensics, GPU library integrity and boot recovery require their own evidence and responsible teams.

Method: AI-assisted defensive analysis prepared November 16, 2026 from Elastic's original research and Ray/NVIDIA documentation. The surface and telemetry matrices are proposed review methods. No malware, affected host or customer cluster was executed or tested.

For a separate AI administration boundary, the GPUStack proxy trust and authentication review explains how to distinguish management access from normal model serving. Its authentication fixes do not validate a compromised GPU host.

General product workflow

The official Penligent 3.0 video demonstrates its general product workflow. It does not reproduce SANSAR or validate GPU host recovery.

記事を共有する
関連記事
jaJapanese