Secrets Management

Secrets management is the practice of storing, delivering, rotating, and revoking the credentials software uses to authenticate: API keys, database passwords, tokens, and SSH keys. Also called secret management, or secrets as a service when hosted, it keeps credentials encrypted outside code and issues them to authorized workloads at runtime, so access can be scoped, audited, and revoked.

A secrets manager such as HashiCorp Vault or AWS Secrets Manager solves storage. It does not control what code runs after a secret is delivered. A secrets manager authenticates the workload, not the code inside it. Every library loaded into your process shares the workload's identity, environment, and file permissions, so a malicious dependency can request the same secrets your application can. The worm in the TanStack compromise in May 2026 did exactly that. It targeted AWS Secrets Manager, SSM Parameter Store, and in-cluster HashiCorp Vault, using credentials it harvested from the instance metadata service and the pod's Kubernetes service account token.

Our position: storage is the solved part of secrets management. The failures in 2026 incidents are delivery and revocation. Secrets reach processes that run untrusted dependency code, and leaked secrets get deleted instead of revoked. The LiteLLM compromise began with an administrative key committed to a public repository on May 27, 2026. The team deleted the file within hours, but, in the words of LiteLLM's postmortem, "nobody followed through on revoking the administrative key." The key still worked 136 days later.

Secrets management is the lifecycle discipline for bearer credentials: create, store, deliver, use, rotate, and revoke. A bearer credential grants access to whoever holds it. API tokens, cloud access keys, database connection strings, and SSH private keys all work this way. No second factor stands between possession and use.

Each lifecycle stage has a job:

• Create: generate the credential with the narrowest scope and shortest lifetime the consumer can tolerate.
• Store: keep it encrypted at rest, usually under a key held in a KMS or HSM, with an access policy and an audit log.
• Deliver: hand it to the workload at runtime as an environment variable, a mounted file, or an API response.
• Rotate: replace it on a schedule, without downtime, so every consumer picks up the new value.
• Revoke: invalidate it immediately when it leaks, which is the step most often skipped.

Secrets as a service is the hosted form of the storage and delivery stages. AWS Secrets Manager, Google Cloud Secret Manager, and Azure Key Vault are managed offerings. HashiCorp Vault and its open source fork OpenBao are the common self-hosted choices. All of them add versioning, access policies, audit logs, and scheduled rotation on top of encrypted storage.

Every secrets manager has to answer one question first: how does the caller prove who it is? If the answer is another stored secret, such as a Vault token in a config file, you have moved the problem rather than solved it. This bootstrap credential is usually called secret zero. The modern answer is platform identity. An EC2 instance gets an IAM role through the instance metadata service at 169.254.169.254. A Kubernetes pod gets a service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. A CI job gets a signed OpenID Connect token from its CI provider. The secrets manager trusts the platform's attestation instead of a stored password.

Platform identity removes secret zero, and it also makes identity ambient. Any code running in the workload can reach the metadata service, read the mounted token, and call the secrets manager. We think the trade is right, because a stored bootstrap secret is worse. It does mean a secrets manager's access policy can never be narrower than the workload's identity.

Secrets managers deliver secrets in one of three ways, and all three end with the plaintext inside your process. The OWASP Secrets Management Cheat Sheet ranks environment variables last, because "environment variables are generally accessible to all processes and may be included in logs or system dumps." Files and runtime API calls are better, but only against different threats.

Environment variables are worse than most guides admit. A common recommendation is to read the secret once at startup and then delete it from the environment. We tested that advice on Linux with Node.js 22:

const password = process.env.DB_PASSWORD; // read the secret once at startup
delete process.env.DB_PASSWORD;           // then scrub it, as many guides advise
console.log('process.env.DB_PASSWORD:', process.env.DB_PASSWORD);
setInterval(() => {}, 1000);              // keep serving requests

$ DB_PASSWORD=s3cr3t-from-vault node app.js &
[1] 618
process.env.DB_PASSWORD: undefined
$ tr '\0' '\n' < /proc/618/environ | grep DB_
DB_PASSWORD=s3cr3t-from-vault

The scrub worked inside the process and failed everywhere else. The kernel copies the environment into a new process's memory when execve(2) starts it. The file /proc/<pid>/environ reads that original copy. The proc_pid_environ(5) man page says later changes made with calls like putenv(3) are not reflected. Python behaves the same way: os.environ.pop() leaves the value in /proc/self/environ.

Access to /proc/<pid>/environ is governed by a ptrace read-mode check, which in practice means the same user ID. In our test, a process running as a second user got "Permission denied." Any process running as the same user, including a child spawned by a malicious package, read the secret.

Attackers read /proc/<pid>/environ directly. The payload in the Trivy GitHub Actions compromise in March 2026 found the runner's processes, read each /proc/<pid>/environ, and kept variables whose names contained env or ssh. On GitHub-hosted runners, which grant passwordless sudo, it went further. It dumped the memory of the Runner.Worker process through /proc/<pid>/mem and searched it for "isSecret":true, the marker GitHub Actions uses for secrets in its own heap.

Figure 1 shows a secret's path from a secrets manager into a running application and where the manager's control stops. The secrets manager encrypts the secret at rest, enforces an access policy, and logs every read. It authenticates the caller by workload identity: an IAM role, a Kubernetes service account token, or a CI OIDC token. It then delivers the secret as an environment variable, a mounted file, or an API response. At that point the manager's control ends. Inside the process, the application code, its direct dependencies, its transitive dependencies, and any child processes they spawn all share one user ID, one cloud identity, and one memory space. Any of them can read process.env, /proc/<pid>/environ, mounted secret files, and the metadata endpoint at 169.254.169.254, and can call the secrets manager with the workload's own identity. A malicious transitive dependency can then send what it collects to an attacker.

The delivery methods still differ in what they expose and for how long:

Verdict: use short-lived federated credentials or dynamic secrets wherever the target system supports them, and deliver static secrets as files rather than environment variables. Fetch each secret in the smallest process that needs it, at the point of use. None of these methods protects a secret from code already running inside the workload. That boundary is set by what you install, not by how you deliver.

Supply chain attacks steal managed secrets by running inside the workload that is authorized to read them. The attacker does not need to break the secrets manager. A malicious package executes as your application and asks for what your application is allowed to have.

The 2026 payloads show the pattern clearly. The TanStack worm implemented the IMDSv2 token handshake correctly, so it worked against instances that had disabled IMDSv1. It queried AWS Secrets Manager and SSM Parameter Store. It also targeted Vault at vault.svc.cluster.local:8200 and 127.0.0.1:8200. Our analysis describes the chain: steal the Kubernetes service account token, use it to authenticate to Vault, then retrieve the secrets Vault manages. The collector in LiteLLM 1.107.0 listed Vault, Terraform, container-mounted secrets, non-empty environment variables, and the environments of other readable processes among its 16 collection modules.

Both payloads are infostealers delivered through trusted packages, and both ran before any application code did. The TanStack variant used a prepare script in a git dependency. LiteLLM 1.107.0 shipped a .pth file that Python executes at interpreter startup, even if the application never imports LiteLLM. Install time and startup time are where most supply chain attacks execute. Default-on install scripts were a design mistake, and npm v12 corrected it in July 2026 by turning dependency lifecycle scripts off by default, following pnpm 10. Python startup hooks and code that runs on import are still open paths.

CI runners are the richest target, because they hold deploy credentials for many systems at once. GitGuardian's State of Secrets Sprawl 2026 found that 59% of machines compromised in the second Shai-Hulud wave were CI/CD runners, not personal workstations.

The defensive consequence is that secrets management and dependency control enforce the same boundary. A secrets manager cannot tell your code from code you installed. Socket's Environment variable access alert flags packages whose code reads process.env, os.environ, or their equivalents, so a reviewer can ask why a color library wants NPM_TOKEN before it is installed. Most packages that read the environment are benign. The alert is a prompt for review, not a verdict.

Automate secrets management in DevOps by replacing stored CI secrets with workload identity federation, then issuing short-lived credentials per job. The CI provider signs an OpenID Connect token that describes the job: repository, branch, environment, and workflow. The cloud provider checks those claims against a trust policy and returns temporary credentials. GitHub's OIDC documentation describes the result: "your cloud provider issues a short-lived access token that is only valid for a single job, and then automatically expires."

Federation removes the long-lived secret from CI storage. It does not remove the credential from the runner. As of October 2026, aws-actions/configure-aws-credentials v6 exports AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN into the environment of every later step in the job, valid for 1 hour by default, according to its README. Any dependency installed or executed after that step inherits them.

So the design rule is job separation. Build and test, which install third-party code, run without any ability to mint cloud credentials. Deployment, which needs credentials, installs nothing:

permissions:
  contents: read            # default for every job: no OIDC

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>
      - run: npm ci && npm test   # third-party code runs here, with no cloud identity
      - run: npm pack
      - uses: actions/upload-artifact@<full-commit-sha>
        with: { name: dist, path: '*.tgz' }

  deploy:
    needs: test
    runs-on: ubuntu-latest
    environment: prod                # the trust policy pins sub to this environment
    permissions:
      id-token: write                # only this job can request an OIDC token
      contents: read
    steps:
      - uses: actions/download-artifact@<full-commit-sha>
        with: { name: dist }
      - uses: aws-actions/configure-aws-credentials@<full-commit-sha>  # v6
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-prod
          aws-region: us-east-1
          role-duration-seconds: 900 # the minimum, instead of the 3600 default
      - run: aws s3 cp *.tgz s3://example-releases/   # no npm install in this job

Job separation keeps credentials away from install-time code. It does not make the artifact trustworthy, because the test job that built it ran every dependency's code. That is a build integrity problem, which provenance documents but does not solve.

On the cloud side, the role's trust policy should match the token's sub claim exactly, for example repo:example-org/example-repo:environment:prod, rather than a wildcard over the whole organization.

Federation has a cost, and the 2026 incidents measured it. In the TanStack compromise, attacker code ran during a workflow's test and cleanup phase, extracted an OIDC token from the runner process's memory, and published malicious versions through the project's own trusted publisher binding. TanStack's postmortem says no npm token was stolen. As we wrote when npm revoked classic tokens, "no long-lived token" does not automatically translate into "no publish risk." For package publishing, pair trusted publishing with npm's staged publishing, generally available since May 2026, where a human approves the release with two-factor authentication before it goes public.

Dynamic secrets extend the same idea to systems without OIDC federation. Vault's database secrets engine creates a fresh database user for each lease and drops it when the lease expires or is revoked. A stolen dynamic credential expires on its own. The trade-off is operational: the database must allow Vault to create users, and every consumer must handle credential renewal. All of this is the secrets half of CI/CD security. The other half is controlling which code runs in the job.

An SSH private key is a bearer credential stored as a file, usually in ~/.ssh, usually with no expiry. Credential stealers routinely collect them. The Trivy payload harvested SSH keys from runners, and the LiteLLM 1.107.0 collector listed SSH and GPG private keys alongside cloud credentials.

A passphrase protects a key file at rest, but ssh-agent holds the decrypted key while it is unlocked, and any process running as the same user can use the agent socket. FIDO2 hardware keys, created with ssh-keygen -t ed25519-sk, require a physical touch for each signature, which defeats silent reuse. SSH certificates attack the lifetime instead. A certificate authority signs a user's public key with a validity window, and servers trust the CA rather than individual keys:

$ ssh-keygen -s user_ca -I alice@example.com -n alice -V +1h id_ed25519.pub
Signed user key id_ed25519-cert.pub: id "alice@example.com" serial 0 for alice valid from 2026-10-31T08:44:00 to 2026-10-31T09:45:33

Each server needs one line in sshd_config, TrustedUserCAKeys /etc/ssh/user_ca.pub. If a stealer copies the key and its certificate, the pair stops working when the hour is up. The cost is that the CA's private key becomes the most valuable secret in the system, and it needs an issuance service, an HSM or KMS, and its own rotation plan.

Rotate secrets after a compromise by revoking every credential the compromised environment could reach, all at once, from a clean machine, before issuing replacements. Deleting a leaked value from a file, a commit, or a log does nothing to the credential. Only revocation does.

The order matters, and each step below exists because an incident went wrong without it:

1. Scope by reach, not by evidence. Treat every secret the process, user, or runner could read as exposed: environment variables, credential files such as ~/.npmrc, ~/.aws, and ~/.ssh, AI tool configuration, metadata endpoints, and every path its identity could fetch from the secrets manager. The two malicious LiteLLM workflows referenced 20 distinct secret names, including database passwords scoped to individual GitHub environments.
2. Revoke from a clean environment. A compromised host may still be running a loader. LiteLLM 1.107.0 started a background loader that retried a remote code fetch every 3,600 seconds, and our analysis advised revoking exposed credentials from a clean environment.
3. Revoke before you reissue, in one step. After a first breach of Trivy's CI in March 2026, maintainers rotated credentials, but the rotation was not fully atomic. The attacker kept access to newly issued credentials and used it to force-push 75 of 76 version tags in aquasecurity/trivy-action.
4. Revoke credentials that mint credentials. A GitHub App private key signs the JWTs that mint installation tokens, so revoking issued tokens without rotating the key leaves the attacker able to mint new ones. The same applies to OIDC trust bindings, SSH CA keys, OAuth refresh tokens, and saved browser sessions. One stolen LiteLLM secret was a Playwright storage state file holding live session cookies.
5. Read the audit logs for the exposure window. Check cloud audit trails, registry publish history, and the source control organization's audit log for use of each credential between the earliest possible exposure and its revocation.

The gap between a leak and its revocation decides the outcome. In 2024, a Python Software Foundation administrator leaked a GitHub token in a public Docker image, with admin access to the repositories of CPython, PyPI, and the PSF. In the PSF token leak, the token was revoked within 17 minutes of the security team being notified. The LiteLLM key stayed valid for 136 days. GitGuardian found that 64% of the valid secrets it detected in public repositories in 2022 were still active four years later. Detection through secret scanning shortens the window only when someone acts on the finding.

Figure 2 reconstructs the LiteLLM attack chain from LiteLLM's postmortem and Socket's analysis of the earlier workflows. Every step after the first depended on a credential that should already have been dead. The table below carries the same timeline, in UTC.

Revoking the May 27 key at any point in the following 136 days would have stopped the chain at step one. Revoking the owner's token after October 10 would not have, because by then the attacker held a GitHub App private key and other maintainers' tokens. A compromise widens with every hour it goes unrevoked, so the cost of late rotation compounds. The supply chain incident response playbook for this is short: revoke wide, revoke early, revoke once.

Teams skip revocation for an operational reason, not a security one. GitGuardian's 2026 report puts it bluntly: teams cannot rotate secrets without risking production outages, so they don't. A secret with an unknown number of consumers cannot be rotated during an incident without breaking something, and so it is not rotated at all.

The fix is to make rotation a routine deployment. Every secret should have a known owner, a known list of consumers, and a rotation that has run in production at least once. Measure time to revoke for each class of secret, the way you measure time to restore for a service. A secret you cannot revoke within an hour is a standing liability, whatever vault it is stored in.

The hardest open problem is the credentials that mint other credentials. GitHub App private keys, SSH CA keys, OIDC trust policies, and the owner tokens that administer them are rarely rotated, because each sits upstream of many systems. They are also exactly what LiteLLM's attacker went after first. The most important secret in your organization is probably the one whose rotation nobody has ever tested.