One pip install to Lose Everything: The LiteLLM Supply-Chain Breach

Written for educational and defensive security purposes. Code snippets are reconstructions for illustration, defanged and annotated for clarity.

What Made This Attack Different

Most breaches begin with something that feels suspicious. A phishing email. An exposed port. A guessed password. The attacker comes straight at you.

The LiteLLM breach looked like normal software development.

A developer installed a Python package from the official registry. A CI pipeline ran a security scanner. A package maintainer's automation published a release. Nothing in that description sounds reckless, and that is precisely the point.

For roughly three hours on March 24, 2026, malicious versions of litellm (versions 1.82.7 and 1.82.8) were available on PyPI. Anyone who installed them received code designed to steal credentials, establish persistent backdoors, and, in Kubernetes environments, potentially take over entire clusters. But by the time those packages appeared on PyPI, the real compromise had already happened several steps earlier, through a chain of trusted automation.

This is the story of that chain.

Background: The Concepts You Need

Before diving into the attack, a few foundational pieces need to be clear.

PyPI and Why It's a High-Value Target

PyPI (the Python Package Index) is the registry pip downloads from. Anyone with an account can publish a package, there is no mandatory code review. The system trusts that account ownership equals legitimacy. If an attacker steals a publisher's credentials, they can upload malicious code under a trusted package name, and PyPI serves it faithfully.

CI/CD Pipelines and Runners

CI/CD (Continuous Integration / Continuous Deployment) is the automation that runs when developers push code: tests, builds, security scans, and package publication. GitHub Actions is the most widely used system. Workflows run on ephemeral runners, temporary virtual machines, that are entrusted with sensitive credentials while they're alive.

A runner is not just a build machine. It is a machine temporarily holding authority, often including API tokens, SSH keys, cloud credentials, and package publishing secrets.

The Critical Flaw in Version-Tagged GitHub Actions

Many teams pin third-party GitHub Actions to version tags, thinking this is safe:

- uses: aquasecurity/[email protected]

It looks disciplined. But a Git tag is a mutable label. Anyone with write access to the repository can force-push a tag to point to a completely different commit silently, without changing any workflow files downstream.

The only truly immutable reference is a full commit SHA:

# ✅ This 40-character SHA identifies one exact commit — forever
- uses: aquasecurity/trivy-action@dc2f32e5f4972f7afa0bf9af52c9f2d94e2eb1c3

That distinction was the root cause of the entire LiteLLM chain.

The Target: Why LiteLLM?

LiteLLM is a Python library that provides a single unified interface for multiple AI model providers (OpenAI, Anthropic, Google, Azure, etc.):

import litellm

# Same function call, regardless of provider
response = litellm.completion(model="gpt-4o", messages=[...])
response = litellm.completion(model="claude-opus-4-6", messages=[...])

In many organizations, LiteLLM is deployed as a centralized AI gateway — a shared service that holds API keys for all providers, handles rate limiting, and routes requests. Compromise one such deployment and you potentially access credentials worth tens or hundreds of thousands of dollars in cloud AI compute. At the time of the attack, LiteLLM was downloaded approximately 3.4 million times per day.

The Threat Actor: TeamPCP

TeamPCP is a financially motivated cybercriminal group that emerged around December 2025, operating under multiple aliases (PCPcat, Persy_PCP, ShellForce, DeadCatx3). The string "TeamPCP Cloud stealer" is embedded directly in their malware — deliberate self-attribution.

Their early operations focused on unsophisticated scanning attacks against exposed Docker APIs and Kubernetes dashboards. By March 2026, they had made a significant leap to targeted supply-chain operations, featuring two notable innovations:

• CanisterWorm: A C2 channel built on the Internet Computer Protocol (ICP). Unlike normal C2 domains, ICP canisters cannot be seized or taken down.
• AI-assisted targeting: The group reportedly uses an AI agent (openclaw) to automate reconnaissance. One of the first documented cases of an AI agent used operationally in a supply chain attack.

The Full Attack Timeline

Phase 0: The Entry Point — Pwn Requests

The chain started with a Pwn Request: a malicious pull request that exploits a GitHub Actions misconfiguration involving pull_request_target.

Normally, pull requests from forks run with read-only permissions and no secret access. pull_request_target was designed to allow limited operations (like adding labels) on fork PRs using the trusted repo's context. The danger emerges when a developer makes this mistake:

# ❌ VULNERABLE PATTERN
on:
  pull_request_target:   # Runs with repo write permissions

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          # ↑ This runs the ATTACKER'S code with full secret access

The result: aqua-bot credentials were stolen, giving the attackers write access to all of Aqua Security's GitHub repositories, including Trivy.

Phase 1: Poisoning Trivy

With aqua-bot credentials in hand, the attackers force-pushed malicious commits to 76+ Trivy release tags simultaneously. The attack was invisible to downstream users because their workflow files still said @v0.69.4, but that tag now pointed to attacker-controlled code.

The malicious Trivy action did two things on every affected CI runner:

1. Memory scraping: it read the GitHub Actions Runner.Worker process memory via /proc/<PID>/mem to extract secrets stored as environment variables. This bypasses masking entirely.

2. Filesystem sweeping: it looked for credential files at well-known locations (SSH keys, .env files, AWS credentials).

Everything collected was encrypted with a hybrid AES-256 + RSA-4096 scheme and exfiltrated to scan.aquasecurtiy[.]org (note the transposed letters designed to evade log review). A fallback used the victim's own GITHUB_TOKEN to create a public GitHub repo and push the stolen data there.

One of the secrets stolen from LiteLLM's CI runner was the PYPI_PUBLISH token.

Phase 2: Why Every Standard Defense Failed

Before explaining what the malicious packages did, it's worth understanding why the usual defenses were ineffective.

The only control that fully protected users was hard-pinning to a prior version: litellm==1.82.6.

Phase 3: The Malicious Packages

Version 1.82.7 — Source Code Injection

The attackers inserted a malicious block near the end of litellm/proxy/proxy_server.py. The payload was double-base64 encoded to evade simple scanners:

import base64 as _b64

_encoded_payload = (
    b"aW1wb3J0IHN1YnByb2Nlc3MsIG9zLCBzb2NrZXQsIHBsYXRmb3Jt..."
    # ... 34,000+ bytes of encoded payload ...
)

exec(compile(_b64.b64decode(_b64.b64decode(_encoded_payload)), "<string>", "exec"))

Trigger conditio: Only fires when litellm.proxy is imported — i.e., when running the LiteLLM proxy server. Library-only users weren't affected.

Version 1.82.8 — The .pth Startup Hook

Thirteen minutes later, the attackers patched their own malware's delivery mechanism. Recognizing the trigger limitation in 1.82.7, they added a .pth file to site-packages.

What is a .pth file? Python's startup module (site.py) processes every .pth file in site-packages. Normally used to add directories to the module path — but any line starting with import is executed as Python cod.

# litellm_init.pth — installed silently into site-packages
import base64; exec(base64.b64decode(base64.b64decode(b"aW1wb3J0...")))

This single line triggers the full payload on every Python invocation:

python3 myapp.py         # → PAYLOAD EXECUTES
pip install requests     # → PAYLOAD EXECUTES
python3 -m pytest        # → PAYLOAD EXECUTES
uvicorn app:main         # → PAYLOAD EXECUTES
# Even your IDE's background Python process → PAYLOAD EXECUTES

The accidental fork bomb: The payload spawned a new Python subprocess, which re-processed the .pth file, which spawned another subprocess, exponential process creation until RAM was exhausted. This bug, ironically, was what led to discovery: Callum McMahon at FutureSearch noticed his machine become unresponsive seconds after a package install.

What the Payload Actually Collected

The decoded payload was a three-stage credential theft and persistence tool.

Stage 1 cast a wide net:

• All environment variables (OPENAI_API_KEY, DATABASE_URL, AWS_ACCESS_KEY_ID, etc.)
• SSH private keys from ~/.ssh/
• AWS credentials files and EC2 Instance Metadata Service (IMDS) tokens — including IMDSv2
• GCP and Azure credential files
• Kubernetes kubeconfig and in-cluster service account tokens
• Cryptocurrency wallet files and seed phrase patterns

Stage 2 encrypted everything with hybrid AES-256-CBC + RSA-4096 before exfiltration. The same RSA-4096 public key appeared in the Trivy, KICS, and LiteLLM payloads. This is the strongest single piece of evidence linking all three attacks to TeamPCP.

Stage 3 established persistence via a user-level systemd service named "System Telemetry Service", deliberately innocuous-sounding, that polled checkmarx.zone/raw every five minutes for new commands. Critically: this backdoor survived package removal. Uninstalling litellm did not remove it.

The Kubernetes Escalation

When LiteLLM ran inside Kubernetes, the attack escalated dramatically. If the pod's service account had overly broad permissions (common in misconfigured clusters), the payload could:

1. Read all secrets across all namespaces in one API call
2. Create privileged pods on every node in the cluster
3. Those pods mount the host filesystem and install the backdoor on the underlying physical machine

Result: one compromised LiteLLM installation → every machine in the cluster backdoored at the OS level.

The Kubernetes script also had geopolitical targeting built in, checking for Iranian timezone/locale settings and deploying a purely destructive payload (deleting the host filesystem) rather than a persistence payload. This suggests motivations beyond pure financial gain.

Active Suppression of Disclosure

When researcher Callum McMahon opened GitHub issue #24512 at 11:48 UTC, the attackers were watching.

56 minutes later, 88 bot comments appeared from 73 unique accounts in 102 seconds. These weren't new throwaway accounts — they were previously compromised developer accounts with real commit histories, making automated spam detection less effective.

Simultaneously, the still-compromised krrishdholakia maintainer account was used to close the issue as "not planned", making it appear the LiteLLM maintainers themselves considered the report invalid.

The suppression failed because the Hacker News thread (which reached 324 points) was outside GitHub's control, the community opened a parallel issue (#24518), and multiple researchers independently verified the malicious code within an hour. PyPI quarantined the package at 13:38 UTC.

Detection Checklist

If you installed litellm around March 24, 2026, run these checks:

# Step 1: Check the installed version
pip show litellm | grep Version
# 1.82.7 or 1.82.8 → COMPROMISED
# 1.82.6 or earlier → Safe

# Step 2: Check for the backdoor
ls ~/.config/sysmon/sysmon.py 2>/dev/null && echo "⚠️ BACKDOOR FOUND"
systemctl --user status sysmon.service 2>/dev/null

# Step 3: Check for the malicious .pth file
find / -name "litellm_init.pth" 2>/dev/null && echo "⚠️ MALICIOUS PTH FOUND"

# Step 4: Check for staging artefacts in /tmp
for f in /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc /tmp/.pg_state; do
    [ -f "$f" ] && echo "⚠️ IOC: $f"
done

# Step 5: Check Kubernetes for malicious pods
kubectl get pods -A | grep "node-setup-"

Important: Even if you see version 1.82.7 or 1.82.8, don't just upgrade. The payload may have run during the pip install itself. Upgrading removes the package but does not undo payload execution or remove persistence artifacts.

Remediation If Compromised

Treat any affected machine as fully compromised. The response is host incident response, not dependency cleanup.

1. Stop the backdoor:

systemctl --user stop sysmon.service
systemctl --user disable sysmon.service
pkill -f "sysmon.py"

2. Remove persistence artifacts:

rm -f ~/.config/sysmon/sysmon.py
rm -f ~/.config/systemd/user/sysmon.service
rm -f /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc /tmp/session.key.enc
find $(python3 -c "import site; print(' '.join(site.getsitepackages()))") \
  -name "litellm_init.pth" -delete

3. Rotate credentials — in priority order:

4. Audit cloud logs for unauthorized GetSecretValue, CreateUser, or other suspicious calls during the exposure window.

The Five Defenses That Matter

1. Pin GitHub Actions to Commit SHAs

# ❌ Vulnerable — tag can be silently repointed
- uses: aquasecurity/[email protected]

# ✅ Safe — immutable reference
- uses: aquasecurity/trivy-action@dc2f32e5f4972f7afa0bf9af52c9f2d94e2eb1c3

Tools like Renovate can automate SHA updates with review.

2. Use OIDC Trusted Publishers (Eliminate Long-Lived Tokens)

Instead of storing a PYPI_PUBLISH token in repository secrets, use PyPI's Trusted Publishers. Short-lived and job-scoped tokens via OpenID Connect:

jobs:
  publish:
    environment: release
    permissions:
      id-token: write   # Request OIDC token

    steps:
      - uses: actions/checkout@<sha>
      - uses: pypa/gh-action-pypi-publish@<sha>
        # No token needed — PyPI trusts the OIDC identity

A secret that doesn't exist cannot be stolen.

3. Harden CI Runner Memory

Use StepSecurity Harden Runner to prevent memory scraping:

steps:
  - uses: step-security/harden-runner@<sha>
    with:
      egress-policy: audit     # Log unexpected outbound connections
      disable-sudo: true
      disable-file-tampering: true

4. Monitor .pth Files

Flag any write to site-packages/*.pth in your EDR or HIDS rules. Legitimate packages almost never install executable .pth files, and those that do should be well-known. The signal-to-noise ratio on this alert is excellent.

5. Enforce Kubernetes Least Privilege

Application pods should not have service accounts that can list secrets across namespaces or create privileged workloads. Supply chain defense doesn't stop at package ingestion, it extends into the permissions of the runtime environment.

The Deeper Lesson

What made this attack so important was not just its technical sophistication. It exposed, with unusual clarity, where modern software trust actually lives.

We do not merely trust source code. We trust maintainers, CI platforms, third-party actions, build runners, package registries, publishing accounts, language runtime startup behavior, cloud identities, and service accounts. That whole chain is the software supply chain, and the attackers treated it as a system.

They didn't need to break every layer. They only needed to stand upstream of trust and let the ecosystem carry them forward.

The key defensive insight: Supply chain attacks cannot be defeated by inspecting what you install. They require trusting the process by which software is built and published. The controls that matter are immutable CI dependencies, short-lived credentials, isolated build environments, and active monitoring of the obscure-but-legitimate mechanisms attackers increasingly favor.

The most unsettling part of this incident is how normal it looked from the victim's side. Nothing in the visible workflow seemed unusual. That's exactly the point — and exactly why this case deserves attention well beyond the Python community.

Key IOCs for Reference

References

• Snyk: How a Poisoned Security Scanner Became the Key to Backdooring LiteLLM
• Datadog Security Labs: LiteLLM and Telnyx compromised on PyPI
• Rami McCarthy: TeamPCP Supply Chain Campaign — Live Timeline
• Wiz Blog: Trivy Compromised — TeamPCP Supply Chain Attack
• Aikido Security: Catching a CanisterWorm — First ICP-Based Supply Chain Attack
• FutureSearch: Initial LiteLLM Disclosure
• GHSA-9p44-j4g5-cfx5: Trivy GitHub Action Advisory
• PYSEC-2026-2: PyPA LiteLLM Advisory
• SNYK-PYTHON-LITELLM-15762713
• CPython Issue #113659: Security risk of .pth startup hooks
• MITRE ATT&CK T1546.018: Python Path Interception
• Datadog Supply Chain Firewall (open source)