Trivy Was the Key the Whole Time

Derek: Okay, okay. Welcome back to Autonomous Autopsy. I'm Derek, and Max, I gotta say, this one has been eating at me all week. The same Derek, same. Because where do you even start with this? So get this. March 24th, 2026. Someone launches a local MCP server through Cursor. The machine starts stuttering. CPU pegged at 100%. That's how we find out LiteLLM, the AI proxy, sitting in front of... In front of basically every LLM API on the planet just got backdoored. Not a scanner, not an alert, a crashed laptop. A crashed laptop discovered by accident. And this is a package with, according to Endor Labs, over 95 million monthly downloads. And Snyk traced how it happened. TeamPCP didn't go after LiteLLM directly. They poisoned Trivy, the security scanner LiteLLM was running. Running unpinned in its own CI pipeline, the scanner was the weapon. The scanner was the weapon. Stole the PyPI publishing token, built malicious packages locally, pushed them straight to PyPI. No Git commit, no release tag, nothing in the official repo. Plot twist on a plot twist. And the payload, Three stages: credential harvesting, Kubernetes lateral movement, persistent systemd back door. Trendmicro's write-up called it one of the most sophisticated multi-ecosystem supply chain campaigns publicly documented. Then there's the blast radius, downstream frameworks, a Mercor breach, and TeamPCP running a GitHub bot operation to suppress the initial disclosure. We'll get into all of it, plus the two structural fixes that could have stopped this cold, no vendor cooperation required. Alright, let's get into the kill chain, Segment One right now. Now March 26, 2026, 10:39 UTC. Two packages land on PyPI, 13 minutes apart, and Production containers start dying. CPU pegged at 100%. OOM kills everywhere. Processes just going haywire. And nobody's security tooling fires. Not one alert. Zero. And here's the kicker. The person who caught it? Not a security engineer. Not a scanner. Callum McMahon at FutureSearch was testing a Cursor MCP plug-in. Wait, really? A Cursor plug-in? Yeah, yeah. LiteLLM came in as a transitive dependency. His machine started acting up, he traced it back, and boom. According to Trendmicro, the malicious versions were discovered not by security tooling but by accident. By accident. That's doing a lot of heavy lifting as a detection strategy. Right? So, okay, let me set the stage here, because the scale matters. Max, tell people what LiteLLM actually is. So LiteLLM is your AI gateway. One interface, 100 plus LLM providers. OpenAI, Anthropic, AWS Bedrock, Google, you name it. And because it sits in the middle of all of that, it holds the keys to everything. Everything. Snyk confirmed it pulls 3.4 million downloads per day, Max: Wow. Derek: and Wiz reported sitting in 36% of cloud environments. So you're not targeting one set of credentials, you're targeting the key ring for an entire org's AI stack. Exactly. And look at the transitive dependency footprint. Trend Micro's report called out CrewAI, DSPy, Browser-Use, MLflow, OpenHands, Microsoft GraphRAG, Google ADK. So if you never installed LiteLLM yourself, you might still be cooked. Your framework pulled it for you. Okay, okay, so that's the target. What's the damage window? Speaker 3: According to Snyk, the malicious versions were on PyPI for approximately three hours before PyPI quarantined them. CVE-2026-33634, CVSS 9.4. Three hours at three point four million downloads a day, I'm doing that math, and it's ugly. Very ugly. And the attack didn't start March twenty sixth. Right, that's the part that keeps me up at night. The packages landed on PyPI without any corresponding GitHub tag, no release in the actual repo. The maintainer's credentials were already gone. Derek: stolen days earlier through a completely different project. So the question is how Speaker 3: How does a package with ninety five million monthly downloads end up backdoored and nobody on the team knew it was coming? Derek: And the answer starts not in March, not even with LiteLLM. Speaker 3: It starts in February with a misconfigured workflow in a security scanner. Derek: Oh, great. The thing watching the watchers. Speaker 3: The thing watching the watchers. That's exactly where we're going. Derek: So the credentials that kicked all this off weren't stolen on March 24th. They were stolen on February 28th. Speaker 3: Three and a half weeks earlier. Derek: And the entry point wasn't LiteLLM's code. It wasn't even LiteLLM's repo. It was Trivy, Aqua Security's open source vulnerability scanner. According to Snyk's write-up, an AI-powered autonomous bot called Hacker Bot Claw exploited a misconfigured pull request target workflow in Trivy's GitHub app. Speaker 3: Hub Actions. Okay, so quick explainer on pull request target because this matters, Please. So it's a GitHub Actions trigger that runs with the base repo's secrets and permissions even when the code comes from an outsider's fork. The idea is it lets maintainers safely run CI against pull requests. But if you check out the PR code inside that workflow, an attacker can run whatever they want with your secrets attached. Derek: So the PR was closed immediately. Didn't matter. Matter: the workflow already ran, Token exfiltrated. Love that for everyone. Aqua disclosed on March first and rotated credentials. Speaker 3: Mm-hmm. Derek: But, and this is where things go sideways, the rotation was not atomic, Meaning they didn't invalidate everything simultaneously. Exactly. TeamPCP observed the refresh and held credentials that survived the gap. Then they just waited. Twenty days. Speaker 3: That's patience, not that they were watching. On March nineteenth at seventeen forty three UTC they came back using the retained Aquabot service account token they force pushed seventy five of seventy six version tags in Aqua Security slash Trivy slash Actions to attacker controlled commits, all seven tags in Setup Trivy same treatment. Derek: Wow. Speaker 3: The malicious Trivy binary Derek: The period versions sixty nine point four went to GitHub releases, Docker Hub, GHCR and Amazon ECR simultaneously, and pipelines referencing those tags by name, not by commit SHA, just silently pulled attacker code on their next build run. No diff, no alert. No. Nothing. Which brings us to LiteLLM specifically. According to Snyk's analysis, LiteLLM's CI pipeline ran Trivy via a shell script. Del script, installed with sudo apt get install Trivy, no pinned version. Oh no. So when v zero point six nine point four dropped, LiteLLM's pipeline pulled it on the next build. The poisoned binary scraped process memory through slash proc slash pid slash mem, which bypasses GitHub's secret masking entirely, and extracted the PYPI_PUBLISH_PASSWORD in plain text. The memory scrape is the key part: GitHub masks secrets in logs. Logs Max: Right. Derek: It does not mask them in memory, so reading the runner process directly sidesteps the entire protection. And five days after that token was stolen, on March twenty fourth, TeamPCP used it to push versions one point eight two point seven and one point eight two point eight straight to PyPI, not through LiteLLM CI, not tied to any GitHub release. Zero corresponding GitHub tags. Zero release artifacts. Facts.—LiteLLM's own security update: the packages just appeared. The exfiltration domain was models.litellm.cloud, a typosquatted domain built to look like legitimate LiteLLM infrastructure. So at every hop, the attacker looked like the legitimate actor; spoofed maintainer commits in Trivy; legitimate PyPI credentials for LiteLLM; exfiltration to a domain that looks like the real thing. Thing.—One incomplete credential rotation in late February cascaded through four weeks and three ecosystems and now here's the thing that sets up everything we're about to get into: the packages themselves. Team PCP pushed two versions, one point eight two point seven then one point eight two point eight thirteen minutes apart. That gap is not an accident. Thirteen minutes—that's the gap between one point eight two point seven and one point eight two point eight. 13 minutes? They were iterating live mid-attack, like a developer fixing a bug in prod, except the bug was not enough machines compromised. Right, and the fix was a .pth file. So Derek, can we talk about that for a sec? Because most people have no idea what a .pth file even is. Speaker 3: Please. Derek: Okay, so Python has this feature. Most developers never think about it. Any .pth file sitting in your site packages directory gets processed automatically by the interpreter at startup. Speaker 4: Wow. Derek: Not when you import a package, not when you call a function. Every time Python runs. Every time Python runs. So in 1.82.7, the payload was embedded in proxyserver.py. You had to actually import the LiteLLM module to trigger it. Contained-ish. Contained-ish. Then they shipped one point eight two point eight. Thirteen minutes later and now it's LiteLLM init dot .pth in site packages. Pip runs, fires your test runner, fires a totally unrelated Python script that has nothing to do with LiteLLM, fires. You don't even have to know LiteLLM is installed. You do not have to know LiteLLM is installed. Okay, so what does it actually do once it fires? Three stages. According to Trend Micro's analysis, stage one is credential harvesting: SSH keys, AWS, GCP, Azure tokens, database strings, .env files, shell history. Every environment variable on the machine. The whole buffet. Stage two: Kubernetes lateral movement. It uses the service account token to enumerate every node in the cluster. in the cluster and deploys a privileged pod to each one. So one infected machine becomes the whole cluster. Stage three: persistent systemd backdoor. Symex flagged it. Sysmon.dot service. Polling checkmarks.dot zone every fifty minutes for additional binaries. And the whole thing was double base sixty four encoded, which is why static scanners missed it entirely." Dryly, the name Sysmon, the classic "look like a legit process" move. Sarcastically extremely subtle." With a hint of dark humor, the attacker also left two earlier payload iterations commented out in proxyserver dot py. You could literally watch them refine the evasion technique across three drafts. (Laughing) A developer with version control habits, you almost have to respect the discipline. Almost. Now flip that picture outward, because all of this landed silently in the dependency trees of Speaker 5: hundreds of others. Derek: of projects that never installed LiteLLM directly. That's where this gets significantly worse. So the PTH payload hit everything Python touched, but here's the part that really lands the scale. Teams that never typed pip install LiteLLM directly were still hit. Speaker 6: Right, because LiteLLM is a transitive dependency for half the AI ecosystem agent. We're talking CrewAI, DSPy, Browser-Use, MLflow, OpenHands, Microsoft GraphRAG, Derek: Wow. Speaker 6: Google ADK, Mem0. Oh, Instructor, Camel-AI, the list goes on. You pull in one of those frameworks, you got LiteLLM too. Didn't matter, Docker images build during the window, CI pipelines running pip install without lock files, MCP servers bootstrapped with UVX, all of them pulled it in. And then you get to Mercor, which is, Okay, this one needs a second. Give it the second. Mercor is the AI recruiting startup valued at $10 billion. Others, clients include OpenAI, Anthropic, Meta. They sit at the structural center of the frontier model training pipeline. They're the company that vets and pays the human experts who label training data for the biggest labs. Breach confirmed March thirty first twenty twenty six. Strike Graph reported over forty thousand contractors exposed. Personal data, source code, AI training methodology. Then Lapsus claims four terabytes total, including nine hundred and thirty nine gigabytes of Mercor source code alone. Derek: Four terabytes. Speaker 6: And Meta paused all Mercor work (five contractor lawsuits filed). And look, none of Meta, OpenAI or Anthropic needed to be breached directly; their training pipeline data spilled out through a third-party vendor because that vendor ran... Derek: LiteLLM." The blast radius just keeps expanding the further you pull back. And here's the part that's almost darkly funny: how did any of this surface publicly? A developer's Cursor MCP plug in started behaving badly. Callum McMahon at FutureSearch, testing a plug in that pulled LiteLLM as a transitive dependency, his machine started eating RAM. No scanner flagged it, no SIEM alert, a misbehaving laptop. Laptop. Not a security tool. A RAM leak. So, he opens issue 2451-2, and this is where it gets genuinely wild. According to BreachedCompany, within 102 seconds, 88 bot comments flood in from 73 compromised accounts, dismissing it, calling it a false positive, claiming the packages are clean. 76% account overlap with the botnet used during the Trivy disclosure. Usher: Same playbook, new target. And then the LiteLLM CEO's own GitHub account, which TeamPCP still controlled, closes the issue as "not planned. NOT PLANNED: Four terabytes of stolen data. NOT PLANNED. The attackers treated the disclosure window as part of the attack surface. That's the line that sticks, and it sets up the question that's been bugging me since we started this: where were the defenders during all of this? Yeah, and the answer is complicated. So all of that downstream carnage, and here's the part that stings, most of it was preventable. Max, walk me through where the Blue Team actually dropped the ball. Three failure modes. First one is Aqua's remediation on March first. According to multiple reports from Palo Alto Networks and others, the credential rotation was not atomic-not all tokens were revoked at the same time. Which means. TeamPCP held a valid token through the rotation window. Max: Wow! Derek: Aqua said they patched. They had not fully patched. That gap is what handed over Trivy's release infrastructure three weeks later. Partial remediation is not remediation. It's just a slower breach. Exactly. Second failure, LiteLLM pulled Trivy from apt without a pinned version. Snyk confirmed this. Speaker 3: Mm-hmm. Derek: One unpinned security tool in the CI pipeline was all it took to hand over the PyPI publishing token on March 24th. And that's the brutal irony, right? The security scanner is the thing that had no security constraints on it. Yeah, the one thing you'd assume someone checked. Now the third failure is the detection gap, and this one genuinely bothers me. Tell me. Nobody caught this. No scanner, no SIEM, no PyPI monitoring tool. Snyk's reporting confirms the malicious versions were discovered by a researcher at FutureSearch because his Cursor plug-in was behaving and he traced it to a 34,628-byte double-base64-encoded .pth file in site-packages. Gasp! A pure accident. Yeah, the payload evaded static scanners by design. 9. Double Base 64 Encoding, but Behavioral Detection? A Python sub-process spawning from pip immediately reading SSH keys and .env files in rapid succession, that's an anomalous pattern regardless of what the package claims to be. So our runtime behavioral profile would have flagged it. Yes, and egress filtering would have caught the exfil to models.litellm.cloud before any data left the network. WORK. That domain was registered the DAY BEFORE the attack. Wait, wait, wait. Registered the DAY BEFORE? March twenty third, the exfil domain went live twenty four hours before the payload dropped—an outbound allow list kills the whole operation right there. And the community signal, issue two five one eight two, the Hacker News thread hitting three hundred and twenty four points. That's how the actual information got out, not a corporate detection tool. Community thread, because the original disclosure was buried under bot spam. Which is the structural lesson. TeamPCP treated the GitHub disclosure window as part of the attack surface. If your early warning depends on watching public issues, you need to account for coordinated suppression. Three Structural Failures. Incomplete rotation, unpinned CI tooling, behavioral blind spots. None of them exotic, all of them fixable before the next build runs. And fixing them doesn't require a vendor to do anything first. First, that's where we're going next. What two changes actually break this chain at the root? So here's the fix that actually changes the structural risk: after the incident, according to Endor Labs, LiteLLM migrated to PyPI Trusted Publisher. Speaker 4: Mm-hmm. Derek: Instead of a static PyPI publish token sitting in secrets forever, you get short-lived OIDC tokens scoped to a specific workflow run. No token to steal. Right; there's nothing to exfiltrate. The credential doesn't exist until the workflow runs, then it expires. TeamPCP's entire play collapses if that's in place from day one. Gone. And that's the structural fix LiteLLM actually made. But the take home for everyone listening right now, the Monday morning thing, is simpler: SHA pinning. SHA pinning. Open your CI workflow YAML, grep for any line with uses and an at v in it. Speaker 6: D. Derek: That's a mutable tag; TeamPCP rewrote seventy-six Trivy tags in one operation-pointedly, seventy-six, "all of them"--and every pipeline using those tags just polled whatever they rewrote. A commit SHA cannot be moved, full stop. One grep, replace "at v " with a commit SHA-takes maybe twenty minutes. That's it. That's the action item. The attack chain that hit thirty-six percent of cloud environments had a t Set a 20-minute fix sitting right there the whole time. Darkly poetic. If you published to PyPI, migrate to Trusted Publisher. If you run GitHub Actions, pin to commit SHAs. Both of them are documented, both are free, and neither requires waiting for a vendor. And that is a wrap on the LiteLLM autopsy. Max, honestly, the detail that keeps lodging in my brain from today, the key ring thing? The key ring thing. You said it perfectly. You're not compromising one API key. You're compromising the key ring for an entire org's AI stack. One dependency. Everything. And the part that keeps coming back for me is how this got caught. Not a scanner, not an alert. Somebody's local machine started melting. Completely by accident. Which, given that the security scanner was literally the attack vector, tracks Beautifully circular. So, the Monday morning action here is concrete: migrate PyPI publishing to Trusted Publisher Speaker 5: infrastructure. Derek: Publisher OIDC tokens, and pin every GitHub Action step to a commit SHA, not a tag, a SHA. Two changes: no vendor cooperation required. Do 'em before lunch! If today's episode saved your pipeline, do us a solid and leave a review. New episodes every Tuesday; subscribe wherever you listen. Thanks for spending the hour with us. Stay patched out there, or at least know what's in your pipeline. Later, everyone.