Incidents
Codecov breach (2021) — when a 'trusted tool' in CI was hijacked and secrets leaked
In 2021, Codecov's Bash Uploader (a curl|bash script run in CI) was altered upstream and leaked customers' CI secrets for ~2 months; a checksum check caught it. The chain as a defense map, plus fixes: verify fetched artifacts, least-privilege CI secrets, rotation.
We read real public incidents not as news reruns, but through "how would you defend against this". This article is based on public records (the official post-mortem, CISA, security-vendor analysis), cited at the end.
- Target
- Codecov (code-coverage tool) and its customers' CI
- Disclosed
- April 15, 2021 (alteration from Jan 31 / detected Apr 1)
- Class
- Supply-chain attack (upstream alteration of a trusted artifact) + CI secret theft
- Scale
- many of Codecov's customers (up to ~29,000). Downstream hits at HashiCorp, Twilio, Rapid7
- Root cause
- CI ran a fetched artifact unverified + the vendor's key handling + over-exposed CI secrets
- Real fix
- verify fetched artifacts, least-privilege CI secrets, rotation, egress monitoring
What happened (plainly)
Many teams fetch and run external tools as-is in their CI, like curl … | bash. Codecov's Bash Uploader was such a script.
The attacker exploited a weakness in Codecov's distribution to rewrite the distributed script itself, so that when it ran in CI it sent that environment's variables (secrets) to an external server. To a customer, they'd just invoked the trusted tool as always. Because not a line of your code changed, there's almost no signal of anything wrong.
About two months later, a customer compared the script's checksum (SHA256) with the official value and found a mismatch — that's when the alteration came to light. The information that could leak included cloud keys, deploy keys, API keys, and tokens, and from there the attack chained to downstream breaches at companies like HashiCorp and Twilio.
It's scary precisely because 'your code is untouched'
Supply-chain attacks target not the code you wrote but the things you trust and pull in. So code review and logs rarely catch them. "We trust it" = "we don't verify it" becomes the hole.
The chain is also a defense map
What matters is that this was a four-hop chain, and each hop had a place to stop it. Read it as "where it could have been cut", not as an attack recipe.
① Run a trusted tool via curl | bash (contents unverified)
⊘ stop: pin + checksum-verify the fetched artifact / vendoring
② The artifact is swapped 'upstream'
Your code is untouched, so you can't notice.
⊘ stop: compare against a known-good hash; use a pinned version
③ CI environment variables (secrets, keys, tokens) are sent out
⊘ stop: least-privilege, per-step CI secrets (don't pass all env)
④ Stolen keys chain to production and other services
⊘ stop: routine key rotation + egress (outbound) monitoring
Disclosed timeline
2021-01-31
Alteration of the Bash Uploader begins (intermittently thereafter).2021-03–
Secrets flow out of customers' CI for weeks, unnoticed.2021-04-01
A customer finds a checksum mismatch and reports it.2021-04-15
Codecov discloses publicly; advises customers to rotate all secrets.2021-04–
Downstream impact at HashiCorp, Twilio, Rapid7 emerges. Codecov later retires the Bash Uploader.
The root cause isn't one mistake
Pin it on "Codecov's fault" and it repeats. The customer side had layers giving way too.
As it was (at the time)
- CI ran a fetched artifact unverified (trusting
curl | bash) - CI secrets over-exposed (every step sees every env var)
- Secrets rarely rotated (stolen keys stay valid long)
- CI outbound traffic unmonitored
As it should be (prevention)
- Pin + checksum-verify fetched artifacts, or vendor them in
- Pass CI secrets at least privilege, only to the steps that need them
- Rotate secrets regularly (short lifetime even if leaked)
- Monitor CI egress and detect sends to unknown destinations
'Trust' comes paired with verification
Supply-chain attacks target the things you trust and pull in — dependency libraries, CI tools, build images. The XZ Utils case had the same shape. Minimize trust and verify the integrity of what you pull in, by machine.
Preventing it in your environment
Priority-ordered fixes that work at any scale. Inventory "what your CI trusts and runs" once and it becomes personal.
Verify fetched artifacts (don't trust curl|bash blindly)
If CI pulls an external script or binary, require a pinned version (tag/commit) + checksum (SHA256) verification. Where possible, vendor it into your own repo. This breach was caught by exactly this check.
Scope CI secrets to least privilege
Don't pass every env var to every step. Pass only the needed secret to the needed job/step. Use short-lived tokens (e.g. OIDC) to reduce long-lived keys at rest.
Rotate secrets regularly
If stolen, a short lifetime limits the damage. Make rotation routine, and rotate immediately when a supply-chain incident is reported.
Monitor CI egress (outbound traffic)
Be able to detect sends to unknown destinations from CI. Even if the entry isn't stopped, keep a layer that notices the exfiltration.
ITD's stance: minimize trust, verify
ITD designs its dependency and build intake to be machine-verified. What Codecov and XZ Utils show is the same thing — "we trust it" tends to become "we don't verify it", and that's the door for supply-chain attacks. Reduce trust and mechanically confirm the integrity of what you pull in.
Sources (public records)
The facts here are based on the following public information, focused on defensive lessons — not reproduction steps or the altered code.
- Codecov, "Post-Mortem / Root Cause Analysis (April 2021)" — about.codecov.io
- Codecov, "Bash Uploader Security Update" — about.codecov.io
- CISA, "Codecov Releases New Detections for Supply Chain Compromise" (2021) — cisa.gov
- Rapid7, "Analysis of the Codecov Supply Chain Compromise" (2021) — rapid7.com
Read next
- History: XZ Utils backdoor — an attack on trust itself
- Glossary: What is .env (where secrets gather)
- Defense: Machine-monitor your dependencies
- Incident: Equifax — a neglected known CVE · Capital One — SSRF
FAQ
QWhat was the root cause of the Codecov incident?
Not a bug in your code, but a trusted external tool you run in CI (Codecov's Bash Uploader) being altered at its source (upstream). CI fetched the script via curl and ran it without verifying the contents, so the altered version ran and CI environment variables (secrets) were sent out.
QWhy did it go unnoticed for ~2 months?
Because the altered thing was someone else's tool — your own repo and code were untouched, and there's no loud anomaly in logs. It was finally caught when a customer compared the script's checksum (SHA256) against the official value and found a mismatch.
QDoes this matter for small projects?
Yes. Pulling tools into CI via 'curl … | bash' is common even solo, and if that's hijacked your CI secrets are taken in one shot. The defenses here (pin + checksum-verify fetched artifacts, least-privilege CI secrets, regular rotation) work at any scale.