The Certificate Was Valid

Let's Call It Custody Engineering

Last week I promised you the MCP as a webhook, unless something more catastrophic came along; something did, and it came with a valid certificate. At 20:57 UTC on Friday, Aug. 28, a network nobody at Softaculous had ever done business with started announcing 162.55.80.0/24, a slice of Hetzner's address space that happened to hold the Virtualizor update endpoint, the client area, and the billing site. For the next 33 hours, in two waves, a meaningful share of the internet's routers believed the shortest path to Virtualizor ran through an attacker's server.

That server asked Let's Encrypt for a certificate covering 26 Virtualizor hostnames; Let's Encrypt did what domain validation does and sent an HTTP challenge down the hijacked route, and the attacker answered it. From that moment every hypervisor that phoned home for an update got a green padlock, a valid chain, and a package with a root SSH key in it. (Thursday's simultaneous ChatGPT, Claude, and Grok outage also qualified as catastrophic for anyone whose pipeline calls one model provider, but the cause isn't public and I'm not going to guess at it.)

The thing I want to take apart this week is the sentence every one of us has said while writing a pipeline step: it's HTTPS from the vendor's own domain, so it's fine. That sentence conflates two different guarantees. TLS vouches for the wire: you're talking to whoever controls that hostname's validation path right now. A signature vouches for the file: this exact set of bytes was produced by someone holding a key you pinned in advance; it says so no matter which server handed it to you or which route the packets took. The Virtualizor incident is what happens when a whole product line rests on the first guarantee and never had the second; what we'll call CUSTODY, the same idea a courtroom uses when it asks not where the evidence was found but who has signed for it at every hand-off since.

Build the update step so the file has to prove where it came from, and stop asking the wire to do it.

Thirty-Three Hours on Someone Else's Route

The routing side of the story is short and unforgiving. The forged announcement came from AS62390, routed through transit provider AS6204, and kept Hetzner's AS24940 as the origin; Hetzner's route origin authorization permits AS24940 to announce that block down to a /24, the forged route was also a /24, so RPKI looked at it and said valid. By hard2bit's read of RIPE RIS data, about 72% of 368 vantage points saw the hijacked path at peak and all 368 saw it at some point, with roughly 10,600 route withdrawals over the 33 hours as the two waves came and went. The first wave ran about 12 hours, there was an 11-hour gap, and the second wave ran about 10 more before the route was withdrawn at 06:10 UTC on Sunday, Aug. 30. Whether the operators of AS62390 were the attackers or a compromised intermediary hasn't been established, and I'm not going to establish it here.

The certificate should bother anyone who's ever trusted a padlock. Let's Encrypt issued it during the first wave, it stayed valid after the routing was restored, and the attacker used it again in the second wave before it was revoked. Nothing malfunctioned. Domain validation is an HTTP request to the hostname being validated; the internet delivered that request to the address the routing table said was correct, and the certificate authority had no way to know the routing table was lying. A DV certificate is a statement that somebody controlled the validation path at issuance time; it's never been a statement about who wrote the software behind it.

The payload was old-fashioned, which is its own lesson. The malicious update announced itself as version 3.2.9.8 while leaving the real install at 3.2.9.7 (the version mismatch is now the quickest indicator of compromise), modified three legitimate Virtualizor files, dropped a root cron job, installed a systemd unit named java-jre-update.service, added an attacker SSH key to root, installed Java if it wasn't present, and created a proxyuser account.

One hosting provider, AlbaHost, reported that 5 of its 34 Virtualizor hypervisors carried the same modifications. Softaculous says a small number of installations were affected and, in the same post, that it can't produce a definitive list, because the update requests it'd need to audit went to a server it doesn't own. The company shipped 3.2.9.9 on Sept. 1 with a scanner for the known indicators and promised to put a code signing mechanism in place for all packages going forward.

Going forward. The fix that would've made the entire 33 hours a non-event is the one item on the list that didn't exist yet.

It's HTTPS From the Vendor's Own Domain. Right?

That is the objection, and I've made it myself more times than I'd like to count while wiring curl | bash into a Dockerfile at 4:30 in the afternoon. Spelled out, the objection is also the reason it fails: HTTPS from the vendor's domain means your client negotiated a session with a server holding a certificate for that name. The certificate was real. The server wasn't the vendor's. Every layer did its job and the composition was still wrong, because the guarantee you needed (these bytes came from Softaculous) was never the guarantee any of those layers offered.

Some of my professional life was spent as DevOps for a suite of ecommerce sites. Certificate health is life-or-death when you're asking users to trust you with their PII and payment info. There were a number of attempts over the years by contracted vendors and malicious bad actors alike to wedge themselves into our cert process; so much so we created a series of release-gate checks that confirmed certificate existence and the full routing history at origin for requests.

The vendor wasn't being lazy in the way that word usually means. Softaculous ran HTTPS, ran an update endpoint, and published versions; that's the standard of care for most self-hosted software in 2026, and it's the standard of care in most of our own pipelines when we pull a release tarball, a GitHub release asset, a Helm chart from a vendor bucket, or an installer script off a CDN.

The gap is that TLS protects the conversation and nothing protects the artifact. When the conversation is with the wrong party, a perfectly protected conversation is the attacker's best friend. The network-side defenses that should've caught the swap (a route origin authorization without a permissive maxLength, an ASPA record on the path, a prefix filter at the transit provider) all belong to your vendor, your vendor's host, and a transit network in another country; you don't operate any of them. The one control you do operate is whether your pipeline will execute a file that can't prove who made it.

The Signed Thing Is Not the Trusted Thing

Two days after the Virtualizor patch shipped, the Symantec Threat Hunter Team published its account of a campaign that's been running since February against government departments, technology companies, and hotels (Symantec ties part of the activity to the initial access broker tracked as KongTuke). In at least one intrusion the attackers had their Cobalt Strike and AdaptixC2 beacons blocked on the way in, so they stopped dropping binaries. Instead they stage the official Node.js runtime, node.exe, a legitimate signed developer tool, and run their implant as JavaScript through it, with a registry Run key relaunching it at every login. Security software rarely flags the runtime because the runtime is what it claims to be. The malicious logic lives in an interpreted script that nobody signed and nobody checks.

Put the two incidents side by side and the pattern is the same one, viewed from opposite ends. Virtualizor's customers trusted a valid certificate on the transport and got an unsigned package. Symantec's victims trusted a valid signature on the launcher and got unsigned code inside it. In both cases the trusted object was real and the trust was misplaced by one layer: the signature covered the wrapper, and the payload rode along underneath. The question a pipeline has to ask of every artifact it pulls is whether the thing that was signed is the thing that'll run, and if the answer is "the wire" or "the interpreter," the answer is no. Custody means the signature is on the bytes you execute, verified against a key you chose before you ever saw those bytes.

The Fix Is a Key You Already Have

The tooling objection to signing has always been that it's a project: a PKI, a key ceremony, a new dependency on every host. That was true once. Since OpenSSH 8.0 in 2019, ssh-keygen -Y sign and ssh-keygen -Y verify produce and check detached Ed25519 signatures over arbitrary files; OpenSSH is on every Linux runner, every Mac, and Windows since 2018.

I looked at minisign first, because it's the purpose-built tool and its signature format is simpler; it wasn't installed on my own machine, though, and a companion script whose first step is "install the verifier" has already lost the argument with the person writing the Dockerfile. ssh-keygen -Y costs nothing to adopt, uses the same Ed25519 math, supports a revocation list and a valid-after/valid-before window on the signer entry, and its allowed_signers file is one line of text you commit to the repo (with the release key's public half, so a compromised release box can't rotate it out from under you).

The publisher side is one command per artifact: ssh-keygen -Y sign -f release_key -n file install.sh writes install.sh.sig next to the file. The consumer side is the companion script, and the design constraints are the whole point. The artifact and the signature land in a scratch directory, never at the destination; a missing signature is a distinct exit code from a bad signature, so an "unsigned artifact" alert and a "tampered artifact" alert page different people; plain http:// and any redirect off https:// are refused before a byte is read; and the script never runs anything, because a fetcher that can also execute is one flag away from being curl | bash with extra steps. The key is pinned to a principal name (release) in the allowed_signers file, so rotating the key is a pull request that a human reads (see the CODEOWNERS rule from Issue #29; the same interlock applies).

None of this stops a BGP hijack. It makes the hijack irrelevant to your pipeline, because the attacker's server can hand you a file with a valid certificate and a perfect TLS handshake, and sigfetch will read the signature, find that it wasn't made with the key you pinned, exit 4, and leave your destination empty. That's the difference between a control you own and one you hope your vendor's hosting provider owns.

#!/usr/bin/env bash
# sigfetch: fetch an artifact and its detached signature, verify the signature
# against a pinned public key, and only then hand the file over. Fails closed.
#
#   sigfetch <artifact-url-or-path> <dest> [signature-url-or-path]
#
# Env:
#   SIGFETCH_SIGNERS  allowed_signers file, committed with your repo
#                     (default: ./allowed_signers)
#   SIGFETCH_IDENTITY principal named in that file (default: release)
#   SIGFETCH_NS       signature namespace (default: file)
#
# Exit: 0 verified and written to <dest>; 2 fetch failed; 3 no signature
#       (unsigned artifact); 4 signature does not verify. Nothing executes here.
set -euo pipefail

src=${1:?artifact url or path}; dest=${2:?destination path}; sig=${3:-$src.sig}
signers=${SIGFETCH_SIGNERS:-./allowed_signers}
identity=${SIGFETCH_IDENTITY:-release}
ns=${SIGFETCH_NS:-file}

[[ -r $signers ]] || { echo "sigfetch: no allowed_signers at $signers" >&2; exit 2; }
work=$(mktemp -d); trap 'rm -rf "$work"' EXIT

fetch() { # source, target: https only (no redirect off-scheme), or a file already on disk
  case $1 in
    https://*) curl --fail --silent --show-error --location --proto '=https' \
                    --proto-redir '=https' --max-time 60 --output "$2" "$1" ;;
    *://*)     echo "sigfetch: refusing $1, https only" >&2; return 1 ;;
    *)         [[ -f $1 ]] && cp "$1" "$2" ;;
  esac
}

fetch "$src" "$work/artifact" || { echo "sigfetch: could not fetch $src" >&2; exit 2; }
fetch "$sig" "$work/artifact.sig" || {
  echo "sigfetch: REFUSED unsigned artifact, no signature at $sig" >&2; exit 3; }

if ssh-keygen -Y verify -f "$signers" -I "$identity" -n "$ns" \
     -s "$work/artifact.sig" < "$work/artifact" >"$work/verify.log" 2>&1; then
  install -m 0644 "$work/artifact" "$dest"
  echo "sigfetch: verified $(basename "$src") as $identity -> $dest"
else
  echo "sigfetch: REFUSED $src, signature does not verify for $identity" >&2
  sed 's/^/sigfetch:   /' "$work/verify.log" >&2
  exit 4
fi

ssh-keygen -t ed25519 -N '' -f release_key -C release
awk '{print "release " $1 " " $2}' release_key.pub > allowed_signers   # commit this
ssh-keygen -Y sign -f release_key -n file install.sh                    # writes install.sh.sig

$ sigfetch ./install.sh /tmp/out.sh
sigfetch: verified install.sh as release -> /tmp/out.sh

$ sigfetch ./evil.sh /tmp/out.sh
sigfetch: REFUSED ./evil.sh, signature does not verify for release
sigfetch:   Signature verification failed: incorrect signature
sigfetch:   Could not verify signature.
$ echo $?
4

Quick Wins

🟢 Easy (~15 min): Grep your pipeline definitions and Dockerfiles for curl and wget and list every remote fetch that executes or installs what it downloads. For each, write down which of the three you're trusting: the wire (HTTPS only), a checksum you copied from the same server (which is the wire again), or a signature against a key you pinned in advance. Most lists come back all wire.

🟡 Medium (~1 hour): Pick the one fetch on that list with root or deploy privileges downstream and route it through sigfetch. If the vendor publishes signatures (GitHub releases with cosign or gpg files, Debian and RPM repos, Homebrew bottles), pin their key. If the vendor publishes nothing, mirror the artifact you've already vetted into storage you control, sign it with your own release key, and fetch it from there; the vendor's server stops being in your trust path at all.

🔴 Advanced (half day): Sign your own release artifacts. Generate the release key on a machine that isn't the build runner, store the private half in your secrets manager with access scoped to the release job, commit allowed_signers with a CODEOWNERS rule on it, and add a verification step to every consumer, including your own deploy scripts. Then run a tabletop against the Virtualizor timeline: an attacker controls your download host for 33 hours with a valid certificate. Every consumer should exit 4.

Next Week

The MCP as a webhook is still in the queue and I still owe it to you, unless the internet hands us another week like this one. More next Tuesday, from the new mailing platform, which is why this issue looks a little different in your inbox.

A hostile network announced a Hetzner prefix for 33 hours; a certificate authority issued a real certificate to the wrong server because the validation traffic followed the same lie; a vendor's update endpoint then delivered root access to an unknown number of hypervisors it can't enumerate because the requests never reached it. Every component in that chain worked as designed; the design assumed the transport was the proof, and a transport can only prove who it's connected to.

A signature travels with the file, and the key that checks it was in your repo before the attacker showed up.

That's custody, and it's one command on each side of the wire.

Bobby R. Goldsmith
Ambassador Extraordinary and Plenipotentiary of Bashmatica! by NodeBridge Automation Solutions

P.S. This is the first issue to arrive from Buttondown after Beehiiv and I parted ways (their idea, then mine), so if it landed in a folder it didn't use to, drag it out and tell your mail client we're friends. Issue #30 covered the runner holding your keys, and this one covers what the runner pulls down with them. If someone forwarded this to you, subscribe at bashmatica.com, and if you know whoever owns the curl | bash line in your Dockerfile, forward it to them.