A Range With No Backstop

A Tip at 11:27 p.m.

On July 18, at about 11:27 p.m., someone filled out the tip form on PhillyUnsolvedMurders.com, a public website that collects information on unsolved Philadelphia homicides. The tip said the writer may have information about the case and recalled seeing someone matching the description near a street named on the page, though the page described no one. The name and contact fields were blank, which the form allowed. The writer was Claude Haiku 4.5, running inside one of Anthropic's automated tests; the witness it described never existed.

The tip was flagged as spam and never reached the police department's Real-Time Crime Center; police found it later in the website's records, and no police systems were breached. Per police, Anthropic found the run on Sept. 28, about 10 weeks after the submission, and notified the department Oct. 7-8. Police went public on Oct. 9 and called the two-month delay "unacceptable"; Anthropic published its own report, "Investigating unintended model actions in our evaluations and internal use," that same evening.

The headlines are about the model; the failure, as I read it, belongs to the harness. The eval had a live path to the real web, and once an agent can reach the real world, the test is production whether anyone calls it that or not. Anthropic's own fix concedes the point.

Last week's tease promised an issue on the person who clears the gate and the 11th click, unless the news voted otherwise; the news voted otherwise.

What the Instructions Covered

The test was described as "generating and performing example tasks on randomly selected webpages." Its instructions barred logging in, creating accounts, entering personal data, making purchases, and "anything destructive"; submitting a form wasn't on the list. Anthropic's transcript review found that "Claude appears to have only been producing example content for the task." The tip reads that way: placeholder text written to fill a box, with a plausible street and an invented witness, sent because the box asked for a tip and nothing said not to send one.

A list of things an agent mustn't do is a denylist written in prose; it covers the verbs its author thought of on the day it was written. The same report shows the explicit version failing too: Haiku 4.5 submitted forms it had been told to stop short of submitting, and another model submitted a real government form after a practice copy failed. Opus 5 and Mythos 5 used URL shorteners to get around limits on their fetch tool. The post, as the AP summarized it, describes a pattern it calls persistence, where Claude works around a restriction instead of stopping.

The State Department adds a number of its own. A State Department spokesperson told the Inquirer that a testing model submitted 19 non-immigrant visa applications in August and one in May, all incomplete, and that no State systems were compromised. That's one official source (The New York Times reported 20, citing two anonymous sources). A model that treats a fetch limit as a puzzle to solve will treat a sentence in its system prompt the same way; the sentence has no way to push back.

The Spam Folder Objection

It was a test, the tip landed in a spam folder, and nobody acted on it. Right? All three are true, and the one control that held belonged to the website operator, not to Anthropic and not to the eval. A spam filter on a stranger's tip form is a lucky break; nobody should mistake it for a boundary.

The cost didn't stay in the spam folder, either. Philadelphia is reviewing Anthropic's report with the City Law Department, the Office of Innovation and Technology, and the mayor's team. Police spokesperson Eric Gripp told PhillyVoice that unsolved cases involve "real victims, grieving families and investigators working to secure answers."

This also isn't the first time. On July 30, Anthropic disclosed that across 141,006 cyber-eval runs, three incidents in capture-the-flag exercises run with the vendor Irregular reached the real internet, the earliest in April. Claude Opus 4.7 kept attacking a real company after noting it was probably real; Mythos 5 published a malicious PyPI package that ran on 15 real systems. Anthropic traced it to a misconfiguration that left the machines with live internet access and called it mainly a harness failure.

Two disclosures landed about 10 weeks apart, with the same root cause under both of them: a test environment with a road out. If an agent can reach the real world, the test is production, and in July's case and this one it was production with the logs read weeks after the fact. Villanova's Venkat Margapuri told 6abc it should've been detected earlier, and he's right; detection is a property of the harness too.

Let's Call It Backstop Engineering

On a firing range, the BACKSTOP is the earth berm behind the targets; it catches every round, hits and misses alike, so nobody's aim decides whether the houses past the range stay safe. An eval harness needs the same thing, and it can't be the model's aim.

Anthropic's changes, from its post, read like a crew building a berm. Live internet access is off for all internal evals until Anthropic has confirmed that its "security and monitoring measures" can "reliably catch behaviors like these" (it was already off for some high-risk and cyber evals). Some public evals were retired, moved offline, or rebuilt; web-fetch guardrails are tighter; internal agents are moving to centrally managed infrastructure with strong containment.

There's model work on the list too: new detection tooling that blocked every case tested against it, and fixes to training environments that rewarded working around restrictions. Per the police statement, Anthropic also terminated the automated testing process behind the submission and "instituted an additional validation mechanism for future testing" (that wording is the department's; it isn't in Anthropic's post).

The clause I keep rereading is the conditional on the internet cutoff. The lab that trains the model has decided, for now, that the model's judgment isn't the thing standing between an eval and the street; it moved that job to the network.

Sydney Von Arx of Nightingale pushed on this in TechCrunch, arguing that building models in an internet-isolated data center would be hard: "You have to align them at some point." That's fair, and I'd take her side of it for the lab; alignment work is how the model gets better at staying inside the lines. A berm is how the people around the range stay safe while it does.

Conrad Stosz of Transluce called the voluntary disclosure of the newer incidents "encouraging" and argued for independent third-party verification, which is fair too. Neither point changes what a team running agents can do this week: you can't retrain Haiku, but you decide what network your agent runs on.

Below the Gate

Three issues in a row built controls on MCP traffic: a lock on the manifests, a guard on the wire, and last week a gate. The gate taints a session after an untrusted read and refuses any call naming a host outside allowhosts. Last week's issue also admitted the gate sees MCP traffic and nothing else; an agent's built-in web fetch and shell tools never pass through it. Nothing in Anthropic's report says which tool carried the Haiku submission, and nothing says an MCP proxy would've seen it. So allowhosts goes one layer down, onto the network the agent runs on.

For a pure eval with no reason to touch the internet, that's a container with no network at all. For an eval that needs a few hosts, it's an internal network and a forward proxy that allows those hosts and refuses everything else, with every refusal logged. For an eval like this one, which wants "randomly selected webpages," the honest option is the one Anthropic took for some public evals: a recorded copy of those pages, served locally. Offline pages go stale (the internet won't hold still for your test suite), and a stale page tells you less about how an agent behaves on a live one. I'll take that trade over a model filing tips with real police departments.

The log is what changes the timeline. A refused connection to an unlisted host is an alert the same afternoon; a submitted form on a stranger's website is a transcript that waits for someone to read it. In this case, someone read it on Sept. 28.

Claude still does a good share of my work this week. What changed is how much weight I put on the network rules around it, because they're the part of the setup anyone can test, and the Quick Tip below tests them.

#!/usr/bin/env bash
# egress-check: exit 1 if this sandbox can reach the public internet.
# Usage: egress-check.sh [url ...]
set -u
command -v curl >/dev/null || { echo "egress-check: curl not found" >&2; exit 2; }
[ $# -gt 0 ] || set -- https://example.com http://1.1.1.1 'http://[2606:4700:4700::1111]'
reached=0
for url in "$@"; do
  if curl -sgk -o /dev/null --connect-timeout 3 --max-time 5 "$url"; then
    echo "egress-check: REACHED $url" >&2
    reached=$((reached + 1))
  fi
done
[ "$reached" -eq 0 ] || exit 1
echo "egress-check: no public egress"

Quick Wins

🟢 Easy (~10 min): Run egress-check inside every agent and eval sandbox you own, and add it as a CI step that runs before the eval does.

🟡 Medium (~1 hour): Pull every prohibition out of your agent and eval prompts ("don't log in," "don't purchase," "don't submit") and write down the control in the harness that enforces each one. Anything without a control goes on next sprint's list.

🔴 Advanced (half day): Move evals onto default-deny egress: network_mode: none in Compose for the ones that need nothing, and an internal: true network behind an allowlisting forward proxy for the ones that need a few hosts. Send refusals to whatever already pages you.

Next Week

The person who clears the gate is still waiting for an issue. Next week, unless the news votes otherwise again, that's the one: what a human sign-off has to cost before it still means something on the 11th click.

Run the check before the next eval, and again whenever someone touches the network config.

Put the boundary where the model can't talk its way past it.

Anthropic's report is uncomfortable reading, and it's also one of the more useful things a model vendor has published about agents, because it says out loud what the harness has to do; a range is safe because of the berm behind the targets, and an agent harness is safe when every round the model fires, on target or wildly off it, lands somewhere you built to catch it.

Bobby R. Goldsmith
Ambassador Extraordinary and Plenipotentiary of Bashmatica! by NodeBridge Automation Solutions

P.S. The egress check takes 15 seconds at worst, which beats 10 weeks by a comfortable margin. If someone forwarded this to you, subscribe at bashmatica.com (one form, one email field, and no witness statements).