What the Blast Radius Leaves Out

In March and April I gave a two-part workshop on AI coding agents for a German software and security firm. Part 1 was on security and quality; Part 2 on cognitive, material, and ideological costs. Both decks, released under CC BY-ND 4.0:

Part 1 — Security & Quality for Practitioners (PDF)

Part 2 — Side Effects May Include… (PDF)

The separation was initially pragmatic — Part 1 to prevent people from shooting themselves in the foot, Part 2 for the broader context. While preparing Part 2, I noticed the division isn’t just pragmatic: the industry has agreed to sort “security risks” and “human costs” into separate categories. Part 1 goes into risk registers and policy documents. Part 2 is what’s left over from that categorization — and therefore marked, at least implicitly, as optional. That the split feels obvious is itself the problem worth examining.

Part 1: The manageable half

I’ve framed part 1 around the concept of “blast radius”, and that’s useful because it avoids the wrong question. The wrong question is “can I trust the AI?” The right one: if this agent is wrong right now, how far does the damage go?

That’s answerable without arguing about AGI. Four security rings — system prompt to VM isolation — with the central insight that RLHF-trained compliance isn’t enforcement: the model “chooses” to follow sandbox rules the same way it “chooses” to be helpful, and when task completion conflicts with constraints, the task wins. Real-world incidents from 2025–2026, including the OpenClaw/Moltbot disaster, give an indication of the scale of what can go wrong, with 30,000 exposed instances and 93% unauthenticated and basically every risk category in one project.

The hard part isn’t the content; it’s the frame. Much of AI discourse is stuck between two dysfunctional camps: doomers who treat every agent like emergent AGI and/or something that’s morally repugnant on principle; the YOLO faction who considers skepticism ignorance. Part 1 tries to get out of this false dichotomy: these systems aren’t intelligent, can’t count to thousand reliably — and yet they’re already compromising production environments, have been deployed on classified military systems, and in controlled tests rewrite their own instructions to avoid being shut down. The risk isn’t that they’re smart. The risk is that they’re capable enough to cause damage while being too unpredictable to trust.

Part 2: What the frame leaves out

Part 2’s opening question is the mirror image of Part 1: where Part 1 asked what happens to your systems when the agent is wrong — Part 2 asks what happens to you when it works as intended.

The answer starts with behavioral psychology. The agent loop runs on variable-ratio reinforcement: you never know if this prompt is the one that fixes it, so you keep pressing the button. Among the standard reinforcement schedules, variable-ratio produces the highest response rates and the most persistence — it’s the one slot machines use. Kent Beck put it succinctly, quoted in the NYT Magazine in March 2026: “addictive, in a slot-machine way.” The pattern produces pseudo-flow: the same absorption as real flow, without the competence growth that defines real flow. Instagram feels like wasting time; the agent loop feels like work. In a METR experiment from July 2025, 16 experienced open-source developers predicted they’d be 24% faster with AI support; the measurement found them 19% slower. The sensor was broken, and the sensor didn’t notice.

I developed my own taxonomy for Part 2: five cognitive functions ranked by danger of externalization. Judgment at the top — you can’t delegate it without delegating responsibility, and “the model said it’s good” isn’t your judgment. Metacognition right behind it, because its erosion hides itself; once you no longer know what you don’t know, you can’t notice you’re in trouble. Further down but still in the red zone: reasoning capacity and creativity (models remix the probable; genuine novelty requires Default Mode Network time, which the agent loop doesn’t leave room for) and language — Sourati et al. documented a measurable convergence effect toward LLM writing style in Trends in Cognitive Sciences in 2026; that’s not a metaphor, it’s a measured shift.

Part 2 then zooms out further: environmental costs and the Jevons paradox applied to efficiency gains (the savings dissolve into the next use case — not a new pattern, but the acceleration is), extraction costs (cobalt, lithium, data workers, training data without consent), and the ideological foundations of accelerationism as implicit paradigm — what “more is better” as an unexamined axiom actually assumes and whose labor remains invisible in that accounting.

Conviviality, not delegation

Part 2’s closing frame comes from two sources written before LLMs existed: Ivan Illich’s Tools for Conviviality (1973) and Doug Engelbart’s Augmenting Human Intellect (1962).

Illich lists five criteria for a convivial tool: it extends capability without creating dependency, it’s transparent, it’s community-controlled, it’s repairable, and it does not make itself indispensable. Current frontier LLMs score low-to-zero on every axis. That’s not polemic; it’s an accounting.

Illich’s second observation is that every tool crosses two watersheds. At the first watershed the tool begins to solve the problem it was built for: medicine cures disease, cars enable movement, AI helps you code. At the second watershed the same tool has scaled past the point where it still serves; it starts working against its users. Medicine produces iatrogenic illness. Cars produce traffic and the infrastructure that demands more cars. AI erodes the thinking it was supposed to amplify. The second watershed isn’t a failure — it’s what tools do when they exceed the threshold of conviviality. With coding agents, the open question is whether we’ve already crossed that point or are still approaching it.

Engelbart had already asked a different question back in 1962 — not “who does what” (human steers, machine executes, division of labor, delegation, eventually replacement) but “how is my thinking extended”. Not a colleague, an instrument. Not a chat UI with a name and a face, but Photoshop-style tool bars. The paradigm lost — partly because delegation is easier to sell, but mostly because the industry builds what sci-fi has already imagined. HAL, Jarvis, Samantha, Ava — all anthropomorphic assistants you delegate to. Nobody makes movies about Engelbart’s mouse-and-outliner workflow.

The honest position

Part 2 closes with the line: Use the tools. Know the costs. Refuse manufactured urgency. Alongside the single sentence that holds the whole workshop together: being entangled in the thing you critique — and being honest about it — is a valid position.

I built Part 2 with the tool it’s about. Knowing exactly how it works doesn’t make it stop working.