Example, not a real testNot a measurement from the IronClaw machine.
A 20B model behind a fail-closed wrapper
I put a 20B open-weight model behind a tool-call wrapper that fails closed. The model can ask for a tool. If it asks for the same tool with the same arguments twice, the wrapper returns an error and stops. No silent retry. No “let me try that again” on my behalf.
The bench is invented for this note: one 24 GB card, one quant, a short system prompt, and a fake toolbox with three tools. Search notes. Read a file. Draft a reply. The file tool only sees a folder of sample notes written for the test. Nothing here is a real host, a real address, or a machine on a home network.
The question was small. Does “be careful” in the prompt stop a loop, or do I have to take the loop away?
I ran the same task ten times.
Prompt-only, four runs called the same tool again after it had already returned. Wrapped, zero runs got a third identical call. Two wrapped runs stopped at the error and wrote a plain answer. One wrapped run returned an empty draft. I count that as a miss, not a save.
| Component | Status | Note |
|---|---|---|
| Runtime | ok | llama.cpp, one model loaded |
| Wrapper | ok | Second identical call returns an error |
| Prompt-only control | warn | Looped on 4 of 10 |
| Empty draft | warn | 1 of 10 wrapped runs |
| Network tools | idle | Not in this test |
Strongest evidence yet that the stop condition belongs in the wrapper, not in a sentence the model can talk its way past.
The catch is that fail-closed also fails boring. A legitimate second read of the same note looks like a loop. I added a one-line exception for the read-only tool and left the write tool strict.
This is a Tuesday test that did what I asked. It is not a safety case, and the ten runs are sample runs.
end of log