IronClaw bench

IronClaw Bench Stats

What this machine loaded, and what a test actually recorded.

Measured
Results from an actual test: speeds, VRAM, pass rates.
Documented
Models loaded, hardware used, components built.
Not yet measured
An experiment worth a future test. Never an invented result.

IronClaw local bench, selected measurements from 2026

Models tested locally: 8+
Conservative documented count
Documented
Tokens/sec: 95.7tok/s
GPT-OSS 20B measured generation speed
Measured
Largest model run: 35B
Qwen3.5 mixture-of-experts, quantized
Documented
GPU VRAM: 16GB
NVIDIA RTX 5060 Ti
Documented

Personal test results, not standardized benchmarks. Speeds vary with context, quantization and runtime settings.

The 8+ count is Llama 3 8B, GPT-OSS 20B, Gemma 4 26B, Gemma 4 12B, Qwen3.5 35B, Qwen3-Coder 30B, Mistral Small 24B, Ministral 3 14B. Variants are not counted separately.

Qwen3.5-35B-A3B was run on 20 June 2026 on the RTX 5060 Ti; about 35B total parameters but only about 3B active per token, so not equivalent to a dense 35B.

Observed local model speeds Measured

Scale, 0 to 120 tok/s

GPT-OSS 20B
MXFP4, llama.cpp
95.72 tok/sMeasured
Qwen3-Coder 30B
quantized Q3 variant
~82 to 84 tok/sMeasured
Gemma 4 26B-A4B
IQ3_XXS
~77 to 78 tok/sMeasured
Qwen3.5 35B-A3B
IQ3 variant
~64 to 68 tok/sMeasured
Gemma 4 12B
QAT Q4
~34 to 49 tok/sMeasured

Rounded recorded observations from different test sessions. Not a standardized comparison. Midpoints are used for ranges only to draw the bars; they are not additional measurements.

Actual IronClaw evaluation results Measured

Direct model test results: 61/64
September GPT-OSS general + IronClaw evaluation
Measured
Tool-contract successes: 9/16
GPT-OSS under the current-contract test
Measured
Median warm latency: 2.54s
direct 64-case evaluation
Measured
95th percentile latency: 6.34s
same evaluation
Measured

Separate internal tests, not a universal accuracy rating or independently validated benchmark. A 61/64 model baseline did not mean Lucy could finish 95% of real tasks; tool selection, source selection and integration exposed different problems.

Not yet measured

Why a good model can make a bad assistant

Upcoming article. Nothing measured here until it is written.

The actual IronClaw setup

Component	Setup	Label
Primary GPU	NVIDIA RTX 5060 Ti, 16GB	Documented
Processor	Intel Core i7	Documented
System RAM	48GB, mixed DDR4 modules	Documented
Operating system	Windows 11 Pro	Documented
Inference engine	llama.cpp / llama-server	Documented
Primary model	GPT-OSS 20B	Documented
Quantization	MXFP4 GGUF	Documented
Model file	approximately 11.28 GiB	Measured
Context configured	131,072 tokens	Documented
Parallel inference slots	1	Documented
GPU allocation	approximately 15GB in recorded loaded state	Measured
Observed generation	~95 to 100 tokens/sec in the main baseline	Measured
Cold prefill example	14.6 seconds with a ~43,313-token prompt	Measured
Warm cached-prefix example	~21 milliseconds	Measured
Primary interface	LibreChat, with Lucy and AgentGate	Documented

Prefill figures describe specific cache tests, not typical end-to-end response time. 131,072 is the configured context limit, not a promise that every request at that length works well.

Long context Measured

Prompt tokens tested: 120K
Large-context GPT-OSS run
Measured
Generated tokens/sec: 61.3
During that long-context test
Measured

Longer context has a performance cost, even when the model can technically accept it.

Not yet measured

Interesting tests. No numbers until they are run.

Not yet measured

GPT-OSS 20B vs Gemma 4 26B on the same tool-calling prompt set

Next testOne shared prompt set, both models, pass or fail written down.

Not yet measured

Speed at 8K vs 32K vs 64K context, same model and quant

Next testOne model, one quant, three context lengths.

Not yet measured

Tool-call pass rate through AgentGate

Next testA fixed set of calls, counted through AgentGate.

Not yet measured

VRAM headroom with two parallel slots

Next testTwo slots loaded, then record what VRAM is left.