A switch is not a win.
VowLock Arm Evidence Engine proves that an Arm optimization actually ran, measures it against an identical baseline, checks deterministic output equivalence, and keeps the result even when it is not flattering.
Verdict: KleidiAI and I8MM were genuinely active, but this registered workload showed no material overall speedup.
One changed variable.
- Freeze the comparison.
Same Qwen2.5 1.5B Q4_0 file and hash, pinnedllama.cpp, four threads, prompt/generation sizes and Arm64 runner class. - Change KleidiAI only.
Build once withGGML_CPU_KLEIDIAI=OFFand once with it enabled. - Prove the backend.
The optimized log must select theCPU_KLEIDIAIbuffer and I8MM kernels; the baseline must not. - Measure and replicate.
Runs 4–6 executed the identical commit. Every run produced the same direction: a small prompt gain and a generation regression. - Verify output separately.
Run 7 requires both runtimes to produce exactlyREADYand rejects equal-but-wrong outputs.
Optimization needs an evidence trail.
Small local models can make private Setup Companion guidance possible where connectivity, specialist support and compute are limited. But enabling an accelerated backend is not evidence that the user experience improved. This repository is a reusable pattern for Arm developers: verify selection, compare fact-equivalent conditions, preserve raw evidence, replicate, and keep regressions visible.
Version 1 claim boundary: the one-token contract is an equivalence canary. It does not establish that free-form device guidance is factual, safe or useful. Version 1 did not independently measure peak process memory; Version 2 below does.
Three runs. One unchanged commit.
No login or workflow dispatch is required to inspect the sealed replication. Each run below links to its original GitHub Actions record.
| Run | Prompt baseline | Prompt KleidiAI | Change | Generation baseline | Generation KleidiAI | Change |
|---|---|---|---|---|---|---|
| 4 | 129.957 | 131.219 | +0.97% | 35.623 | 35.429 | −0.54% |
| 5 | 129.688 | 130.712 | +0.79% | 34.004 | 33.111 | −2.63% |
| 6 | 130.214 | 131.313 | +0.84% | 35.814 | 35.232 | −1.62% |
| Pooled | 129.953 | 131.081 | +0.87% | 35.147 | 34.591 | −1.58% |
Throughput is tokens per second. Runs 4–6 share experiment commit cf3ab598. Run 7 repaired the semantic verifier separately and is not pooled.
VERSION 2 · COMPLETE
From one benchmark to a reusable laboratory.
Version 2 was registered before execution. It compared Q8 and Q4, 1.5B and 0.5B models, three workload shapes and two native Arm Ubuntu images. It measured exact model size, cold-process peak RSS, cold first output and complete synthetic explanations.
Decision: retain Qwen2.5 1.5B Q4_0 on the baseline Arm CPU path. It passed all three fixtures with byte-identical explanations across both runtimes and both host images. Reject the 0.5B model: it was 58–69% faster but failed quality. Reject the tested KleidiAI switch: it did not improve cold explanation latency.
| Condition | Ubuntu 22.04 | Ubuntu 24.04 | Decision |
|---|---|---|---|
| 1.5B Q4 baseline | 3/3 · 5.77s mean | 3/3 · 5.78s mean | Retain |
| 1.5B Q4 + KleidiAI | 3/3 · 5.84s mean | 3/3 · 5.82s mean | Reject switch |
| 1.5B Q8 baseline | 0/3 · invalid text | 2/3 · contradiction found | Reject |
| 0.5B Q4 | 0–1/3 | 1/3 | Reject |
Separation rule: Version 2 is a new question and is never pooled into Version 1. Neither hosted runner exposed a real power counter, so power is reported as unavailable. All fixtures are synthetic; no device command was generated or executed.
COMPLETE · NATIVE ARM
The best model was no model.
Version 3 showed that runtime tuning could not repair a model that failed the product contract. Version 4 moved every decision, evidence identifier and next action into deterministic code. The model could only explain that immutable envelope; rejected prose fell back once to a deterministic template.
The hybrid passed 12/12 registered automatic checks only because six cases fell back. Its model server reached about 2.05 million KiB peak RSS. A disclosed qualitative audit also found that the surface verifier accepted misleading prose, so automatic acceptance is not presented as factual safety.
Decision: retain the deterministic template. Removing the model eliminates seconds of latency, roughly two gigabytes of resident model-server memory and an unverified natural-language failure surface. No human-comprehension advantage has been demonstrated.
COMPLETE · EXHAUSTIVE DECLARED DOMAIN
One permissible state. All 648 checked.
Version 5 replaced fixture sampling with a compiler over the complete registered seven-field state space. Every result carries its authoritative decision, decisive rules, exact evidence, stable state ID and deterministic output hash. No model writes or judges the policy.
| Rendering | Outputs/second | Full-domain p50 | Median bytes |
|---|---|---|---|
| B0 · brief | 50,378 | 12.856 ms | 799.5 |
| D0 · detailed | 43,359 | 14.947 ms | 1,052 |
| P0 · progressive | 33,351 | 19.439 ms | 1,600 |
The declared policy contained 600 STOP states, 47 REQUEST_EVIDENCE states and exactly one PASS. The native Arm process peaked at 23,168 KiB RSS and used zero model bytes or network calls.
Claim boundary: this is complete for the registered finite domain, not theorem-prover formal verification, real-device safety or evidence that people prefer or understand any rendering.