Arm Create 2026 · Cloud AI

A switch is not a win.

VowLock Arm Evidence Engine proves that an Arm optimization actually ran, measures it against an identical baseline, checks deterministic output equivalence, and keeps the result even when it is not flattering.

PROMPT PROCESSING+0.87%129.95 → 131.08 tokens/s
TOKEN GENERATION−1.58%35.15 → 34.59 tokens/s

Verdict: KleidiAI and I8MM were genuinely active, but this registered workload showed no material overall speedup.

The experiment

One changed variable.

  1. Freeze the comparison.
    Same Qwen2.5 1.5B Q4_0 file and hash, pinned llama.cpp, four threads, prompt/generation sizes and Arm64 runner class.
  2. Change KleidiAI only.
    Build once with GGML_CPU_KLEIDIAI=OFF and once with it enabled.
  3. Prove the backend.
    The optimized log must select the CPU_KLEIDIAI buffer and I8MM kernels; the baseline must not.
  4. Measure and replicate.
    Runs 4–6 executed the identical commit. Every run produced the same direction: a small prompt gain and a generation regression.
  5. Verify output separately.
    Run 7 requires both runtimes to produce exactly READY and rejects equal-but-wrong outputs.
Why it matters

Optimization needs an evidence trail.

Small local models can make private Setup Companion guidance possible where connectivity, specialist support and compute are limited. But enabling an accelerated backend is not evidence that the user experience improved. This repository is a reusable pattern for Arm developers: verify selection, compare fact-equivalent conditions, preserve raw evidence, replicate, and keep regressions visible.

Version 1 claim boundary: the one-token contract is an equivalence canary. It does not establish that free-form device guidance is factual, safe or useful. Version 1 did not independently measure peak process memory; Version 2 below does.

Browser evidence replay · Version 1

Three runs. One unchanged commit.

No login or workflow dispatch is required to inspect the sealed replication. Each run below links to its original GitHub Actions record.

RunPrompt baselinePrompt KleidiAIChangeGeneration baselineGeneration KleidiAIChange
4129.957131.219+0.97%35.62335.429−0.54%
5129.688130.712+0.79%34.00433.111−2.63%
6130.214131.313+0.84%35.81435.232−1.62%
Pooled129.953131.081+0.87%35.14734.591−1.58%

Throughput is tokens per second. Runs 4–6 share experiment commit cf3ab598. Run 7 repaired the semantic verifier separately and is not pooled.

Registered follow-up

VERSION 2 · COMPLETE

From one benchmark to a reusable laboratory.

Version 2 was registered before execution. It compared Q8 and Q4, 1.5B and 0.5B models, three workload shapes and two native Arm Ubuntu images. It measured exact model size, cold-process peak RSS, cold first output and complete synthetic explanations.

Q8 → Q4 ARTIFACT−43.72%1.895 GB → 1.066 GB
Q8 → Q4 PEAK RSS−34.68%replicated across both host images

Decision: retain Qwen2.5 1.5B Q4_0 on the baseline Arm CPU path. It passed all three fixtures with byte-identical explanations across both runtimes and both host images. Reject the 0.5B model: it was 58–69% faster but failed quality. Reject the tested KleidiAI switch: it did not improve cold explanation latency.

ConditionUbuntu 22.04Ubuntu 24.04Decision
1.5B Q4 baseline3/3 · 5.77s mean3/3 · 5.78s meanRetain
1.5B Q4 + KleidiAI3/3 · 5.84s mean3/3 · 5.82s meanReject switch
1.5B Q8 baseline0/3 · invalid text2/3 · contradiction foundReject
0.5B Q40–1/31/3Reject

Separation rule: Version 2 is a new question and is never pooled into Version 1. Neither hosted runner exposed a real power counter, so power is reported as unavailable. All fixtures are synthetic; no device command was generated or executed.

Authority allocation · Version 4

COMPLETE · NATIVE ARM

The best model was no model.

Version 3 showed that runtime tuning could not repair a model that failed the product contract. Version 4 moved every decision, evidence identifier and next action into deterministic code. The model could only explain that immutable envelope; rejected prose fell back once to a deterministic template.

DETERMINISTIC TEMPLATE12/120.0816 ms median · no model artifact
CONSTRAINED MODEL6/123.0608 s median · 1.066 GB artifact

The hybrid passed 12/12 registered automatic checks only because six cases fell back. Its model server reached about 2.05 million KiB peak RSS. A disclosed qualitative audit also found that the surface verifier accepted misleading prose, so automatic acceptance is not presented as factual safety.

Decision: retain the deterministic template. Removing the model eliminates seconds of latency, roughly two gigabytes of resident model-server memory and an unverified natural-language failure surface. No human-comprehension advantage has been demonstrated.

Verified explanation compiler · Version 5

COMPLETE · EXHAUSTIVE DECLARED DOMAIN

One permissible state. All 648 checked.

Version 5 replaced fixture sampling with a compiler over the complete registered seven-field state space. Every result carries its authoritative decision, decisive rules, exact evidence, stable state ID and deterministic output hash. No model writes or judges the policy.

REGISTERED COVERAGE648/648three renderings · 1,944 distinct outputs
CORRUPTIONS REJECTED8/8authority, evidence and provenance mutations
RenderingOutputs/secondFull-domain p50Median bytes
B0 · brief50,37812.856 ms799.5
D0 · detailed43,35914.947 ms1,052
P0 · progressive33,35119.439 ms1,600

The declared policy contained 600 STOP states, 47 REQUEST_EVIDENCE states and exactly one PASS. The native Arm process peaked at 23,168 KiB RSS and used zero model bytes or network calls.

Claim boundary: this is complete for the registered finite domain, not theorem-prover formal verification, real-device safety or evidence that people prefer or understand any rendering.