Magnitude

Magnitude competes on a claim the rest of the local-inference world mostly ignores: that the engine should adapt to your hardware, not the other way around. Its published numbers are specific — 466 to 507 tok/s prefill and 30 to 57 tok/s decode on an M4 Pro 48 GB against llama.cpp, and 2,033 to 2,507 tok/s prefill with 49 to 58 tok/s decode on a DGX Spark, measured on Qwen 3.6 35B A3B at 4-bit, 64k context, with no speculative decoding. Those are single-machine benchmarks run by the vendor, so treat them as direction rather than gospel, but the mechanism behind them is sound: kernels compiled and tuned on the actual device before a model runs, rather than precompiled for a hardware class. The operational details are where it earns its keep day to day. Concurrent sessions share prefix caches so a second agent doesn't make the first one crawl, memory per agent is 27% lower than the baseline it measures against, and memory is released when an agent stops — that's the difference between a background inference server you can leave running and one you keep killing. Models load on demand and unload when idle. Once a model is on disk, everything works offline, prompts and files included. The honest limits are about breadth, not quality. Magnitude hand-writes optimized kernels for popular open-weight families, and its own FAQ says the model list lives at magnitude.dev/models — if your model isn't in that catalog, you're outside the guided path. Any hardware works, from an Apple Silicon laptop to an NVIDIA or AMD GPU to CPU only, but there's no fixed minimum: smaller machines run smaller models, and how much memory you have decides how much model you can run. And if you already have a tuned Ollama or LM Studio pipeline, Magnitude is asking you to trade flexibility for tuning and a picker that does the configuration for you.