Modular

Modular's argument is architectural. Most inference stacks are assembled from a serving engine here, an optimizer there, a custom-kernel toolchain somewhere else, and each seam is a place things break. Modular built MAX (serving), a PyTorch-like modeling layer, and Mojo kernels as one stack, and because MAX generates kernels per accelerator, the same container and OpenAI-compatible base_url follow you across NVIDIA, AMD, Google TPU, AWS Trainium, Qualcomm, Intel, ARM, and Apple silicon. That is the genuine differentiator: hardware leverage without a rewrite. The numbers Modular publishes are the kind you can check against your own workload rather than take on faith — 2x performance over vLLM on diverse hardware, sub-500ms mean time to first token in production, up to 70% total cost savings on AMD compute from one customer, and 15+ CPU+GPU architectures cited by an AWS VP in a customer quote. The rate card is public per model, including cache-hit pricing (DeepSeek V4 at $0.145 per 1M cached input tokens), which is unusual at this layer of the stack and makes pre-sales math possible. Where it gets harder: this is a platform, not a plug-in. Standing up MAX and Mojo assumes engineering resources, and writing custom kernels means learning Mojo — mitigated somewhat by the August 2026 Apache 2.0 open sourcing and the 26.6 release's Mojo 1.1 compiler, which invites external contributions, plus the new Community Grant Program funding ecosystem projects. On the pure self-hosted tier, autoscaling, burst capacity, and reliability are yours to run. Shared endpoints limit you to the hosted model list; custom and fine-tuned models need dedicated endpoints or BYOC. Forecasting spend means modeling your own traffic instead of reading a simple rate card, because dedicated capacity is billed per minute and reserved. Where it fits: infrastructure teams whose serving bill is large enough that hardware choice moves the total, enterprises serving frontier open models across multiple GPU vendors from one codebase, and conversational or real-time products that need sub-second TTFT with inference pinned inside their own VPC. Where it doesn't: one model on one GPU vendor, a team with no GPU or Mojo appetite that wants a fully managed no-code serving product, or anyone looking for training and fine-tuning — Modular is inference. Quiz: a FLUX.2-klein-4B user paying $1 per 1K megapixels on shared endpoints probably doesn't need the kernel layer; a DeepSeek V4 host at scale on mixed AMD and NVIDIA almost certainly does.