Runtime ISA Dispatch

gemmkit ships one engine. It picks the instruction set it runs on when your program starts, not when you compile it. Take a build made on a laptop and copied to a server. That binary uses the server’s AVX-512 if the server has it. The same binary, run on an older machine, quietly falls back to a narrower kernel. You do not select a backend, gate on a cfg, or rebuild per host. The first GEMM call detects the CPU’s features and caches the winning kernel. Every later call is then a plain indirect call through that cached pointer.

The backend roster

The set of kernels a build carries depends on the target architecture. Which one runs depends on the CPU. From fastest to slowest, the candidates are:

• AVX-512F on x86-64, the widest float kernel. 2 dot-product specializations sit alongside it for the narrow element types: AVX-512 VNNI (vpdpbusd) for i8 -> i32, and AVX-512 BF16 (vdpbf16ps) for bf16. These require the int8 and half features respectively, and the CPU must report the matching feature bit.
• FMA / AVX2 on x86-64, the widen-FMA kernel for machines without AVX-512.
• NEON on aarch64, where SIMD is baseline (every aarch64 CPU has it), so there is nothing to detect at runtime.
• simd128 on wasm32, chosen at compile time rather than runtime (see below).
• scalar, the portable floor. It exists on every target and runs when nothing better is available. A correct, if unaccelerated, result is always reachable.

Tile geometry is the one thing that changes per (element type, ISA) pair. The microkernel computes an MR x NR register tile, sized to the ISA’s vector width. For f32 the shipped tiles are:

All 5 run the same generic float microkernel. Only the tile shape differs. MR is MR_REG * LANES, so a wider vector buys a taller tile. f64 halves the lane count and therefore halves MR (AVX-512F f64 is 16 x 12, and so on). The VNNI and BF16 dot kernels use their own depth-grouped geometry, covered under Element Types. This table is background for reasoning about why a kernel packs and blocks the way it does. It is not a knob you set.

Automatic selection

Each element type owns a single dispatch slot: a OnceLock holding a typed function pointer. On the first call for that type, the selection ladder runs feature detection once and picks the best available kernel. It then stores that kernel’s monomorphized entry points (plain, prepacked, fused), plus the tile geometry. Whichever call happens first pays this one-time cost: the is_x86_feature_detected! probe and the OnceLock initialization. From then on, dispatch is a cached pointer load and an indirect call, with no per-call branching on the ISA. There is no transmute and no atomic pointer juggling behind this, just a typed slot per type.

A consequence worth stating plainly: there is no public API that reports which ISA was selected. The choice is an internal detail of the memoized slot. If you need to be certain a specific kernel is live, do not try to read it back. Pin it instead (next section), and let a mismatch fail loudly.

Pinning a kernel with GEMMKIT_REQUIRE_ISA

Setting the environment variable GEMMKIT_REQUIRE_ISA forces exactly one kernel end to end, instead of auto-selecting. The accepted values (case-insensitive, surrounding whitespace trimmed) are:

The avx512vnni and avx512bf16 pins select the dot kernel for their one narrow type. Everything else runs the plain AVX-512F path, so a mixed workload under one of these pins still runs correctly for its other types.

The contract is panic, not fallback. Dispatch panics, rather than silently running a different kernel, whenever the requested ISA is unavailable. 3 cases count as unavailable:

• the CPU does not report the feature
• the value names an ISA that does not exist on this target architecture (neon on x86, avx512f on aarch64)
• the value is an outright typo

This is deliberate, and it is exactly what you want for CI. A job whose whole purpose is to exercise the AVX-512 VNNI path must not pass by quietly testing the scalar fallback instead. That could happen if a feature flag was misspelled, or if an emulator was misconfigured. gemmkit’s own CI pins each kernel this way: it runs the x86 dot kernels under Intel SDE, NEON on aarch64, and simd128 on wasm. A broken pin then turns into a red build, instead of a false green.

The value is read once, before the first dispatch, and memoized alongside the kernel choice. Set it in the process environment before any GEMM runs. Changing it mid-process has no effect, because the slot is already populated. An unrecognized value is a hard error, precisely so it cannot be mistaken for auto and slip through.

WebAssembly is compile-time

wasm32 has no runtime feature detection, so simd128 is not chosen by probing the machine. It is selected by a compile-time cfg, and the build must actually enable it with -C target-feature=+simd128. Forget the flag, and the wasm build silently uses the scalar floor. Pinning GEMMKIT_REQUIRE_ISA=simd128 turns that silent degradation into an assertion: the build panics if the SIMD path is not live. This is why the wasm CI jobs pin it. See no_std and WebAssembly for the full wasm build story, including the threaded target.