ML System Bottleneck Analyzer

Weight precision. Lower = smaller memory, faster decode, slight accuracy loss.

Sequence length (tokens) Total prompt + generated tokens per request. Click a preset or type a custom value.

Advanced model internals

Total Parameters (B) Hidden Size Number of Layers Number of Heads Architecture Family Transformer Dense Transformer MoE Transformer Hybrid SSM + Transformer State Space / Mamba-like Encoder-Decoder Multimodal Transformer Diffusion / DiT Audio / Speech Transformer

KV Heads

Intermediate Size

Active Params (B)

Experts

Active Experts

Routing Auto Dense MoE

Attention Auto Standard Grouped-Query Multi-Query MLA Hybrid Linear Sliding Window Hybrid SSM

Data Type float32 bfloat16 float16 int8 fp8 q4 Batch size 1 — interactive (single user) 4 — small batch 8 — medium batch 16 — large batch 32 — high throughput 64 — max throughput Concurrent requests processed together. Higher = better GPU utilization but more memory.

How to distribute across devices AUTO — pick the best strategy Pipeline (layers split across devices) Tensor (each layer split across devices) Data (replica per device, separate requests) Expert (MoE experts split across devices) Sequence parallelism Context parallelism Hybrid TP+PP Hybrid TP+DP AUTO enumerates all valid strategies and picks the highest decode rate. Inference optimization Standard inference Speculative decoding (2–3x speedup) Continuous batching PagedAttention (vLLM) Flash Attention EXO phase split (prefill/decode) Runtime framework AUTO — match hardware llama.cpp MLX / MLX-LM Ollama vLLM SGLang TensorRT-LLM EXO / MLX Distributed