Embla 1: A reasoning model built for phones

At 4 bits Embla 1 averages 57.4 on BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 and answers in 1.77 s end to end on an iPhone 18 Pro.^* In our measurements, the models that scored higher took at least twice as long on the same phone.

Embla 1 is a reasoning model built to run on a phone, private, offline and free per request. We designed it to be fast and to use few tokens per answer, and trained it on 16 nodes of 8× H100.

Architecture

Embla 1 is built to be fast on a phone in both prefill and decode. It has 32 layers of width 2048, with 8 of 96 routed experts and one shared expert per token. We use 15 sliding-window layers of 128 tokens, 12 sparse attention layers and 5 full attention layers. For more detail see The Architecture of Embla 1. It is built around the following:

• Sevren Sparse Attention. Full-attention layers use ReLU² instead of softmax. About 95% of attention weights are exactly zero and decode skips their values.
• Shared K/V caches. Sparse layers keep no cache of their own. They read from the preceding full-attention layer. K/V bridges build every second-half cache after layer 15 so prefill skips layers 16 to 31.
• Engram. The last two and three tokens are hashed into a 3B-parameter table. Rows they hit form a second input next to the token embedding for every layer.

Training

The model is pretrained on 1.4T tokens followed by a 20B-token extension to 256K context. For post-training we run SFT on 123B tokens. During the last 2,000 SFT steps we train with quantization-aware training (QAT) [1] and all later stages run at 4 bits. For RL we run 100 steps of on-policy distillation (OPD) [2], 800 + 200 steps of GRPO [3] and 300 steps of multi-teacher on-policy distillation (MOPD) [4]. Every stage before GRPO distils from a larger model [5].

Stage	Tokens or steps	Teacher or signal
Pretraining	1.4T tokens	Qwen3.5-35B-A3B-Base
Long-context extension	20B tokens	Qwen3.5-35B-A3B
SFT with QAT	123B tokens	Qwen3.5-35B-A3B
OPD	100 steps	Qwen3.8-27B and Qwen3.5-35B-A3B
GRPO	800 + 200 steps	Verifiers and a judge
MOPD	300 steps	GRPO-trained domain specialists

Pretraining

We pretrain at 4K context in two phases: 1T tokens without document masking, then 400B tokens with it. We use Qwen3.8's tokenizer pruned to 65,536 tokens. From the first step the model distils from Qwen3.5-35B-A3B-Base with 0.9 × KL on its top-256 logits plus 0.1 × cross-entropy. Its embeddings and output head start from the teacher's top principal components [6]. Muon [7] trains the weight matrices and AdamW the rest with 4M-token batches and a constant learning rate. We then extend the context to 256K over 20B tokens with Qwen3.5-35B-A3B Instruct as teacher.

Pretraining	Phase 1	Phase 2
Web text, filtered and synthetic	50%	43%
Code	19%	19%
STEM, reasoning and QA (synthetic)	10%	10%
Math	9%	8%
PDFs	8%	14%
Multilingual web (10 European languages)	5%	5%

Post-training

SFT. We train on 123B tokens in 32K packed sequences with Qwen3.5-35B-A3B Instruct as teacher on assistant tokens. Reasoning traces in math, code and STEM are capped at 8K tokens. One in ten reasoning prompts gets no reasoning. The last 2,000 steps run QAT in the phone runtime's 4-bit format: round-to-nearest in groups of 64 with a bf16 scale and offset per group.

SFT	Share
Chat	25.5%
Math	16%
Tool use	14%
Instruction following	12%
Code	12%
STEM	10%
Multilingual	6%
Long context, safety, other	4.5%

OPD. We run 100 steps with answers up to 8K tokens. Qwen3.8-27B and Qwen3.5-35B-A3B score each answer token by token. Both are pruned to our vocabulary and keep 99.1% of their probability mass.

GRPO. We run 800 steps with 16 answers per prompt and no reference model. Verifiers score math, code, tool calls and instruction following. Another 200 steps add knowledge questions judged by gpt-oss-120b [8]: +1 correct, 0 abstain, −1 wrong. The last 180 of them add a loop penalty [9] of −10⁻⁴ per token in a repeated 40-gram of a correct answer up to 0.5. Answers hitting the 16K limit fell from 20.7% to 1.8%.

MOPD. We run 300 steps with GRPO-trained specialists as teachers for four domains: tool use, instruction following, math and abstention.

Results

Score is the mean of BFCL, IFBench, AA-Omniscience, GPQA and MATH-500 with up to 16K tokens per answer and reasoning on. All models run at 4 bits: Embla 1 in the phone runtime's format, the others from their official Q4_K_M GGUF weights. Each model uses its lab's recommended sampling settings. Latency uses 4-bit builds on an iPhone 18 Pro with a 1,024-token prompt and 256 output tokens.

These are our own measurements: one phone, one evaluation harness and five benchmarks. Each average carries an uncertainty of about ±1 point, and other setups may rank the models differently.

Five of the twelve models score higher than Embla 1. LFM2.5-8B-A1B is 1.4 points ahead and takes 2.0× as long. LFM2.5-2.6B, Granite 4.2 3B, Gemma 4 E4B and Qwen3.5-4B are 2.6 to 5.3 points ahead and take 3.0 to 5.5× as long. Of the reasoning models we tested, Embla 1 writes the fewest tokens per answer:

Model	Mean tokens per answer	vs Embla 1	Score
Embla 1	888	1.0×	57.4
Gemma 4 E4B	1,577	1.8×	60.4
Gemma 4 E2B	1,611	1.8×	52.8
LFM2.5-8B-A1B	3,129	3.5×	58.8
LFM2.5-2.6B	3,557	4.0×	62.7
Granite 4.2 3B	5,633	6.3×	61.0
MiniCPM5-1B	6,461	7.3×	43.1
Qwen3.5-4B	6,619	7.5×	60.0

Embla 1 is clearly behind on math: 76.1 on MATH-500 against 89.6 for LFM2.5-8B-A1B and 90.5 for LFM2.5-2.6B. There is still a lot to improve.

The 4-bit weights of Embla 1 are available on Hugging Face.

References

1. Benoit Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", CVPR, 2018.
2. Rishabh Agarwal et al., "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes", ICLR, 2024.
3. Zhihong Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", arXiv:2402.03300, 2024.
4. Xiaomi LLM-Core Team, "MiMo-V2-Flash Technical Report", arXiv:2601.02780, 2026.
5. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, "Distilling the Knowledge in a Neural Network", arXiv:1503.02531, 2015.
6. Zhiqiu Xu et al., "Initializing Models with Larger Ones", ICLR, 2024.
7. Keller Jordan et al., "Muon: An optimizer for hidden layers in neural networks", blog post, 2024.
8. OpenAI, "gpt-oss-120b & gpt-oss-20b Model Card", arXiv:2508.10925, 2025.
9. Edward Yeo et al., "Demystifying Long Chain-of-Thought Reasoning in LLMs", arXiv:2502.03373, 2025.