The Architecture of Embla 1

Embla 1 is a 10B A1B mixture-of-experts model built for phones. Notable architectural decisions include:

• Sevren Sparse Attention saving ~60% of V bytes read at long context
• Prefill being ~2x faster due to KV bridges
• Engram table adding parameters without comparably more compute or memory
• ReLU² MLP activation saving ~78% of down-projection bytes read at B=1

1. Overview

   The model consists of 32 layers:
   15 Sliding window
   5 Full
   12 Sparse

   More on the layer schedule

2. 3 attention types

   Sliding-window attention processes local information and understands the position of previous tokens.

   Full attention sees the whole context and picks out the important tokens for Sparse attention.

   How each one works

3. KV bridge and reuse

   The Full layers in the second half of the model get their KV cache through a bridge, while the Sparse layers reuse the KV cache from the previous Full layer.

   More on the bridge

4. Prefill exits early

   Because of the KV bridges and reuse there is no need to run the second half of the model during prefill. This makes prefill roughly twice as fast.

   More on prefill

5. Attention residuals

   Layers come in blocks of four. Before its attention and again before its MoE, each layer reads a learned mix of the embedding, the engram, every finished block, and its own block so far.

   More on attention residuals

6. Engram

   A second input next to the embedding: rows looked up from the last two tokens and the last three tokens. Every layer from layer 1 on can read it.

   More on the engram

7. Experts

   Layer 0 is a dense MLP. Every other layer sends each token to 8 of 96 experts, plus one shared expert.

   More on the experts

The layer schedule

Embla 1 has 32 layers of width 2048 and a vocabulary of 65,536 tokens. The first half of the model uses sliding-window attention except for layer 7, which uses full attention. The second half is four blocks of four: a full layer (16, 20, 24 and 28) followed by three sparse layers that lean on it. That makes 15 window layers, 5 full and 12 sparse.

The full layers are the ones that become expensive at long context, so the layer schedule limits their use.

Three kinds of attention

Sliding window. Each window layer reads the 128 most recent tokens with an ordinary softmax, so every key in the window gets some weight, however small. These are the only layers that encode position with RoPE, so everything that needs word order locally gets it here.

Full. A full layer reads every token in the context and encodes no position (NoPE). It uses Sevren Sparse Attention, where ReLU² is the scoring function instead of softmax, which creates lossless sparsity. Read more about Sevren Sparse Attention here.

Sparse. A sparse layer has no keys or values of its own. It reuses its block's full layer's cache, restricted to the 1,024 keys that layer scored highest plus the 128 most recent. The selection goes by score, not by whether a key survived ReLU²: when fewer than 1,024 keys get a nonzero weight, the rest are keys the full layer scored just below zero, which ReLU² sent to exactly zero but the sparse layer can still use. When more than 1,024 survive, only the 1,024 highest-scoring are passed on. Over that selection it puts its own softmax, so unlike the full layer it never gives a key exactly zero: most get a small weight, a few get most of it. Like the full layers, it encodes no position.

Sevren Sparse Attention learns to be sparse

Sevren Sparse Attention is attention whose scoring function is ReLU² instead of softmax. For a query, a full layer gives each key a weight and returns the weighted sum of the values, divided by the sum of and the weights:

Here and are the RMS-normalised query and key. Any key whose score is zero or below gets exactly zero weight. A softmax never does that: every key gets some weight, however small, so every value has to be read.

At initialisation the normalised queries and keys point in random directions, so a score is as likely to be negative as positive and half of the weights are zero. We measured 50.8% on a randomly initialised copy of the model. Nothing in training asks for more: there is no sparsity loss term, no learned threshold and no top-k. Yet the model converges to about 95% sparsity during training.

Skipping V

Generating a token is limited by memory bandwidth, and at long context most of the bytes a full layer reads are its KV cache. A decode kernel has to read every key to score it, but it only needs a key's value when that key's weight is above zero. So a Sevren Sparse Attention kernel can read K, score, and fetch V only for the keys that survive.

In Embla 1's full layers, 8 query heads share each KV head, so a value can be skipped only when all 8 give its key zero weight. The heads that share a cache learn to ignore the same tokens, so this happens far more often than chance. Our kernel checks 8 keys at a time and skips about 60% of value reads at long context.

KV bridge, KV reuse, and prefill

In an ordinary decoder, a layer's keys and values for a token are projected from that layer's own input.

Three of Embla 1's second-half full layers break that link. Layer 20 projects its K and V from an input mixed at the same point in the model as layer 16. Layers 24 and 28 mix their input at the same point as layer 7.

The sparse layers add no cache at all, since they reuse their full layer's. Only the five full caches grow with the context, and by just 2.6 KB per token when KV is kept in 8-bit precision.

This matters most for prefill, where the bulk of the work is filling the KV caches and, in an ordinary decoder, every prompt token has to run through every layer to do it.

In Embla 1, every cache in layers 16 to 31 can be constructed after layer 15. That skips half of the layers for every prompt token, which on a phone is where most of the time to first token goes. Only the last prompt token, whose output is the first prediction, goes on through all 32 layers, as does every new token after it.

This follows YOCO [1], HySparse2 [2] and the Causal Encoder-Decoder of DeepSeek-V4.1-Flash [3], which also build later layers' caches from earlier hidden states so that prefill can stop early.

Attention residuals

In a plain residual stream, each layer adds its output to a running sum and the next layer reads that sum. This means the residual stream has to cater to every later layer, instead of each layer building the input it needs.

Embla 1 uses block attention residuals, introduced by the Kimi Team [4]. The 32 layers are grouped into 8 blocks of 4 layers.

The attention and the MoE of a layer each get their own mix. Before the attention, and again before the MoE, the model builds the input as a softmax-weighted mix of a few sources: the embedding, the engram, the output of every finished block, and the partial sum of its own block so far. For the MoE, that partial sum already includes the output of the same layer's attention. Each of the two mixes has its own learned query vector, so attention and MoE can weigh the sources differently. The weights depend on the token and sum to one, so a deep layer can reach straight back to an early block without it being diluted by everything in between.

Working by block keeps this cheap. A sublayer weighs at most ten sources instead of up to 65, and only one summed output per finished block has to be kept around, not one per sublayer.

Engram

The engram is a second input to the model, next to the embedding. The last two tokens and the last three tokens are hashed into a 3B-parameter table, and the rows they land on make up the engram. The lookup is ready once layer 0 has run, and from layer 1 on it is one more source for the attention residuals: it is never added to the residual stream, and each layer decides for itself how much of it to use. It is a lookup, so it costs a memory read rather than compute, which is why the table can be nearly half the size of the layers without changing how much work a token takes. The table doesn’t have to be held in memory either: it can be stored on SSD, and only the rows a token lands on are read.

DeepSeek-V4.1-Flash [3] and Qwen3.8-Flash-Next [5] use the same kind of n-gram memory, with tables prefetched from host memory.

Experts

Layer 0 has a dense MLP. Every other layer has 96 routed experts, 8 of them active per token, plus 1 shared expert.

Exact Quantile Balancing (EQB) keeps the experts in use. Each expert has a bias that is added to its router score only to choose the top 8; the weights the chosen experts get come from the unbiased scores. Instead of nudging the biases a fixed step after every batch, as in DeepSeek-V3 [6], EQB sets each bias after every training step to the exact cutoff that would have given its expert its share of the batch's tokens: one in 12, with 96 experts and 8 per token. A two-pass radix select over the bf16 scores finds that cutoff exactly across GPUs, at the cost of two small all-reduces per layer per step. There is no step size to tune and no auxiliary loss.

EQB comes from Aleph Alpha, who use it to train the 384 experts per layer of Kolibri-1; see the Kolibri technical report [7] (section 2.1.5) and Neitemeier et al. [8].

ReLU² activation

Embla 1's MLPs use a ReLU² activation: negative values become zero and positive values are squared. For any given token most neurons end up exactly zero, and a zero neuron means the matching row of the down projection contributes nothing and never has to be read. At batch size 1, where decoding is limited by memory bandwidth, a kernel can read only the rows of the neurons that fired. In Embla 1 this saves about 78% of the down-projection bytes read at B=1. This is the same idea as skipping values in attention. It is described in more detail in our Skipping dead neurons post, where it is applied to Nemotron 3.5 Lightning.

References

1. Yutao Sun et al., "You Only Cache Once: Decoder-Decoder Architectures for Language Models", arXiv:2405.05254, 2024.
2. Jianyu Wei et al., "HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing", arXiv:2609.26368, 2026.
3. DeepSeek-AI, "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression", arXiv:2609.19969, 2026.
4. Kimi Team, "Attention Residuals", arXiv:2603.15031, 2026.
5. Qwen Team, "Qwen3.8-Flash-Next Technical Report", GitHub repository, 2026.
6. DeepSeek-AI, "DeepSeek-V3 Technical Report", arXiv:2412.19437, 2024.
7. Aleph Alpha, "Kolibri: A Sovereign European Model on the Pareto Frontier", Technical report, 2026.
8. Pit Neitemeier et al., "Exact Quantile Balancing and Load-Error Injection for Mixture-of-Experts", arXiv:2609.28053, 2026.