---

_{Opening view of the survey. The canonical decoder fixes a pre-norm residual block, a dense feed-forward network, the same depth for every token, and a token-level key–value cache that grows with generation. The four panels are the directions developed in Section 15, Beyond Components: A Unified View of Architectural Design.}

⚡ Processing Flexible and conditional: who runs, and how much work a token receives.

🧠 State Selective and flexible: what history is kept, and how that state is updated.

🔭 Lookahead Delayed commitment: predict further ahead, then decide what to keep.

✨ Dynamics Refined transforms: the nonlinearity and the path across depth.

---

🎯 The question

Large language models have improved through scale, data, and training recipes. Their architectures have changed with them. Most deployed systems are still variations of the decoder-only Transformer in Section 4, Basic Modules in Transformer: multi-head self-attention, a position-wise feed-forward network, positional encoding, residual connections, and normalization, with tokenization as the interface to raw text.

Section 1, Introduction states the constraint that organizes the rest of the survey. Self-attention is quadratic in sequence length, and the key–value cache grows linearly with context. Dense activation, a fixed layer schedule, and one-token-at-a-time decoding add further training and serving cost. The survey asks how an architecture trades representational power, optimization stability, and computational cost when it relaxes one of those choices and keeps the others.

It differs from a component survey in three ways, all set out in Section 1:

1. It covers the full decoder, from tokenization and position through attention, feed-forward layers, normalization, residuals, the output objective, and depth allocation.
2. Section 14, Popular Architectures records how open-weight families actually combine those pieces.
3. Section 15 rereads the same mechanisms as assumptions of the canonical decoder that each design preserves, refines, or relaxes.

📈 Where open-weight models stand

Section 1 closes with two snapshots. MMLU against release date covers roughly 2021–2025 and is reproduced in the literature map. By 2026 that benchmark was nearing saturation, so the survey turns to Humanity’s Last Exam for the recent open-weight generation.

_{Humanity’s Last Exam versus open-weight model size (Section 1). The plot is concentrated on models released in 2026; GLM-4.7, from late 2025, is kept as a reference. The horizontal axis is total parameters on a log scale. Marker area grows with active parameters for mixture-of-experts models and with total parameters for dense models. Dashed and dotted curves are the observed Pareto frontiers with and without tools. Both start from Qwen3.6-27B (Dense); that model’s tool-use protocol is unspecified. Scores that mix tool-enabled and non-tool protocols should be compared with caution.}

Four earlier trends, read off the MMLU snapshot in Section 1, still frame this picture.

Nearly all of the models on the HLE figure are sparse mixture-of-experts models, and scores still rise with total parameter count along the observed frontiers.

---

🗺️ Survey map

Section 2, Overview is the roadmap. Sections 1 and 2 frame the tree; they are not a third technical family. The figure below is the taxonomy in Section 2, numbered as in the survey. Color follows the role of each block in the stack.

The color legend is the one used by the taxonomy in Section 2.

Sections 5 and 6 are two answers to the same cost: cheapen mixing over time. Sections 12 and 13 are two answers to another: decide how much computation a generated token receives.

📖 Chapters

Reading order is the order of the survey. Each note below is the claim of that chapter, not a second taxonomy.

🧭 1–2 · Frame

Section 1, Introduction places architecture next to scaling laws. Loss tracks model size, data, and compute fairly smoothly, but the architecture decides complexity, optimization dynamics, and what kind of state the model is allowed to keep. The section also locates modern attention in a longer line: LSTM pretrained models, then the 2017 Transformer, whose attention mechanism the survey traces to the 1991 unnormalized linear Transformer.

Section 2, Overview is the reading map drawn above. After the component chapters, the survey turns from layers to generation: how many tokens one forward pass is asked to predict, and how many layers each token actually traverses.

🧱 3–4 · Foundations

Section 3, History of Language Modeling Architectures starts from language as a stochastic process. Markov’s counts and Shannon’s n-gram source are the probabilistic problem. The architectural answers that follow are statistical n-grams, recurrent networks, fast-weight programmers, LSTM’s protected additive state, and then self-attention as a way to select over an explicit history instead of compressing it into a fixed state.

Section 4, Basic Modules in Transformer fixes the notation every later chapter uses.

💠 5 · Efficient Attention and KV Cache Management

This chapter keeps quadratic attention and reduces the key–value state that is stored or read. The four axes are orthogonal in intent and not freely stackable in practice.

Section 5.5, Future Directions singles out two open problems. Token reduction is the axis that can depend on the input, because sequence length is not an architectural constant. Combining axes is attractive and constrained: latent attention already shares one state across heads, so it does not sit cleanly on top of grouped-query attention or head-wise budgets.

🌊 6 · Linear Language Models

These models do not sparsify the attention matrix. They avoid explicit pairwise interactions, so cost scales linearly with length. Section 6 treats four families, then one skeleton.

Section 6.6, Future Directions puts the limit on the transition. An input-independent transition, or an input-dependent but diagonal one, cannot route information between state channels. Under the survey’s bounded-depth, log-precision assumptions, that places many of these models in TC⁰ and outside NC¹-hard state tracking such as permutation composition. Richer non-diagonal transitions, depth, and hybrids restore some of that power. A practical pattern already at large scale is to interleave linear layers with softmax attention, often near a 3:1 ratio. Training and prefill use chunkwise parallel kernels; decoding uses a fused recurrent update. Sequence recurrence in this chapter is not the depth recurrence in Section 13.

🟣 7 · Mixture of Experts

An expert layer is a router plus a bank of networks, usually feed-forward blocks. Only the selected experts run, so cost tracks active experts rather than the size of the bank. Section 7.1, Evolution of MoE in LLMs is one subsection with three overlapping developments, not three separate chapters.

Section 7.2, Future Directions leaves three problems open: better routing, where to place experts across depth, and heterogeneous experts. Current training systems assume experts have the same shape.

🍑 8–9 · Inside the block

Section 8, Activation Functions follows the feed-forward nonlinearity through four stages in Section 8.1: rectification (ReLU and its variants); smooth gates (GELU, SiLU, and Swish); gated linear units, with SwiGLU the default after strong empirical results; and scale-driven departures, including squared ReLU in some hybrid Mamba–Transformer families and clamped SwiGLU where unbounded branches threaten numerics. Section 8.2 expects the next step to be layer- or task-specific, hardware-aware, and not confined to the feed-forward network. Gating has already been tried on attention outputs.

Section 9, Normalization and Residual Connections separates the normalizer from the wiring.

Pre-RMSNorm is the wiring most current dense decoders share. That fact becomes one of the reference choices in Section 14.

🟢 10–11 · The sequence interface

Section 10, Positional Encoding supplies order to an operation that would otherwise be permutation-equivariant.

Section 11, Tokenization treats the vocabulary as architecture.

🪸 12–13 · Compute per generated token

Section 12, Multi-Token Prediction changes the objective before it changes the serving stack. Next-token training supervises one future token. Multi-token training supervises a horizon of K future tokens, which includes the ordinary next token. The extra predictors can be dropped at inference, leaving a standard model, or kept as a drafter.

The classification is by factorization. Joint pre-training versus a retrofit is a separate choice of when the predictor is attached.

Section 13, Adaptive and Recurrent Depth varies how many layers a token executes. If a binary mask says whether token i runs layer ℓ, effective depth is the sum of that mask. The standard decoder sets every entry to one.

Depth recurrence adds compute by revisiting a shared block. It does not summarize the sequence. That job belongs to Section 6.

🟡 14–16 · From components to a design space

Section 14, Popular Architectures stops listing modules and reads released models. Section 14.1 tells the history as four shifts: foundations and open replication; convergence on a reusable dense baseline; diversification of capacity and of state; then joint configuration of capacity, state, and prediction. Section 14.2 names the four patterns in Patterns. The survey’s dense-model table and mixture-of-experts table are the model-by-model record behind those patterns.

Section 15 is the teaser at the top of this page, written out as assumptions and alternatives. The dimensions are in Design space. Section 15.3, Interdependencies and Joint Design is the reason a module list is not enough: operator choice sets the form of history; sparse activation and compressed history spend different budgets inside one serving stack; tokenization sets the unit of both memory and the prediction horizon; residual paths have to stay stable when depth or the sequence operator changes; a longer draft creates state that may be discarded.

Section 16, Concluding Remarks draws three conclusions.

1. The canonical dense decoder, with a fixed schedule and the usual Pre-RMSNorm, SwiGLU, and rotary positions, is still the reference. Most of the survey relaxes one assumption and keeps the rest.
2. The valuable comparisons are joint configurations. Sparse capacity with compressed attention state, hybrid sequence and state processing, and prediction-augmented backbones show up because their pieces interact.
3. The usable design space is bounded by load balance, kernels, credit assignment, and caching. At least one hybrid family returned to full attention when the infrastructure was immature, and later moved to block-sparse attention. Those decisions get revisited.

The open measurement problems Section 16 names are scaling laws for prediction horizon, recurrent depth, and state size under matched training and inference compute; stable optimization of conditional and recurrent computation; and evaluations that report expressivity, efficiency, and stability separately.

---

🧩 Design space

Section 15.1 states the canonical computational structure. Every token uses a fixed tokenization and the same dense blocks. History is an append-only, token-level key–value cache. Each state is trained to predict the next token, and that token is committed before a longer continuation is compared.

Section 15.2 decomposes the four directions in the teaser into ten design dimensions.

A single mechanism can move more than one row. Mixture-of-experts changes parameter activation. Speculative decoding changes output commitment and, while candidates are alive, state update. The teaser draws the rows as four panels because that is how the survey asks the question, not because a model may enter only one panel.

🏗️ Patterns in released models

Section 14.2, Recurring Architectural Design Patterns defines a pattern as a combination. The four are not a partition. One backbone can instantiate more than one.

📐 14.2.1 · Canonical dense decoders Decoder-only stack, dense feed-forward blocks, fixed depth. Pre-RMSNorm, SwiGLU, and rotary positions became the usual way to fill that skeleton. LLaMA is the reference instance. Later dense families change the number of key–value heads or the attention span, as Mistral does with grouped-query attention and a sliding window, without leaving the dense path.

🎛️ 14.2.2 · Sparse capacity, compressed attention state Expert routing and cache compression spend different budgets. DeepSeek-V2 and DeepSeek-V3 pair fine-grained experts and a shared expert with multi-head latent attention. Other releases pair the same expert layout with grouped-query attention, or with alternating local and global attention. Co-occurrence is not evidence that either piece improves the modeling quality of the other.

🔀 14.2.3 · Hybrid sequence and state processing Softmax attention keeps an addressable history. A recurrent or linear operator compresses history as it is built. Griffin, Jamba, Nemotron, Qwen3-Next, and Kimi Linear interleave the two across layers. Hymba runs them inside a layer. LoLCATs converts a pretrained attention model after the fact. How much softmax attention remains decides how much of the cache is actually gone.

🔭 14.2.4 · Prediction-augmented backbones A multi-token objective or module sits on top of a dense, sparse, or hybrid trunk. DeepSeek-V3, MiMo-7B, LongCat-Flash, and MiMo-V2-Flash are instances. The module changes which future tokens supervise the representation. It does not, by itself, decide how a draft is verified or committed. That decision is Section 12.3.

---

🚀 Entry points

A longer, section-numbered literature map is in README_detailed.md. The rows below are the shortest path through the survey.

---

📎 Citation

If you find our paper helpful, please cite it. Thank you.

@article{llmarchsurvey2026,
    doi = {10.20944/preprints202610.1532.v1},
    url = {https://doi.org/10.20944/preprints202610.1532.v1},
    year = 2026,
    month = {October},
    publisher = {Preprints},
    author = {Yao Teng and Guoxuan Chen and Yimeng Chen and Han Shi and Piotr Piękos and Jie Ran and Yutao Zeng and Chuanyang Zheng and Jingyao Li and Li Nanbo and Haoli Bai and Chao Huang and Xihui Liu and Di He and Weiyang Liu and Jürgen Schmidhuber},
    title = {Towards Next-Generation Architectures of Large Language Models: A Survey},
    journal = {Preprints}
}

Paper on Preprints.org · doi:10.20944/preprints202610.1532.v1

👥 Authors

Yao Teng^1,* · Guoxuan Chen^1,* · Yimeng Chen^2,† · Han Shi^3,† · Piotr Piękos² · Jie Ran³ · Yutao Zeng⁴ · Chuanyang Zheng³ · Jingyao Li⁵ · Li Nanbo²
Haoli Bai³ · Chao Huang¹ · Xihui Liu¹ · Di He⁶ · Weiyang Liu^5,7 · Jürgen Schmidhuber^2,8

¹The University of Hong Kong  ·  ²Center of Excellence for Generative AI, KAUST  ·  ³Huawei Technologies  ·  ⁴ByteDance Seed
⁵The Chinese University of Hong Kong  ·  ⁶Peking University  ·  ⁷Max Planck Institute for Intelligent Systems  ·  ⁸The Swiss AI Lab, IDSIA-USI/SUPSI

_{* Equal contribution.    † Corresponding authors.}

---