LLM Architecture Explained: Why Nearly Every Open Model Uses the Same Four Blocks

In February 2020, Noam Shazeer proposed SwiGLU, a new feed-forward layer for transformers, in a short paper that ends with an unusual admission: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence." Six years later, that layer runs in almost every major open model, from Meta's Llama to OpenAI's gpt-oss. In Qwen3.8-27B, the open model that beats Claude Opus 4.6 Max at agentic coding, SwiGLU holds almost two-thirds of the parameters.

SwiGLU is one of four blocks these models share, all published between 2017 and 2021. Since then, labs have improved their models mainly with better data and more compute, not with new blocks. Learn these four blocks once and you can make sense of almost any open model. When a model modifies one of them, as Qwen3.8-27B does with attention, you can quickly understand what that change does and why it matters.

The four blocks open models share

Today's open LLMs are decoder-only transformers: a stack of layers with the same structure, in which each token can only look at itself and the tokens before it. Each layer takes one vector per token, refines it and passes it on to the next. Four blocks do that work:

In each layer, the blocks run in two steps: attention first, then the feed-forward block. RMSNorm rescales the input of each step. Around each step sits a skip connection, also called a residual connection: the step's result is added to the token's vector instead of replacing it, so every layer refines what the previous ones computed. Stack a few dozen of these layers, add an embedding table at the bottom and an output layer at the top that predicts the next token, and you have the skeleton of most major open LLMs released since 2023.

None of these blocks was invented for this recipe. Meta assembled the combination for its first Llama models in February 2023, taking each part from earlier models. What made Llama special was not its design but its release: Meta shared the trained model with researchers. Its recipe quickly became everyone's starting point.

How closely later models follow Llama's recipe

Look at the models that came after Llama, and you find the same blocks again:

• Mistral 7B describes itself as Llama with "a few changes", such as sliding-window attention, which limits how far back each layer can look.
• DeepSeek LLM states that its design largely follows Llama: RMSNorm before each block, SwiGLU and rotary embeddings.
• gpt-oss, OpenAI's open model, uses RMSNorm, rotary embeddings and its own variant of SwiGLU.
• Gemma, from Google, uses RMSNorm and rotary embeddings, and replaces SwiGLU with GeGLU, its sibling from the same 2020 paper.
• Qwen3.8-27B keeps RMSNorm and SwiGLU in all 64 layers, but replaces attention itself in 48 of them, a redesign that helps it compete with much larger closed-source models such as Claude Opus 4.6 Max.

Within a model family, a bigger model is usually the same architecture, scaled up. Going from DeepSeek LLM 7B to 67B mostly means editing a configuration file: three times as many layers and vectors twice as wide. Build these four blocks once, and they carry you from one model to the next.

Where the parameters go

The four blocks are far from equal in size. Counting Qwen3.8-27B's 26.9 billion language-model parameters block by block gives this split:

The feed-forward share is no accident. Since the original transformer, the feed-forward block has been designed with twice as many parameters as attention, which makes it two-thirds of a standard layer. SwiGLU kept that size, and Qwen3.8-27B makes the block wider still: about 70% of each layer, or 63.6% of the whole model once the embedding table and output layer are counted. Meanwhile, two of the four blocks cost almost nothing: the normalization layers hold well under a million parameters, and rotary embeddings have none at all.

So what do all those feed-forward parameters do? In large part, they act as the model's memory. The input side of the block works like thousands of pattern detectors: each one fires when a token, read in its context, matches a pattern the model has learned. The output side stores what goes with each pattern and pushes the prediction toward the words that usually follow it. A 2021 study found exactly this structure in trained models. Later work showed where a model keeps a specific fact. When GPT completes "The Space Needle is located in the city of" with "Seattle", the decisive step happens in the feed-forward blocks of its middle layers: editing just one of those blocks is enough to make the model answer "Paris" instead. Attention moves information between tokens; the feed-forward blocks store much of what the model knows.

Why the recipe has barely changed

Researchers have proposed plenty of changes to the original transformer. In 2021, a team at Google reimplemented dozens of them in a single codebase and tested them side by side. Most made no real difference: in their words, "most modifications do not meaningfully improve performance." Two of the few that did help are now in nearly every open model: RMSNorm and gated feed-forward blocks like SwiGLU. Both improved results without slowing training down.

Once a recipe has passed that kind of test, labs change it only for a strong reason. A frontier training run costs too much to gamble on an untested block, and a trick that helps a small model may not hold up at full scale. Meta's Llama 3 paper shows the same caution. It sticks to a standard architecture and credits its progress to data and scale instead: "our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale."

Where labs do change the recipe

When labs do have a strong reason, they usually change one block and keep the other three. In practice, the block they change is almost always the feed-forward block or attention.

Labs change the feed-forward block to add capacity. Mixture-of-experts models replace it with many separate feed-forward blocks, the "experts", and a router sends each token to only a few of them. Since the feed-forward blocks store much of what a model knows, experts let that memory grow to hundreds of billions of parameters while each token uses only a small fraction of them.

Labs change attention for a different reason: it is the only block whose cost per token grows with the length of the input, because each new token is compared with every token before it. Qwen3.8-27B tackles this head-on. In 48 of its 64 layers, it replaces attention with Gated DeltaNet, a layer that keeps a fixed-size memory of everything it has read instead of looking back at every earlier token. This keeps long inputs cheap, which matters most for agents: every file an agent opens and every command it runs stays in its context. This redesign is part of how a 27B model beats Claude Opus 4.6 Max at agentic coding. Training is the other part, as I explain in Qwen3.8-27B vs Claude Opus 4.6 Max.

Learn four blocks, read almost any open model

Four blocks published between 2017 and 2021 still make up most major open models. Learn them once, and every new release looks familiar: you recognize the standard recipe at a glance and can focus on the one block the model changes, such as attention in Qwen3.8-27B.

Common questions

Do all LLMs use the same architecture?

Most open LLMs released since 2023 share one skeleton: a decoder-only transformer built from RMSNorm, attention with rotary position embeddings, and a gated feed-forward block such as SwiGLU. The exceptions usually change a single block: mixture-of-experts models replace the feed-forward block with many experts, and models built for very long inputs, like Qwen3.8-27B, rework attention.

Do GPT and Claude use the same architecture as open models?

Nobody outside OpenAI and Anthropic knows, because closed labs don't publish their architectures. The best clue is what OpenAI releases in the open: its gpt-oss models use RMSNorm, rotary embeddings and SwiGLU. Anthropic hasn't released an open model, so for Claude there is no such clue.

What is SwiGLU?

SwiGLU is the feed-forward block used by most open LLMs. It projects each token's vector twice, passes one copy through the Swish activation and multiplies the two together, so one branch acts as a gate on the other. A third matrix then projects the result back to the model's width.

My course explains SwiGLU in depth, and you implement it yourself, along with every other block of Qwen3.8-27B. See the course →

Do mixture-of-experts models use the same blocks?

Yes. They keep RMSNorm, attention and rotary embeddings, and replace the single feed-forward block with many experts, plus a small router that picks a few of them for each token. OpenAI's gpt-oss-120b, for example, has 128 experts in each layer and uses four of them for each token, so only about 5 billion of its 117 billion parameters are active at a time.

Sources

• Noam Shazeer, “GLU Variants Improve Transformer” (2020).
• Biao Zhang and Rico Sennrich, “Root Mean Square Layer Normalization” (2019).
• Ashish Vaswani et al., “Attention Is All You Need” (2017).
• Jianlin Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021).
• Hugo Touvron et al., “LLaMA: Open and Efficient Foundation Language Models” (2023).
• Albert Q. Jiang et al., “Mistral 7B” (2023).
• DeepSeek-AI, “DeepSeek LLM: Scaling Open-Source Language Models with Longtermism” (2024).
• OpenAI, “gpt-oss-120b & gpt-oss-20b Model Card” (2025).
• Gemma Team, “Gemma: Open Models Based on Gemini Research and Technology” (2024).
• Mor Geva et al., “Transformer Feed-Forward Layers Are Key-Value Memories” (2021).
• Kevin Meng et al., “Locating and Editing Factual Associations in GPT” (2022).
• Sharan Narang et al., “Do Transformer Modifications Transfer Across Implementations and Applications?” (2021).
• Llama Team, AI @ Meta, “The Llama 3 Herd of Models” (2024).
• Qwen team, Qwen3.8-27B configuration file (2026).