decoder-layer — Transformer Decoder Layer (Pure C)

Part 12 of: Transformer from Scratch in Pure C

Goal: Implement the full "Attention is All You Need" paper in pure C, building up from primitives to a complete machine translation pipeline.

decoder-layer breaks the symmetry of the architecture, introducing Masked Self-Attention and Encoder-Decoder Cross Attention to construct the full decoder block.

---

Architecture Context

This project implements the exact Decoder Block architecture described in the original 2017 paper:

                  Decoder Input
                        │
                        ▼
            Masked Self Attention
                        │
                   + Residual
                        │
                        ▼
                    LayerNorm
                        │
                        ▼
                 Cross Attention
                 (Q ← Decoder)
                (K,V ← Encoder)
                        │
                   + Residual
                        │
                        ▼
                    LayerNorm
                        │
                        ▼
                  Feed Forward
                        │
                   + Residual
                        │
                        ▼
                    LayerNorm
                        │
                        ▼
                 Decoder Output

---

Highlights

• Pure C (C99), orchestrating the primitives into an asymmetric block.
• Generalized Attention API — Exposes explicit Q, K, V routing via mha_forward_generalized to cleanly isolate Cross Attention without modifying earlier projects.
• Causal Masking — Mathematically ensures auto-regressive generation capabilities by setting upper-triangle attention probabilities strictly to zero (-1e9f pre-softmax).
• Three-Stage Normalization — Adheres perfectly to the paper's triplet LayerNorm architecture.
• Built on encoder-stack ← | Foundation for transformer →

---

What decoder-layer provides

A single public header (include/decoder_layer.h) exposes:

Decoder Layer

typedef struct {
    MultiHeadAttention *masked_mha;
    MultiHeadAttention *cross_mha;
    LayerNorm          *ln1;
    LayerNorm          *ln2;
    LayerNorm          *ln3;
    FeedForward        *ffn;

    size_t d_model;
    size_t num_heads;
    size_t d_ff;
} DecoderLayer;

DecoderLayer* decoder_layer_create(size_t d_model, size_t num_heads, size_t d_ff);
Matrix*       decoder_layer_forward(DecoderLayer *decoder, const Matrix *decoder_input, const Matrix *encoder_output);
size_t        decoder_layer_num_parameters(const DecoderLayer *decoder);
void          decoder_layer_free(DecoderLayer *decoder);

Generalized MHA

Matrix* mha_forward_generalized(MultiHeadAttention *mha, 
                                const Matrix *query, 
                                const Matrix *key, 
                                const Matrix *value, 
                                bool causal);

---

Example usage

#include "decoder_layer.h"

int main(void) {
    size_t d_model = 512;
    size_t heads = 8;
    size_t d_ff = 2048; 
    
    DecoderLayer *decoder = decoder_layer_create(d_model, heads, d_ff);

    /* Encodings from the external encoder stack */
    Matrix *encoder_output = matrix_create(15, d_model);
    
    /* Shifted auto-regressive inputs */
    Matrix *decoder_input = matrix_create(10, d_model);

    /* Run the complete 3-stage residual pipeline */
    Matrix *output = decoder_layer_forward(decoder, decoder_input, encoder_output);

    matrix_free(output);
    matrix_free(decoder_input);
    matrix_free(encoder_output);
    decoder_layer_free(decoder);
    
    return 0;
}

---

Directory structure

decoder-layer/
├── include/
│   ├── decoder_layer.h        # Public API for Decoder Layer
│   └── generalized_mha.h      # API for Cross Attention / Masking
├── src/
│   ├── decoder_layer.c        # Orchestrator and memory ownership logic
│   ├── generalized_mha.c      # Causal masking and decoupled Q, K, V
│   └── demo.c                 # Visual pipeline checkpoint tracer
├── tests/
│   └── test_decoder_layer.c   # 13-test suite
├── Makefile                   # Links 7 previous upstream projects
├── README.md
└── Analysis.md

---

Build & Run

Requirements

• GCC (C99 compatible)
• GNU Make
• libm (math library)
• Seven previous projects: mini-tensor, neural-net, attention, multi-head-attention, layernorm, and feed-forward must be adjacent in the directory tree.

Build

make

Run tests

make test

Run visual demo

make demo

---

Pipeline Demonstration Output

=========================================
  Decoder Layer — Architecture Pipeline
=========================================

--- Model Details ---
d_model : 8
heads   : 2
d_ff    : 32

Masked MHA      : 256
Cross MHA       : 256
LayerNorm 1     : 16
LayerNorm 2     : 16
LayerNorm 3     : 16
Feed-Forward    : 552
--------------------------------
Total Parameters: 1112

--- Forward Pass Tracing ---

Encoder Output Shape : 5x8
Decoder Input Shape  : 3x8

Masked Self Attention
  ↓
Residual
  ↓
LayerNorm
  ↓
Cross Attention
  ↓
Residual
  ↓
LayerNorm
  ↓
Feed Forward
  ↓
Residual
  ↓
LayerNorm
  ↓
Decoder Output successfully generated (Shape: 3x8)!

---

Test suite

=== Decoder Layer Test Suite ===
Running test_decoder_create...
Running test_forward_shape...
Running test_masked_attention_shape...
Running test_cross_attention_shape...
Running test_mask_correctness_and_diagonal...
Running test_residual_connection...
Running test_layernorm_statistics...
Running test_parameter_count...
Running test_invalid_configuration...
[decoder-layer] ERROR: dimensions must be > 0.
[decoder-layer] ERROR: dimensions must be > 0.
[decoder-layer] ERROR: dimensions must be > 0.
[decoder-layer] ERROR: d_model must be divisible by num_heads.
Running test_cross_attention_dependency...
Running test_causal_property...
Running test_component_reuse...
All tests passed!

---

What's explicitly NOT in this project

Missing feature	Why	Where it belongs
Decoder Stack	Focus is solely on building a single asymmetric block	Part 13 — Transformer
Embeddings	Input to the layer must already be embedded	Part 04 / Part 05
Output Projection	Generating vocabulary logits is separate	Part 13 — Transformer

---

Engineering Design © 2026 Shahid Ul Islam.
Built with passion for Mathematical Rigor and Technical Excellence.