Ml Llm Wiki

Machine Learning Wiki

A self-contained markdown knowledge base on transformer architectures, attention cost and efficiency, and long-context scaling. This skill is for _querying_ it: the knowledge is already compiled into articles under wiki/, so read those rather than re-deriving from scratch.

Keep this current: as the wiki grows, update the name and description above so they describe what it actually covers and trigger on the right questions.

(Sample note: this example wiki lives in examples/ within the llm-wiki repo. To load it as a skill, place the directory in your skills path named ml-llm-wiki, so the directory matches the name above.)

Maintenance and deeper analysis - ingesting sources, superseding stale knowledge, linting, auditing, critiquing reasoning - is not done here. Use the llm-wiki skill, which owns the write workflow and the file format. The llm-wiki skill is required to keep this wiki current; without it the wiki is still readable, but do not hand-edit articles outside the conventions in wiki/README.md.

What's inside

One topic so far, machine-learning: how attention works, why its memory cost was once thought to be a hard quadratic limit and why that turned out to be an implementation artefact, and what makes long context practical.

How to query

1. Read wiki/index.md - the catalogue, grouped by topic. Start here to find relevant articles. 2. Read the articles it points to. Follow body links for related material; grep -rl "<article>.md" wiki/ lists pages that link to a given article (backlinks). 3. If a local/ directory exists, search it too and fold in any relevant personal notes, labelling each hit as local/ (uncommitted) so it is never mistaken for shared, committed knowledge. local/ is the user's own, gitignored and absent from the index. 4. Answer from the wiki's content in preference to general knowledge. Cite articles with markdown links, e.g. [Attention Efficiency](wiki/machine-learning/attention-efficiency.md). 5. If a cited article has status: stale, say so and point to its replacement. Here, attention-cost.md is stale and superseded by attention-efficiency.md. 6. If the wiki has no answer, check wiki/gaps.md - the question may already be a tracked gap. Recording a new gap is a write, so it goes through the llm-wiki skill, not here.

Conventions

wiki/README.md explains the format - frontmatter, the raw/wiki split, and supersession-not-deletion - for anyone reading without a skill. Articles carry status: current | stale; stale pages are kept on purpose and point at their replacement.

Updating

To add a source, change an article, supersede knowledge, lint, audit, or critique, invoke the llm-wiki skill. It is required for all writes and keeps the format consistent. This skill deliberately does not modify the wiki.

Tips

• Use sub-agents with well defined goals, scope and context to parallelise work and reduce context rot in the main conversation.

Machine Learning Wiki

<!-- Note this is an EXAMPLE CLAUDE.md file for the EXAMPLE llm-wiki -->

This repository is an LLM wiki: a knowledge base on transformer architectures, attention cost and efficiency, and long-context scaling, compiled into plain markdown under wiki/ and maintained by an AI agent at the user's direction - the user curates sources and asks questions; the agent does the bookkeeping.

To work with it, start by reading SKILL.md in this directory - it explains how to query the wiki and the conventions to follow. The article catalogue is wiki/index.md; the format is described in wiki/README.md.

If the `llm-wiki` skill is available and not already active, activate it. It owns the workflow for every change to the wiki - ingesting sources, superseding stale knowledge, linting, auditing - and keeps the format consistent. Without it, treat the wiki as read-only and do not hand-edit articles outside the wiki/README.md conventions. raw/ is immutable source material.

General rules

• Always use Australian English spelling.
• Always use plain ASCII punctuation (straight quotes, single hyphens), Do NOT use any "smart" formatting such as smart quotes, em-dashes, en-dashes or non-breaking spaces.

Git commits

When you have completed all work, you may wish to ask the user if they'd like you to commit your changes to git, if doing so ensure your git commit message is clear and concise as to what has been added or improved.

(Sample note: this example wiki lives in examples/ within the llm-wiki repo.)

Prep: internal talk on attention scaling

Personal notes for the team session. This file lives in local/, so it is gitignored and stays in this clone only. The prep/2026-06-04-... path follows the optional dating hint: date-prefix notes tied to an event so they stay sortable. It links into the shared wiki (allowed); nothing in the wiki links back here (not allowed).

Angle

Open with the quadratic-cost framing, then show why it stopped being the binding limit. The wiki already has the through-line:

• Why Transformers Scale - the narrative spine for the talk.
• Attention Efficiency - the IO-aware result that reframed the cost.
• Transformer Architectures - background for anyone new.

To do before the session

• Pull two concrete sequence-length numbers from the FlashAttention raw source.
• Decide whether to mention the superseded Attention Memory Cost page as a "here is how the framing shifted" aside.

If any of this hardens into something worth keeping, promote it through a normal ingest rather than leaving it here.

Attention Is All You Need

Vaswani et al., 2017. Introduces the Transformer, a sequence model built entirely on attention, with no recurrence and no convolution.

The core component is scaled dot-product attention: queries, keys, and values are projected, and the output is a weighted sum of the values where the weight of each value comes from the dot product of the query with the corresponding key, scaled by the square root of the key dimension. Multi-head attention runs several of these in parallel on different learned projections, then concatenates the results.

Because the model has no recurrence, position is injected through fixed sinusoidal positional encodings added to the input embeddings. The architecture is an encoder-decoder: the encoder maps the input sequence to a continuous representation, and the decoder generates the output one token at a time, attending to both the encoder output and its own previous outputs (with masking so it cannot attend to future positions).

The paper's headline result is that attention alone, without recurrence, both trains faster (it parallelises across sequence positions) and reaches higher quality than the recurrent and convolutional models that preceded it. The authors note that self-attention relates any two positions in constant path length, which helps the model learn long-range dependencies.

A known cost: self-attention compares every position with every other position, so compute and memory grow with the square of the sequence length. The paper frames this as the main scaling concern for very long sequences.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Dao et al., 2022. Argues that the bottleneck in attention on modern GPUs is not arithmetic but memory movement between the slow high-bandwidth memory (HBM) and the fast on-chip SRAM. Standard implementations materialise the full attention matrix in HBM, which is what makes memory grow with the square of the sequence length.

FlashAttention computes the same exact attention without ever writing the full matrix to HBM. It tiles the computation, streams blocks of queries, keys, and values into SRAM, and uses the online softmax trick to combine partial results, so the output is identical to standard attention. In the backward pass it recomputes attention blocks rather than storing them, trading a little extra compute for a large reduction in memory traffic.

The result: memory use becomes linear in sequence length rather than quadratic, and wall-clock time drops because the kernel moves far less data. This makes much longer context windows practical on the same hardware.

The reframing matters as much as the numbers. The quadratic memory cost of attention had been treated as an algorithmic property of the method. FlashAttention shows it was largely an artefact of how the computation was mapped onto the memory hierarchy: an IO problem, not an inherent limit of attention.

Example sample vault

An illustrative wiki in the format this skill produces. It is small and self-consistent so you can see how the pieces fit, including supersession, a crystallised query, and a knowledge-gap register. The content is a worked example, not a real knowledge base.

Open this examples/ directory as an Obsidian vault, or read the files directly. The relative links resolve within the tree.

Layout

examples/
├── SKILL.md                                 (load the wiki as a query-only skill)
├── raw/machine-learning/
│   ├── 2017-06-12-attention-is-all-you-need.md
│   └── 2022-05-27-flashattention.md
└── wiki/
    ├── README.md
    ├── index.md
    ├── log.md
    ├── gaps.md
    └── machine-learning/
        ├── transformer-architectures.md     (current)
        ├── attention-efficiency.md          (current, supersedes attention-cost)
        ├── attention-cost.md                (stale, superseded)
        └── why-transformers-scale.md        (archive)

What each file shows

Raw vs compiled

The raw files keep the source text as collected. The wiki articles distill and reorganise that material, add provenance and cross-references, and stay current as new sources arrive. The pair attention-cost.md (stale) and attention-efficiency.md (current) shows knowledge being superseded rather than overwritten: the old page remains, marked stale and pointing at its replacement, and git carries the full history.

Knowledge Gaps

Known unknowns. Open gaps are ranked by evidence of demand, never by a score.

machine-learning

[open] question | How does FlashAttention-2 improve on the original?

• Raised by: Attention Efficiency
• Asked: 2026-04-06, 2026-04-08

[open] wanted | Online softmax

• Referenced by: Attention Efficiency
• Noted: 2026-04-05

[resolved] question | Is attention's quadratic memory cost a hard limit? -> Attention Efficiency (2026-04-05)

Knowledge Base Index

machine-learning

How sequence models work and what governs their cost and scaling.

Wiki Log

<!-- Format: ## [YYYY-MM-DD] op | title. Sub-items note created, updated, and superseded articles by title. grep "^## \[" log.md | tail -5 for recent activity. -->

[2026-04-03] ingest | Transformer Architectures

• Created: Attention Memory Cost (separate concept from the same source)

[2026-04-05] ingest | Attention Efficiency

• Updated: Transformer Architectures
• Superseded: Attention Memory Cost -> Attention Efficiency

[2026-04-06] query | Archived: Why Transformers Scale to Long Context

[2026-04-06] lint | 1 issue found, 1 auto-fixed

[2026-04-06] audit | transformer-architectures.md: 4 claims, 4 supported, 0 unsupported/partial

[2026-04-06] critique | attention-efficiency.md: reasoning sound, no key issues

Attention Memory Cost

[!warning] Superseded by Attention Efficiency (2026-04-05). Kept for history.

Sources: Vaswani et al., 2017-06-12

Raw: Attention Is All You Need

Overview

This page recorded the early view that the quadratic memory cost of self-attention is an inherent property of the method and the binding limit on sequence length. That view no longer holds; see Attention Efficiency. The page is kept because it explains why long-context work was framed as blocked before 2022.

The original claim

Self-attention compares every position with every other position, so the attention score matrix has a size that grows with the square of the sequence length. The original Transformer work treated this as the main scaling concern for long sequences, with no practical workaround that preserved exact attention.

Why it is now stale

The claim conflated an algorithmic property with an implementation detail. The quadratic term is real for the score matrix, but materialising that matrix in slow memory is a choice, not a requirement. Attention Efficiency covers the IO-aware approach that computes the same result in linear memory.

Attention Efficiency

Sources: Dao et al., 2022-05-27; Vaswani et al., 2017-06-12

Raw: FlashAttention; Attention Is All You Need

Overview

The memory cost of attention is not the hard limit it was once taken to be. Standard implementations materialise the full attention score matrix, which makes memory grow with the square of sequence length (Attention Is All You Need). FlashAttention shows this is an artefact of how the computation is mapped onto GPU memory rather than a property of attention itself (FlashAttention). This page supersedes the earlier Attention Memory Cost.

The IO-aware view

On modern GPUs the bottleneck is moving data between slow high-bandwidth memory and fast on-chip SRAM, not arithmetic. FlashAttention tiles the computation, streams blocks of queries, keys, and values through SRAM, and combines partial results with an online softmax, so the full matrix is never written to slow memory. The output is exact, identical to standard attention. The backward pass recomputes blocks instead of storing them, trading a little compute for a large drop in memory traffic.

Result

Memory use becomes linear in sequence length rather than quadratic, and wall-clock time drops because far less data moves. Longer context windows become practical on the same hardware. The reframing is the durable lesson: a cost treated as algorithmic turned out to be an IO problem.

See Also

• Transformer Architectures
• Attention Memory Cost (superseded)

Transformer Architectures

Sources: Vaswani et al., 2017-06-12

Raw: Attention Is All You Need

Overview

The Transformer is a sequence model built entirely on attention, with no recurrence and no convolution. Removing recurrence lets the model process all positions in parallel during training, which is the main reason it trains faster than the recurrent models it replaced while reaching higher quality.

Components

Scaled dot-product attention takes projected queries, keys, and values and returns a weighted sum of the values, where each weight is the dot product of a query with a key, scaled by the square root of the key dimension. Multi-head attention runs several of these in parallel on different learned projections and concatenates the results, so the model can attend to several kinds of relationship at once.

Because there is no recurrence, position is supplied explicitly through sinusoidal positional encodings added to the input embeddings.

Encoder-decoder structure

The encoder maps the input sequence to a continuous representation. The decoder generates output one token at a time, attending to both the encoder output and its own previous tokens, with masking that prevents it from attending to future positions.

Scaling characteristic

Self-attention relates any two positions in constant path length, which helps with long-range dependencies. The cost is that every position is compared with every other position, so the naive implementation grows with the square of the sequence length. Whether that quadratic cost is a hard limit is taken up in Attention Efficiency.

See Also

• Attention Efficiency

Why Transformers Scale to Long Context

Sources: Transformer Architectures; Attention Efficiency

Archived: 2026-04-06

Question

If self-attention compares every position with every other, why are long context windows practical today?

Findings

Two separate properties combine. First, the Transformer drops recurrence, so all positions are processed in parallel and any two positions sit at constant path length, which is what makes attention good at long-range dependencies in the first place (Transformer Architectures).

Second, the memory cost that once capped sequence length was an implementation artefact, not an algorithmic limit. IO-aware methods compute exact attention in linear memory by never materialising the full score matrix (Attention Efficiency). The earlier framing that the quadratic cost was a hard ceiling has been superseded.

So the short answer: attention gives the modelling benefit, and IO-aware kernels remove the memory penalty that used to make long context impractical.

Map

Two paths converge on practical long context; the superseded view is shown dashed and grey. Archive snapshot as of 2026-04-06.

flowchart LR
    arch["Transformer Architectures"]:::focus
    eff["Attention Efficiency"]:::current
    cost["Attention Memory Cost"]:::stale
    out(["Long context is practical"]):::archive

    arch -->|"enables long-range modelling"| out
    arch -->|"naive cost is quadratic"| cost
    cost -->|"superseded by"| eff
    eff -->|"removes the memory penalty"| out

    classDef focus    fill:#cfe2ff,stroke:#2f6fb0,stroke-width:2px,color:#0d2a44;
    classDef current  fill:#e8eef6,stroke:#4a5b6e,color:#16202c;
    classDef stale    fill:#ececec,stroke:#9aa0a6,stroke-dasharray:4 3,color:#5f6368;
    classDef archive  fill:#fdebcf,stroke:#c8862a,color:#5a3e12;

Lessons

• A cost that looks algorithmic can turn out to be a systems problem; check where the data actually moves before treating a bound as fundamental.
• "Exact but cheaper" beats "approximate" when an implementation change can deliver it.

See Also

• Transformer Architectures
• Attention Efficiency

About this wiki

A personal knowledge base of plain markdown, built and maintained with the llm-wiki skill (part of https://github.com/sammcj/agentic-coding). It is meant to stay readable and editable without that skill or any other tool: every file is standard markdown with YAML frontmatter and relative links, so it renders on GitHub and opens as an Obsidian vault.

Structure

• raw/<topic>/ - immutable source material, one markdown file per source. Read, never edited after capture.
• wiki/<topic>/<article>.md - compiled articles, distilled from raw/ and owned by whoever maintains the wiki.
• wiki/index.md - the catalogue: one row per article, grouped by topic. Start here.
• wiki/log.md - append-only history of ingests, queries, lints, and audits (recent activity kept; full history in git).
• wiki/gaps.md - register of known unknowns: concepts the wiki references but has not written, and questions it cannot answer yet.
• local/ - optional, gitignored. Personal notes kept in this clone only and never committed. It may link into wiki/ and raw/, but nothing committed links into it.

Conventions, if you maintain this by hand

• Every article carries frontmatter: title, type (concept | entity | archive), topic, created, updated, status (current | stale), and superseded_by. Bump updated whenever an article's content changes, and keep the matching index.md row in step.
• Replace outdated knowledge by superseding, not deleting: set the old article's status: stale and superseded_by, add a > [!warning] Superseded by ... callout, and write the replacement as a new current article. Keeping the old page is the point - it explains why the current state exists.
• Cite sources, do not score them. Link the raw/ file, and for a load-bearing claim point at the exact spot (a section, page, or timestamp), rather than attaching a confidence number a reader cannot check.
• Keep the wiki in git. History, rollback, and "how did this get here" come from version control, not from bespoke fields.

For the full ingest, query, lint, and audit workflow, use the llm-wiki skill.