XSLM

Public definition · since October 2026

XSLM Extra Small Language Model

XSLM / noun

A language model with millions to a few hundred million parameters — one step smaller than LLMs and SLMs. It runs on small devices, costs almost nothing to run, and is built to do one job well.

A term coined by SONUB (sonub.com).

Build one in your browser Source on GitHub

How small is it — LLM, SLM, XSLM

Model size is the number of learned numbers (parameters). Each step down is about ten to a thousand times smaller.

Model Size (log scale) Parameters Where it runs
LLMLarge Language Model tens to hundreds of billions GPU servers in data centers
SLMSmall Language Model around a few billion a powerful computer or a single GPU
XSLMExtra Small Language Model millions to a few hundred million smartphones, wearables, IoT devices
Referencethe learning model in this repository 1,248 Python 3, no GPU

Three features

1

On-device

Light enough to run on its own, without a network connection, on smartphones, wearables and IoT equipment.

2

Very low cost

An SLM needs several GB of VRAM just to load. An XSLM runs in a few hundred MB of memory or less, so deploying and running it costs almost nothing.

3

A specialist for one job

Not an encyclopedia like an LLM, but one task done very well: summarizing, classifying text, function calling, or routing for an AI agent.

Models of this size in real use

  • SmolLM2-135M (Hugging Face, 2024) — about 135 million parameters. A GPT-style model that continues text and runs on smartphones.
  • MobileBERT (Google, 2020) — about 25 million parameters. Understands sentences to classify them or find an answer in a passage.

Shrinking a big model to XSLM size

  • Knowledge distillation — a small student model learns to follow the probabilities of a big teacher model.
  • Quantization — each parameter is stored as an 8-bit or 4-bit number instead of 32-bit. Memory shrinks 4 to 8 times.

How to build an XSLM — the basic method

An XSLM is a chain of small programs. Each one reads the file made by the previous one and makes the next file. Change an earlier file, and rebuild everything after it.

  1. 1

    Data

    prepare.py

    data/raw.txt → train.txt, val.txt. Q/A lines, split into training and validation text. The data format decides what the model becomes good at.

  2. 2

    Tokenizer

    tokenizer.py

    Character-level BPE. Learns merge rules and turns text into token IDs and back — decode(encode(text)) == text.

  3. 3

    Model

    engine.py model.py

    Autograd (Value, backward) and a GPT: token + position embedding, causal attention, MLP, residuals, output head.

  4. 4

    Training

    train.py

    Next-token prediction with cross-entropy loss and gradient descent. Saves a resumable checkpoint after every step.

  5. 5

    Generation

    generate.py

    Predict the next token, pick one, append, repeat — with temperature, top-k and seed.

  6. 6

    Runner

    runner.py

    Loads the weights once and serves an HTTP API: GET /health, POST /generate.

  7. 7

    XChat

    xchat.py

    A simple chatbot for testing the XSLM you built, in your web browser. It is only a test tool. What it is for

  8. 8

    Publish

    export.py

    Writes model.safetensors, config.json and tokenizer.json for Hugging Face.

What XChat is for

XChat is just for testing. It is a simple chatbot that shows whether the XSLM you built can answer. It is not the final product. Once the test goes well, the model moves on:

  1. 1

    Test

    XChat

    Chat with your model in the web browser and check its answers.

  2. 2

    Publish

    Hugging Face

    Upload the model so anyone can download it.

  3. 3

    Try it in common tools

    LM Studio, Ollama, llama.cpp

    Download the model and test it the way people test any other model. These tools only load the formats and architectures they support, usually GGUF, which export.py does not produce.

  4. 4

    Embed it

    ONNX and similar formats

    Convert the model and run it inside your own app.

Tokenizer — learn one BPE merge
pairs = Counter(zip(ids, ids[1:]))
pair, frequency = pairs.most_common(1)[0]
new_id = len(vocab)
vocab.append(vocab[pair[0]] + vocab[pair[1]])
ids = merge(ids, pair, new_id)
Model — causal attention at one position
q = linear(h, w['query'])
keys.append(linear(h, w['key']))
values.append(linear(h, w['value']))
scores = [dot(q, k) / width ** 0.5 for k in keys]
attention = softmax(scores)   # only past positions
Training — one step
loss = model.loss(window[:-1], window[1:])
loss.backward()
for p in model.parameters:
    p.data -= learning_rate * p.grad * scale
Generation — sample the next token
logits = model.forward(ids[-context:])[-1]
candidates = ranked[:top_k]
weights = [exp((logits[i] - peak) / temperature)
           for i in candidates]
token = rng.choices(candidates, weights=weights)[0]

Run the reference model

The reference/ folder is the complete course code. Pure Python 3, standard library only — no pip, no GPU, no API key, no pretrained weights.

Terminal
git clone https://github.com/thruthesky/xslm
cd xslm/reference
python3 prepare.py
python3 tokenizer.py
python3 engine.py
python3 train.py
python3 generate.py
python3 runner.py &
python3 xchat.py
What you will see
train 12
val 3
vocab: 40 merges: 12
roundtrip: OK [29, 13, 14, 1, 30]
y = 15.0 dy/dx = 8.0
parameters: 1248 resume: 0
validation: 3.7568
step: 1 loss: 3.6875
...
saved: checkpoints/model.json
XSLM Runner: ('127.0.0.1', 8765)
You> hi

The 1,248 parameters

WeightsShapeCount
Token embedding40 × 8320
Position embedding12 × 896
Query, key, value, project4 × (8 × 8)256
MLP expand, contract16 × 8 + 8 × 16256
Output head40 × 8320
Total — vocabulary 40, width 8, context 12, 1 block, 1 head1,248

After 20 training steps the answers are still mostly random — that is expected. The model is shrunk to the smallest size so you can read every calculation, but the parts are the same. Grow the same structure and train it on data for one job, and it becomes a practical XSLM.

Learn it step by step — in your web browser

The full course types and trains an XSLM inside a Linux terminal that runs in the web browser. Nothing to install.

  1. XSLM overview
  2. Development environment
  3. Project structure and data
  4. BPE tokenizer
  5. GPT model
  6. Training and checkpoints
  7. Text generation and sampling
  8. Runner
  9. XChat → Runner → XSLM
  10. Publish your model

Origin of the term

The term XSLM (Extra Small Language Model) was coined by SONUB (sonub.com) in October 2026, as the name of its course for learning to build an AI from scratch. This page is the public definition of the term.

How to cite

XSLM (Extra Small Language Model). SONUB, 2026. https://thruthesky.github.io/xslm/