Udgam Labs · EvaluationMeasured 19 Oct 2026

Small Model Board

27 open language models up to 400M parameters, each at a pinned revision, scored on eight standard benchmarks with one harness and one set of settings. These are our own runs, measured on 19 October 2026: not a certified leaderboard.

Our five models are in the table like any other. Laghu-Base is 5th of the 19 models up to 200M on the 5-task average, level within noise with 4th and 6th; Gargi-Base is 2nd of the 8 models from 200M to 400M, 7.3 points behind SmolLM2-360M.

Models27
Benchmarks8
Items per model46,098
Harnesslm-eval 0.4.13
Click a column to sort. Ranks are within each size group.
# Model · pinned revision Params
(M)		 ARC-E ARC-C HellaSwag PIQA WinoGrande Avg
(5)		 MMLU
letter		 MMLU
cloze		 Cloze −
letter		 GSM8K
5-shot
InstructInstruct or chat model (per its own card), scored without its chat template.
CodeRuns its own modeling code: we read it before running it and pinned that commit.
CalculatorQ-164M has a built-in calculator, so its GSM8K score is tool-assisted.
Udgam LabsOur models. Laghu-Base was evaluated from our training repo; its files are byte-identical to the linked revision.
ARC-C, HellaSwag and PIQA use length-normalised accuracy. Avg (5) is the mean of the first five columns.
Cloze − letter: MMLU cloze minus MMLU letter on the same 14,042 questions. Negative (bold) means letter format scores higher.

How these numbers were made

One harness, one setting

  • lm-eval 0.4.13, transformers 5.19.0, torch 2.14.1, one NVIDIA RTX PRO 6000 Blackwell GPU, bf16, batch size 64, seed 1234. Every model ran on 18–19 October 2026 (UTC) in one session.
  • Zero-shot ARC-Easy and ARC-Challenge (test), HellaSwag, PIQA and WinoGrande (validation), MMLU test in two formats; GSM8K test 5-shot, exact match after lm-eval's flexible-extract filter. lm-eval's default task configurations, no chat template, no system prompt.
  • MMLU letter shows the question with options A–D and compares the probability of each letter. MMLU cloze shows only the question and compares the probability of each full answer.

The BOS rule

  • Our models were trained with a <bos> token, so they ran with add_bos_token=True. Every other model ran with its own tokenizer defaults.

Batch size

  • Every model ran at batch size 64. In bf16, a log-likelihood can change slightly with the other sequences in its batch, so a re-run on other hardware or at another batch size may flip a few close items. We did not re-run to measure that.

Pinned and checked

  • Each model links to the exact commit we ran. Ten models need their own modeling code; we read that code without running it, checked it for network, shell, eval, pickle and file-write calls, and pinned the reviewed commit.
  • Every one of the 1,244,646 item results was recomputed from the logged log-likelihoods: all equal the per-item values lm-eval logged, and all 216 benchmark scores equal lm-eval's own results.
  • Parameters are as loaded by the harness, counting tied input and output weights once.

The data

  • Per-item right/wrong for 25 of the models, with ids but no question text: udgamlabs/small-model-board-scores. MobileLLM-R1-140M-base and nltower-100m-base have aggregate scores only, for licence reasons.

Where this board falls short

  • Our own runs, not a certified leaderboard. One harness version, one GPU type, one batch size. Other harnesses, prompts or hardware give different numbers; compare scores within this board, not across boards.
  • Instruct and chat models were scored without their chat templates, as plain text completers. That is not how they are meant to be used, and it can understate them.
  • Dated 19 October 2026. Newer models are not in it, for example Boris-2-Preview-2 (21 October). Three models could not run in the harness: lille-130m-base (no modeling code), GoLLeM-v5-128M (no model type in its config) and needle3 (a tool-calling model with no language-model head).
  • Small gaps are noise. Two 5-task averages less than about a point apart are not distinguishable. One model's MMLU score has a 95% interval of about ±0.7 to ±0.8 points.
  • Letter-format MMLU mostly measures format skill at this size. Always answering "D" scores 26.9%; 23 of the 27 models score at or below that in letter format, and several pick the same letter almost every time. Cloze is the more informative MMLU column here.
  • GSM8K is near zero for most models, so small differences mean little, and Q-164M's maths uses its built-in calculator.
  • We did not test for contamination. These benchmarks are public and old.
  • Size is parameters as loaded. Models count embeddings differently, so two models of similar size can differ in compute per token.