Building a 200M Parameter Time Series LLM from Scratch

Decoder-only patched transformer for zero-shot forecasting

Fareed Khan

51 min read

·

Jun 29, 2026

--

9

Listen

Share

Building a single forecasting model that works on a series it has never seen, instead of training a fresh model for every dataset, remains a hard problem. The field is converging on one answer, treat forecasting like language modeling. Decoder-only transformers, the architecture behind modern language models, have become one of the strongest approaches to forecasting, and Google’s recently released decoder-based model reaches an accuracy that classical methods cannot match. So I set out to solve it from the ground up, building one from scratch, the model and the full training pipeline both, and scaling it to 200 million parameters.

The model we are building is a patched decoder, and it has six parts:

• Patch tokenizer: turns each 32-number window into one token.
• RevIN normalization: rescales each window so its units and size do not matter.
• Decoder stack: transformer layers where each token looks back at the earlier ones.
• Quantile head: outputs an uncertainty band, not a single number.
• All-positions training: every token predicts the stretch that follows it.
• Autoregressive…