↓Skip to main content
Zeitgeist — a spike by Chris Gathercole
  1. Creators/
  2. Simon Willison — Creator of Datasette; co-creator of Django/

ttok 1.0

Blog · Watch/Read · (gathered 2026-10-14)

  • Simon Willison released ttok 1.0, a tool designed to count and truncate text based on tokens, primarily to update its default tokenizer to align with the latest OpenAI GPT models.
  • The core change in ttok 1.0 is switching the default tokenizer from GPT-4 to the GPT-5/GPT-6 family.
  • This update is supported by an experiment by William Liu, which found that GPT-5.x and GPT-6 models (including 5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) likely share the same tokenizer, showing consistent token counts across a test corpus.
  • Users of ttok can now expect token counts that are consistent with OpenAI’s most recent and upcoming large language models, providing more accurate estimations for LLM interactions.