↓Skip to main content
Zeitgeist — a spike by Chris Gathercole
  1. Creators/
  2. Simon Willison — Creator of Datasette; co-creator of Django/

SQLite compressed text-history prototypes

Blog · Watch/Read

  • Simon Willison proposes an efficient method for storing text revision histories in SQLite by compressing a JSON array of all prior document versions using Zlib or Zstandard.
  • Initial prototypes demonstrated significant compression, reducing 20.4 MB of raw revision text (1,000 revisions) to just 80.3 KB when using Zstandard-compressed JSON.
  • To optimize performance and avoid decompressing the entire history on every edit, the scheme can be refined by breaking the compressed history into multiple SQLite rows, each holding a maximum of 128 revisions or 3MB of uncompressed JSON.