- /
- Creators/
- Simon Willison — Creator of Datasette; co-creator of Django/
- SQLite compressed text-history prototypes/
SQLite compressed text-history prototypes
Blog · Watch/Read
- Simon Willison proposes an efficient method for storing text revision histories in SQLite by compressing a JSON array of all prior document versions using Zlib or Zstandard.
- Initial prototypes demonstrated significant compression, reducing 20.4 MB of raw revision text (1,000 revisions) to just 80.3 KB when using Zstandard-compressed JSON.
- To optimize performance and avoid decompressing the entire history on every edit, the scheme can be refined by breaking the compressed history into multiple SQLite rows, each holding a maximum of 128 revisions or 3MB of uncompressed JSON.