PelatihDictionary encoding stores a column’s distinct values once and each value as an index into them. Run-length encoding stores each run of equal consecutive values as the value and the run’s length. Delta encoding stores the first value and then each value’s difference from the previous one, usually as the excess over the smallest difference in a block. Bit packing stores small non-negative integers in the fewest bits that hold the largest of them, with no byte boundaries between values.
| column (50 000 rows) | plain | dictionary | run-length | delta | best |
|---|---|---|---|---|---|
| date | 200 000 | 13 | 15 | 449 | dictionary |
| symbol, capture order | 400 008 | 37 917 | 404 432 | – | dictionary |
| symbol, sorted by symbol | 400 008 | 37 917 | 480 | – | run-length |
| timestamp | 400 000 | 500 009 | 406 259 | 201 085 | delta |
| bid | 400 000 | 99 097 | 410 505 | 137 947 | dictionary |
| bid size | 200 000 | 37 705 | 208 072 | 87 948 | dictionary |
| venue | 400 008 | 12 549 | 318 436 | – | dictionary |
firm.colfile’s encodings, before any compression; strings are stored plain as offsets and bytes, and delta encoding applies to integers only. Data: pl_colfmt.encodings_by_column.