Time-Series & IoT Storage Engines: TSM Trees, Gorilla Compression & Beyond
"What tree do IoT databases use?" has a short answer and a long one. The short answer: almost none of them use a plain B-tree for the data itself. They use log-structured, time-partitioned, columnar layouts. Think of an LSM tree, specialised for data that arrives roughly in time order and is almost never updated. InfluxDB calls its version a TSM (Time-Structured Merge) tree. Prometheus calls it blocks and chunks. VictoriaMetrics and ClickHouse call it a MergeTree. TimescaleDB is the odd one out. It does use Postgres B-trees, and it gets away with it because it cuts the table into small time chunks first.
The long answer is this page. Time-Series Databases compares the products. This page opens them up and shows how the bytes are laid out on disk, how a float shrinks from 16 bytes to about 1.4, and where each design breaks.
Prerequisites
This page assumes you know how B-trees and LSM trees work: memtables, SSTables, compaction, write amplification. If not, read Storage Engines first. Everything below is a variation on those two designs.
Why Time-Series Workloads Break General-Purpose Engines
A general-purpose OLTP engine like InnoDB or Postgres heap+B-tree is built for a workload where any row can be read, updated or deleted at any time, with about equal probability. Time-series and IoT data looks nothing like that.
| Property | OLTP (orders, users) | Time-series / IoT (metrics, sensors) | Consequence for the engine |
|---|---|---|---|
| Write pattern | Inserts + updates + deletes | ~99.9% inserts, append-mostly | MVCC versions, undo logs and in-place updates are wasted work |
| Key order of arrival | Random (UUIDs, user IDs) | Nearly sorted by time within a series | Sorted runs are almost free to produce |
| Write volume | 10Β²β10β΄ rows/s | 10β΅β10β· points/s | Per-row overhead (24-byte tuple header in Postgres) dominates |
| Row width | Wide, heterogeneous | Narrow: (series, ts, value) = 16 bytes of payload | Per-row metadata can be bigger than the data |
| Cardinality | Millions of rows, few "kinds" | Millions of distinct series, each a tiny stream | The index of series becomes the bottleneck, not the data |
| Read pattern | Point lookups by PK | Range scans of one series over time, aggregates over many | Data should sit contiguously by (series, time) |
| Hotness | Uniform-ish | Last 1β24 h is >90% of reads | Keep recent data in RAM, old data on cheap storage |
| Deletion | Row-by-row | Bulk "drop everything older than 30 days" | Deletes should be unlink() of a file, not a tombstone per row |
The last row matters a lot. A system that holds 30 days of data and ingests 1M points/s must delete 1M points/s forever, just to stay level. If deletion means writing a tombstone per row and compacting it away later, deletion costs as much as ingestion. Every successful TSDB partitions by time so that retention becomes "delete the directory for day N-30".
Why a Single B-Tree Struggles
Picture the obvious schema: one table metrics(series_id, ts, value) with a B-tree primary key on (series_id, ts). That key order is what range queries want, since all of one series' points sit together.
Now look at the write pattern. 100,000 sensors each report every 10 seconds. In any given second, ~10,000 different series get one new point each. With key (series_id, ts), each new point belongs at the end of its own series' range, and those ranges are spread over the whole tree:
graph TD
Root["Root page"] --> I1["Internal: series 0-25k"]
Root --> I2["Internal: series 25k-50k"]
Root --> I3["Internal: series 50k-75k"]
Root --> I4["Internal: series 75k-100k"]
I1 --> L1["Leaf: s=00017 ... ts=now"]
I1 --> L2["Leaf: s=08341 ... ts=now"]
I2 --> L3["Leaf: s=31002 ... ts=now"]
I3 --> L4["Leaf: s=66120 ... ts=now"]
I4 --> L5["Leaf: s=99871 ... ts=now"]
W["10,000 inserts/sec,<br/>one per series"] -.->|"random leaf"| L1
W -.->|"random leaf"| L2
W -.->|"random leaf"| L3
W -.->|"random leaf"| L4
W -.->|"random leaf"| L5
style W fill:#ff6b6b,color:#fff
Each insert touches a different leaf page. Once the working set of "current tail leaves" (100k series Γ an 8 KB page = 800 MB) is bigger than the buffer pool, every insert is a random read-modify-write. Leaves fill up and split. Splits leave pages ~50β70% full, which bloats the tree, and each split writes 2β3 pages to the WAL. In Postgres you also pay the 24-byte heap tuple header plus a ~16-byte index tuple to store a 16-byte payload. That is roughly 3Γ overhead before any compression.
Flip the key to (ts, series_id) and inserts become append-only at the right edge of the tree. Writes get fast again, but now "give me series 66120 for the last week" has to skip over 100k series per timestamp. A single B-tree cannot be sorted for both the write pattern (by time) and the read pattern (by series).
Why LSM-Style Designs Fit
An LSM tree breaks that tension. Writes go into an in-memory sorted structure. That structure can be sorted by (series, time) cheaply because it is in RAM. It gets flushed as an immutable sorted file. The write is sequential on disk and the file is laid out for range reads. Time-series data then hands LSM designs three extra gifts:
- No updates. Merges rarely have to reconcile versions. Compaction is mostly concatenation plus re-compression.
- Time locality. If files are also partitioned by time window, a query for "last hour" opens only the files for the last hour, and retention drops whole files.
- Homogeneous columns. One series' timestamps are nearly evenly spaced and its values change slowly. That is ideal for the delta and XOR encodings below.
graph LR
subgraph "Generic LSM (RocksDB)"
A1["Memtable<br/>skiplist of key/value"] --> A2["SSTable L0"] --> A3["L1 ... Ln<br/>leveled compaction"]
end
subgraph "Time-series LSM (TSM, Prometheus, MergeTree)"
B1["In-memory buffer<br/>per-series columns"] --> B2["Immutable file<br/>for time window T"]
B2 --> B3["Compacted larger<br/>time window"]
B3 --> B4["Retention: unlink<br/>whole file"]
end
style B4 fill:#51cf66,color:#fff
The rest of this page shows how each major engine fills in that template.
InfluxDB's TSM Tree (1.x / 2.x)
InfluxDB started on LevelDB, then moved to BoltDB (a B+tree), then gave up on both. Their 2015 write-up says it plainly: LevelDB couldn't delete old data cheaply because it had no time sharding, and BoltDB's random writes collapsed once the database grew past a few GB. The fix was the Time-Structured Merge tree, released in 0.9.5 and the default engine for all of 1.x and 2.x.
Shards: Time Partitioning First
Before any tree comes into play, InfluxDB cuts data by time into shard groups. Each shard is a fully independent TSM engine with its own WAL, cache, TSM files and index.
| Retention policy duration | Default shard group duration |
|---|---|
| < 2 days | 1 hour |
| 2 days β 6 months | 1 day |
| > 6 months | 7 days |
Retention enforcement is then rm -rf of a shard directory. That is O(1) in the number of points. It is the single most important design choice in the engine.
The Write Path: WAL β Cache β TSM Files
sequenceDiagram
participant C as Client (line protocol)
participant W as WAL segment
participant Ca as Cache (in-memory)
participant S as Snapshotter
participant T as TSM files
participant K as Compactor
C->>W: append batch (snappy-compressed)
W-->>C: fsync then 204 No Content
C->>Ca: insert points into per-series value arrays
Note over Ca: cache-snapshot-memory-size 25 MB or 10 min cold
Ca->>S: snapshot cache
S->>T: write new level-1 TSM file
S->>W: delete WAL segments covered by snapshot
K->>T: merge level-1 files into level-2, then 3, then full
- WAL. Each write batch is appended to the current WAL segment (10 MB per segment), snappy-compressed, then fsynced. Only then is the write acknowledged. The WAL is purely for crash recovery and is never read on the query path. See Write-Ahead Logging for the general idea.
- Cache. The same points go into an in-memory map from
series key + fieldto a slice of(timestamp, value). Queries read the cache, so new data is visible right away. The cache is not a sorted skiplist. Each series' slice is sorted lazily at snapshot time, because data almost always arrives in order already. - Snapshot. When the cache passes
cache-snapshot-memory-size(default 25 MB), or the shard has been write-cold forcache-snapshot-write-cold-duration(default 10 min), the cache is written out as a new TSM file and the matching WAL segments are deleted. If the cache hitscache-max-memory-size(default 1 GB) before a snapshot finishes, writes are rejected withcache-max-memory-size exceeded. That is the engine's backpressure.
Inside a TSM File
A TSM file is an immutable, sorted, columnar SSTable with a twist. Each block holds up to 1,000 points of one field of one series, and timestamps and values are compressed separately within the block.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Header: magic 0x16D116D1 (4 B) β version 1 (1 B) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Block 1: CRC32 β type β len(ts) β ts bytes β value bytes β β cpu,host=a#usage_user
β Block 2: CRC32 β type β len(ts) β ts bytes β value bytes β β cpu,host=a#usage_user (next 1000 pts)
β Block 3: ... β β cpu,host=b#usage_user
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Index (sorted by key): β
β key_len β key β type β count β
β [min_time β max_time β offset β size] Γ count β
β ... β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Footer: index offset (8 B) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The index sits at the end. It maps each series key to a list of (min_time, max_time, offset, size) entries, one per block. Reading cpu,host=b#usage_user for 10:00β11:00 goes like this:
- Binary-search the index (mmapped) for the key.
- Walk that key's block entries and keep those whose
[min_time, max_time]overlaps 10:00β11:00. - Read and decompress only those blocks.
There are no bloom filters as in RocksDB. Instead, the min/max time ranges per block do the pruning. That is cheaper and more selective for time-range queries.
Per-block encodings depend on the column type:
| Column | Encoding | Typical size |
|---|---|---|
| Timestamps | Delta β if all deltas equal: RLE (first, delta, count). Else simple8b packing of deltas, scaled by the largest common divisor (e.g. 10βΉ for second-precision ns timestamps) | Regular interval: ~0 bits/point. Jittery: 1β2 bytes |
| Floats | Gorilla XOR (see below) | 1β2 bytes |
| Integers | ZigZag β delta β simple8b, or RLE if constant delta | 1β3 bytes |
| Booleans | Bit-packed | 1 bit |
| Strings | Snappy over the concatenated block | Varies |
Compaction Levels
Snapshots produce many small level-1 files. The compactor merges them in tiers:
graph LR
S["Snapshot<br/>from cache"] --> L1["Level 1<br/>small files"]
L1 -->|"4 L1 files"| L2["Level 2"]
L2 -->|"4 L2 files"| L3["Level 3"]
L3 -->|"shard cold"| F["Full / optimize<br/>compaction"]
F --> O["1-N large files<br/>blocks fully packed<br/>1000 pts each"]
style O fill:#51cf66,color:#fff
- Level compactions (1β2β3) merge groups of files by concatenating and re-sorting blocks. They mostly copy blocks over and only decode and re-encode when blocks overlap in time.
- Full compaction runs when a shard goes cold, for example yesterday's shard once today's takes the writes. It rewrites everything into a few files of max size (~2 GB each) with blocks fully packed to 1,000 points. Packing is where the best compression comes from.
- Optimize compaction packs fragmented blocks when there are too many files at the top level.
Write amplification is low in practice, around 2β4Γ. Data mostly arrives in order, and a shard stops receiving writes once its time window closes. Compare that with RocksDB leveled compaction at 10β30Γ, as covered in Storage Engines.
Deletes, the rare ones, write a .tombstone file next to the TSM file. The next compaction drops the covered points.
The Series Index: from inmem to TSI
TSM files answer "give me points for this exact series key". InfluxQL queries look like WHERE region='eu' AND host=~/web-.*/. Answering that needs an inverted index from tag pairs to series. That index is where InfluxDB lives or dies.
inmem index (β€ 1.4 default). Every shard built a full in-RAM map of measurement β tag key β tag value β series IDs at startup by scanning every TSM file. Memory use was roughly 1β2 KB per series per shard. 10M series meant tens of GB of RAM and startups that took many minutes.
TSI (Time Series Index, 1.5+ default, the only index in 2.x) moves the index onto disk with the same LSM shape as the data:
graph TD
New["New series on write"] --> SF["Series file (per DB)<br/>series key to series ID<br/>mmapped, Robin Hood hash"]
New --> LOG["TSI log file (.tsl)<br/>append-only, in-memory copy"]
LOG -->|"max-index-log-file-size 1 MB"| IDX["TSI index file (.tsi)<br/>immutable, sorted"]
IDX -->|"compaction"| IDX2["Larger .tsi files"]
IDX2 --- HLL["HyperLogLog++ sketches<br/>per measurement and tag<br/>for fast cardinality estimates"]
style SF fill:#ffd43b,color:#000
- The series file gives every series key a compact 64-bit ID, shared across shards.
- Each shard's TSI is split into 8 partitions by series hash, so concurrent writes don't contend.
- A
.tsifile holds sorted blocks:measurement β tag key β tag value β sorted series ID set(roaring-bitmap-like). Only the pages a query touches are paged in through mmap. - HyperLogLog sketches answer
SHOW SERIES CARDINALITYwithout counting.
How High Cardinality Hurts TSM
Cardinality is the number of distinct series: measurement Γ every distinct combination of tag values. It costs you in four places at once:
- Series file and TSI grow without bound. Each new
host=β¦,container_id=β¦writes a new series entry that lives forever, or at least until the shard expires. - Blocks get tiny. 1M points/s spread over 10 series gives full 1,000-point blocks quickly. Spread over 10M series, each series gets one point per 10 s, so a block per series per snapshot holds ~2 points. The compression ratio collapses and the TSM index (one entry per block) balloons.
- Queries fan out.
GROUP BY container_iddecodes millions of tiny blocks. - Startup and compaction cost scales with series count, not just bytes.
The practical ceiling was low tens of millions of series per node. Putting request IDs, user IDs or pod UIDs in tags is still the number-one cause of InfluxDB outages.
InfluxDB 3.0: Why They Threw TSM Away
In 2020 InfluxData started IOx, a full rewrite in Rust, which shipped as InfluxDB 3.0 (Cloud first, then 3 Core / Enterprise GA in 2025). It drops TSM completely and is built on what they call the FDAP stack:
| Layer | Project | Role |
|---|---|---|
| Flight | Apache Arrow Flight (gRPC) | Moves Arrow record batches over the network without serialisation |
| DataFusion | Apache DataFusion | Vectorised SQL query engine (and InfluxQL front-end) |
| Arrow | Apache Arrow | In-memory columnar format for buffered writes and query execution |
| Parquet | Apache Parquet | On-disk columnar files, written to object storage (S3, GCS, local) |
graph LR
W["Line protocol writes"] --> WAL["WAL<br/>flushed to object store<br/>every 1 s by default"]
WAL --> Buf["In-memory Arrow buffer<br/>queryable immediately"]
Buf -->|"persist every ~10 min"| PQ["Parquet files<br/>partitioned by table and day<br/>sorted by tags then time"]
PQ --> OS[("Object storage<br/>S3 / GCS / disk")]
Q["SQL / InfluxQL"] --> DF["DataFusion<br/>predicate + time pruning<br/>via Parquet stats"]
DF --> Buf
DF --> PQ
style OS fill:#74c0fc,color:#000
Why they did it:
- Cardinality stops mattering much. There is no per-series index at all. A tag is just a dictionary-encoded column in a Parquet file sorted by tags then time. A new
container_idvalue costs a few bytes in a dictionary, not a new series entry plus a new index entry. The cost moved from write time and RAM to scan time, and vectorised scans of sorted, dictionary-encoded columns are very fast. - Object storage as the source of truth. Separating compute from storage makes nodes stateless. Ten years of data on S3 costs ~$23/TB-month instead of $100+/TB-month on replicated SSDs.
- SQL and ecosystem. Parquet files can be read by Spark, DuckDB, Snowflake, pandas. TSM files could only be read by InfluxDB.
- Not building a query engine alone. Arrow and DataFusion are maintained by a large community.
The trade-off: per-series point lookups over long ranges no longer have a tiny dedicated index. The engine relies on file and row-group pruning by time and min/max tag statistics instead. Broad analytical queries got much faster. Narrow "one series, last 30 days" queries depend on good sort order.
The industry converged
InfluxDB 3 (Parquet), VictoriaMetrics (MergeTree-like), Mimir and Thanos (Prometheus blocks on S3), ClickHouse for metrics, and TimescaleDB's columnstore all end up in the same place. Data is columnar, sorted by series then time, partitioned by time, compressed per column, and pushed to object storage when cold. The differences lie in the index and the compression codecs.
Facebook Gorilla Compression
In 2015 Facebook published "Gorilla: A Fast, Scalable, In-Memory Time Series Database" (Pelkonen et al., VLDB 2015). Gorilla was an in-memory write-through cache in front of their HBase-based ODS monitoring system. It had to hold the last 26 hours of ~2 billion time series in RAM. Raw, a point is a 64-bit timestamp + a 64-bit float = 16 bytes. Gorilla got that down to an average of 1.37 bytes per point, about 12Γ smaller. Its two tricks now show up in InfluxDB TSM, Prometheus, VictoriaMetrics, TimescaleDB, M3, QuestDB and IoTDB.
The scheme works on one series at a time, in blocks aligned to 2-hour windows. Timestamps and values are compressed as separate streams.
Timestamps: Delta-of-Delta
Monitoring data arrives on a schedule, say every 60 s, with small jitter. Deltas between timestamps are almost constant, so the delta of the deltas is almost always zero.
Algorithm for a block whose header timestamp is tββ (start of the 2-hour window):
- Store the header
tββ(64 bits). - Store the first delta
tβ β tββin 14 bits (enough for 2 hours plus some slack at 1-second precision). - For each following timestamp, compute
D = (tβ β tβββ) β (tβββ β tβββ)and emit:
| Range of D | Prefix | Value bits | Total bits |
|---|---|---|---|
| D = 0 | 0 | β | 1 |
| β63 β¦ 64 | 10 | 7 | 9 |
| β255 β¦ 256 | 110 | 9 | 12 |
| β2047 β¦ 2048 | 1110 | 12 | 16 |
| anything else | 1111 | 32 | 36 |
Worked example, block starting at 02:00:00, a sensor that reports about once a minute:
| Timestamp | Delta | Delta-of-delta D | Encoded as | Bits |
|---|---|---|---|---|
| 02:01:02 | 62 | β (first) | 14-bit delta 00000000111110 | 14 |
| 02:02:02 | 60 | β2 | 10 + 7-bit β2 = 10 1111110 | 9 |
| 02:03:02 | 60 | 0 | 0 | 1 |
| 02:04:02 | 60 | 0 | 0 | 1 |
| 02:05:01 | 59 | β1 | 10 + 1111111 | 9 |
| 02:06:01 | 60 | +1 | 10 + 0000001 | 9 |
Six timestamps take 43 bits after the header, versus 384 bits raw. In Facebook's production data, 96% of timestamps compressed to a single bit.
Values: XOR of Consecutive Floats
Neighbouring values in a series are usually similar, often identical. Two close IEEE-754 doubles share their sign, their exponent and the top bits of their mantissa. XOR them and the result is mostly zeros: a run of leading zeros, a short "meaningful" middle, and a run of trailing zeros.
Encoding for each value after the first, which is stored raw in 64 bits:
x = bits(vβ) XOR bits(vβββ).- If
x == 0, emit0(1 bit). The value repeated. - Otherwise emit
1, then:- Control
0: the meaningful bits ofxfit inside the previous window (same or more leading zeros and same or more trailing zeros). Emit only the bits in that window. No header needed. - Control
1: emit 5 bits giving the number of leading zeros, 6 bits giving the length of the meaningful bits, then the meaningful bits. This becomes the new window.
- Control
Worked Bit-Level Example
Series: 12.0, 12.0, 24.0, 15.5, 14.5, 14.5
First, the raw IEEE-754 bits (sign β 11-bit exponent β 52-bit mantissa, shown as hex):
| Value | = | Hex bits |
|---|---|---|
| 12.0 | 1.5 Γ 2Β³ | 0x4028000000000000 |
| 24.0 | 1.5 Γ 2β΄ | 0x4038000000000000 |
| 15.5 | 1.9375 Γ 2Β³ | 0x402F000000000000 |
| 14.5 | 1.8125 Γ 2Β³ | 0x402D000000000000 |
Now encode:
| # | Value | XOR with previous | Leading / trailing zeros | Window? | Emitted | Bits |
|---|---|---|---|---|---|---|
| 0 | 12.0 | β | β | β | raw 64 bits | 64 |
| 1 | 12.0 | 0x0000000000000000 | β | β | 0 | 1 |
| 2 | 24.0 | 0x0010000000000000 | 11 / 52 (1 meaningful bit) | no window yet | 1 1 01011 000001 1 | 14 |
| 3 | 15.5 | 0x0017000000000000 | 11 / 48 (5 meaningful bits) | needs trailing β₯ 52, has 48 β doesn't fit | 1 1 01011 000101 10111 | 18 |
| 4 | 14.5 | 0x0002000000000000 | 14 / 49 | window is [11 lead, 48 trail]. 14 β₯ 11 and 49 β₯ 48 β fits | 1 0 + 5 window bits 00010 | 7 |
| 5 | 14.5 | 0x0000000000000000 | β | β | 0 | 1 |
Total: 105 bits for six values versus 384 bits raw, 3.7Γ even on this tiny, jumpy sample. Real sensor data is much smoother: a temperature that changes in the 3rd significant digit, or a gauge that sits still for minutes. Facebook measured this distribution of value encodings:
| Case | Share of values | Avg bits |
|---|---|---|
Identical to previous (0) | ~51% | 1 |
Fits previous window (10) | ~30% | ~26.6 |
New window (11) | ~19% | ~36.9 |
Together with the ~1-bit timestamps, that gives 1.37 bytes/point on average over their 2-hour blocks. A day of a 1-minute series (1,440 points) shrinks from ~23 KB to ~2 KB.
Reference Implementation
This is the full encoder. It reproduces the bit counts in both tables above:
import struct
class BitWriter:
def __init__(self): self.bits = []
def write(self, value, n):
for i in range(n - 1, -1, -1):
self.bits.append((value >> i) & 1)
def f2u(x): return struct.unpack(">Q", struct.pack(">d", x))[0]
def encode_timestamps(header, ts, w):
w.write(header, 64)
delta = ts[0] - header
w.write(delta, 14)
prev, prev_delta = ts[0], delta
for t in ts[1:]:
delta = t - prev
dod = delta - prev_delta
if dod == 0: w.write(0b0, 1)
elif -63 <= dod <= 64: w.write(0b10, 2); w.write(dod & 0x7F, 7)
elif -255 <= dod <= 256: w.write(0b110, 3); w.write(dod & 0x1FF, 9)
elif -2047 <= dod <= 2048: w.write(0b1110, 4); w.write(dod & 0xFFF, 12)
else: w.write(0b1111, 4); w.write(dod & 0xFFFFFFFF, 32)
prev, prev_delta = t, delta
def encode_values(vals, w):
prev = f2u(vals[0]); w.write(prev, 64)
lead = trail = None
for v in vals[1:]:
cur = f2u(v); x = cur ^ prev; prev = cur
if x == 0:
w.write(0, 1); continue
w.write(1, 1)
l = min(64 - x.bit_length(), 31) # 5-bit field caps at 31
t = (x & -x).bit_length() - 1 # trailing zeros
if lead is not None and l >= lead and t >= trail:
w.write(0, 1); w.write(x >> trail, 64 - lead - trail)
else:
lead, trail = l, t
w.write(1, 1); w.write(l, 5); w.write(64 - l - t, 6); w.write(x >> t, 64 - l - t)
if __name__ == "__main__":
w = BitWriter(); encode_values([12.0, 12.0, 24.0, 15.5, 14.5, 14.5], w)
assert len(w.bits) == 105
w = BitWriter(); encode_timestamps(0, [62, 122, 182, 242, 301, 361], w)
assert len(w.bits) == 64 + 43
print("ok")
Where Gorilla compresses badly
- High-precision noisy floats. A float with random low-order bits, like
23.418723β¦from a raw ADC reading divided by a calibration constant, XORs into ~50 meaningful bits. Rounding at the edge (round(v, 2)) often cuts storage 3β5Γ. - Counters stored as floats that jump by large amounts. Store them as integers, with delta + zigzag + simple8b.
- Irregular timestamps. Event-driven IoT data, such as "door opened", has no regular interval. Delta-of-delta then gives 12β36 bits per timestamp.
- Random access. The stream can only be decoded from the start of a block. That is why everyone caps blocks at ~120β1,000 points.
Descendants and Improvements
| Codec | Used by | Idea | vs Gorilla |
|---|---|---|---|
| Gorilla XOR | Prometheus, InfluxDB TSM, M3, TimescaleDB floats, IoTDB GORILLA | As above | Baseline |
| Chimp / Chimp128 (VLDB 2022) | Research, some engines | Uses the trailing zeros better and can XOR against one of the last 128 values | ~1.3β2Γ smaller, faster decode |
| Delta + simple8b / FastPFOR | TSM integers, Timescale ints | Bit-pack many small integers per 64-bit word | Better for integers |
| ALP (SIGMOD 2024) | DuckDB | Turn decimal-like floats into integers with an exponent, then bit-pack | Often 2β3Γ better on real-world "decimal" floats |
| VictoriaMetrics | VictoriaMetrics | Convert floats to scaled int64 (decimal mantissa + exponent), then delta + zstd | Commonly 0.4β1 byte/sample |
| ZSTD / LZ4 over blocks | ClickHouse (Delta, ZSTD, Gorilla, DoubleDelta, T64), Parquet | Generic entropy coding after a specialised transform | Good general fallback |
Prometheus TSDB
Prometheus' storage engine (TSDB v3, written by Fabian Reinartz for Prometheus 2.0 in 2017) is the most widely run time-series engine in the world. It is a log-structured, time-partitioned design: an in-memory head block for the last ~2β3 hours, and immutable on-disk blocks for everything older. The operational view is in Prometheus Deep Dive. This section is about the bytes.
graph TD
Scrape["Scrape / remote-write<br/>samples"] --> WAL["WAL<br/>128 MB segments<br/>+ checkpoints"]
Scrape --> Head["Head block (memory)<br/>series map + postings<br/>active chunk per series"]
Head -->|"full chunk: 120 samples"| MM["chunks_head/<br/>mmapped full chunks"]
Head -->|"every 2 h, cut oldest 2 h"| B1["Block 01H...<br/>2 h, immutable"]
B1 --> C["Compactor"]
B2["Block 2 h"] --> C
B3["Block 2 h"] --> C
C --> BL["Block 6 h / 18 h / 54 h ...<br/>up to 10% of retention, max 31 d"]
BL -->|"older than retention"| Del["delete directory"]
style Head fill:#ffd43b,color:#000
style Del fill:#ff6b6b,color:#fff
Head Block, Chunks and the WAL
- Every series (a unique label set) in the head has an in-memory struct: labels, a reference ID, and its currently open chunk.
- A chunk holds up to 120 samples of one series, Gorilla-encoded. At a 15 s scrape interval, 120 samples is 30 minutes. Prometheus uses the XOR value encoding unchanged but widens the timestamp buckets for millisecond precision: delta-of-delta buckets of 14, 17, 20 bits, else 64 bits. Native histograms use their own chunk encoding.
- Since v2.19, full chunks are mmapped from
chunks_head/instead of kept on the heap, which cut head memory by 30β50%. - Every sample is also appended to the WAL (
wal/, 128 MB segments, records for series, samples, tombstones, exemplars). On restart, Prometheus replays the WAL to rebuild the head. Long replays (minutes for tens of millions of series) are a classic restart pain, which is what WAL checkpoints and the v2.30 in-memory snapshot on shutdown address.
Blocks: 2-Hour Immutable Directories
Every 2 hours the head cuts its oldest 2 hours into a block. Each block is a directory named by a ULID:
data/
βββ 01JC8X2Y9M0WQ8KX3TZ0E9G4AB/ β block covering [t, t+2h)
β βββ meta.json β minTime, maxTime, stats, compaction level, sources
β βββ index β symbol table, series, label index, postings
β βββ chunks/
β β βββ 000001 β up to 512 MB of concatenated chunks
β β βββ 000002
β βββ tombstones β deletion intervals (series ref, mint, maxt)
βββ chunks_head/ β mmapped full chunks of the head
βββ wal/
β βββ 00000941
β βββ 00000942
β βββ checkpoint.00000940/
βββ lock
The index file has four parts:
- Symbol table. Every distinct label name and value string, stored once and referenced by offset.
job,instance,__name__,http_requests_totalappear once no matter how many series use them. - Series section. For each series, sorted by label set: its labels as symbol references, plus a list of
(mint, maxt, chunk ref)for its chunks. - Postings lists. For every label pair
name=value, a sorted list of series IDs that have it. This is the inverted index. - Postings offset table and TOC to find each list.
The Inverted Index: How a Selector Is Answered
A PromQL selector like
http_requests_total{job="api", status=~"5..", region!="us-east"}
resolves like a search engine query:
graph LR
P1["postings(__name__=http_requests_total)<br/>[2, 5, 9, 14, 21, 30]"] --> I["Intersect<br/>(merge of sorted lists)"]
P2["postings(job=api)<br/>[5, 9, 11, 14, 30]"] --> I
P3["union of postings(status=500, 502, 503)<br/>[9, 14, 22, 30]"] --> I
I --> R1["[9, 14, 30]"]
R1 --> S["Subtract postings(region=us-east)<br/>[14]"]
S --> R["Series [9, 30]<br/>then read chunks overlapping query range"]
style R fill:#51cf66,color:#fff
The engine intersects sorted ID lists with a merge, O(sum of list lengths). For a regex matcher it first scans the label values for that name and unions their postings. That is why {status=~".*"} or matchers on a label with 10M values are slow. Series IDs are assigned in label-sorted order within a block, so the resulting series are also read in sorted order.
Compaction and Retention
- Vertical compaction merges overlapping blocks, for example after backfill or out-of-order ingestion (
out_of_order_time_window, added in v2.39). - Horizontal compaction merges adjacent blocks into bigger ranges: 2 h β 6 h β 18 h β 54 h β¦ with each step 3Γ the last, up to 10% of the retention period or 31 days. Bigger blocks mean fewer index lookups for long queries and a shared symbol table.
- Deletes through the admin API write
tombstones. The data is physically removed at the next compaction. - Retention (
--storage.tsdb.retention.time=15dor.size=500GB) simply deletes whole block directories whosemaxTimeis past the window. Once again: retention isunlink().
Typical numbers: ~1.3β2 bytes per sample on disk, about 200β500k samples/s ingested per core, and 1β3 KB of disk index per series per block.
Why Cardinality Explosions Kill Prometheus
The head keeps every active series in memory with its labels, postings entries and an open chunk. A rough rule: ~3β8 KB of RAM per active series including Go GC overhead, varying with label size and churn.
| Active series | Approximate head RAM | Notes |
|---|---|---|
| 100k | 0.5 GB | A small cluster |
| 1M | 4β8 GB | Comfortable single Prometheus |
| 10M | 40β80 GB | Upper limit of one instance, long WAL replays |
| 50M | β | Shard it, or use Mimir or VictoriaMetrics |
A single bad label wrecks this:
http_requests_total{path="/users/8812734/orders"} β user ID in the path
http_requests_total{pod="api-7d9f8c6b5-x2k9q"} β pod name: churn on every deploy
http_requests_total{trace_id="4bf92f3577b34da6"} β one series per request
Every new label value creates a new series. It gets a struct in memory, a new postings entry, a new WAL series record and a chunk that will hold one or two samples, which gives terrible compression. Churn hurts as much as raw cardinality. A Kubernetes rollout that renames 5,000 pods creates 5,000 Γ (metrics per pod) new series. The old ones stay in the head until they age out, for up to 2β3 h, and stay in the blocks' indexes for the full retention. Defences: metric_relabel_configs to drop labels, sample_limit and label_limit per scrape, the prometheus_tsdb_head_series alert, and promtool tsdb analyze to find the worst labels. See Metrics Design.
Long-Term Storage: Remote Write, Thanos, Mimir, VictoriaMetrics
Prometheus is single-node, and local retention is usually 15β30 days. For years of data and a global view, there are three main architectures:
graph TD
subgraph "Thanos (sidecar model)"
P1["Prometheus"] --- SC["Sidecar"]
SC -->|"upload each 2 h block"| S3A[("Object storage")]
SG["Store Gateway"] --> S3A
Q["Querier<br/>fan-out + dedup"] --> SC
Q --> SG
CP["Compactor<br/>+ downsample 5 m and 1 h"] --> S3A
end
subgraph "Mimir / Cortex (remote-write model)"
P2["Prometheus / Agent"] -->|"remote_write<br/>snappy protobuf"| D["Distributor<br/>hash ring"]
D --> IN["Ingesters<br/>per-tenant TSDB head<br/>RF=3"]
IN -->|"2 h blocks"| S3B[("Object storage")]
SGB["Store Gateway"] --> S3B
end
- Thanos. Each Prometheus keeps writing its normal TSDB. A sidecar uploads finished 2 h blocks to S3/GCS. Queries fan out to the sidecars (recent data) and store gateways (historical data) and deduplicate HA pairs. The compactor merges blocks in object storage and downsamples to 5-minute and 1-hour resolution, so a 1-year dashboard doesn't read raw 15 s samples.
- Cortex β Grafana Mimir. Prometheus
remote_writes to a horizontally scaled, multi-tenant service. Ingesters each run a Prometheus TSDB head per tenant, replicated 3Γ, cut 2 h blocks and ship them to object storage. Mimir adds a split-and-merge compactor and query sharding, and has been tested at 1 billion active series. It is the same storage format, partitioned across many machines. - VictoriaMetrics. Its own engine, explicitly inspired by ClickHouse's MergeTree. Incoming samples are buffered and written as small sorted parts (columns of
TSID, timestamps, values, each compressed). Background merges combine them into bigger parts, exactly like ClickHouse MergeTree. The label index (indexdb) is a separate mergeset, an LSM of sorted byte strings mappinglabel=value β metricIDsandmetricID β TSID, partitioned per day. Floats are converted to scaled integers before delta encoding and zstd. In practice VictoriaMetrics reports ~0.4β1 byte/sample and several times less RAM per active series than Prometheus. The cost is its own, less standard file format.
TimescaleDB: Making B-Trees Work by Partitioning Time
TimescaleDB is the counterexample to "B-trees can't do time series". It is Postgres: heap tables, B-tree indexes, the normal WAL. It solves the random-insert problem by changing the shape of the problem rather than the data structure. (The company behind it renamed itself TigerData in 2025. The extension is still timescaledb.)
Hypertables and Chunks
A hypertable is a parent table that is automatically split into chunks, ordinary Postgres child tables, each covering one time interval and optionally a hash of a space column.
CREATE TABLE readings (
time timestamptz NOT NULL,
device_id int NOT NULL,
temperature double precision,
humidity double precision
);
SELECT create_hypertable('readings', by_range('time', INTERVAL '1 day'));
CREATE INDEX ON readings (device_id, time DESC);
graph TD
H["Hypertable: readings"] --> C1["_hyper_1_101_chunk<br/>2026-11-06<br/>own heap + B-trees"]
H --> C2["_hyper_1_102_chunk<br/>2026-11-07<br/>own heap + B-trees"]
H --> C3["_hyper_1_103_chunk<br/>2026-11-08 (hot)<br/>fits in shared_buffers"]
W["INSERTs (now)"] --> C3
Q["WHERE time > now() - 2h"] -->|"chunk exclusion"| C3
R["drop_chunks older than 30 d"] -.->|"DROP TABLE, instant"| C1
style C3 fill:#ffd43b,color:#000
style C1 fill:#adb5bd,color:#000
Why the B-tree is fine now:
- The hot B-tree is small. Only today's chunk takes inserts. Its
(device_id, time DESC)index covers one day, not three years, so its pages fit inshared_buffers. The random-leaf problem from earlier is still there, but every leaf is in RAM, so a "random" insert is a memory write plus a WAL record. Timescale's sizing rule: pickchunk_time_intervalso the most recent chunk and its indexes fit in ~25% of memory. - Chunk exclusion. The planner, and the executor for
now()-relative predicates, skips chunks whose time range doesn't overlap the query. That is partition pruning on steroids. - Retention is
DROP TABLE.drop_chunks('readings', INTERVAL '30 days')oradd_retention_policyremoves whole chunks. NoDELETE, no vacuum storm, no bloat. - Postgres stays Postgres. JOINs against
devices, foreign data wrappers, PostGIS, logical replication,pg_dumpall keep working.
The cost: each row still pays Postgres' ~24-byte tuple header plus ~16 bytes in each index, so raw uncompressed storage is 3β5Γ larger than a dedicated TSDB. That is why compression matters.
Native Columnar Compression (Hypercore Columnstore)
Older chunks are converted from row-oriented heap into a columnar layout. Internally, rows are grouped into batches of up to 1,000 rows per segmentby value. Each column of a batch is stored as one compressed array in a single row of an internal table:
ALTER TABLE readings SET (
timescaledb.compress,
timescaledb.compress_segmentby = 'device_id',
timescaledb.compress_orderby = 'time DESC'
);
SELECT add_compression_policy('readings', INTERVAL '7 days');
Before (heap, 1 row per reading): After (1 row per 1,000 readings of a device):
time device temp hum device β time[] β temp[] β hum[] β _min/_max time
2026-11-01 00:00:00 42 21.5 40.1 42 β DoD+simple8b β Gorilla bytes β Gorilla bytes β metadata for pruning
2026-11-01 00:00:10 42 21.5 40.2 43 β ... β ... β ... β
... Γ1000 per device
| Column type | Algorithm |
|---|---|
| Integers, timestamps | Delta-of-delta + simple-8b + run-length |
| Floats | Gorilla XOR |
| Low-cardinality text | Dictionary + RLE |
| Everything else | Array + LZ (pglz) |
Real-world ratios of 90β98% (10β50Γ smaller) are typical for sensor and metric data. segmentby columns are stored once per batch, and the min/max metadata lets queries skip batches. Writes to compressed chunks are supported (DML on the columnstore, since 2.11), but they are slower. The intended pattern is "rowstore for hot, columnstore for cold".
Continuous Aggregates
A continuous aggregate is a materialized view on a hypertable that refreshes incrementally. Only buckets whose underlying chunks changed get recomputed. It is the built-in downsampling mechanism:
CREATE MATERIALIZED VIEW readings_1h
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', time) AS bucket,
device_id,
avg(temperature) AS avg_temp,
min(temperature) AS min_temp,
max(temperature) AS max_temp,
count(*) AS n
FROM readings
GROUP BY bucket, device_id;
SELECT add_continuous_aggregate_policy('readings_1h',
start_offset => INTERVAL '3 days',
end_offset => INTERVAL '1 hour',
schedule_interval => INTERVAL '30 minutes');
The continuous aggregate is itself a hypertable, so it can be compressed, and it can be retained longer than the raw data, e.g. raw for 30 days and hourly for 5 years. "Real-time aggregation" can union the materialized buckets with a live computation over the not-yet-materialized tail.
Columnar Approaches: QuestDB, ClickHouse, IoTDB
QuestDB: Append-Only Column Files
QuestDB (Java and C++, zero-GC on the hot path) takes the simplest possible layout. A table is a directory of time partitions, and each partition holds one file per column. A row is the same index across all column files.
db/sensors/
βββ 2026-11-07/ β PARTITION BY DAY
β βββ ts.d β designated timestamp, int64, sorted
β βββ device.d β SYMBOL: int32 keys into a dictionary
β βββ temperature.d β float64, raw (8 B Γ n)
β βββ humidity.d
βββ 2026-11-08/
β βββ ...
βββ device.c / device.o / device.k β symbol dictionary + optional bitmap index
βββ _txn β committed row counts per partition
- Ingestion via the InfluxDB Line Protocol over TCP or HTTP. Writes are memcpy-appends into mmapped column files. Commits bump a row count in
_txn. TSBS benchmarks put single-node ingestion at several million rows/second. - Out-of-order (O3) data is the hard part for append-only files. QuestDB sorts each incoming batch, then splits partitions or merges in place when late rows land inside an existing partition. Since 7.x, tables use a WAL with deduplication (
DEDUP UPSERT KEYS), and the WAL is applied to column files asynchronously. - Queries are SIMD scans over contiguous column arrays plus time-partition pruning.
SAMPLE BY 1handLATEST ON ts PARTITION BY deviceare the time-series primitives. - Compression: the native format stores columns uncompressed for raw mmap speed. Users rely on filesystem compression such as ZFS lz4/zstd for 3β6Γ. Recent releases can convert cold partitions to Parquet, which compresses and is readable by other tools.
It is the opposite philosophy to Gorilla: spend disk to make scans and ingestion as cheap as possible.
ClickHouse MergeTree for Metrics
ClickHouse isn't a TSDB, but its MergeTree is one of the best time-series engines available. Many observability backends store metrics this way, including SigNoz, Uptrace, and the metrics paths of several large companies. The general mechanics (parts, sparse primary index, merges) are in ClickHouse Internals. The time-series-specific settings:
CREATE TABLE metrics
(
metric LowCardinality(String),
tags_hash UInt64, -- hash of sorted labels
ts DateTime64(3) CODEC(DoubleDelta, ZSTD(1)),
value Float64 CODEC(Gorilla, ZSTD(1))
)
ENGINE = MergeTree
PARTITION BY toDate(ts)
ORDER BY (metric, tags_hash, ts)
TTL toDateTime(ts) + INTERVAL 30 DAY TO VOLUME 'cold',
toDateTime(ts) + INTERVAL 365 DAY DELETE;
ORDER BY (metric, tags_hash, ts)makes each series contiguous inside each part. That is the same "sorted by series then time" property as TSM blocks.- The
DoubleDeltacodec is Gorilla's delta-of-delta. TheGorillacodec is the XOR float encoding. ZSTD on top captures what remains. - Series labels usually live in a separate table (
tags_hash β labels map) to avoid repeating them per row, which mirrors the TSM series index. - Rollups use an
AggregatingMergeTreetarget fed by a materialized view, orGraphiteMergeTree, which downsamples by retention rules during merges.
Compressed sizes of 1β2 bytes per point with scans of billions of rows per second per node are typical. The weakness is the same as all MergeTree designs: small frequent inserts create too many parts. Batch inserts (β₯ 10k rows or β₯ 1 s), or use async inserts.
Apache IoTDB and the TsFile Format
Apache IoTDB (from Tsinghua University, Apache top-level since 2020) was built specifically for industrial IoT: wind turbines, power grids, factories, with millions of devices and thousands of sensors each. Its storage format, TsFile, is organised around the device β measurement hierarchy:
TsFile
βββ Magic "TsFile" + version
βββ ChunkGroup (device root.plant1.turbine7)
β βββ Chunk (measurement: wind_speed, encoding GORILLA, compression LZ4)
β β βββ ChunkHeader
β β βββ Page 1: PageHeader [count, min_t, max_t, min, max, first, last, sum] + data
β β βββ Page 2: ...
β βββ Chunk (measurement: rotor_rpm, encoding TS_2DIFF)
βββ ChunkGroup (device root.plant1.turbine8)
β βββ ...
βββ Index: TimeseriesMetadata per series (with statistics)
β βββ MetadataIndex tree (B+tree-like: device nodes β measurement nodes)
βββ Footer: index offset + bloom filter over series paths
Notable ideas:
- Statistics at every level (page, chunk, series in file): count, min/max time, min/max value, first, last, sum.
SELECT max(wind_speed) β¦ GROUP BY ([t1, t2), 1h)can often be answered from page headers alone without decoding the data. This is the same idea as Parquet column statistics, pushed further. - Aligned time series. Sensors on one device that sample together can share a single time column ("aligned"), which removes duplicate timestamp storage.
- Sequence vs unsequence files. In-order data goes to sequence TsFiles. Late or out-of-order data goes to separate unsequence files, and compaction merges them later. This keeps the common path append-only.
- Encodings:
TS_2DIFF(second-order delta, i.e. delta-of-delta),GORILLA,RLE,DICTIONARY,ZIGZAG,CHIMP,SPRINTZ, plus LZ4, ZSTD and Snappy on top.
The write path is the familiar LSM one: WAL β memtable per storage group β flush to TsFile β compaction.
IoT Edge and Embedded Storage
Everything above runs in a data centre on SSDs. The other end of an IoT system is a gateway or sensor node: a Raspberry Pi, an industrial PC, an ESP32. It has 64 MBβ4 GB of RAM, an SD card or eMMC, intermittent connectivity, and sometimes power that can vanish mid-write. The storage engine choice there is driven by flash wear and power loss, not query speed.
Flash Wear and Write Amplification
NAND flash can only be erased in big erase blocks (128 KBβ4 MB) and each block survives a limited number of program/erase cycles:
| Media | Typical P/E cycles | Notes |
|---|---|---|
| SLC NAND / industrial SD | 50,000β100,000 | Expensive, used in industrial-grade cards |
| pSLC (MLC in SLC mode) | ~20,000β30,000 | Common "industrial" eMMC compromise |
| MLC | 3,000β10,000 | |
| TLC (consumer SD, cheap eMMC) | 500β3,000 | Most Raspberry Pi SD cards |
| QLC | 100β1,000 | Avoid for write-heavy logging |
The flash translation layer (FTL) on a cheap SD card is simple. A 4 KB random write can force a read-modify-write of a whole erase block, which can mean 100Γ+ device-level write amplification. Multiply that by filesystem journaling (ext4 metadata journal) and database journaling (SQLite rollback journal or WAL, plus checkpoint), and a naive "INSERT one row per reading with fsync" logger can wear out a consumer SD card in months.
Back-of-envelope: 10 sensors Γ 1 reading/s, each a separate SQLite transaction. That is ~3 page writes of 4 KB each (WAL page + checkpoint + metadata):
With 10β50Γ FTL amplification, that becomes 100β500 GB/day hitting an 8 GB TLC card rated for ~1,000 cycles, i.e. ~8 TB total lifetime writes. That is 2β12 weeks. Batching the same data into one transaction per minute cuts it by ~60Γ.
Engine Choices on the Edge
| Option | Structure | Good for | Watch out for |
|---|---|---|---|
| SQLite (WAL mode) | B-tree pages + WAL | Gateways that need local SQL, config + buffered readings | Batch inserts in transactions. PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;. Use larger checkpoint intervals. Time-partition into one table or file per day so retention is DROP / unlink |
| RocksDB / LevelDB | LSM | High-rate buffering on Linux gateways (e.g. Azure IoT Edge, some historians) | Compaction write amplification (10β30Γ leveled) adds to FTL amplification. Prefer universal/FIFO compaction. compaction_style=kCompactionStyleFIFO is a natural time-series ring buffer |
| Embedded TSDB (e.g. InfluxDB OSS on a gateway, QuestDB, IoTDB edge, TDengine) | TSM / columnar | Local dashboards and aggregation at the edge | RAM footprint (hundreds of MB), more moving parts |
| Fixed-size ring buffer file | Pre-allocated circular log of fixed-size records | Microcontrollers, "last N hours of readings", black-box recorders | You design the record format, CRC and recovery |
| Raw flash KV / log FS (LittleFS, SPIFFS, NVS on ESP32) | Log-structured with wear levelling | MCUs with no OS filesystem | Small capacity, no query capability |
A ring buffer is often the right answer on a microcontroller. It pre-allocates N fixed-size slots, writes sequentially, wraps around, and puts a sequence number + CRC in every record so a reboot can find the head:
// Fixed-record ring buffer on raw flash / a pre-allocated file
typedef struct __attribute__((packed)) {
uint32_t seq; // monotonically increasing; max seq = newest record
uint32_t ts; // unix seconds
uint16_t sensor_id;
float value;
uint16_t crc16; // over the preceding fields; torn writes fail the check
} record_t; // 16 bytes
#define SLOTS (4 * 1024 * 1024 / sizeof(record_t)) // 4 MB region = 262,144 readings
uint32_t slot_for(uint32_t seq) { return seq % SLOTS; }
// On boot: scan for the record with the highest seq and a valid CRC -> that's the head.
// Append: write record at slot_for(head_seq + 1). Writes are sequential, so the
// flash erases each block once per wrap, which is ideal wear levelling.
Store-and-Forward with MQTT
Edge devices lose connectivity: cellular dead zones, broker restarts, WAN outages lasting days. The standard pattern is store-and-forward. Every reading is persisted locally first, then published upstream, and only removed or marked sent once the broker acknowledges it.
sequenceDiagram
participant S as Sensor
participant G as Gateway (local store)
participant B as MQTT broker
participant I as Ingest service
participant T as TSDB
S->>G: reading (Modbus / BLE / serial)
G->>G: append to local log (seq 1042)
G->>B: PUBLISH QoS 1, packet id 77, batch seq 1000-1042
Note over G,B: link down, no PUBACK received
G->>G: keep buffering seq 1043-9000 on disk
G->>B: reconnect (clean start false, same client id)
G->>B: re-PUBLISH unacked batches, DUP flag set
B-->>G: PUBACK 77
G->>G: advance acked cursor to 1042
B->>I: deliver (at least once)
I->>T: idempotent write keyed by device + seq
Rules that make this work:
- Persist before publishing, and advance an "acked up to seq N" cursor only on
PUBACK(QoS 1). QoS 2 is rarely worth its 4-packet handshake for telemetry. See MQTT QoS levels. - Use a persistent session (
clean_start=false, stable client ID, MQTT 5Session Expiry Interval) so the broker keeps queued QoS 1 messages for subscribers that are offline too. - Because QoS 1 is at-least-once, make ingestion idempotent. Include
(device_id, seq)or(device_id, ts)and dedupe in the TSDB: InfluxDB overwrites identical series+timestamp, QuestDBDEDUP UPSERT KEYS, ClickHouseReplacingMergeTree, TimescaleON CONFLICT DO NOTHINGon a unique index. - Batch on reconnect: one MQTT message carrying 500 readings, compressed (CBOR or protobuf + zstd), instead of 500 messages.
- Bound the local buffer. When the disk fills, drop the oldest data or downsample it on-device (keep 1-minute averages instead of 1-second raw), and expose a "data gap" metric.
- Send timestamps from the device and correct for clock drift. Gateways backfilling a 3-day outage produce out-of-order writes at the TSDB, so pick an engine that accepts them: Prometheus OOO window, IoTDB unsequence files, QuestDB O3, Timescale (any order).
The full pipeline end to end, sensor to dashboard, is built in Project: IoT Sensor Pipeline.
Downsampling, Rollups, Retention and Tiered Storage
Raw resolution is only valuable while it's fresh. Nobody looks at 1-second CPU samples from last March, but they do look at last March's hourly p95. Every production time-series system ends up with a resolution pyramid:
graph LR
R["Raw<br/>10 s resolution<br/>keep 15-30 days<br/>NVMe SSD"] -->|"rollup job / cont. aggregate"| M["5-minute rollups<br/>keep 6-13 months<br/>SSD or HDD"]
M -->|"rollup"| H["1-hour rollups<br/>keep 2-10 years<br/>object storage (Parquet)"]
R -.->|"drop partition"| X1["deleted"]
M -.->|"drop partition"| X2["deleted"]
style R fill:#ff6b6b,color:#fff
style M fill:#ffd43b,color:#000
style H fill:#74c0fc,color:#000
Storage math for 100k series at 10 s intervals, ~1.5 bytes/point:
| Tier | Points/series/day | Bytes/day (100k series) | Retention | Total |
|---|---|---|---|---|
| Raw 10 s | 8,640 | 1.3 GB | 30 d | ~39 GB |
| 5 min (min, max, sum, count = 4 values) | 288 Γ 4 | 0.17 GB | 395 d | ~68 GB |
| 1 h (4 values) | 24 Γ 4 | 0.014 GB | 10 y | ~52 GB |
| Raw for 10 years, for comparison | ~4.7 TB |
Rollups Done Right
- Store mergeable aggregates, not averages. Keep
sumandcount(so avg = sum/count), plusminandmax. An average of averages is wrong whenever buckets have different counts, for example after a device outage. - Percentiles don't roll up. p99 of p99s is meaningless. Store a mergeable sketch: t-digest, DDSketch, HDR histogram, or Prometheus native histograms. Timescale has
percentile_agg()in its toolkit, and ClickHouse hasquantilesTDigestStatein anAggregatingMergeTree. - Counters: roll up
increase()or the last value per bucket, neveravg()of a monotonically increasing counter. - Late data: rollup jobs must re-process a trailing window (Timescale
start_offset, Thanos compactor waits for blocks to be complete) or out-of-order points never reach the rollup.
Retention
The theme of this page: retention must be a metadata operation. Every engine here does it by dropping a time partition.
| Engine | Unit dropped | Command / config |
|---|---|---|
| InfluxDB 1.x/2.x | Shard (1 h / 1 d / 7 d) | Retention policy / bucket retention |
| InfluxDB 3 | Parquet files per day | Retention period per database |
| Prometheus | 2 hβ31 d block directory | --storage.tsdb.retention.time |
| TimescaleDB | Chunk (child table) | add_retention_policy() / drop_chunks() |
| ClickHouse | Partition or part | TTL β¦ DELETE, ALTER TABLE β¦ DROP PARTITION |
| QuestDB | Partition directory | ALTER TABLE β¦ DROP PARTITION WHERE ts < β¦, TTL |
| VictoriaMetrics | Monthly partition | -retentionPeriod |
Tiered Storage: Hot SSD β Object Storage
Cold time-series data is the ideal tenant for object storage. It's immutable, read rarely, read sequentially, and easily large enough (100 MBβ2 GB files) to amortise per-request latency.
| Tier | Media | ~Cost / TB-month | Latency | Holds |
|---|---|---|---|---|
| Hot | Local NVMe | $80β200 (replicated) | Β΅s | Last hoursβdays, head blocks, active chunks |
| Warm | Network SSD / HDD | $40β80 | ms | Last weeks |
| Cold | S3 / GCS standard | ~$21β23 | 10β100 ms first byte | Monthsβyears, Parquet / blocks |
| Archive | S3 Glacier / Deep Archive | $1β4 | minutesβhours | Compliance data |
How each engine gets there: ClickHouse TTL β¦ TO VOLUME 'cold' with an S3 disk. Timescale tiered storage moves chunks to S3 Parquet (managed cloud). Thanos and Mimir use object storage as the primary long-term store. InfluxDB 3 is object-storage-native. QuestDB converts partitions to Parquet. The S3 side is covered in S3 & CloudFront.
Geospatial IoT: A Side Note on R-Trees, Geohash and H3
A lot of IoT data has a location: vehicle fleets, delivery riders, asset trackers, drones. The time axis is handled by everything above. The space axis needs a different index, covered fully in Spatial Indexing Deep Dive and the Geospatial section.
| Approach | Structure | Fits time-series engines how |
|---|---|---|
| R-tree (PostGIS GiST, SQLite R*Tree module) | Tree of nested bounding boxes | Works in TimescaleDB because each chunk has its own GiST index: CREATE INDEX ON pings USING GIST (geom). Queries prune by time chunk first, then spatially. Poor fit for append-only columnar engines |
| Geohash | Base-32 string, shared prefix = nearby | Store as a column or tag at a fixed precision (6 chars β 1.2 km Γ 0.6 km). Prefix filters work with sorted keys in any LSM/MergeTree. Edge effects at cell boundaries |
| H3 (Uber) | Hierarchical hexagon cell IDs as UInt64 | Store h3_index (res 8 β 0.74 kmΒ² cells, res 9 β 0.1 kmΒ²) as an integer column, include it in ORDER BY or a skip index (ClickHouse has geoToH3, h3kRing). Uniform neighbour distance helps heatmaps and "vehicles near X" |
| S2 (Google) | Hilbert-curve cell IDs | Same idea as H3 with square cells. Good covering of arbitrary regions |
Don't make location a tag
Putting lat / lon or a fine-resolution geohash/H3 cell into an InfluxDB tag or Prometheus label turns every movement into a new series, which is the cardinality explosion from earlier. Store coordinates as fields or values. If you must group by area, use a coarse cell (H3 res 5β7, a few hundred to a few thousand distinct values per region) as the tag.
Decision Table: Which Engine (and Which Tree) for Which Workload
| Workload | Recommended engine | Underlying structure | Ingest (single node, order of magnitude) | Compressed size | Why |
|---|---|---|---|---|---|
| Kubernetes / infra metrics, < 10M active series | Prometheus (+ Thanos for history) | Head + WAL + 2 h blocks, Gorilla chunks, postings index | 0.5β1M samples/s | ~1.3β2 B/sample | Ecosystem default, PromQL, alerting |
| Metrics at 10Mβ1B active series, multi-tenant | Grafana Mimir or VictoriaMetrics | Prometheus blocks on S3 / MergeTree-like parts | Horizontally scaled; VM ~1β2M samples/s per vCPU-heavy node | 0.4β2 B/sample | Prometheus-compatible, scales out |
| IoT telemetry + relational context (devices, customers, geofences) | TimescaleDB | Time-chunked Postgres heap + B-trees, columnar for cold | 100kβ1M+ rows/s with batching | 10β50Γ compression | SQL JOINs, PostGIS, one database |
| Industrial IoT, millions of devices Γ many sensors, edge + cloud | Apache IoTDB (or TDengine) | TsFile: device/measurement chunks, page statistics | Millions of points/s | Gorilla/TS_2DIFF + LZ4 | Device hierarchy, aligned series, edge-cloud sync |
| Market ticks, high-rate ingestion, SQL analytics | QuestDB | Append-only column files per time partition | Several M rows/s | Uncompressed (FS or Parquet for cold) | Fastest ingest and scan, simplest layout |
| Metrics + logs + traces in one analytical store, ad-hoc high-cardinality queries | ClickHouse | MergeTree parts, DoubleDelta / Gorilla + ZSTD codecs | 1M+ rows/s per node in batches | 1β2 B/point | Arbitrary SQL, cardinality isn't a series index problem |
| Existing InfluxDB / Telegraf estates, moderate cardinality | InfluxDB 2.x (TSM) | TSM tree + TSI | 0.5β1M points/s | ~1β2 B/point | Mature, Telegraf integration |
| New InfluxDB deployments, high cardinality, long retention | InfluxDB 3 (FDAP) | Arrow buffer β Parquet on object storage | High, scales with nodes | Parquet (dictionary + zstd) | Unlimited cardinality, SQL, cheap retention |
| Gateway / edge buffer (Linux) | SQLite (WAL, batched) or RocksDB (FIFO compaction) | B-tree / LSM | 10kβ100k rows/s batched | β | Small, robust, survives power loss |
| Microcontroller logger | Ring buffer on flash / LittleFS | Circular log | Bounded by flash write speed | Fixed record size | Wear levelling by design, no OS needed |
Rules of thumb:
- Cardinality is your first question, not throughput. Under ~10M series, any engine works. Above that, pick a columnar/MergeTree/Parquet design that doesn't keep a per-series in-memory structure.
- Need JOINs with business data? TimescaleDB, or ClickHouse with dictionaries.
- Need PromQL and alerting? The Prometheus family.
- Retention > 1 year? Make sure the engine can put cold data on object storage and keep rollups.
- On a device? Optimise for write amplification and power-loss recovery, not query speed.
Interview Questions
Q1. "What tree does a time-series database use, and why not a B-tree?"
Most use a log-structured, time-partitioned design: InfluxDB's TSM tree, Prometheus' head + blocks, MergeTree in ClickHouse and VictoriaMetrics. Time-series ingestion is append-mostly across many series at once. A B-tree keyed by (series, time) turns that into random leaf writes and page splits once the hot leaves exceed RAM. A B-tree keyed by time makes per-series range reads scattered. LSM designs buffer in memory, sort by (series, time) there, and flush sequentially. Time partitioning makes retention a file delete. TimescaleDB uses B-trees, but only on small time chunks whose indexes fit in memory.
Q2. "Explain Gorilla compression."
Two streams per series per block. Timestamps are stored as delta-of-delta: regular intervals give D = 0, which is 1 bit, and 96% of Facebook's timestamps were 1 bit. Values are XORed with the previous value. An identical value is 1 bit. Otherwise you store only the meaningful middle bits, either reusing the previous leading/trailing-zero window (10 + bits) or with a new 5-bit lead + 6-bit length header (11). Result: ~1.37 bytes/point vs 16 bytes raw, about 12Γ.
Q3. "Your Prometheus is OOM-killed after a deploy. What happened and how do you fix it?"
Probably a cardinality or churn spike: a new label with unbounded values (user ID, request path, pod hash) or a rollout that created thousands of new series. Each active series costs a few KB in the head. Check prometheus_tsdb_head_series, run promtool tsdb analyze, look at topk(10, count by (__name__)({__name__=~".+"})). Fix with metric_relabel_configs to drop or replace the label, add sample_limit per target, and for scale, shard or move to Mimir or VictoriaMetrics.
Q4. "How do you delete 30-day-old data at 1M points/s without hurting writes?"
Never row deletes. Partition by time and drop whole partitions: Timescale drop_chunks, Prometheus block deletion, Influx shard expiry, ClickHouse DROP PARTITION / TTL. It is O(1) in rows and costs no compaction or vacuum.
Q5. "Why did InfluxDB 3.0 move to Arrow/Parquet/DataFusion?" TSM's per-series index (TSI) and per-series blocks made high cardinality expensive in RAM, startup time and compression. TSM files were also proprietary and tied to local disk. Columnar Parquet sorted by tags then time removes the per-series index, so cardinality becomes a scan-cost problem rather than a memory problem. It enables object storage and stateless compute, and lets them reuse a community query engine (DataFusion) and a format other tools can read.
Q6. "Design storage for 1M IoT devices reporting every 10 s, with offline gateways."
That is 100k points/s. Edge: buffer to local storage (SQLite WAL or a ring buffer), publish batched over MQTT QoS 1 with persistent sessions, advance the cursor on PUBACK. Ingest: MQTT broker β Kafka β idempotent writer (dedupe on device_id + seq). Storage: TimescaleDB (if you need relational joins) or ClickHouse/IoTDB, partitioned by day, columnar compression after 1β7 days, continuous aggregates at 5 min and 1 h, raw kept 30 days, rollups kept for years on object storage. The engine must accept out-of-order writes from gateways backfilling outages.
Q7. "How does Prometheus find all series matching {job="api", status=~"5.."}?"
Through postings lists, an inverted index from label=value to a sorted list of series IDs. It unions the postings of every status value matching the regex, intersects with job=api and the metric name by merging the sorted lists, then reads only the chunks of the matching series whose time range overlaps the query.
Q8. "Why might a flash-based edge device die after a few months of logging, and how do you prevent it?"
Write amplification stacks up: small synchronous writes, then filesystem journaling, then database journaling and checkpoints, then the cheap FTL's erase-block read-modify-write. A TLC card with ~1,000 P/E cycles burns through its lifetime writes in weeks. Prevent it by batching writes (one transaction per minute), using a sequential ring buffer or FIFO-compacted LSM, using industrial pSLC/SLC media, mounting with noatime, keeping temp data in tmpfs, and downsampling on-device.
Q9. "TimescaleDB uses B-trees. Doesn't that contradict everything you just said?" No. The B-tree problem is a large index with random inserts that don't fit in memory. Hypertables keep each chunk's index small, sized so the hot chunk fits in ~25% of RAM, so inserts hit cached pages. Cold chunks are converted to columnar compressed batches anyway. The partitioning does the job that the LSM's memtable does elsewhere.
Q10. "How do you downsample without losing correctness?"
Store mergeable aggregates (sum, count, min, max) rather than averages. Use mergeable sketches (t-digest, DDSketch, native histograms) for percentiles. Use increase / last for counters. Re-process a trailing window to absorb late data.
Further Reading
- Storage Engines β B-trees, LSM trees, compaction and amplification, the foundation for every engine here
- Time-Series Databases β product-level comparison of TimescaleDB, InfluxDB and QuestDB, data modelling and tag design
- ClickHouse Internals β MergeTree parts, sparse primary index, codecs and TTL
- Write-Ahead Logging β the durability mechanism every engine above relies on
- SQLite Internals β the most common edge database, its B-tree pages and WAL mode
- Cassandra Internals β a general-purpose LSM often used (with TWCS) for time series
- MQTT for IoT β QoS, persistent sessions and brokers for store-and-forward
- Prometheus Deep Dive and PromQL Cheat Sheet β operating the engine described above
- Metrics Design β label design that avoids cardinality explosions
- Spatial Indexing Deep Dive β R-trees, geohash, S2 and H3 in depth
- ClickHouse vs Druid vs Pinot β real-time analytical stores that also serve time series
- Project: IoT Sensor Pipeline β an end-to-end implementation
- Pelkonen et al. (2015): "Gorilla: A Fast, Scalable, In-Memory Time Series Database", VLDB β the compression scheme
- Fabian Reinartz (2017): "Writing a Time Series Database from Scratch" β the design of Prometheus TSDB v3
- InfluxData (2016): "The New InfluxDB Storage Engine: Time Structured Merge Tree" and (2023) "FDAP: Flight, DataFusion, Arrow, Parquet"
- Liakos et al. (2022): "Chimp: Efficient Lossless Floating Point Compression for Time Series Databases", VLDB
- Afroozeh et al. (2024): "ALP: Adaptive Lossless floating-Point Compression", SIGMOD