On this page

Time-Series & IoT Storage Engines: TSM Trees, Gorilla Compression & Beyond

"What tree do IoT databases use?" has a short answer and a long one. The short answer: almost none of them use a plain B-tree for the data itself. They use log-structured, time-partitioned, columnar layouts. Think of an LSM tree, specialised for data that arrives roughly in time order and is almost never updated. InfluxDB calls its version a TSM (Time-Structured Merge) tree. Prometheus calls it blocks and chunks. VictoriaMetrics and ClickHouse call it a MergeTree. TimescaleDB is the odd one out. It does use Postgres B-trees, and it gets away with it because it cuts the table into small time chunks first.

The long answer is this page. Time-Series Databases compares the products. This page opens them up and shows how the bytes are laid out on disk, how a float shrinks from 16 bytes to about 1.4, and where each design breaks.

Prerequisites

This page assumes you know how B-trees and LSM trees work: memtables, SSTables, compaction, write amplification. If not, read Storage Engines first. Everything below is a variation on those two designs.

Why Time-Series Workloads Break General-Purpose Engines

A general-purpose OLTP engine like InnoDB or Postgres heap+B-tree is built for a workload where any row can be read, updated or deleted at any time, with about equal probability. Time-series and IoT data looks nothing like that.

PropertyOLTP (orders, users)Time-series / IoT (metrics, sensors)Consequence for the engine
Write patternInserts + updates + deletes~99.9% inserts, append-mostlyMVCC versions, undo logs and in-place updates are wasted work
Key order of arrivalRandom (UUIDs, user IDs)Nearly sorted by time within a seriesSorted runs are almost free to produce
Write volume10²–10⁴ rows/s10⁡–10⁷ points/sPer-row overhead (24-byte tuple header in Postgres) dominates
Row widthWide, heterogeneousNarrow: (series, ts, value) = 16 bytes of payloadPer-row metadata can be bigger than the data
CardinalityMillions of rows, few "kinds"Millions of distinct series, each a tiny streamThe index of series becomes the bottleneck, not the data
Read patternPoint lookups by PKRange scans of one series over time, aggregates over manyData should sit contiguously by (series, time)
HotnessUniform-ishLast 1–24 h is >90% of readsKeep recent data in RAM, old data on cheap storage
DeletionRow-by-rowBulk "drop everything older than 30 days"Deletes should be unlink() of a file, not a tombstone per row

The last row matters a lot. A system that holds 30 days of data and ingests 1M points/s must delete 1M points/s forever, just to stay level. If deletion means writing a tombstone per row and compacting it away later, deletion costs as much as ingestion. Every successful TSDB partitions by time so that retention becomes "delete the directory for day N-30".

Why a Single B-Tree Struggles

Picture the obvious schema: one table metrics(series_id, ts, value) with a B-tree primary key on (series_id, ts). That key order is what range queries want, since all of one series' points sit together.

Now look at the write pattern. 100,000 sensors each report every 10 seconds. In any given second, ~10,000 different series get one new point each. With key (series_id, ts), each new point belongs at the end of its own series' range, and those ranges are spread over the whole tree:

graph TD
    Root["Root page"] --> I1["Internal: series 0-25k"]
    Root --> I2["Internal: series 25k-50k"]
    Root --> I3["Internal: series 50k-75k"]
    Root --> I4["Internal: series 75k-100k"]
    I1 --> L1["Leaf: s=00017 ... ts=now"]
    I1 --> L2["Leaf: s=08341 ... ts=now"]
    I2 --> L3["Leaf: s=31002 ... ts=now"]
    I3 --> L4["Leaf: s=66120 ... ts=now"]
    I4 --> L5["Leaf: s=99871 ... ts=now"]
    W["10,000 inserts/sec,<br/>one per series"] -.->|"random leaf"| L1
    W -.->|"random leaf"| L2
    W -.->|"random leaf"| L3
    W -.->|"random leaf"| L4
    W -.->|"random leaf"| L5
    style W fill:#ff6b6b,color:#fff

Each insert touches a different leaf page. Once the working set of "current tail leaves" (100k series Γ— an 8 KB page = 800 MB) is bigger than the buffer pool, every insert is a random read-modify-write. Leaves fill up and split. Splits leave pages ~50–70% full, which bloats the tree, and each split writes 2–3 pages to the WAL. In Postgres you also pay the 24-byte heap tuple header plus a ~16-byte index tuple to store a 16-byte payload. That is roughly 3Γ— overhead before any compression.

Flip the key to (ts, series_id) and inserts become append-only at the right edge of the tree. Writes get fast again, but now "give me series 66120 for the last week" has to skip over 100k series per timestamp. A single B-tree cannot be sorted for both the write pattern (by time) and the read pattern (by series).

Why LSM-Style Designs Fit

An LSM tree breaks that tension. Writes go into an in-memory sorted structure. That structure can be sorted by (series, time) cheaply because it is in RAM. It gets flushed as an immutable sorted file. The write is sequential on disk and the file is laid out for range reads. Time-series data then hands LSM designs three extra gifts:

  1. No updates. Merges rarely have to reconcile versions. Compaction is mostly concatenation plus re-compression.
  2. Time locality. If files are also partitioned by time window, a query for "last hour" opens only the files for the last hour, and retention drops whole files.
  3. Homogeneous columns. One series' timestamps are nearly evenly spaced and its values change slowly. That is ideal for the delta and XOR encodings below.
graph LR
    subgraph "Generic LSM (RocksDB)"
        A1["Memtable<br/>skiplist of key/value"] --> A2["SSTable L0"] --> A3["L1 ... Ln<br/>leveled compaction"]
    end
    subgraph "Time-series LSM (TSM, Prometheus, MergeTree)"
        B1["In-memory buffer<br/>per-series columns"] --> B2["Immutable file<br/>for time window T"]
        B2 --> B3["Compacted larger<br/>time window"]
        B3 --> B4["Retention: unlink<br/>whole file"]
    end
    style B4 fill:#51cf66,color:#fff

The rest of this page shows how each major engine fills in that template.

InfluxDB's TSM Tree (1.x / 2.x)

InfluxDB started on LevelDB, then moved to BoltDB (a B+tree), then gave up on both. Their 2015 write-up says it plainly: LevelDB couldn't delete old data cheaply because it had no time sharding, and BoltDB's random writes collapsed once the database grew past a few GB. The fix was the Time-Structured Merge tree, released in 0.9.5 and the default engine for all of 1.x and 2.x.

Shards: Time Partitioning First

Before any tree comes into play, InfluxDB cuts data by time into shard groups. Each shard is a fully independent TSM engine with its own WAL, cache, TSM files and index.

Retention policy durationDefault shard group duration
< 2 days1 hour
2 days – 6 months1 day
> 6 months7 days

Retention enforcement is then rm -rf of a shard directory. That is O(1) in the number of points. It is the single most important design choice in the engine.

The Write Path: WAL β†’ Cache β†’ TSM Files

sequenceDiagram
    participant C as Client (line protocol)
    participant W as WAL segment
    participant Ca as Cache (in-memory)
    participant S as Snapshotter
    participant T as TSM files
    participant K as Compactor
    C->>W: append batch (snappy-compressed)
    W-->>C: fsync then 204 No Content
    C->>Ca: insert points into per-series value arrays
    Note over Ca: cache-snapshot-memory-size 25 MB or 10 min cold
    Ca->>S: snapshot cache
    S->>T: write new level-1 TSM file
    S->>W: delete WAL segments covered by snapshot
    K->>T: merge level-1 files into level-2, then 3, then full
  1. WAL. Each write batch is appended to the current WAL segment (10 MB per segment), snappy-compressed, then fsynced. Only then is the write acknowledged. The WAL is purely for crash recovery and is never read on the query path. See Write-Ahead Logging for the general idea.
  2. Cache. The same points go into an in-memory map from series key + field to a slice of (timestamp, value). Queries read the cache, so new data is visible right away. The cache is not a sorted skiplist. Each series' slice is sorted lazily at snapshot time, because data almost always arrives in order already.
  3. Snapshot. When the cache passes cache-snapshot-memory-size (default 25 MB), or the shard has been write-cold for cache-snapshot-write-cold-duration (default 10 min), the cache is written out as a new TSM file and the matching WAL segments are deleted. If the cache hits cache-max-memory-size (default 1 GB) before a snapshot finishes, writes are rejected with cache-max-memory-size exceeded. That is the engine's backpressure.

Inside a TSM File

A TSM file is an immutable, sorted, columnar SSTable with a twist. Each block holds up to 1,000 points of one field of one series, and timestamps and values are compressed separately within the block.

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Header: magic 0x16D116D1 (4 B) β”‚ version 1 (1 B)             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Block 1: CRC32 β”‚ type β”‚ len(ts) β”‚ ts bytes β”‚ value bytes     β”‚  ← cpu,host=a#usage_user
β”‚ Block 2: CRC32 β”‚ type β”‚ len(ts) β”‚ ts bytes β”‚ value bytes     β”‚  ← cpu,host=a#usage_user (next 1000 pts)
β”‚ Block 3: ...                                                 β”‚  ← cpu,host=b#usage_user
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Index (sorted by key):                                       β”‚
β”‚   key_len β”‚ key β”‚ type β”‚ count                               β”‚
β”‚     [min_time β”‚ max_time β”‚ offset β”‚ size] Γ— count            β”‚
β”‚   ...                                                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Footer: index offset (8 B)                                   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The index sits at the end. It maps each series key to a list of (min_time, max_time, offset, size) entries, one per block. Reading cpu,host=b#usage_user for 10:00–11:00 goes like this:

  1. Binary-search the index (mmapped) for the key.
  2. Walk that key's block entries and keep those whose [min_time, max_time] overlaps 10:00–11:00.
  3. Read and decompress only those blocks.

There are no bloom filters as in RocksDB. Instead, the min/max time ranges per block do the pruning. That is cheaper and more selective for time-range queries.

Per-block encodings depend on the column type:

ColumnEncodingTypical size
TimestampsDelta β†’ if all deltas equal: RLE (first, delta, count). Else simple8b packing of deltas, scaled by the largest common divisor (e.g. 10⁹ for second-precision ns timestamps)Regular interval: ~0 bits/point. Jittery: 1–2 bytes
FloatsGorilla XOR (see below)1–2 bytes
IntegersZigZag β†’ delta β†’ simple8b, or RLE if constant delta1–3 bytes
BooleansBit-packed1 bit
StringsSnappy over the concatenated blockVaries

Compaction Levels

Snapshots produce many small level-1 files. The compactor merges them in tiers:

graph LR
    S["Snapshot<br/>from cache"] --> L1["Level 1<br/>small files"]
    L1 -->|"4 L1 files"| L2["Level 2"]
    L2 -->|"4 L2 files"| L3["Level 3"]
    L3 -->|"shard cold"| F["Full / optimize<br/>compaction"]
    F --> O["1-N large files<br/>blocks fully packed<br/>1000 pts each"]
    style O fill:#51cf66,color:#fff
  • Level compactions (1β†’2β†’3) merge groups of files by concatenating and re-sorting blocks. They mostly copy blocks over and only decode and re-encode when blocks overlap in time.
  • Full compaction runs when a shard goes cold, for example yesterday's shard once today's takes the writes. It rewrites everything into a few files of max size (~2 GB each) with blocks fully packed to 1,000 points. Packing is where the best compression comes from.
  • Optimize compaction packs fragmented blocks when there are too many files at the top level.

Write amplification is low in practice, around 2–4Γ—. Data mostly arrives in order, and a shard stops receiving writes once its time window closes. Compare that with RocksDB leveled compaction at 10–30Γ—, as covered in Storage Engines.

Deletes, the rare ones, write a .tombstone file next to the TSM file. The next compaction drops the covered points.

The Series Index: from inmem to TSI

TSM files answer "give me points for this exact series key". InfluxQL queries look like WHERE region='eu' AND host=~/web-.*/. Answering that needs an inverted index from tag pairs to series. That index is where InfluxDB lives or dies.

inmem index (≀ 1.4 default). Every shard built a full in-RAM map of measurement β†’ tag key β†’ tag value β†’ series IDs at startup by scanning every TSM file. Memory use was roughly 1–2 KB per series per shard. 10M series meant tens of GB of RAM and startups that took many minutes.

TSI (Time Series Index, 1.5+ default, the only index in 2.x) moves the index onto disk with the same LSM shape as the data:

graph TD
    New["New series on write"] --> SF["Series file (per DB)<br/>series key to series ID<br/>mmapped, Robin Hood hash"]
    New --> LOG["TSI log file (.tsl)<br/>append-only, in-memory copy"]
    LOG -->|"max-index-log-file-size 1 MB"| IDX["TSI index file (.tsi)<br/>immutable, sorted"]
    IDX -->|"compaction"| IDX2["Larger .tsi files"]
    IDX2 --- HLL["HyperLogLog++ sketches<br/>per measurement and tag<br/>for fast cardinality estimates"]
    style SF fill:#ffd43b,color:#000
  • The series file gives every series key a compact 64-bit ID, shared across shards.
  • Each shard's TSI is split into 8 partitions by series hash, so concurrent writes don't contend.
  • A .tsi file holds sorted blocks: measurement β†’ tag key β†’ tag value β†’ sorted series ID set (roaring-bitmap-like). Only the pages a query touches are paged in through mmap.
  • HyperLogLog sketches answer SHOW SERIES CARDINALITY without counting.

How High Cardinality Hurts TSM

Cardinality is the number of distinct series: measurement Γ— every distinct combination of tag values. It costs you in four places at once:

  1. Series file and TSI grow without bound. Each new host=…,container_id=… writes a new series entry that lives forever, or at least until the shard expires.
  2. Blocks get tiny. 1M points/s spread over 10 series gives full 1,000-point blocks quickly. Spread over 10M series, each series gets one point per 10 s, so a block per series per snapshot holds ~2 points. The compression ratio collapses and the TSM index (one entry per block) balloons.
  3. Queries fan out. GROUP BY container_id decodes millions of tiny blocks.
  4. Startup and compaction cost scales with series count, not just bytes.

The practical ceiling was low tens of millions of series per node. Putting request IDs, user IDs or pod UIDs in tags is still the number-one cause of InfluxDB outages.

InfluxDB 3.0: Why They Threw TSM Away

In 2020 InfluxData started IOx, a full rewrite in Rust, which shipped as InfluxDB 3.0 (Cloud first, then 3 Core / Enterprise GA in 2025). It drops TSM completely and is built on what they call the FDAP stack:

LayerProjectRole
FlightApache Arrow Flight (gRPC)Moves Arrow record batches over the network without serialisation
DataFusionApache DataFusionVectorised SQL query engine (and InfluxQL front-end)
ArrowApache ArrowIn-memory columnar format for buffered writes and query execution
ParquetApache ParquetOn-disk columnar files, written to object storage (S3, GCS, local)
graph LR
    W["Line protocol writes"] --> WAL["WAL<br/>flushed to object store<br/>every 1 s by default"]
    WAL --> Buf["In-memory Arrow buffer<br/>queryable immediately"]
    Buf -->|"persist every ~10 min"| PQ["Parquet files<br/>partitioned by table and day<br/>sorted by tags then time"]
    PQ --> OS[("Object storage<br/>S3 / GCS / disk")]
    Q["SQL / InfluxQL"] --> DF["DataFusion<br/>predicate + time pruning<br/>via Parquet stats"]
    DF --> Buf
    DF --> PQ
    style OS fill:#74c0fc,color:#000

Why they did it:

  • Cardinality stops mattering much. There is no per-series index at all. A tag is just a dictionary-encoded column in a Parquet file sorted by tags then time. A new container_id value costs a few bytes in a dictionary, not a new series entry plus a new index entry. The cost moved from write time and RAM to scan time, and vectorised scans of sorted, dictionary-encoded columns are very fast.
  • Object storage as the source of truth. Separating compute from storage makes nodes stateless. Ten years of data on S3 costs ~$23/TB-month instead of $100+/TB-month on replicated SSDs.
  • SQL and ecosystem. Parquet files can be read by Spark, DuckDB, Snowflake, pandas. TSM files could only be read by InfluxDB.
  • Not building a query engine alone. Arrow and DataFusion are maintained by a large community.

The trade-off: per-series point lookups over long ranges no longer have a tiny dedicated index. The engine relies on file and row-group pruning by time and min/max tag statistics instead. Broad analytical queries got much faster. Narrow "one series, last 30 days" queries depend on good sort order.

The industry converged

InfluxDB 3 (Parquet), VictoriaMetrics (MergeTree-like), Mimir and Thanos (Prometheus blocks on S3), ClickHouse for metrics, and TimescaleDB's columnstore all end up in the same place. Data is columnar, sorted by series then time, partitioned by time, compressed per column, and pushed to object storage when cold. The differences lie in the index and the compression codecs.

Facebook Gorilla Compression

In 2015 Facebook published "Gorilla: A Fast, Scalable, In-Memory Time Series Database" (Pelkonen et al., VLDB 2015). Gorilla was an in-memory write-through cache in front of their HBase-based ODS monitoring system. It had to hold the last 26 hours of ~2 billion time series in RAM. Raw, a point is a 64-bit timestamp + a 64-bit float = 16 bytes. Gorilla got that down to an average of 1.37 bytes per point, about 12Γ— smaller. Its two tricks now show up in InfluxDB TSM, Prometheus, VictoriaMetrics, TimescaleDB, M3, QuestDB and IoTDB.

The scheme works on one series at a time, in blocks aligned to 2-hour windows. Timestamps and values are compressed as separate streams.

Timestamps: Delta-of-Delta

Monitoring data arrives on a schedule, say every 60 s, with small jitter. Deltas between timestamps are almost constant, so the delta of the deltas is almost always zero.

Algorithm for a block whose header timestamp is t₋₁ (start of the 2-hour window):

  1. Store the header t₋₁ (64 bits).
  2. Store the first delta tβ‚€ βˆ’ t₋₁ in 14 bits (enough for 2 hours plus some slack at 1-second precision).
  3. For each following timestamp, compute D = (tβ‚™ βˆ’ tₙ₋₁) βˆ’ (tₙ₋₁ βˆ’ tβ‚™β‚‹β‚‚) and emit:
Range of DPrefixValue bitsTotal bits
D = 00β€”1
βˆ’63 … 641079
βˆ’255 … 256110912
βˆ’2047 … 204811101216
anything else11113236

Worked example, block starting at 02:00:00, a sensor that reports about once a minute:

TimestampDeltaDelta-of-delta DEncoded asBits
02:01:0262β€” (first)14-bit delta 0000000011111014
02:02:0260βˆ’210 + 7-bit βˆ’2 = 10 11111109
02:03:0260001
02:04:0260001
02:05:0159βˆ’110 + 11111119
02:06:0160+110 + 00000019

Six timestamps take 43 bits after the header, versus 384 bits raw. In Facebook's production data, 96% of timestamps compressed to a single bit.

Values: XOR of Consecutive Floats

Neighbouring values in a series are usually similar, often identical. Two close IEEE-754 doubles share their sign, their exponent and the top bits of their mantissa. XOR them and the result is mostly zeros: a run of leading zeros, a short "meaningful" middle, and a run of trailing zeros.

Encoding for each value after the first, which is stored raw in 64 bits:

  1. x = bits(vβ‚™) XOR bits(vₙ₋₁).
  2. If x == 0, emit 0 (1 bit). The value repeated.
  3. Otherwise emit 1, then:
    • Control 0: the meaningful bits of x fit inside the previous window (same or more leading zeros and same or more trailing zeros). Emit only the bits in that window. No header needed.
    • Control 1: emit 5 bits giving the number of leading zeros, 6 bits giving the length of the meaningful bits, then the meaningful bits. This becomes the new window.

Worked Bit-Level Example

Series: 12.0, 12.0, 24.0, 15.5, 14.5, 14.5

First, the raw IEEE-754 bits (sign β”‚ 11-bit exponent β”‚ 52-bit mantissa, shown as hex):

Value=Hex bits
12.01.5 Γ— 2Β³0x4028000000000000
24.01.5 Γ— 2⁴0x4038000000000000
15.51.9375 Γ— 2Β³0x402F000000000000
14.51.8125 Γ— 2Β³0x402D000000000000

Now encode:

#ValueXOR with previousLeading / trailing zerosWindow?EmittedBits
012.0β€”β€”β€”raw 64 bits64
112.00x0000000000000000β€”β€”01
224.00x001000000000000011 / 52 (1 meaningful bit)no window yet1 1 01011 000001 114
315.50x001700000000000011 / 48 (5 meaningful bits)needs trailing β‰₯ 52, has 48 β†’ doesn't fit1 1 01011 000101 1011118
414.50x000200000000000014 / 49window is [11 lead, 48 trail]. 14 β‰₯ 11 and 49 β‰₯ 48 β†’ fits1 0 + 5 window bits 000107
514.50x0000000000000000β€”β€”01

Total: 105 bits for six values versus 384 bits raw, 3.7Γ— even on this tiny, jumpy sample. Real sensor data is much smoother: a temperature that changes in the 3rd significant digit, or a gauge that sits still for minutes. Facebook measured this distribution of value encodings:

CaseShare of valuesAvg bits
Identical to previous (0)~51%1
Fits previous window (10)~30%~26.6
New window (11)~19%~36.9

Together with the ~1-bit timestamps, that gives 1.37 bytes/point on average over their 2-hour blocks. A day of a 1-minute series (1,440 points) shrinks from ~23 KB to ~2 KB.

Reference Implementation

This is the full encoder. It reproduces the bit counts in both tables above:

import struct

class BitWriter:
    def __init__(self): self.bits = []
    def write(self, value, n):
        for i in range(n - 1, -1, -1):
            self.bits.append((value >> i) & 1)

def f2u(x): return struct.unpack(">Q", struct.pack(">d", x))[0]

def encode_timestamps(header, ts, w):
    w.write(header, 64)
    delta = ts[0] - header
    w.write(delta, 14)
    prev, prev_delta = ts[0], delta
    for t in ts[1:]:
        delta = t - prev
        dod = delta - prev_delta
        if dod == 0:                 w.write(0b0, 1)
        elif -63 <= dod <= 64:       w.write(0b10, 2);   w.write(dod & 0x7F, 7)
        elif -255 <= dod <= 256:     w.write(0b110, 3);  w.write(dod & 0x1FF, 9)
        elif -2047 <= dod <= 2048:   w.write(0b1110, 4); w.write(dod & 0xFFF, 12)
        else:                        w.write(0b1111, 4); w.write(dod & 0xFFFFFFFF, 32)
        prev, prev_delta = t, delta

def encode_values(vals, w):
    prev = f2u(vals[0]); w.write(prev, 64)
    lead = trail = None
    for v in vals[1:]:
        cur = f2u(v); x = cur ^ prev; prev = cur
        if x == 0:
            w.write(0, 1); continue
        w.write(1, 1)
        l = min(64 - x.bit_length(), 31)          # 5-bit field caps at 31
        t = (x & -x).bit_length() - 1             # trailing zeros
        if lead is not None and l >= lead and t >= trail:
            w.write(0, 1); w.write(x >> trail, 64 - lead - trail)
        else:
            lead, trail = l, t
            w.write(1, 1); w.write(l, 5); w.write(64 - l - t, 6); w.write(x >> t, 64 - l - t)

if __name__ == "__main__":
    w = BitWriter(); encode_values([12.0, 12.0, 24.0, 15.5, 14.5, 14.5], w)
    assert len(w.bits) == 105
    w = BitWriter(); encode_timestamps(0, [62, 122, 182, 242, 301, 361], w)
    assert len(w.bits) == 64 + 43
    print("ok")

Where Gorilla compresses badly

  • High-precision noisy floats. A float with random low-order bits, like 23.418723… from a raw ADC reading divided by a calibration constant, XORs into ~50 meaningful bits. Rounding at the edge (round(v, 2)) often cuts storage 3–5Γ—.
  • Counters stored as floats that jump by large amounts. Store them as integers, with delta + zigzag + simple8b.
  • Irregular timestamps. Event-driven IoT data, such as "door opened", has no regular interval. Delta-of-delta then gives 12–36 bits per timestamp.
  • Random access. The stream can only be decoded from the start of a block. That is why everyone caps blocks at ~120–1,000 points.

Descendants and Improvements

CodecUsed byIdeavs Gorilla
Gorilla XORPrometheus, InfluxDB TSM, M3, TimescaleDB floats, IoTDB GORILLAAs aboveBaseline
Chimp / Chimp128 (VLDB 2022)Research, some enginesUses the trailing zeros better and can XOR against one of the last 128 values~1.3–2Γ— smaller, faster decode
Delta + simple8b / FastPFORTSM integers, Timescale intsBit-pack many small integers per 64-bit wordBetter for integers
ALP (SIGMOD 2024)DuckDBTurn decimal-like floats into integers with an exponent, then bit-packOften 2–3Γ— better on real-world "decimal" floats
VictoriaMetricsVictoriaMetricsConvert floats to scaled int64 (decimal mantissa + exponent), then delta + zstdCommonly 0.4–1 byte/sample
ZSTD / LZ4 over blocksClickHouse (Delta, ZSTD, Gorilla, DoubleDelta, T64), ParquetGeneric entropy coding after a specialised transformGood general fallback

Prometheus TSDB

Prometheus' storage engine (TSDB v3, written by Fabian Reinartz for Prometheus 2.0 in 2017) is the most widely run time-series engine in the world. It is a log-structured, time-partitioned design: an in-memory head block for the last ~2–3 hours, and immutable on-disk blocks for everything older. The operational view is in Prometheus Deep Dive. This section is about the bytes.

graph TD
    Scrape["Scrape / remote-write<br/>samples"] --> WAL["WAL<br/>128 MB segments<br/>+ checkpoints"]
    Scrape --> Head["Head block (memory)<br/>series map + postings<br/>active chunk per series"]
    Head -->|"full chunk: 120 samples"| MM["chunks_head/<br/>mmapped full chunks"]
    Head -->|"every 2 h, cut oldest 2 h"| B1["Block 01H...<br/>2 h, immutable"]
    B1 --> C["Compactor"]
    B2["Block 2 h"] --> C
    B3["Block 2 h"] --> C
    C --> BL["Block 6 h / 18 h / 54 h ...<br/>up to 10% of retention, max 31 d"]
    BL -->|"older than retention"| Del["delete directory"]
    style Head fill:#ffd43b,color:#000
    style Del fill:#ff6b6b,color:#fff

Head Block, Chunks and the WAL

  • Every series (a unique label set) in the head has an in-memory struct: labels, a reference ID, and its currently open chunk.
  • A chunk holds up to 120 samples of one series, Gorilla-encoded. At a 15 s scrape interval, 120 samples is 30 minutes. Prometheus uses the XOR value encoding unchanged but widens the timestamp buckets for millisecond precision: delta-of-delta buckets of 14, 17, 20 bits, else 64 bits. Native histograms use their own chunk encoding.
  • Since v2.19, full chunks are mmapped from chunks_head/ instead of kept on the heap, which cut head memory by 30–50%.
  • Every sample is also appended to the WAL (wal/, 128 MB segments, records for series, samples, tombstones, exemplars). On restart, Prometheus replays the WAL to rebuild the head. Long replays (minutes for tens of millions of series) are a classic restart pain, which is what WAL checkpoints and the v2.30 in-memory snapshot on shutdown address.

Blocks: 2-Hour Immutable Directories

Every 2 hours the head cuts its oldest 2 hours into a block. Each block is a directory named by a ULID:

data/
β”œβ”€β”€ 01JC8X2Y9M0WQ8KX3TZ0E9G4AB/      ← block covering [t, t+2h)
β”‚   β”œβ”€β”€ meta.json                    ← minTime, maxTime, stats, compaction level, sources
β”‚   β”œβ”€β”€ index                        ← symbol table, series, label index, postings
β”‚   β”œβ”€β”€ chunks/
β”‚   β”‚   β”œβ”€β”€ 000001                   ← up to 512 MB of concatenated chunks
β”‚   β”‚   └── 000002
β”‚   └── tombstones                   ← deletion intervals (series ref, mint, maxt)
β”œβ”€β”€ chunks_head/                     ← mmapped full chunks of the head
β”œβ”€β”€ wal/
β”‚   β”œβ”€β”€ 00000941
β”‚   β”œβ”€β”€ 00000942
β”‚   └── checkpoint.00000940/
└── lock

The index file has four parts:

  1. Symbol table. Every distinct label name and value string, stored once and referenced by offset. job, instance, __name__, http_requests_total appear once no matter how many series use them.
  2. Series section. For each series, sorted by label set: its labels as symbol references, plus a list of (mint, maxt, chunk ref) for its chunks.
  3. Postings lists. For every label pair name=value, a sorted list of series IDs that have it. This is the inverted index.
  4. Postings offset table and TOC to find each list.

The Inverted Index: How a Selector Is Answered

A PromQL selector like

http_requests_total{job="api", status=~"5..", region!="us-east"}

resolves like a search engine query:

graph LR
    P1["postings(__name__=http_requests_total)<br/>[2, 5, 9, 14, 21, 30]"] --> I["Intersect<br/>(merge of sorted lists)"]
    P2["postings(job=api)<br/>[5, 9, 11, 14, 30]"] --> I
    P3["union of postings(status=500, 502, 503)<br/>[9, 14, 22, 30]"] --> I
    I --> R1["[9, 14, 30]"]
    R1 --> S["Subtract postings(region=us-east)<br/>[14]"]
    S --> R["Series [9, 30]<br/>then read chunks overlapping query range"]
    style R fill:#51cf66,color:#fff

The engine intersects sorted ID lists with a merge, O(sum of list lengths). For a regex matcher it first scans the label values for that name and unions their postings. That is why {status=~".*"} or matchers on a label with 10M values are slow. Series IDs are assigned in label-sorted order within a block, so the resulting series are also read in sorted order.

Compaction and Retention

  • Vertical compaction merges overlapping blocks, for example after backfill or out-of-order ingestion (out_of_order_time_window, added in v2.39).
  • Horizontal compaction merges adjacent blocks into bigger ranges: 2 h β†’ 6 h β†’ 18 h β†’ 54 h … with each step 3Γ— the last, up to 10% of the retention period or 31 days. Bigger blocks mean fewer index lookups for long queries and a shared symbol table.
  • Deletes through the admin API write tombstones. The data is physically removed at the next compaction.
  • Retention (--storage.tsdb.retention.time=15d or .size=500GB) simply deletes whole block directories whose maxTime is past the window. Once again: retention is unlink().

Typical numbers: ~1.3–2 bytes per sample on disk, about 200–500k samples/s ingested per core, and 1–3 KB of disk index per series per block.

Why Cardinality Explosions Kill Prometheus

The head keeps every active series in memory with its labels, postings entries and an open chunk. A rough rule: ~3–8 KB of RAM per active series including Go GC overhead, varying with label size and churn.

Active seriesApproximate head RAMNotes
100k0.5 GBA small cluster
1M4–8 GBComfortable single Prometheus
10M40–80 GBUpper limit of one instance, long WAL replays
50Mβ€”Shard it, or use Mimir or VictoriaMetrics

A single bad label wrecks this:

http_requests_total{path="/users/8812734/orders"}   ← user ID in the path
http_requests_total{pod="api-7d9f8c6b5-x2k9q"}       ← pod name: churn on every deploy
http_requests_total{trace_id="4bf92f3577b34da6"}     ← one series per request

Every new label value creates a new series. It gets a struct in memory, a new postings entry, a new WAL series record and a chunk that will hold one or two samples, which gives terrible compression. Churn hurts as much as raw cardinality. A Kubernetes rollout that renames 5,000 pods creates 5,000 Γ— (metrics per pod) new series. The old ones stay in the head until they age out, for up to 2–3 h, and stay in the blocks' indexes for the full retention. Defences: metric_relabel_configs to drop labels, sample_limit and label_limit per scrape, the prometheus_tsdb_head_series alert, and promtool tsdb analyze to find the worst labels. See Metrics Design.

Long-Term Storage: Remote Write, Thanos, Mimir, VictoriaMetrics

Prometheus is single-node, and local retention is usually 15–30 days. For years of data and a global view, there are three main architectures:

graph TD
    subgraph "Thanos (sidecar model)"
        P1["Prometheus"] --- SC["Sidecar"]
        SC -->|"upload each 2 h block"| S3A[("Object storage")]
        SG["Store Gateway"] --> S3A
        Q["Querier<br/>fan-out + dedup"] --> SC
        Q --> SG
        CP["Compactor<br/>+ downsample 5 m and 1 h"] --> S3A
    end
    subgraph "Mimir / Cortex (remote-write model)"
        P2["Prometheus / Agent"] -->|"remote_write<br/>snappy protobuf"| D["Distributor<br/>hash ring"]
        D --> IN["Ingesters<br/>per-tenant TSDB head<br/>RF=3"]
        IN -->|"2 h blocks"| S3B[("Object storage")]
        SGB["Store Gateway"] --> S3B
    end
  • Thanos. Each Prometheus keeps writing its normal TSDB. A sidecar uploads finished 2 h blocks to S3/GCS. Queries fan out to the sidecars (recent data) and store gateways (historical data) and deduplicate HA pairs. The compactor merges blocks in object storage and downsamples to 5-minute and 1-hour resolution, so a 1-year dashboard doesn't read raw 15 s samples.
  • Cortex β†’ Grafana Mimir. Prometheus remote_writes to a horizontally scaled, multi-tenant service. Ingesters each run a Prometheus TSDB head per tenant, replicated 3Γ—, cut 2 h blocks and ship them to object storage. Mimir adds a split-and-merge compactor and query sharding, and has been tested at 1 billion active series. It is the same storage format, partitioned across many machines.
  • VictoriaMetrics. Its own engine, explicitly inspired by ClickHouse's MergeTree. Incoming samples are buffered and written as small sorted parts (columns of TSID, timestamps, values, each compressed). Background merges combine them into bigger parts, exactly like ClickHouse MergeTree. The label index (indexdb) is a separate mergeset, an LSM of sorted byte strings mapping label=value β†’ metricIDs and metricID β†’ TSID, partitioned per day. Floats are converted to scaled integers before delta encoding and zstd. In practice VictoriaMetrics reports ~0.4–1 byte/sample and several times less RAM per active series than Prometheus. The cost is its own, less standard file format.

TimescaleDB: Making B-Trees Work by Partitioning Time

TimescaleDB is the counterexample to "B-trees can't do time series". It is Postgres: heap tables, B-tree indexes, the normal WAL. It solves the random-insert problem by changing the shape of the problem rather than the data structure. (The company behind it renamed itself TigerData in 2025. The extension is still timescaledb.)

Hypertables and Chunks

A hypertable is a parent table that is automatically split into chunks, ordinary Postgres child tables, each covering one time interval and optionally a hash of a space column.

CREATE TABLE readings (
    time        timestamptz      NOT NULL,
    device_id   int              NOT NULL,
    temperature double precision,
    humidity    double precision
);
SELECT create_hypertable('readings', by_range('time', INTERVAL '1 day'));
CREATE INDEX ON readings (device_id, time DESC);
graph TD
    H["Hypertable: readings"] --> C1["_hyper_1_101_chunk<br/>2026-11-06<br/>own heap + B-trees"]
    H --> C2["_hyper_1_102_chunk<br/>2026-11-07<br/>own heap + B-trees"]
    H --> C3["_hyper_1_103_chunk<br/>2026-11-08 (hot)<br/>fits in shared_buffers"]
    W["INSERTs (now)"] --> C3
    Q["WHERE time > now() - 2h"] -->|"chunk exclusion"| C3
    R["drop_chunks older than 30 d"] -.->|"DROP TABLE, instant"| C1
    style C3 fill:#ffd43b,color:#000
    style C1 fill:#adb5bd,color:#000

Why the B-tree is fine now:

  • The hot B-tree is small. Only today's chunk takes inserts. Its (device_id, time DESC) index covers one day, not three years, so its pages fit in shared_buffers. The random-leaf problem from earlier is still there, but every leaf is in RAM, so a "random" insert is a memory write plus a WAL record. Timescale's sizing rule: pick chunk_time_interval so the most recent chunk and its indexes fit in ~25% of memory.
  • Chunk exclusion. The planner, and the executor for now()-relative predicates, skips chunks whose time range doesn't overlap the query. That is partition pruning on steroids.
  • Retention is DROP TABLE. drop_chunks('readings', INTERVAL '30 days') or add_retention_policy removes whole chunks. No DELETE, no vacuum storm, no bloat.
  • Postgres stays Postgres. JOINs against devices, foreign data wrappers, PostGIS, logical replication, pg_dump all keep working.

The cost: each row still pays Postgres' ~24-byte tuple header plus ~16 bytes in each index, so raw uncompressed storage is 3–5Γ— larger than a dedicated TSDB. That is why compression matters.

Native Columnar Compression (Hypercore Columnstore)

Older chunks are converted from row-oriented heap into a columnar layout. Internally, rows are grouped into batches of up to 1,000 rows per segmentby value. Each column of a batch is stored as one compressed array in a single row of an internal table:

ALTER TABLE readings SET (
    timescaledb.compress,
    timescaledb.compress_segmentby = 'device_id',
    timescaledb.compress_orderby   = 'time DESC'
);
SELECT add_compression_policy('readings', INTERVAL '7 days');
Before (heap, 1 row per reading):                After (1 row per 1,000 readings of a device):
 time                 device temp   hum          device β”‚ time[]       β”‚ temp[]        β”‚ hum[]         β”‚ _min/_max time
 2026-11-01 00:00:00  42     21.5   40.1          42    β”‚ DoD+simple8b β”‚ Gorilla bytes β”‚ Gorilla bytes β”‚ metadata for pruning
 2026-11-01 00:00:10  42     21.5   40.2          43    β”‚ ...          β”‚ ...           β”‚ ...           β”‚
 ... Γ—1000 per device
Column typeAlgorithm
Integers, timestampsDelta-of-delta + simple-8b + run-length
FloatsGorilla XOR
Low-cardinality textDictionary + RLE
Everything elseArray + LZ (pglz)

Real-world ratios of 90–98% (10–50Γ— smaller) are typical for sensor and metric data. segmentby columns are stored once per batch, and the min/max metadata lets queries skip batches. Writes to compressed chunks are supported (DML on the columnstore, since 2.11), but they are slower. The intended pattern is "rowstore for hot, columnstore for cold".

Continuous Aggregates

A continuous aggregate is a materialized view on a hypertable that refreshes incrementally. Only buckets whose underlying chunks changed get recomputed. It is the built-in downsampling mechanism:

CREATE MATERIALIZED VIEW readings_1h
WITH (timescaledb.continuous) AS
SELECT time_bucket('1 hour', time) AS bucket,
       device_id,
       avg(temperature) AS avg_temp,
       min(temperature) AS min_temp,
       max(temperature) AS max_temp,
       count(*)         AS n
FROM readings
GROUP BY bucket, device_id;

SELECT add_continuous_aggregate_policy('readings_1h',
    start_offset      => INTERVAL '3 days',
    end_offset        => INTERVAL '1 hour',
    schedule_interval => INTERVAL '30 minutes');

The continuous aggregate is itself a hypertable, so it can be compressed, and it can be retained longer than the raw data, e.g. raw for 30 days and hourly for 5 years. "Real-time aggregation" can union the materialized buckets with a live computation over the not-yet-materialized tail.

Columnar Approaches: QuestDB, ClickHouse, IoTDB

QuestDB: Append-Only Column Files

QuestDB (Java and C++, zero-GC on the hot path) takes the simplest possible layout. A table is a directory of time partitions, and each partition holds one file per column. A row is the same index across all column files.

db/sensors/
β”œβ”€β”€ 2026-11-07/                    ← PARTITION BY DAY
β”‚   β”œβ”€β”€ ts.d                       ← designated timestamp, int64, sorted
β”‚   β”œβ”€β”€ device.d                   ← SYMBOL: int32 keys into a dictionary
β”‚   β”œβ”€β”€ temperature.d              ← float64, raw (8 B Γ— n)
β”‚   └── humidity.d
β”œβ”€β”€ 2026-11-08/
β”‚   └── ...
β”œβ”€β”€ device.c / device.o / device.k ← symbol dictionary + optional bitmap index
└── _txn                           ← committed row counts per partition
  • Ingestion via the InfluxDB Line Protocol over TCP or HTTP. Writes are memcpy-appends into mmapped column files. Commits bump a row count in _txn. TSBS benchmarks put single-node ingestion at several million rows/second.
  • Out-of-order (O3) data is the hard part for append-only files. QuestDB sorts each incoming batch, then splits partitions or merges in place when late rows land inside an existing partition. Since 7.x, tables use a WAL with deduplication (DEDUP UPSERT KEYS), and the WAL is applied to column files asynchronously.
  • Queries are SIMD scans over contiguous column arrays plus time-partition pruning. SAMPLE BY 1h and LATEST ON ts PARTITION BY device are the time-series primitives.
  • Compression: the native format stores columns uncompressed for raw mmap speed. Users rely on filesystem compression such as ZFS lz4/zstd for 3–6Γ—. Recent releases can convert cold partitions to Parquet, which compresses and is readable by other tools.

It is the opposite philosophy to Gorilla: spend disk to make scans and ingestion as cheap as possible.

ClickHouse MergeTree for Metrics

ClickHouse isn't a TSDB, but its MergeTree is one of the best time-series engines available. Many observability backends store metrics this way, including SigNoz, Uptrace, and the metrics paths of several large companies. The general mechanics (parts, sparse primary index, merges) are in ClickHouse Internals. The time-series-specific settings:

CREATE TABLE metrics
(
    metric     LowCardinality(String),
    tags_hash  UInt64,                                   -- hash of sorted labels
    ts         DateTime64(3)   CODEC(DoubleDelta, ZSTD(1)),
    value      Float64         CODEC(Gorilla, ZSTD(1))
)
ENGINE = MergeTree
PARTITION BY toDate(ts)
ORDER BY (metric, tags_hash, ts)
TTL toDateTime(ts) + INTERVAL 30 DAY TO VOLUME 'cold',
    toDateTime(ts) + INTERVAL 365 DAY DELETE;
  • ORDER BY (metric, tags_hash, ts) makes each series contiguous inside each part. That is the same "sorted by series then time" property as TSM blocks.
  • The DoubleDelta codec is Gorilla's delta-of-delta. The Gorilla codec is the XOR float encoding. ZSTD on top captures what remains.
  • Series labels usually live in a separate table (tags_hash β†’ labels map) to avoid repeating them per row, which mirrors the TSM series index.
  • Rollups use an AggregatingMergeTree target fed by a materialized view, or GraphiteMergeTree, which downsamples by retention rules during merges.

Compressed sizes of 1–2 bytes per point with scans of billions of rows per second per node are typical. The weakness is the same as all MergeTree designs: small frequent inserts create too many parts. Batch inserts (β‰₯ 10k rows or β‰₯ 1 s), or use async inserts.

Apache IoTDB and the TsFile Format

Apache IoTDB (from Tsinghua University, Apache top-level since 2020) was built specifically for industrial IoT: wind turbines, power grids, factories, with millions of devices and thousands of sensors each. Its storage format, TsFile, is organised around the device β†’ measurement hierarchy:

TsFile
β”œβ”€β”€ Magic "TsFile" + version
β”œβ”€β”€ ChunkGroup (device root.plant1.turbine7)
β”‚   β”œβ”€β”€ Chunk (measurement: wind_speed, encoding GORILLA, compression LZ4)
β”‚   β”‚   β”œβ”€β”€ ChunkHeader
β”‚   β”‚   β”œβ”€β”€ Page 1: PageHeader [count, min_t, max_t, min, max, first, last, sum] + data
β”‚   β”‚   └── Page 2: ...
β”‚   └── Chunk (measurement: rotor_rpm, encoding TS_2DIFF)
β”œβ”€β”€ ChunkGroup (device root.plant1.turbine8)
β”‚   └── ...
β”œβ”€β”€ Index: TimeseriesMetadata per series (with statistics)
β”‚   └── MetadataIndex tree (B+tree-like: device nodes β†’ measurement nodes)
└── Footer: index offset + bloom filter over series paths

Notable ideas:

  • Statistics at every level (page, chunk, series in file): count, min/max time, min/max value, first, last, sum. SELECT max(wind_speed) … GROUP BY ([t1, t2), 1h) can often be answered from page headers alone without decoding the data. This is the same idea as Parquet column statistics, pushed further.
  • Aligned time series. Sensors on one device that sample together can share a single time column ("aligned"), which removes duplicate timestamp storage.
  • Sequence vs unsequence files. In-order data goes to sequence TsFiles. Late or out-of-order data goes to separate unsequence files, and compaction merges them later. This keeps the common path append-only.
  • Encodings: TS_2DIFF (second-order delta, i.e. delta-of-delta), GORILLA, RLE, DICTIONARY, ZIGZAG, CHIMP, SPRINTZ, plus LZ4, ZSTD and Snappy on top.

The write path is the familiar LSM one: WAL β†’ memtable per storage group β†’ flush to TsFile β†’ compaction.

IoT Edge and Embedded Storage

Everything above runs in a data centre on SSDs. The other end of an IoT system is a gateway or sensor node: a Raspberry Pi, an industrial PC, an ESP32. It has 64 MB–4 GB of RAM, an SD card or eMMC, intermittent connectivity, and sometimes power that can vanish mid-write. The storage engine choice there is driven by flash wear and power loss, not query speed.

Flash Wear and Write Amplification

NAND flash can only be erased in big erase blocks (128 KB–4 MB) and each block survives a limited number of program/erase cycles:

MediaTypical P/E cyclesNotes
SLC NAND / industrial SD50,000–100,000Expensive, used in industrial-grade cards
pSLC (MLC in SLC mode)~20,000–30,000Common "industrial" eMMC compromise
MLC3,000–10,000
TLC (consumer SD, cheap eMMC)500–3,000Most Raspberry Pi SD cards
QLC100–1,000Avoid for write-heavy logging

The flash translation layer (FTL) on a cheap SD card is simple. A 4 KB random write can force a read-modify-write of a whole erase block, which can mean 100Γ—+ device-level write amplification. Multiply that by filesystem journaling (ext4 metadata journal) and database journaling (SQLite rollback journal or WAL, plus checkpoint), and a naive "INSERT one row per reading with fsync" logger can wear out a consumer SD card in months.

Back-of-envelope: 10 sensors Γ— 1 reading/s, each a separate SQLite transaction. That is ~3 page writes of 4 KB each (WAL page + checkpoint + metadata):

With 10–50Γ— FTL amplification, that becomes 100–500 GB/day hitting an 8 GB TLC card rated for ~1,000 cycles, i.e. ~8 TB total lifetime writes. That is 2–12 weeks. Batching the same data into one transaction per minute cuts it by ~60Γ—.

Engine Choices on the Edge

OptionStructureGood forWatch out for
SQLite (WAL mode)B-tree pages + WALGateways that need local SQL, config + buffered readingsBatch inserts in transactions. PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;. Use larger checkpoint intervals. Time-partition into one table or file per day so retention is DROP / unlink
RocksDB / LevelDBLSMHigh-rate buffering on Linux gateways (e.g. Azure IoT Edge, some historians)Compaction write amplification (10–30Γ— leveled) adds to FTL amplification. Prefer universal/FIFO compaction. compaction_style=kCompactionStyleFIFO is a natural time-series ring buffer
Embedded TSDB (e.g. InfluxDB OSS on a gateway, QuestDB, IoTDB edge, TDengine)TSM / columnarLocal dashboards and aggregation at the edgeRAM footprint (hundreds of MB), more moving parts
Fixed-size ring buffer filePre-allocated circular log of fixed-size recordsMicrocontrollers, "last N hours of readings", black-box recordersYou design the record format, CRC and recovery
Raw flash KV / log FS (LittleFS, SPIFFS, NVS on ESP32)Log-structured with wear levellingMCUs with no OS filesystemSmall capacity, no query capability

A ring buffer is often the right answer on a microcontroller. It pre-allocates N fixed-size slots, writes sequentially, wraps around, and puts a sequence number + CRC in every record so a reboot can find the head:

// Fixed-record ring buffer on raw flash / a pre-allocated file
typedef struct __attribute__((packed)) {
    uint32_t seq;          // monotonically increasing; max seq = newest record
    uint32_t ts;           // unix seconds
    uint16_t sensor_id;
    float    value;
    uint16_t crc16;        // over the preceding fields; torn writes fail the check
} record_t;                // 16 bytes

#define SLOTS  (4 * 1024 * 1024 / sizeof(record_t))   // 4 MB region = 262,144 readings

uint32_t slot_for(uint32_t seq) { return seq % SLOTS; }

// On boot: scan for the record with the highest seq and a valid CRC -> that's the head.
// Append: write record at slot_for(head_seq + 1). Writes are sequential, so the
// flash erases each block once per wrap, which is ideal wear levelling.

Store-and-Forward with MQTT

Edge devices lose connectivity: cellular dead zones, broker restarts, WAN outages lasting days. The standard pattern is store-and-forward. Every reading is persisted locally first, then published upstream, and only removed or marked sent once the broker acknowledges it.

sequenceDiagram
    participant S as Sensor
    participant G as Gateway (local store)
    participant B as MQTT broker
    participant I as Ingest service
    participant T as TSDB
    S->>G: reading (Modbus / BLE / serial)
    G->>G: append to local log (seq 1042)
    G->>B: PUBLISH QoS 1, packet id 77, batch seq 1000-1042
    Note over G,B: link down, no PUBACK received
    G->>G: keep buffering seq 1043-9000 on disk
    G->>B: reconnect (clean start false, same client id)
    G->>B: re-PUBLISH unacked batches, DUP flag set
    B-->>G: PUBACK 77
    G->>G: advance acked cursor to 1042
    B->>I: deliver (at least once)
    I->>T: idempotent write keyed by device + seq

Rules that make this work:

  • Persist before publishing, and advance an "acked up to seq N" cursor only on PUBACK (QoS 1). QoS 2 is rarely worth its 4-packet handshake for telemetry. See MQTT QoS levels.
  • Use a persistent session (clean_start=false, stable client ID, MQTT 5 Session Expiry Interval) so the broker keeps queued QoS 1 messages for subscribers that are offline too.
  • Because QoS 1 is at-least-once, make ingestion idempotent. Include (device_id, seq) or (device_id, ts) and dedupe in the TSDB: InfluxDB overwrites identical series+timestamp, QuestDB DEDUP UPSERT KEYS, ClickHouse ReplacingMergeTree, Timescale ON CONFLICT DO NOTHING on a unique index.
  • Batch on reconnect: one MQTT message carrying 500 readings, compressed (CBOR or protobuf + zstd), instead of 500 messages.
  • Bound the local buffer. When the disk fills, drop the oldest data or downsample it on-device (keep 1-minute averages instead of 1-second raw), and expose a "data gap" metric.
  • Send timestamps from the device and correct for clock drift. Gateways backfilling a 3-day outage produce out-of-order writes at the TSDB, so pick an engine that accepts them: Prometheus OOO window, IoTDB unsequence files, QuestDB O3, Timescale (any order).

The full pipeline end to end, sensor to dashboard, is built in Project: IoT Sensor Pipeline.

Downsampling, Rollups, Retention and Tiered Storage

Raw resolution is only valuable while it's fresh. Nobody looks at 1-second CPU samples from last March, but they do look at last March's hourly p95. Every production time-series system ends up with a resolution pyramid:

graph LR
    R["Raw<br/>10 s resolution<br/>keep 15-30 days<br/>NVMe SSD"] -->|"rollup job / cont. aggregate"| M["5-minute rollups<br/>keep 6-13 months<br/>SSD or HDD"]
    M -->|"rollup"| H["1-hour rollups<br/>keep 2-10 years<br/>object storage (Parquet)"]
    R -.->|"drop partition"| X1["deleted"]
    M -.->|"drop partition"| X2["deleted"]
    style R fill:#ff6b6b,color:#fff
    style M fill:#ffd43b,color:#000
    style H fill:#74c0fc,color:#000

Storage math for 100k series at 10 s intervals, ~1.5 bytes/point:

TierPoints/series/dayBytes/day (100k series)RetentionTotal
Raw 10 s8,6401.3 GB30 d~39 GB
5 min (min, max, sum, count = 4 values)288 Γ— 40.17 GB395 d~68 GB
1 h (4 values)24 Γ— 40.014 GB10 y~52 GB
Raw for 10 years, for comparison~4.7 TB

Rollups Done Right

  • Store mergeable aggregates, not averages. Keep sum and count (so avg = sum/count), plus min and max. An average of averages is wrong whenever buckets have different counts, for example after a device outage.
  • Percentiles don't roll up. p99 of p99s is meaningless. Store a mergeable sketch: t-digest, DDSketch, HDR histogram, or Prometheus native histograms. Timescale has percentile_agg() in its toolkit, and ClickHouse has quantilesTDigestState in an AggregatingMergeTree.
  • Counters: roll up increase() or the last value per bucket, never avg() of a monotonically increasing counter.
  • Late data: rollup jobs must re-process a trailing window (Timescale start_offset, Thanos compactor waits for blocks to be complete) or out-of-order points never reach the rollup.

Retention

The theme of this page: retention must be a metadata operation. Every engine here does it by dropping a time partition.

EngineUnit droppedCommand / config
InfluxDB 1.x/2.xShard (1 h / 1 d / 7 d)Retention policy / bucket retention
InfluxDB 3Parquet files per dayRetention period per database
Prometheus2 h–31 d block directory--storage.tsdb.retention.time
TimescaleDBChunk (child table)add_retention_policy() / drop_chunks()
ClickHousePartition or partTTL … DELETE, ALTER TABLE … DROP PARTITION
QuestDBPartition directoryALTER TABLE … DROP PARTITION WHERE ts < …, TTL
VictoriaMetricsMonthly partition-retentionPeriod

Tiered Storage: Hot SSD β†’ Object Storage

Cold time-series data is the ideal tenant for object storage. It's immutable, read rarely, read sequentially, and easily large enough (100 MB–2 GB files) to amortise per-request latency.

TierMedia~Cost / TB-monthLatencyHolds
HotLocal NVMe$80–200 (replicated)Β΅sLast hours–days, head blocks, active chunks
WarmNetwork SSD / HDD$40–80msLast weeks
ColdS3 / GCS standard~$21–2310–100 ms first byteMonths–years, Parquet / blocks
ArchiveS3 Glacier / Deep Archive$1–4minutes–hoursCompliance data

How each engine gets there: ClickHouse TTL … TO VOLUME 'cold' with an S3 disk. Timescale tiered storage moves chunks to S3 Parquet (managed cloud). Thanos and Mimir use object storage as the primary long-term store. InfluxDB 3 is object-storage-native. QuestDB converts partitions to Parquet. The S3 side is covered in S3 & CloudFront.

Geospatial IoT: A Side Note on R-Trees, Geohash and H3

A lot of IoT data has a location: vehicle fleets, delivery riders, asset trackers, drones. The time axis is handled by everything above. The space axis needs a different index, covered fully in Spatial Indexing Deep Dive and the Geospatial section.

ApproachStructureFits time-series engines how
R-tree (PostGIS GiST, SQLite R*Tree module)Tree of nested bounding boxesWorks in TimescaleDB because each chunk has its own GiST index: CREATE INDEX ON pings USING GIST (geom). Queries prune by time chunk first, then spatially. Poor fit for append-only columnar engines
GeohashBase-32 string, shared prefix = nearbyStore as a column or tag at a fixed precision (6 chars β‰ˆ 1.2 km Γ— 0.6 km). Prefix filters work with sorted keys in any LSM/MergeTree. Edge effects at cell boundaries
H3 (Uber)Hierarchical hexagon cell IDs as UInt64Store h3_index (res 8 β‰ˆ 0.74 kmΒ² cells, res 9 β‰ˆ 0.1 kmΒ²) as an integer column, include it in ORDER BY or a skip index (ClickHouse has geoToH3, h3kRing). Uniform neighbour distance helps heatmaps and "vehicles near X"
S2 (Google)Hilbert-curve cell IDsSame idea as H3 with square cells. Good covering of arbitrary regions

Don't make location a tag

Putting lat / lon or a fine-resolution geohash/H3 cell into an InfluxDB tag or Prometheus label turns every movement into a new series, which is the cardinality explosion from earlier. Store coordinates as fields or values. If you must group by area, use a coarse cell (H3 res 5–7, a few hundred to a few thousand distinct values per region) as the tag.

Decision Table: Which Engine (and Which Tree) for Which Workload

WorkloadRecommended engineUnderlying structureIngest (single node, order of magnitude)Compressed sizeWhy
Kubernetes / infra metrics, < 10M active seriesPrometheus (+ Thanos for history)Head + WAL + 2 h blocks, Gorilla chunks, postings index0.5–1M samples/s~1.3–2 B/sampleEcosystem default, PromQL, alerting
Metrics at 10M–1B active series, multi-tenantGrafana Mimir or VictoriaMetricsPrometheus blocks on S3 / MergeTree-like partsHorizontally scaled; VM ~1–2M samples/s per vCPU-heavy node0.4–2 B/samplePrometheus-compatible, scales out
IoT telemetry + relational context (devices, customers, geofences)TimescaleDBTime-chunked Postgres heap + B-trees, columnar for cold100k–1M+ rows/s with batching10–50Γ— compressionSQL JOINs, PostGIS, one database
Industrial IoT, millions of devices Γ— many sensors, edge + cloudApache IoTDB (or TDengine)TsFile: device/measurement chunks, page statisticsMillions of points/sGorilla/TS_2DIFF + LZ4Device hierarchy, aligned series, edge-cloud sync
Market ticks, high-rate ingestion, SQL analyticsQuestDBAppend-only column files per time partitionSeveral M rows/sUncompressed (FS or Parquet for cold)Fastest ingest and scan, simplest layout
Metrics + logs + traces in one analytical store, ad-hoc high-cardinality queriesClickHouseMergeTree parts, DoubleDelta / Gorilla + ZSTD codecs1M+ rows/s per node in batches1–2 B/pointArbitrary SQL, cardinality isn't a series index problem
Existing InfluxDB / Telegraf estates, moderate cardinalityInfluxDB 2.x (TSM)TSM tree + TSI0.5–1M points/s~1–2 B/pointMature, Telegraf integration
New InfluxDB deployments, high cardinality, long retentionInfluxDB 3 (FDAP)Arrow buffer β†’ Parquet on object storageHigh, scales with nodesParquet (dictionary + zstd)Unlimited cardinality, SQL, cheap retention
Gateway / edge buffer (Linux)SQLite (WAL, batched) or RocksDB (FIFO compaction)B-tree / LSM10k–100k rows/s batchedβ€”Small, robust, survives power loss
Microcontroller loggerRing buffer on flash / LittleFSCircular logBounded by flash write speedFixed record sizeWear levelling by design, no OS needed

Rules of thumb:

  1. Cardinality is your first question, not throughput. Under ~10M series, any engine works. Above that, pick a columnar/MergeTree/Parquet design that doesn't keep a per-series in-memory structure.
  2. Need JOINs with business data? TimescaleDB, or ClickHouse with dictionaries.
  3. Need PromQL and alerting? The Prometheus family.
  4. Retention > 1 year? Make sure the engine can put cold data on object storage and keep rollups.
  5. On a device? Optimise for write amplification and power-loss recovery, not query speed.

Interview Questions

Q1. "What tree does a time-series database use, and why not a B-tree?" Most use a log-structured, time-partitioned design: InfluxDB's TSM tree, Prometheus' head + blocks, MergeTree in ClickHouse and VictoriaMetrics. Time-series ingestion is append-mostly across many series at once. A B-tree keyed by (series, time) turns that into random leaf writes and page splits once the hot leaves exceed RAM. A B-tree keyed by time makes per-series range reads scattered. LSM designs buffer in memory, sort by (series, time) there, and flush sequentially. Time partitioning makes retention a file delete. TimescaleDB uses B-trees, but only on small time chunks whose indexes fit in memory.

Q2. "Explain Gorilla compression." Two streams per series per block. Timestamps are stored as delta-of-delta: regular intervals give D = 0, which is 1 bit, and 96% of Facebook's timestamps were 1 bit. Values are XORed with the previous value. An identical value is 1 bit. Otherwise you store only the meaningful middle bits, either reusing the previous leading/trailing-zero window (10 + bits) or with a new 5-bit lead + 6-bit length header (11). Result: ~1.37 bytes/point vs 16 bytes raw, about 12Γ—.

Q3. "Your Prometheus is OOM-killed after a deploy. What happened and how do you fix it?" Probably a cardinality or churn spike: a new label with unbounded values (user ID, request path, pod hash) or a rollout that created thousands of new series. Each active series costs a few KB in the head. Check prometheus_tsdb_head_series, run promtool tsdb analyze, look at topk(10, count by (__name__)({__name__=~".+"})). Fix with metric_relabel_configs to drop or replace the label, add sample_limit per target, and for scale, shard or move to Mimir or VictoriaMetrics.

Q4. "How do you delete 30-day-old data at 1M points/s without hurting writes?" Never row deletes. Partition by time and drop whole partitions: Timescale drop_chunks, Prometheus block deletion, Influx shard expiry, ClickHouse DROP PARTITION / TTL. It is O(1) in rows and costs no compaction or vacuum.

Q5. "Why did InfluxDB 3.0 move to Arrow/Parquet/DataFusion?" TSM's per-series index (TSI) and per-series blocks made high cardinality expensive in RAM, startup time and compression. TSM files were also proprietary and tied to local disk. Columnar Parquet sorted by tags then time removes the per-series index, so cardinality becomes a scan-cost problem rather than a memory problem. It enables object storage and stateless compute, and lets them reuse a community query engine (DataFusion) and a format other tools can read.

Q6. "Design storage for 1M IoT devices reporting every 10 s, with offline gateways." That is 100k points/s. Edge: buffer to local storage (SQLite WAL or a ring buffer), publish batched over MQTT QoS 1 with persistent sessions, advance the cursor on PUBACK. Ingest: MQTT broker β†’ Kafka β†’ idempotent writer (dedupe on device_id + seq). Storage: TimescaleDB (if you need relational joins) or ClickHouse/IoTDB, partitioned by day, columnar compression after 1–7 days, continuous aggregates at 5 min and 1 h, raw kept 30 days, rollups kept for years on object storage. The engine must accept out-of-order writes from gateways backfilling outages.

Q7. "How does Prometheus find all series matching {job="api", status=~"5.."}?" Through postings lists, an inverted index from label=value to a sorted list of series IDs. It unions the postings of every status value matching the regex, intersects with job=api and the metric name by merging the sorted lists, then reads only the chunks of the matching series whose time range overlaps the query.

Q8. "Why might a flash-based edge device die after a few months of logging, and how do you prevent it?" Write amplification stacks up: small synchronous writes, then filesystem journaling, then database journaling and checkpoints, then the cheap FTL's erase-block read-modify-write. A TLC card with ~1,000 P/E cycles burns through its lifetime writes in weeks. Prevent it by batching writes (one transaction per minute), using a sequential ring buffer or FIFO-compacted LSM, using industrial pSLC/SLC media, mounting with noatime, keeping temp data in tmpfs, and downsampling on-device.

Q9. "TimescaleDB uses B-trees. Doesn't that contradict everything you just said?" No. The B-tree problem is a large index with random inserts that don't fit in memory. Hypertables keep each chunk's index small, sized so the hot chunk fits in ~25% of RAM, so inserts hit cached pages. Cold chunks are converted to columnar compressed batches anyway. The partitioning does the job that the LSM's memtable does elsewhere.

Q10. "How do you downsample without losing correctness?" Store mergeable aggregates (sum, count, min, max) rather than averages. Use mergeable sketches (t-digest, DDSketch, native histograms) for percentiles. Use increase / last for counters. Re-process a trailing window to absorb late data.

Further Reading

  • Storage Engines β€” B-trees, LSM trees, compaction and amplification, the foundation for every engine here
  • Time-Series Databases β€” product-level comparison of TimescaleDB, InfluxDB and QuestDB, data modelling and tag design
  • ClickHouse Internals β€” MergeTree parts, sparse primary index, codecs and TTL
  • Write-Ahead Logging β€” the durability mechanism every engine above relies on
  • SQLite Internals β€” the most common edge database, its B-tree pages and WAL mode
  • Cassandra Internals β€” a general-purpose LSM often used (with TWCS) for time series
  • MQTT for IoT β€” QoS, persistent sessions and brokers for store-and-forward
  • Prometheus Deep Dive and PromQL Cheat Sheet β€” operating the engine described above
  • Metrics Design β€” label design that avoids cardinality explosions
  • Spatial Indexing Deep Dive β€” R-trees, geohash, S2 and H3 in depth
  • ClickHouse vs Druid vs Pinot β€” real-time analytical stores that also serve time series
  • Project: IoT Sensor Pipeline β€” an end-to-end implementation
  • Pelkonen et al. (2015): "Gorilla: A Fast, Scalable, In-Memory Time Series Database", VLDB β€” the compression scheme
  • Fabian Reinartz (2017): "Writing a Time Series Database from Scratch" β€” the design of Prometheus TSDB v3
  • InfluxData (2016): "The New InfluxDB Storage Engine: Time Structured Merge Tree" and (2023) "FDAP: Flight, DataFusion, Arrow, Parquet"
  • Liakos et al. (2022): "Chimp: Efficient Lossless Floating Point Compression for Time Series Databases", VLDB
  • Afroozeh et al. (2024): "ALP: Adaptive Lossless floating-Point Compression", SIGMOD