Tiered Storage

Tiered Storage

Long retention used to mean big broker disks, which every replacement broker must copy. Tiered storage (KIP-405, early access in Kafka 3.6 129 , production-ready since 3.9) splits a topic's log in two. Recent segments stay on local disk; once a segment is closed, the broker's remote log manager copies it to a remote tier such as S3, records the copy in the internal __remote_log_metadata topic, and deletes the local file when local.retention.ms or local.retention.bytes is exceeded. retention.ms still decides when data disappears altogether. Consumers notice nothing except latency: a fetch for an old offset is served from the remote tier.

A tiered topic: the active and newest segments on the broker, the history in object storage

Kafka defines the plug-in interfaces but ships no production storage plug-in, only a test one. BookNest uses Aiven's open-source tiered storage plug-in (https://github.com/Aiven-Open/tiered-storage-for-apache-kafka 241 ) (Apache 2.0, v1.1.1), whose S3, Google Cloud 1 Storage and Azure 6 backends chunk, compress and optionally encrypt segments; here its filesystem backend, a directory mounted at /remote, stands in for a bucket:

compose/tiered.yaml (excerpt): enabling tiered storage on the brokerYAML

      KAFKA_REMOTE_LOG_STORAGE_SYSTEM_ENABLE: "true"
      KAFKA_REMOTE_LOG_METADATA_MANAGER_LISTENER_NAME: DOCKER
      KAFKA_REMOTE_LOG_STORAGE_MANAGER_CLASS_NAME: io.aiven.kafka.tieredstorage.RemoteStorageManager
      KAFKA_REMOTE_LOG_STORAGE_MANAGER_CLASS_PATH: /opt/tiered/core/*:/opt/tiered/filesystem/*
      KAFKA_RSM_CONFIG_STORAGE_BACKEND_CLASS: >-
        io.aiven.kafka.tieredstorage.storage.filesystem.FileSystemStorage
      KAFKA_RSM_CONFIG_STORAGE_ROOT: /remote

A topic opts in with remote.storage.enable=true. l0615_tiered.sh creates booknest.order-history with 1 MB segments and 2 MB of local retention per partition, loads all 390,737 events, waits 40 seconds, and looks:

Output of 89

earliest       booknest.order-history:0:0
earliest-local booknest.order-history:0:110168
latest         booknest.order-history:0:133560
local:  9 segments, 7.1M
remote: 38 segments, 38M
{"event_id":1,"ts":"2025-01-01T00:02:38Z","type":"order_placed","order_id":1,"customer_id":2518
  ,"total":40.2}
{"event_id":2,"ts":"2025-01-01T00:04:32Z","type":"order_placed","order_id":2,"customer_id":1463
  ,"total":106.8}

Partition 0 still starts at offset 0, but its first local offset is 110,168: everything older lives only in the remote tier. Three segments per partition (7.1 MB) remain local, while 38 segments, the newest of them still also on disk, sit in the remote tier, each as a log file, an indexes file and a manifest. The console consumer read offset 0 from the remote tier without knowing it. Compacted topics cannot be tiered, and turning tiering off for a topic needs remote.log.delete.on.disable=true. MSK (MSK Provisioned Clusters), Aiven and Confluent Cloud 17,245 sell tiered storage as a managed feature.

MENU