Notes

References:

• https://grafana.com/docs/loki/latest/get-started/architecture/
• https://grafana.com/docs/loki/latest/get-started/components/
• https://grafana.com/docs/loki/latest/get-started/deployment-modes

Basics

• Grafana Loki is a horizontally scalable, multi-tenant log aggregation system inspired by Prometheus. Unlike traditional logging systems, Loki only indexes metadata (labels) rather than the full log content, making it significantly more cost-effective and easier to operate. The actual log content is stored in compressed chunks in an object store (S3, GCS, Azure Blob, etc.).
• All data is stored in a single object storage backend (S3, GCS, Azure Blob, etc.), which became the standard with Loki 2.0. This replaces the older model of separate backends for indexes and chunks.
• Loki has two main data file types:
  • Index — a table of contents that maps a set of labels to where logs live. The recommended index format is TSDB (Time Series Database), originally from Prometheus. The older BoltDB format is now deprecated.
  • Chunks — compressed containers of log lines for a specific label set and time range. Each chunk is unique per tenant and per label set.
• A stream is the core unit in Loki — it's a set of logs that share the same tenant ID and unique label set. All log lines in a stream must arrive in timestamp-ascending order (though out-of-order writes can be enabled via config).
• Loki uses LogQL as its query language, similar to PromQL. LogQL supports both log queries (filtering and viewing raw log lines) and metric queries (computing metrics like rate, count_over_time from logs).
• Multi-tenancy is supported out of the box. When multi-tenant mode is enabled, all data is partitioned by tenant ID pulled from the X-Scope-OrgID HTTP header. When disabled, the tenant ID defaults to fake.
• Loki uses a consistent hash ring (via Memberlist or a KV store like Consul) to distribute work across ingesters. All ingesters register themselves with a set of tokens in this ring, and distributors use those tokens to determine which ingesters should receive a given stream.
• The replication factor (typically 3) controls how many ingester instances receive a copy of each stream. A quorum of floor(replication_factor / 2) + 1 acknowledgements must be received before a write is considered successful. This means for replication_factor = 3, two writes must succeed.
• Loki's code compiles all microservice components into a single binary. The target flag controls which component(s) run at startup, enabling flexible deployment from single-process all the way to fully distributed microservices.

Architecture