Is Promtail still the right agent to ship logs to Loki?

No. Promtail reached end of life on March 2, 2026 — commercial support ended and Grafana Labs has stated all future development happens in Grafana Alloy instead. Alloy ships a migration tool that converts an existing Promtail config in one command, so a Promtail setup built before 2026 should be migrated, not extended.

Why is a request ID a bad label in Loki, when it's a perfectly normal field to log?

Loki indexes label sets, not log text — every distinct combination of label values creates its own stream, and Loki's ingesters cap how many streams a tenant can have open at once (the real default is 5000, this lesson's demo config lowers it to 10 to hit it fast). A request ID is different on every line, so promoting it to a label creates one new stream per request. This lesson's run hit that exact limit after only 8 of 20 new request IDs got a stream — confirmed via Loki's own loki_discarded_samples_total metric and Alloy's loki_write_batch_retries_total, both of which keep climbing as long as the bad label stays in place.

What's the difference between a Loki label selector and a line filter like |=?

A label selector such as {service="orders", level="ERROR"} picks which indexed streams to read — it's effectively free, because Loki already knows which streams match without scanning any log text. A line filter like |= "ERROR" runs after that, scanning the actual text of every line in the already-selected streams. Selecting on a label that already exists is always cheaper than filtering for the same thing in the text, because it skips the scan entirely.

What does the | json stage do in a LogQL query, and why did fields come back named level_extracted and service_extracted?

| json parses the current log line as JSON and makes every top-level field available for further filtering, without those fields becoming indexed labels. When a parsed field's name collides with a label that already exists on the stream — this lesson's level and service are both real labels — Loki suffixes the parsed version with _extracted instead of silently overwriting the label's value, so both the original label and the freshly-parsed field stay visible in the result.

How does Loki compare to the ELK stack for log aggregation?

Elasticsearch full-text-indexes every field of every log line by default, which makes arbitrary ad-hoc search fast but costs far more storage and indexing CPU at scale. Loki indexes only the label set and stores the log text itself as compressed, unindexed chunks, which is why LogQL's line filters (|=, |~) have to scan text at query time instead of hitting an index — cheaper to run, slower on a query that can't be expressed as a label selector. The practical tradeoff: Loki favors cases where you already know which service, level, or environment you're looking at (the common case for anyone coming from Prometheus's label model); ELK favors cases where the search itself is the unknown.

Loki and Alloy: a High-Cardinality Label Is a Real Error, Not a Style Nit

Loki does not index the text of a log line. It indexes the label set a line arrives with — and the number of distinct label combinations a reader picks has a real, enforced ceiling.

A Compose stack: Loki and Grafana Alloy

docker-compose.yml runs two containers on one network: Loki itself, and Grafana Alloy, the agent that tails a log file and pushes to it. Alloy’s own data — including where it has read up to in each tailed file — is persisted to a mounted volume via --storage.path, which matters later in this lesson.

services:
  loki:
    image: grafana/loki:3.7.9
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml

  alloy:
    image: grafana/alloy:v1.21.2
    ports:
      - "12345:12345"
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy
      - ./logs:/var/log/app
      - ./alloy-storage:/var/lib/alloy/data
    command: run --storage.path=/var/lib/alloy/data --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
    depends_on:
      - loki

loki-config.yaml runs Loki as a single binary with local filesystem storage — one setting in it matters for this lesson and is called out below:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  allow_structured_metadata: true
  max_global_streams_per_user: 10   # real default is 5000 -- lowered so this lesson's bug is fast to hit
  reject_old_samples: true
  reject_old_samples_max_age: 168h

Alloy’s loki.source.file target has to exist as a real file before Alloy starts, or it fails to find it and never retries — so the log file is created, empty, before up:

$ mkdir -p logs && touch logs/orders.log
$ docker compose up -d 2>&1 | grep -E "Started|Running"
 Container loki-clean-loki-1 Started
 Container loki-clean-alloy-1 Started
$ curl -s http://localhost:3100/ready
ready
$ curl -s http://localhost:3100/loki/api/v1/status/buildinfo
{"version":"3.7.9","revision":"d3b4a2f7","branch":"release-3.7.x","buildUser":"root@buildkitsandbox","buildDate":"2026-10-20T21:43:00Z","goVersion":""}
$ docker exec loki-clean-alloy-1 alloy --version
alloy, version v1.21.2 (branch: HEAD, revision: a69b1ae)
  build user:       root@buildkitsandbox
  build date:       2026-10-22T21:20:03Z
  go version:       go1.27.3
  platform:         linux/amd64
  tags:             gore2regex,netgo,embedalloyui,promtail_journal_enabled

Alloy, not Promtail — Promtail reached end of life on March 2, 2026 (commercial support ended, no further releases), and Grafana Labs’ own guidance is that all new development happens in Alloy. A Promtail config from before that date migrates with a tool Alloy ships for exactly this; this lesson starts from Alloy directly. Every image tag above is pinned; the version strings printed are what this lesson’s output actually came from.

A JSON log line, straight from the app

app.py is a plain Python script with no logging framework — it writes one JSON object per line to a file, which is the only contract Alloy needs on the other end.

# app.py
import json, random, sys, time, uuid

ROUTES = ["/orders", "/orders/checkout", "/orders/cancel"]


def make_line():
    status = 500 if random.random() < 0.15 else 200
    return {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "level": "ERROR" if status == 500 else "INFO",
        "service": "orders",
        "path": random.choice(ROUTES),
        "status": status,
        "request_id": str(uuid.uuid4()),
        "msg": "request failed" if status == 500 else "request handled",
    }


if __name__ == "__main__":
    out_path = sys.argv[1] if len(sys.argv) > 1 else "logs/orders.log"
    count = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    with open(out_path, "a", newline="\n") as f:
        for _ in range(count):
            f.write(json.dumps(make_line()) + "\n")
            f.flush()
            time.sleep(0.05)

$ python app.py logs/orders.log 40
$ head -n 3 logs/orders.log
{"ts": "2026-10-27T08:55:00Z", "level": "INFO", "service": "orders", "path": "/orders/cancel", "status": 200, "request_id": "8b5b67e6-2b0c-498a-b107-bb511b7fde06", "msg": "request handled"}
{"ts": "2026-10-27T08:55:00Z", "level": "INFO", "service": "orders", "path": "/orders/cancel", "status": 200, "request_id": "964ac144-e350-4e74-8bda-6b8408930ee3", "msg": "request handled"}
{"ts": "2026-10-27T08:55:00Z", "level": "INFO", "service": "orders", "path": "/orders/cancel", "status": 200, "request_id": "11433af5-3cbc-4cde-abcb-ea6bd3c80c74", "msg": "request handled"}

~15% of requests come back as a 500, logged at ERROR, so every run produces a real mix of both levels without anything being staged.

Alloy tails the file and extracts two labels

alloy-config.alloy reads /var/log/app/orders.log, parses each line as JSON, and promotes exactly two of its fields to labels — level and service. Alloy’s configuration language comments with //, not # — a # header here is a parse error, not a comment.

// alloy-config.alloy (good: only level and service promoted to labels)
loki.source.file "orders" {
  targets = [
    {__path__ = "/var/log/app/orders.log"},
  ]
  forward_to = [loki.process.orders.receiver]
}

loki.process "orders" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      level   = "level",
      service = "service",
    }
  }

  stage.labels {
    values = {
      level   = "level",
      service = "service",
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Checking what actually landed, through Loki’s own series and labels APIs rather than trusting the config did what it says:

$ curl -s --data-urlencode 'match[]={service="orders"}' http://localhost:3100/loki/api/v1/series
{"status":"success","data":[{"filename":"/var/log/app/orders.log","level":"ERROR","service":"orders","service_name":"orders"},{"filename":"/var/log/app/orders.log","level":"INFO","service":"orders","service_name":"orders"}]}
$ curl -s http://localhost:3100/loki/api/v1/labels
{"status":"success","data":["filename","level","service","service_name"]}

Two streams — one per level value — for all 40 lines, and exactly four real labels: level and service came from this lesson’s own stage.labels block; the other two did not. filename is set by loki.source.file itself, from the target’s __path__, with no config on this lesson’s side asking for it. service_name is added by Loki, not Alloy — Loki’s discover_service_name feature copies the first non-empty value it finds from a fixed priority list (service, app, application, app_name, …), and this app’s service label is first on that list, so it got copied verbatim.

Label selectors and line filters

A label selector picks streams; everything after it scans the text of whatever lines those streams already matched. start/end are computed from the current time in the command itself, so this works whenever it’s actually run:

$ curl -s --data-urlencode 'query={service="orders"}' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=2" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "values": [["1793091302944137850", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"90f0686c-a964-4295-af16-2094aa1e97fd\", \"msg\": \"request failed\"}"]]}, {"stream": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "values": [["1793091302944129985", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"INFO\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 200, \"request_id\": \"798f1eab-f0e3-4a8a-a288-eb4d8c771aba\", \"msg\": \"request handled\"}"]]}]

detected_level showed up in that result’s stream object ("info"/"error", lowercased) even though it isn’t in the /labels list above — it’s structured metadata Loki computes by scanning each line’s content, attached to query results for convenience, but it never becomes part of the indexed label set and never counts toward stream cardinality.

|= "ERROR" is a line filter — it runs after the label selector, checking the raw text of every line {service="orders"} already matched:

$ curl -s --data-urlencode 'query={service="orders"} |= "ERROR"' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=2" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "values": [["1793091302944137850", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"90f0686c-a964-4295-af16-2094aa1e97fd\", \"msg\": \"request failed\"}"], ["1793091302944106281", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders/cancel\", \"status\": 500, \"request_id\": \"e38ebd8e-1d98-4c09-aae1-610cf5d6b41a\", \"msg\": \"request failed\"}"]]}]

!= excludes lines whose text matches — != "checkout" drops every line mentioning that path, keeping only /orders and /orders/cancel:

$ curl -s --data-urlencode 'query={service="orders"} != "checkout"' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=2" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "values": [["1793091302944137850", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"90f0686c-a964-4295-af16-2094aa1e97fd\", \"msg\": \"request failed\"}"]]}, {"stream": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "values": [["1793091302944129985", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"INFO\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 200, \"request_id\": \"798f1eab-f0e3-4a8a-a288-eb4d8c771aba\", \"msg\": \"request handled\"}"]]}]

Both returned lines’ path field is /orders — neither mentions checkout anywhere in the line, confirming != matched on the whole line’s text, not just the path field specifically.

|~ runs a regex instead of a literal match — "cancel|checkout" matches either alternative:

$ curl -s --data-urlencode 'query={service="orders"} |~ "cancel|checkout"' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=2" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "values": [["1793091302944106281", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders/cancel\", \"status\": 500, \"request_id\": \"e38ebd8e-1d98-4c09-aae1-610cf5d6b41a\", \"msg\": \"request failed\"}"]]}, {"stream": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "values": [["1793091302689564019", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"INFO\", \"service\": \"orders\", \"path\": \"/orders/checkout\", \"status\": 200, \"request_id\": \"9fa54fcf-056e-4dd3-8fe0-d50914aad2b6\", \"msg\": \"request handled\"}"]]}]

Parsing JSON and filtering on a parsed field

path and status were never promoted to labels — they’re still just text inside each line. | json parses that text and makes every field available to filter on, without any of them becoming part of the index:

$ curl -s --data-urlencode 'query={service="orders"} | json | status >= 500' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=1" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "level_extracted": "ERROR", "msg": "request failed", "path": "/orders", "request_id": "90f0686c-a964-4295-af16-2094aa1e97fd", "service": "orders", "service_extracted": "orders", "service_name": "orders", "status": "500", "ts": "2026-10-27T08:55:02Z"}, "values": [["1793091302944137850", "{\"ts\": \"2026-10-27T08:55:02Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"90f0686c-a964-4295-af16-2094aa1e97fd\", \"msg\": \"request failed\"}"]]}]

Every field from the JSON body shows up once | json has parsed it — but level and service came back as level_extracted and service_extracted, not plain level and service. That’s not a bug: level and service are already real labels on this stream, and | json refuses to silently overwrite a label’s value with a parsed one of the same name — it suffixes the parsed version with _extracted instead, so both the original label and the newly-parsed field survive in the result. path, status and request_id have no such collision, so they keep their plain names. status >= 500 runs as a numeric comparison on a never-indexed field precisely because | json parsed it fresh for this one query.

From log lines to a number

The same label selector, wrapped in a range-vector function, turns into a metric query instead of a log query:

$ curl -s --data-urlencode 'query=count_over_time({service="orders"}[5m])' http://localhost:3100/loki/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "value": [1793091344.683, "8"]}, {"metric": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "value": [1793091344.683, "32"]}]

8 ERROR lines, 32 INFO — one count per stream, which is exactly the 40-line run from earlier split by level. Wrapping a line filter the same way turns it into a rate:

$ curl -s --data-urlencode 'query=rate({service="orders"} |= "ERROR" [5m])' http://localhost:3100/loki/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "value": [1793091344.834, "0.02666666666666667"]}]

And sum by collapses the per-stream labels Loki keeps (filename, service_name, detected_level) down to just the one that matters for this question:

$ curl -s --data-urlencode 'query=sum by (level) (count_over_time({service="orders"}[5m]))' http://localhost:3100/loki/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"level": "ERROR"}, "value": [1793091344.994, "8"]}, {"metric": {"level": "INFO"}, "value": [1793091344.994, "32"]}]

Same 8 and 32, now carrying only level — sum by (level) added nothing here because there was only one stream per level already, but it’s the same aggregation a real multi-instance deployment needs once several pods all log with the same service label.

The bug: promoting request_id to a label

The request ID looks like a perfectly reasonable thing to want as a label — until it’s one of a kind on every single line. Before changing anything, the file has exactly the 40 lines written above and the usual 2 streams:

$ wc -l logs/orders.log
40 logs/orders.log
$ wc -c logs/orders.log
7499 logs/orders.log
$ curl -s --data-urlencode 'match[]={service="orders"}' http://localhost:3100/loki/api/v1/series | python -c "import json,sys; d=json.load(sys.stdin); print(len(d['data']))"
2

Changing only stage.json and stage.labels to include request_id, then recreating the Alloy container so the new config takes effect:

// alloy-config.alloy (bad: request_id promoted to a label too)
loki.source.file "orders" {
  targets = [
    {__path__ = "/var/log/app/orders.log"},
  ]
  forward_to = [loki.process.orders.receiver]
}

loki.process "orders" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      level      = "level",
      service    = "service",
      request_id = "request_id",
    }
  }

  stage.labels {
    values = {
      level      = "level",
      service    = "service",
      request_id = "request_id",
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

$ docker compose up -d --force-recreate alloy 2>&1 | grep -E "Recreated|Started"
 Container loki-clean-alloy-1 Recreated
 Container loki-clean-alloy-1 Started

A config change means recreating the container, which would normally mean Alloy forgets how far into orders.log it had already read — and re-reads the whole file from byte 0, re-ingesting all 40 existing lines under the new, bad label config. That’s why docker-compose.yml mounts ./alloy-storage to --storage.path: Alloy’s file-position tracking persists there, independent of the container, and survives a recreate. Confirmed directly, not assumed — right after the restart, the saved position already matches the file’s exact byte size, meaning nothing gets replayed:

$ cat alloy-storage/loki.source.file.orders/positions.yml
positions:
  ? path: /var/log/app/orders.log
    labels: '{}'
  : "7499"
$ wc -c logs/orders.log
7499 logs/orders.log

With the position confirmed intact, 20 genuinely new lines — each with its own unique request_id — are the only thing this next step adds:

$ python app.py logs/orders.log 20
$ curl -s --data-urlencode 'match[]={service="orders"}' http://localhost:3100/loki/api/v1/series | python -c "import json,sys; d=json.load(sys.stdin); print(len(d['data']))"
10

20 new requests, each with a unique request_id, should mean up to 20 new streams on top of the 2 already there. The series count stopped at exactly 10 — this lesson’s max_global_streams_per_user limit — not 22. Alloy’s own loki.write metrics show the rejection directly, and so does Loki’s (anchoring the grep to the start of the line skips the # HELP/# TYPE comment lines the plain metric name would otherwise match):

$ curl -s http://localhost:12345/metrics | grep '^loki_write_batch_retries_total'
loki_write_batch_retries_total{component_id="loki.write.default",component_path="/",host="loki:3100"} 4
$ curl -s http://localhost:3100/metrics | grep '^loki_discarded_samples_total'
loki_discarded_samples_total{format="",policy="",reason="stream_limit",retention_hours="0",tenant="fake"} 60

Alloy keeps retrying a batch Loki has already refused — that counter keeps climbing the longer the bad config stays running, with no new traffic at all; checking again a minute later with nothing else changed shows 84, not 60. Only 8 of the 20 new request_id values ever got a stream before the cap stopped accepting new ones — confirmed directly:

$ curl -s "http://localhost:3100/loki/api/v1/label/request_id/values" | python -c "import json,sys; d=json.load(sys.stdin); print(len(d['data']))"
8

2 pre-existing streams (level×service) plus 8 new per-request streams lands on exactly 10 — the configured ceiling. No amount of retrying creates a stream Loki has already refused.

The fix: proven on a fresh Loki, not at the cap

before: 10 and after: 10 wouldn’t prove anything here — the limit is already saturated by the streams the bug above just created, so a flat number is expected either way. Proving the fix for real means starting completely over — new containers, the log file and Alloy’s saved position both cleared, so the only way orders.log gets any content again is the command that’s about to run:

$ docker compose down -v 2>&1 | grep -E "Removed"
 Container loki-clean-alloy-1 Removed
 Container loki-clean-loki-1 Removed
 Network loki-clean_default Removed
$ rm -rf alloy-storage logs/orders.log
$ mkdir -p logs && touch logs/orders.log
$ docker compose up -d 2>&1 | grep -E "Started|Running"
 Container loki-clean-loki-1 Started
 Container loki-clean-alloy-1 Started

alloy-config.alloy is back to just level and service — the file from the “Alloy tails the file” section above, unchanged. 150 lines through it, on a Loki that has never seen a request_id label at all:

$ python app.py logs/orders.log 150
$ curl -s --data-urlencode 'match[]={service="orders"}' http://localhost:3100/loki/api/v1/series | python -c "import json,sys; d=json.load(sys.stdin); print(len(d['data']))"
2
$ curl -s http://localhost:3100/metrics | grep -c '^loki_discarded_samples_total'
0
$ curl -s "http://localhost:3100/loki/api/v1/label/request_id/values"
{"status":"success"}

Still 2 streams for 150 lines. loki_discarded_samples_total doesn’t exist at all — Prometheus-style counters aren’t created until the first increment, so its total absence is itself proof nothing was ever discarded. And the request_id label query comes back with no "data" field whatsoever: Loki has never heard of a label by that name on this fresh instance, because nothing in this config ever asked for one.

Loki vs the ELK stack

Elasticsearch (the “E” in ELK) full-text-indexes every field of every document by default, which is why Kibana can search almost anything about a log line without knowing in advance what you’ll search for — at the cost of indexing CPU and storage that scale with every field you ingest, whether you ever query it or not. Loki inverts that trade: it indexes only the label set, stores the log text as compressed, unindexed chunks, and makes a line filter like |= "ERROR" do a real text scan at query time instead of an index lookup. That scan is Loki’s accepted cost for not paying Elasticsearch’s indexing cost on fields nobody labeled. The choice between them tracks how predictable your queries are: Loki rewards knowing the label (service, level, environment) you’re after before you query; Elasticsearch rewards not having to know anything in advance.

Practice

1. Write a burst, wait over a minute, write a second burst, then compare count_over_time over a 1-minute window against a 5-minute window.

Two 20-line bursts 81 real seconds apart (2026-10-27T08:58:32Z and 2026-10-27T08:59:53Z), on top of the 150 lines this Loki instance had already ingested from the fix section above:

$ python app.py logs/orders.log 20
$ curl -s --data-urlencode 'query=count_over_time({service="orders"}[1m])' http://localhost:3100/loki/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "value": [1793091599.715, "1"]}, {"metric": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "value": [1793091599.715, "19"]}]
$ curl -s --data-urlencode 'query=count_over_time({service="orders"}[5m])' http://localhost:3100/loki/api/v1/query | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"metric": {"detected_level": "error", "filename": "/var/log/app/orders.log", "level": "ERROR", "service": "orders", "service_name": "orders"}, "value": [1793091599.864, "24"]}, {"metric": {"detected_level": "info", "filename": "/var/log/app/orders.log", "level": "INFO", "service": "orders", "service_name": "orders"}, "value": [1793091599.864, "166"]}]

The 1-minute window counts 1 + 19 = 20 — exactly the second burst, nothing older. The 5-minute window counts 24 + 166 = 190 — every line this Loki instance has ingested so far, both bursts plus everything before them. Same metric, same instant, two different answers, because count_over_time’s window decides how far back it’s willing to look.

2. Instead of request_id, promote path to a label (3 possible values) alongside level and service, and compare the resulting stream count to the request_id case.

The target file also changes, to a fresh one, so this starts from zero streams on a fresh stack rather than adding a third label onto the existing orders.log history:

// alloy-config.alloy (path promoted, not request_id)
loki.source.file "orders" {
  targets = [
    {__path__ = "/var/log/app/orders-path.log"},
  ]
  forward_to = [loki.process.orders.receiver]
}

loki.process "orders" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      level   = "level",
      service = "service",
      path    = "path",
    }
  }

  stage.labels {
    values = {
      level   = "level",
      service = "service",
      path    = "path",
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

$ docker compose down -v 2>&1 | grep -E "Removed"
 Container loki-clean-alloy-1 Removed
 Container loki-clean-loki-1 Removed
 Network loki-clean_default Removed
$ rm -rf alloy-storage logs/orders-path.log
$ mkdir -p logs && touch logs/orders-path.log
$ docker compose up -d 2>&1 | grep -E "Started|Running"
 Container loki-clean-loki-1 Started
 Container loki-clean-alloy-1 Started
$ python app.py logs/orders-path.log 60
$ curl -s --data-urlencode 'match[]={service="orders"}' http://localhost:3100/loki/api/v1/series | python -c "import json,sys; d=json.load(sys.stdin); print(len(d['data']))"
6

6 streams for 60 lines — 3 path values times 2 level values, and it stays at 6 no matter how much more volume is sent, because path only ever takes one of 3 fixed values. That’s the actual dividing line this lesson’s bug crossed: a label is safe when its number of distinct values is small and bounded, not because of what the field is named. request_id has no such bound — every single line introduces a brand new value.

3. On this path-labeled stack, query {service="orders", level="ERROR"} directly (a label selector) instead of {service="orders"} |= "ERROR" (a line filter), and confirm they return the same line.

$ curl -s --data-urlencode 'query={service="orders", level="ERROR"}' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=1" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders-path.log", "level": "ERROR", "path": "/orders", "service": "orders", "service_name": "orders"}, "values": [["1793091651443078725", "{\"ts\": \"2026-10-27T09:00:51Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"e28051e2-f88b-42fc-895c-9900d739a134\", \"msg\": \"request failed\"}"]]}]
$ curl -s --data-urlencode 'query={service="orders"} |= "ERROR"' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" --data-urlencode "limit=1" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; print(json.dumps(json.load(sys.stdin)['data']['result']))"
[{"stream": {"detected_level": "error", "filename": "/var/log/app/orders-path.log", "level": "ERROR", "path": "/orders", "service": "orders", "service_name": "orders"}, "values": [["1793091651443078725", "{\"ts\": \"2026-10-27T09:00:51Z\", \"level\": \"ERROR\", \"service\": \"orders\", \"path\": \"/orders\", \"status\": 500, \"request_id\": \"e28051e2-f88b-42fc-895c-9900d739a134\", \"msg\": \"request failed\"}"]]}]

Identical line both ways, now with path along for the ride as a third real label. The results match because level is already a label — but the selector version never has to scan any log text to get there, since Loki already knows exactly which stream level="ERROR" is.

4. Try using detected_level as a label selector, the same way level was used all through this lesson, and see what happens.

$ curl -s --data-urlencode 'query={service="orders", detected_level="error"}' --data-urlencode "start=$(( $(date +%s) - 120 ))000000000" --data-urlencode "end=$(date +%s)000000000" http://localhost:3100/loki/api/v1/query_range | python -c "import json,sys; d=json.load(sys.stdin); print(json.dumps({'status':d['status'],'resultType':d['data']['resultType'],'result':d['data']['result']}))"
{"status": "success", "resultType": "streams", "result": []}

No error — just an empty result, "status": "success" included. Loki doesn’t reject a selector naming a label that was never indexed; it just can’t match anything against a label no stream actually carries, the same way a Prometheus query for a metric that was never scraped comes back empty rather than invalid. detected_level only ever appears inside a query result’s per-entry grouping, never in /labels, and a selector built on it behaves exactly like a selector built on a label name that was simply never used.

Programming

Web Development

Generative AI

Machine Learning

Cybersecurity

Data Engineering

Cloud

DevOps

System Design

GPU & Parallel

Databases

Testing

Interviews