Log Ingestion Costs: Why Your Observability Bill Outgrew Your Cluster

Walk into almost any cloud-native cost review and you'll find the same surprise sitting near the top of the bill: log ingestion. It rarely starts there. A small cluster generates a manageable trickle of logs, the observability bill is a rounding error, and nobody thinks about it. Then services multiply, replicas scale, sidecars proliferate, and somewhere along the way the logging line item quietly outgrew the compute it was meant to observe.

This is one of the most common patterns in cloud cost — and, fortunately, one of the most fixable. The same dynamics show up regardless of provider, because nearly every managed logging platform bills on the same axis: volume ingested. The product names differ across GCP, AWS, and Azure; the cost model and the fix do not.

Why Ingestion Volume Creeps Up

Log costs grow through accumulation, not through any single bad decision.

Verbosity left on. Debug logging is invaluable during development and a disaster in production. A service left at DEBUG can emit 5-10x the volume of the same service at INFO, and almost none of those extra lines are ever read.

Multiplication by scale. Logging is per-replica. A chatty service that logs a few lines per request becomes a firehose when it scales to 40 pods under load. The per-pod volume looks fine; the aggregate does not.

High-frequency, low-value events. Health-check pings, readiness probes, successful access logs, and verbose framework startup chatter often make up the majority of log lines and almost none of the value. You pay to ingest, index, and store events you will never query.

Sidecars and the platform itself. Service mesh sidecars, ingress controllers, and platform components each contribute their own streams. It is common for infrastructure logs to rival application logs in volume.

Retention as a hidden multiplier. Ingestion is the headline cost, but keeping 90 days of fully indexed, searchable logs when investigations almost never reach past two weeks multiplies storage cost on top of it.

Because each contribution is individually small, log waste evades the usual cost reviews. The fix is to stop looking at the total and start looking at the sources.

Find the Noisy Sources

Before cutting anything, find out where the volume comes from. Most logging platforms can break ingestion down by source — namespace, workload, log name, or severity. Sort by volume and the picture is usually stark: a handful of sources account for the bulk of ingestion.

From the cluster side, you can spot the obvious offenders quickly. Check which workloads are emitting the most lines:

bash
# Rough per-pod log line counts over a short window
for p in $(kubectl get pods -n production -o name); do
  count=$(kubectl logs "$p" -n production --since=5m 2>/dev/null | wc -l)
  echo "$count  $p"
done | sort -rn | head

The workloads at the top of that list are where your ingestion budget is going. Combine that with a severity breakdown from the logging platform — a service whose stream is 95% INFO and DEBUG is a prime candidate for turning down.

Cut the Bill Without Losing What Matters

The goal is not "fewer logs." It is "stop paying to ingest, index, and store logs nobody reads," while keeping full fidelity on the logs that actually drive incident response and compliance.

1. Turn down production verbosity at the source

The single highest-leverage change. Because billing happens at ingestion, dropping the log level stops the spend before the data is ever sent:

bash
kubectl set env deployment/api LOG_LEVEL=info -n production

Do this across the noisiest workloads first and re-measure. This change alone frequently cuts ingestion by a third or more.

2. Filter and sample before ingestion

Most logging agents (Fluent Bit, the OpenTelemetry Collector, and provider agents) support dropping or sampling records before they leave the node. Drop health checks and successful probe logs outright; sample high-volume successful access logs at, say, 10%; keep 100% of errors and warnings. A Fluent Bit filter to drop health-check noise looks like:

ini
[FILTER]
    Name    grep
    Match   kube.*
    Exclude log /healthz|/readyz|/livez

Filtering at the agent is strictly better than filtering after ingestion, because you never pay to bring the data in.

3. Tier retention by value

Keep errors, warnings, and audit logs in the searchable hot tier with full retention. Route high-volume operational logs to a shorter hot-retention window and transition anything you must keep for compliance into cheap cold or archive storage. You retain the ability to investigate and the records you're required to keep, at a fraction of the searchable-storage cost.

4. Frame it as Current → Proposed

As with any cost change, record the current ingestion volume and monthly cost, the proposed volume after filtering and verbosity changes, and the projected saving. That makes the change reviewable up front and the saving auditable afterward — and it keeps the conversation about value, not just "less logging."

The Same Story as the Rest of the Bill

Log ingestion is a textbook example of the broader pattern in cloud cost: a managed service billing on a usage axis (here, volume) that creeps up silently because no single increment looks expensive. It sits alongside over-provisioned managed databases and caches and over-provisioned pod requests as part of the same discipline: compare what you're paying for against what you actually use, and close the gap source by source.

Logs are there to help you when something breaks. Paying to ingest, index, and store the 90% that never gets read doesn't make incidents easier to resolve — it just makes the bill harder to explain.

---

See Where Your Telemetry Spend Goes

Want a clearer view of where your cloud-service spend concentrates — including the observability pipeline around your cluster? Create a free KorPro account to surface waste across compute, storage, and managed services. Prefer a guided walkthrough? Contact our team.