Most observability comparisons rank tools on features. That framing misleads, because Grafana and Datadog both do metrics, logs, traces, and dashboards perfectly well. The features converged years ago.
What hasn't converged is the answer to two questions: who operates this thing at 3am, and what does the bill scale with. Those are the decisions you're actually making. This guide compares the options on that basis.
Monitoring vs observability: a distinction worth keeping
Monitoring answers questions you predicted. You knew disk space could fill up, so you built a disk-space alert. Observability is the ability to answer questions you didn't predict, from data you already collected — "why is checkout slow, but only for users on the new pricing plan, and only in eu-west?"
The practical difference is dimensionality. A monitoring system stores disk_free = 12GB. An observability system stores disk_free{host="db-3", region="eu-west", role="primary"} = 12GB, and lets you slice by any of those dimensions after the fact. Every architecture and pricing decision below follows from that one idea.
The three pillars
- Metrics — numeric measurements aggregated over time. Cheap to store, cheap to query, lossy by design. Good for "is it broken" and "how broken".
- Logs — discrete events with detail. Expensive to store, expensive to search, high fidelity. Good for "what exactly happened to this one request".
- Traces — the causal path of one request across services, with timing per hop. Good for "which of my 14 services is the slow one".
You need all three eventually, but not on day one, and the order matters. Metrics first, because they're cheapest and catch the most. Logs second. Traces when you have enough services that you can no longer guess which one is at fault.
Datadog: buy the operations away
Datadog is a fully managed SaaS platform. You install an agent, it discovers what's running, and data appears in a polished UI with a large library of integrations for common infrastructure.
The honest pitch for Datadog is not "better dashboards." It's that you employ zero people to run it. There is no retention policy to tune, no storage volume that fills at 2am, no upgrade that breaks a query. For a team without dedicated platform engineers, that is worth real money, and teams routinely underestimate how much.
How the billing actually works
This is the part that surprises people, so it's worth being precise about the shape even though the numbers change — always price your own workload against the current Datadog pricing page rather than a blog post.
Datadog is not one price. It's a set of separately-billed products that each meter something different:
- Infrastructure — billed per host, per month. Predictable, and the part everyone budgets for.
- Custom metrics — billed per metric time series. This is the one that bites (see the cardinality section below).
- Logs — billed separately for ingestion and for indexing, with retention tiers. You can ingest a log cheaply and still pay again to make it searchable.
- APM / traces — billed per host, often with limits on indexed spans.
- Synthetics, RUM, security, and others — each its own meter.
Per-host billing interacts badly with elastic infrastructure. If you autoscale to handle a traffic spike, you scale your bill with it. Teams running bursty workloads on Kubernetes should model the cost at peak host count, not average, and check how their plan counts short-lived hosts.
When Datadog is the wrong choice
- Your data can't leave your infrastructure for regulatory reasons.
- You're log-heavy and cost-sensitive — verbose applications generate log volume that becomes the dominant line item fast.
- You have platform engineers with spare capacity and a strong cost mandate; at large scale, self-hosting is genuinely cheaper.
Grafana: the middle ground, with a caveat
Grafana is really two things, and conflating them causes confusion.
Grafana the visualization layer is the dashboard tool that queries dozens of data source types. It is near-universal — even shops that pay for other platforms often keep Grafana around.
The Grafana stack (often called LGTM) is a set of purpose-built backends:
| Component | Role | Design idea | |---|---|---| | Loki | Logs | Index only labels, not full text | | Grafana | Visualization | Query anything, one UI | | Tempo | Traces | Trace lookup straight from object storage | | Mimir | Long-term metrics | Scalable Prometheus-compatible storage |
The unifying design choice: push the bulk of your data into cheap object storage (S3 and friends) and keep the index small. Loki is the clearest example. Classic log platforms build a full-text index over every log line, which is why they're fast and why they're expensive. Loki indexes only the labels you attach — {app="api", env="prod"} — and stores the log bodies as compressed chunks. You then grep within the label-selected chunks at query time.
That trade is excellent when you know roughly where to look and awkward when you don't. A query scoped to one app for one hour is fast and cheap. A naive "find this string anywhere in the last 30 days" scans a lot of chunks. If your debugging style leans on broad unscoped text search, a full-text engine like the ELK stack or Elasticsearch fits those habits better — at the storage cost that buys.
Grafana Labs relicensed Grafana, Loki, and Tempo from Apache 2.0 to AGPLv3 in 2021. For the overwhelming majority of teams — running it internally — this changes nothing. If you plan to offer Grafana as part of a service to your own customers, get the AGPL's network-use clause in front of your legal team before you build on it. It's cheap to check now and expensive to discover later.
Grafana Cloud is the managed version of that same stack, with a free tier that is genuinely usable for small projects. Its main strategic advantage over Datadog is portability: the query languages (PromQL, LogQL, TraceQL) and the dashboards are the same ones the open-source stack uses. If the bill grows past what you'll pay, you can move to self-hosted and keep your dashboards, your alerts, and your team's muscle memory. That exit option has real value, and it's the strongest argument for the Grafana ecosystem.
Self-hosted open source: cheapest at scale, never free
The standard stack is Prometheus for metrics, Grafana for dashboards, Loki for logs, and Tempo or Jaeger for traces.
Prometheus is a CNCF-graduated project and the effective default for metrics in the Kubernetes world. Its model is worth understanding because it explains its limits:
- Pull-based. Prometheus scrapes an HTTP endpoint on your services on an interval. Your app doesn't push anywhere; it just exposes
/metrics. This makes "is the target up?" a free side effect of scraping, and it makes local development trivial. - Dimensional. Every series is a metric name plus labels. PromQL queries those labels.
- Single-node, local storage. A Prometheus server stores to its own disk. It is explicitly not a clustered long-term store — for that you add Mimir, Thanos, or Cortex, or use remote write to a hosted backend.
A minimal scrape config:
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "alerts/*.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
scrape_configs:
- job_name: "api"
metrics_path: /metrics
static_configs:
- targets: ["api-1:8080", "api-2:8080"]
labels:
env: prod
The honest cost accounting: the software is free, the servers are not, and the engineer-hours are the real line item. Someone has to own retention tuning, upgrades, storage growth, and the alerting pipeline. Self-hosting pays off when you're big enough that the SaaS bill exceeds a meaningful fraction of an engineer's salary, or when data residency makes it non-negotiable. It's a poor trade for a five-person team who'd rather ship product.
If you want the open-source model with less assembly, OTel-native platforms like SigNoz and OpenObserve are worth a look — they bundle metrics, logs, and traces into a single deployable rather than four.
OpenTelemetry is the decision that outlives the others
Pick your vendor for a year; pick your instrumentation for a decade. Instrument with OpenTelemetry and the vendor becomes a config change instead of a rewrite.
This is the most valuable idea in this article. The hard part of observability was never the backend — it's the instrumentation scattered across your codebase. Historically that instrumentation was vendor-specific, so switching platforms meant touching every service. That was the lock-in, and it was brutal.
OpenTelemetry (OTel) is a CNCF project providing vendor-neutral SDKs and a wire protocol (OTLP). You instrument once against OTel, run a collector, and point the collector wherever you want. Datadog, Grafana Cloud, New Relic, and the self-hosted stack all ingest OTLP.
A collector config that fans the same data out to two backends — which is exactly how you run a migration without a flag day:
# otel-collector.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 10s
memory_limiter:
check_interval: 1s
limit_percentage: 80
exporters:
otlphttp/grafana:
endpoint: https://otlp-gateway.grafana.net/otlp
headers:
authorization: "Basic ${GRAFANA_TOKEN}"
datadog:
api:
key: ${DD_API_KEY}
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/grafana, datadog]
Run that for a couple of weeks, compare the two UIs on real incidents, and make an informed decision instead of a procurement guess. The collector also gives you one place to scrub PII, drop noisy spans, and sample — before the data leaves your network and before you pay to ingest it.
Cardinality: the thing that actually decides your bill
Every platform above — SaaS or self-hosted — is priced, directly or indirectly, on cardinality: the number of unique time series you produce. A metric's cardinality is the product of its label values.
http_requests_total{method, status, endpoint}
= 5 methods x 8 statuses x 40 endpoints
= 1,600 series. Fine.
http_requests_total{method, status, endpoint, user_id}
= 1,600 x 50,000 users
= 80,000,000 series. An outage, or an invoice, or both.
That's not a rounding error, it's an incident. On Datadog it shows up as a custom-metrics bill nobody approved. On self-hosted Prometheus it shows up as the server OOM-killing itself, because the index lives in memory.
The rule is simple and near-universal: never put an unbounded value in a metric label. No user IDs, no request IDs, no email addresses, no raw URLs with IDs baked in — /orders/84213 should be recorded as the route /orders/:id. High-cardinality identifiers belong in logs and trace attributes, which are built to hold them. Metric labels are not.
What to monitor first
Skip the 400-panel dashboard. Nobody reads it during an incident.
The four golden signals (from Google's SRE practice) — latency, traffic, errors, saturation — cover the majority of real failures. For request-driven services, the RED method (Rate, Errors, Duration) is the practical form. For resources, USE (Utilization, Saturation, Errors) is the counterpart.
Real PromQL for the ones that matter:
# Error rate as a fraction of all requests, last 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# p95 latency from a histogram, per service
histogram_quantile(
0.95,
sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
)
And an alert rule that won't page you for a blip:
# alerts/api.yml
groups:
- name: api
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.01
for: 10m
labels:
severity: page
annotations:
summary: "5xx rate above 1% for 10 minutes"
runbook: "https://wiki.internal/runbooks/api-errors"
Two details there earn their keep. for: 10m means the condition must hold continuously before firing — this is what separates an alert from a nuisance. And the runbook annotation means the person woken at 3am gets a link instead of a puzzle. Alert on symptoms users feel, not on causes; "CPU is at 90%" is not an incident if nobody noticed.
Choosing: an honest summary
| Factor | Datadog | Grafana Cloud | Self-hosted OSS | |---|---|---|---| | Time to first dashboard | Minutes | Hours | Days | | Ongoing ops burden | None | Low | Real and permanent | | Cost at small scale | Moderate | Low (free tier) | Server cost only | | Cost at large scale | Highest | Middle | Lowest in $, highest in hours | | Data residency | Vendor cloud | Vendor cloud or self | Full control | | Exit cost | High without OTel | Low — same stack | N/A |
- Solo devs and small teams: Grafana Cloud's free tier, plus Sentry for application errors. Errors and uptime catch most of what will actually hurt you; don't over-build.
- Funded startups without platform engineers: Datadog, deliberately. You are buying back engineer time, which is your scarcest resource. Instrument with OTel anyway so the door stays open.
- Teams with a platform function and a cost mandate: Grafana Cloud until the bill justifies migration, then self-host the same stack. This is the path with the fewest one-way doors.
- Regulated or air-gapped: Self-hosted, and budget the headcount honestly up front.
Whatever you pick, do these two things: instrument with OpenTelemetry, and keep unbounded IDs out of your metric labels. Get those right and the platform choice is reversible. Get them wrong and no platform will save you from the bill.