Grafana Monitoring Notes (for Newbies): Metrics, Dashboards, and Alerts

<strong>Grafana</strong> is a UI for <strong>observability</strong>.

• It <strong>shows graphs</strong> (dashboards) for time-series data.
• It <strong>creates alerts</strong> when metrics cross thresholds.
• It can also display logs and traces depending on your setup.

Important: Grafana does not “monitor” by itself.

• Something else must <strong>collect metrics/logs/traces</strong> (commonly <strong>Prometheus</strong>, <strong>Loki</strong>, <strong>Tempo</strong>).
• Grafana then <strong>queries</strong> those data sources and renders dashboards.

If you remember one rule: &gt; <strong>Metrics come from somewhere; Grafana only visualizes them.</strong>

A connection Grafana uses to fetch data. Common beginners’ choices:

• <strong>Prometheus</strong>: metrics (CPU, memory, request latency)
• <strong>Loki</strong>: logs
• <strong>Tempo</strong>: traces

A collection of panels.

• A <strong>panel</strong> = one chart/table
• A <strong>dashboard</strong> = many panels + layout + shared filters (variables)

Rules that evaluate queries over time and notify you. Typical workflow:

• Query metric in a panel
• Convert query into an alert rule
• Choose conditions + duration + notification channel

Your system has these “symptoms.” Monitoring should help you answer:

1. <strong>Is it healthy?</strong> (up/down, error rates, latency)
2. <strong>Is it getting worse?</strong> (trends + change detection)
3. <strong>Why is it failing?</strong> (logs + traces + related metrics)
4. <strong>What should I do next?</strong> (alerts with actionable context)

Grafana helps mainly with (1) and (2), while logs/traces help with (3).

Start small. Beginners often graph everything and learn nothing.

For each service / API:

• <strong>Request rate</strong> (requests/sec)
• <strong>Error rate</strong> (4xx/5xx per second)
• <strong>Latency</strong> (p50/p95/p99; or at least average + max)

• <strong>CPU usage</strong>
• <strong>Memory usage</strong> (watch for OOM risk)
• <strong>Disk usage</strong> (and disk I/O)
• <strong>Network throughput / drops</strong>

• Database latency + error rate
• Cache hit rate + errors
• Queue depth / consumer lag (if you have async jobs)

• <strong>Time series line chart</strong>: latency, CPU, memory
• <strong>Single stat / gauge</strong>: current error rate or p95 latency
• <strong>Heatmap (optional)</strong>: request patterns by hour/region
• <strong>Table</strong>: top slow endpoints, top erroring instances

Grafana variables help you filter by:

• environment: dev/stage/prod
• service name
• region / cluster

Beginner tip:

• Create <strong>one dashboard per system</strong>, not per chart.
• Add variables only after you have 3–6 solid panels.

Instead of alerting on:

• CPU &gt; 90%

Alert on:

• requests failing
• p95 latency too high
• error rate increasing

CPU is useful, but it’s often a symptom.

A common mistake:

• “If metric crosses threshold, alert instantly”

Better:

• “If it stays bad for <strong>N minutes</strong>”

This reduces false alarms from brief spikes.

For example:

• Alert when p95 latency &gt; 500ms
• Resolve when p95 latency &lt; 350ms

This prevents alert flapping.

If your metric has labels like:

• <code>pod</code>, <code>instance</code>, <code>userId</code>

Alerts can explode into thousands of rules. Beginner rule:

• Alert using labels that identify <strong>the service/cluster</strong>, not the individual request/user.

Average latency can look fine while p95/p99 explode. Fix:

• Track <strong>p95/p99</strong> (or whatever percentile your app supports)

Dashboards are passive. Fix:

• Add alerts for the metrics you care about.

When everything is on a dashboard, nothing is actionable. Fix:

• Make one <strong>“system overview”</strong> dashboard.
• Make smaller <strong>“deep dive”</strong> dashboards only when you need them.

A graph alone rarely tells you the cause. Fix:

• Keep consistent labels (service name, environment)
• Link from Grafana panels to logs/traces

---

• Confirm Prometheus/Loki/Tempo is reachable
• Add Grafana data source

• Request rate
• Error rate
• p95 latency

• CPU
• Memory

• DB latency/errors
• Cache hit rate

• Example: error rate &gt; 1% for 5 minutes

• Example: p95 latency &gt; threshold for 10 minutes

• So you can debug fast when alerts fire

Use consistent names:

• Services: <code>orders-api</code>, <code>payments-api</code>
• Environments: <code>prod</code>, <code>staging</code>
• Metrics: include units when possible (<code>http_request_duration_ms</code>)

Dashboards:

• Start with the system name
• Example: “Orders Service / Overview”

A good alert has:

• <strong>Query</strong>: the data you evaluate
• <strong>Condition</strong>: threshold + operator
• <strong>Duration</strong>: “for 5m/10m”
• <strong>Severity</strong>: warning vs critical
• <strong>Message</strong>: who/what to do next

Beginner message template:

• “High error rate detected in <code>orders-api</code> on <code>prod</code>. Check recent deploys and logs for 5xx spikes.”

Grafana is where you turn raw metrics into decisions. If you start with:

• a small set of service metrics (rate, errors, latency)
• a focused dashboard
• a couple of good alerts (with time windows)

…you will already be doing “real monitoring,” not just making charts.