# Watch It! Monitoring Lessons from Two Decades in Computing - Markus Krüger - NDC Oslo 2026

## Executive summary

This talk provides a comprehensive overview of modern observability practices, emphasizing the shift from reactive debugging to proactive monitoring. The core message is that robust monitoring requires collecting three types of telemetry—metrics, logs, and traces—and implementing sophisticated systems to detect issues before end-users are impacted. Key recommendations include adopting OpenTelemetry for standardization, using tools like Prometheus and Grafana for visualization, and leveraging AI for anomaly detection and automated incident response.

## Key takeaways

- Focus on Observability Pillars: Effective monitoring relies on collecting three distinct signals: Metrics (numerical measures over time, e.g., CPU load), Logs (arbitrary text providing detail), and Traces (tracking request execution across distributed components).
- Adopt OpenTelemetry for Standardization: OpenTelemetry is a vendor-agnostic framework for generating, exporting, and collecting telemetry, allowing organizations to switch backends without rewriting application code.
- Alert on Symptoms, Not Causes: Alerting should focus on the end result (the symptom, e.g., 'server not responding') rather than the underlying cause, which can be numerous and unpredictable. Alerts must be actionable and concise.
- Implement Proactive Testing: To detect failures before customers do, implement synthetic testing (exercising functionality in production) and Real User Monitoring (RUM) to capture client-side errors.
- Leverage AI for Incident Management: AI can assist in anomaly detection (identifying deviations from expected behavior) and, in advanced setups, can propose or even execute fixes for incidents, though caution regarding access control is advised.

## Technical details

- Telemetry Types: Metrics are efficient for overall state tracking but are coarse-grained. Logs provide deep, arbitrary text detail. Traces are essential for distributed systems, allowing tracking of a request's flow across multiple services and components.
- Metric Types and Best Practices: Metrics include Counters (monotonically increasing values, e.g., request count), Gauges (values that can increase or decrease, e.g., CPU load), and Histograms (counting by buckets, necessary for calculating percentiles like the 95th percentile).
- Logging Systems: Logs should be aggregated and indexed (e.g., using the ELK/EFK stack: Elasticsearch, Kibana, Fluent). Structured logging (e.g., JSON format) is highly recommended for easier querying and alerting.
- Visualization and Dashboarding: Visualization tools like Grafana are recommended because they unify data from multiple sources (Prometheus, Elasticsearch, Loki) without storing data themselves. Dashboards should be simple and focused.
- Advanced Monitoring Techniques: Techniques include Synthetic Testing (automated checks of functionality) and Real User Monitoring (RUM) to capture client-side experience, which is critical as backend metrics may miss front-end issues.

## Practical implications

- Make monitoring a core part of the 'Definition of Done' for all features.
- Implement monitoring systems outside of the application server to ensure data persistence even during outages.
- Treat alert configuration as Infrastructure as Code (IaC) to ensure version control and repeatability across environments.
- When using AI agents, strictly define the scope of their actions and ensure they are trained on system-specific knowledge to prevent incorrect fixes or data leakage.

## Topics

Observability, Monitoring, Telemetry, Distributed Systems, SRE, OpenTelemetry, AI in DevOps, Prometheus, Grafana, ELK/EFK Stack, Jaeger, Site Reliability Engineering, Accelerate

Source: https://www.youtube.com/watch?v=0rD3X-RHEQI