Grafana & Friends Taipei #2: From an 800-Person War Room at TSMC to a €6/Month Homelab

EN #observability#grafana#kubernetes#community#taipei

Meetup at 5倍學院, Taipei. Three talks with a common thread: collecting metrics is the easy part; the hard part is knowing you lost the signal the moment you lose it.

TSMC: the incident no alert saw

Chuck (TSMC) told the story of the “FAB99 X System Service Down” case: two outages on the same day (08:30 and 15:30), each lasting more than 10 minutes, and a war room that peaked at ~800 people — the highest figure ever recorded for the service.

The interesting part is how it resolved. Network, K8s, and Infra checked their dashboards and found nothing (“不是我的問題” — not my problem). The final root cause: an application-team metric that had no alert rule configured. The problem was invisible to everyone because it never showed up on the alerting radar.

The lessons they drew themselves: alerting worked for what was configured and the cross-team response was fast, but there was too much noise and no correlation between systems — every team staring at their own dashboards without cross-referencing.

Their roadmap to fix it runs from a custom observability SDK using OTel Semantic Conventions, all the way to unifying metrics, logs, and traces in ClickHouse as a single storage backend, with an AI agent in Grafana for natural-language search, anomaly detection (which would have caught the metric without an alert), and root cause analysis by cross-referencing signals via trace_id.

A technical detail I care about because of the Java stack: in Java, the OTel Agent auto-instruments via bytecode injection (JDBC, Redis, HTTP clients — “for free”); in Go there’s no such thing — you have to instrument every client explicitly.

Edward Oo: CPU at 66%… but steal at 62%

The second talk was a homelab debugging case that applies to any cloud VM. A CPU alert fires: 66% usage. But breaking it down: steal 62%, user 12%. The VM wasn’t getting real CPU cycles because the hypervisor was giving them to another VM (a noisy neighbor) — kubectl top only showed 10% usage, with the real health hidden in node_cpu_seconds_total{mode='steal'}.

The cascade was elegant in its perversity: PostgreSQL lived on the node with steal, the SSO server (authentik) on another node, and every DB call crossed the overlay network with mTLS only to die in timeouts. Result: 33 restarts in 12 hours with Exit Code: 0 — these weren’t crashes, it was the liveness probe killing the pod because the DB wasn’t responding in time. The fix: a nodeSelector to co-locate the DB and the server. Zero restarts.

His 5 lessons, which hold for any virtualized environment:

  1. CPU Usage ≠ CPU Steal — kubectl top doesn’t show steal
  2. ExitCode 0 = probe kill, not an app crash
  3. Co-locate stateful workloads — the Helm chart doesn’t ship with a nodeSelector by default
  4. Cloudflare Proxy isn’t universal — machine-to-machine traffic doesn’t need a CDN (it cost him 4 weeks of silent 403s on remote_write)
  5. Losing data doesn’t mean it’s gone — vmctl backfill recovered 336M samples in 15 minutes from Prometheus’s local TSDB

His stack is a solid argument for VictoriaMetrics over Thanos/Mimir at small scale: a single binary, under 500MB of RAM, 100% PromQL-compatible, native backfill. And the entire observability plane running on a separate VPS for ~€6/month — if the cluster has a serious problem, the dashboards stay alive.

Blueswen: SRE where the rats chew the cables

Blueswen (Grafana Champion) closed with two years of SRE on edge devices for fast-food restaurants in the US — one edge server per restaurant turning the Drive Thru into metrics with camera AI. The edge “lives on site, not in the datacenter”: borrowed network from the client, power outages, heat, and yes, rats chewing cables.

Three patterns I’m taking with me:

  • Backfill after a network outage: the agent buffers to disk and resends once the network is back — but it arrives with old timestamps (out-of-order). If the storage doesn’t have out_of_order_time_window configured, those metrics get silently dropped.
  • Dual-path offline detection: remote write as the primary signal + a heartbeat to S3 queried via Athena as a backup, cross-referenced with a Grafana 11 SQL Expression in the same alert rule. Offline only gets declared if both paths go silent — goodbye false positives from the client’s firewall.
  • Node Exporter’s Textfile Collector for metrics with no exporter: a cron + script that writes a file in Prometheus format, scraped “on the side” without touching the data flow.

The meetup’s closing line sums it all up: on site, the hard part was never collecting the metrics — it was knowing you lost the signal the moment you lose it.