Pickknow API Metrics

This page is the live nerve center for the API process. It is not a marketing dashboard: every number is meant to answer a concrete question — is traffic healthy, is the box drowning, did a deploy make things worse, and which route is the smoking gun?

Data arrives over SSE from /ops/stream every few seconds. Charts can zoom from 1 minute to ~12 hours (history from SQLite ops_ticks). Route tables and window compare use a shorter in-memory buffer (~30 minutes). Recent errors and request logs live in SQLite and survive restarts.

Don’t jump straight into one chart. Walk top → bottom so you don’t mis-blame a route when the host is OOM, or scale the box when a single endpoint is broken.

• 1. KPIs
  Status, throughput, p95, Err%, DB RTT, Host RAM, Process CPU, Event loop. If Status is Degraded, or Err% / p95 / Host RAM are red — you have an incident signal. Note whether everything looks bad or only one KPI.
• 2. Saturation
  High score → the machine/process is drowning (in-flight pile-up, event-loop lag, RAM/CPU/heap/DB). Scale, free memory, fix DB pressure first — route “bugs” may be symptoms. Low score + bad Err% or p95 → capacity is fine; dig into routes.
• 3. Window compare
  Last 15 minutes vs the previous 15. Red Δ on err% or p95 right after a deploy line usually means that release. Green Δ means recovery or quieter traffic — don’t overreact to absolute numbers alone.
• 4. Charts
  Align spikes: did req/s jump (traffic), in-flight climb with flat req/s (waiting), Host RAM climb (OOM risk), DB RTT climb (database)? Teal dashed deploy markers mark process starts — nearby error/latency jumps are release suspects.
• 5. Click a route
  Top / Failing / Slowest rows open a drilldown: that path’s latency and error timeline (~30m). Confirms whether one endpoint owns the pain. Then use Recent errors + Admin → Server logs for stacks, diagnose pack, and staff alerts.

These cards are the 10-second health check. Colors follow the same green / amber / red rules as charts.

• Status
  Healthy vs Degraded for this process. PID is the OS process id — if it changes, the API restarted (you should also see a deploy marker).
• Throughput
  rps = requests per second over the rolling window; rpm = per minute; in-flight = started but not finished. High in-flight with modest rps often means work is stuck waiting (DB, external APIs).
• Latency p95
  95% of requests finished faster than this. Better than average for “user pain.” Subline shows avg and p99. ≥500ms caution · ≥2000ms slow. High p99 with low avg = occasional stalls, not everyone slow.
• Error rate
  Share of 4xx+5xx in the window. Subline splits 4xx (often clients/auth/validation) vs 5xx (server failures — urgent). 0% ideal · any errors amber · ≥10% critical on this console.
• DB RTT
  Cheap SELECT 1 round-trip. High with healthy process CPU → database or network to DB, not “app busy computing.” FAIL = probe failed — treat as outage-class.
• Host RAM
  Whole machine memory. ≥80% watch · ≥92% critical. Near full is a classic cause of flapping restarts / OOM kills. Process RSS (elsewhere) is only this API. Click Host RAM for a breakdown: rolled-up names (docker / bun / …), Docker container Mem%, and top processes by RSS.
• Process CPU
  This API’s CPU (one-core style %). Sustained high with rising latency = compute-bound work or busy loops. Click Process CPU for top host CPU consumers (and the same resource modal as Host RAM).
• Event loop p99
  How blocked the main loop is. High lag means the process can’t start/finish HTTP work promptly even if Host RAM looks fine — often heavy sync work or overload.

• req/s (tick)
  Throughput this SSE tick — finished requests in the last ~2.5s as requests/second. Spikes = bursts or retries. Flat near 0 = quiet (or stream stuck — check Live pill).
• in-flight
  Concurrency. Rising in-flight with flat req/s → backlog waiting on DB/APIs. Sustained high feeds Saturation.
• Deploy markers
  Teal dashed vertical lines = API process starts. Nearby restarts may cluster as deploy×N with slanted labels. Always ask: did errors/latency jump after this line?

• avg / p95 (tick)
  Per-tick speed. Avg = typical; p95 = tail users feel. Align jumps with deploy lines and status/error charts.
• 2xx / 4xx / 5xx
  Counts per tick by status class. 2xx success · 4xx client · 5xx server. A 5xx spike with quiet 4xx is usually our bug or dependency failure.
• Error rate per tick
  %(4xx+5xx) each tick. Aligned with deploy → check release. With high saturation → capacity. On one route only (tables/drilldown) → that path.

• Host RAM %
  Machine-wide. Climbing toward 100% before restart flaps is a smoking gun for under-provisioned VMs or memory leaks elsewhere on the host.
• Heap / CPU / loop / DB / RSS
  Process heap pressure, CPU, event-loop lag, DB ping, process RSS. Pattern: slow DB + calm CPU → database pressure. Rising heap without recovery → possible leak.

These two panels answer “is this new?” and “is the box drowning?” before you touch route tables.

• Window compare
  Previous 15m vs last 15m for volume, err%, p95, avg. Green Δ = better · red Δ = worse. Use after deploys and when someone asks “did it just get worse?”
• Saturation score
  0–100 blend of in-flight, event loop, RAM, CPU, heap, DB. High → scale / free resources / fix DB. Low + errors → application or route bug — click drilldowns.

Every row in Top routes, Failing routes, and Slowest is clickable. The drilldown shows that path’s own latency and error timeline so you can confirm timing and severity without guessing from global charts.

• Top by volume
  Busiest paths in the window (count, avg, p95, err%). High volume + high err% hurts the most users — start here when Saturation is low.
• Failing routes
  Paths that produced errors — count, last status, last seen. Best “what is broken right now?” list.
• Slowest (p95)
  Worst tails even if they aren’t failing. Click to see whether slowness is continuous or spiky.
• Path shaping
  UUIDs / numeric ids collapse to :id so the same logical route groups. /ops/stream is excluded from latency so the console doesn’t measure itself.

• Recent errors
  SQLite-backed uncaught errors and 5xx samples — survive restart. For full stacks, filters, diagnose pack, and staff email/Telegram/push: Admin → Server logs.
• What survives restart
  Chart ticks (~12h), deploy markers, recent errors, ops alerts, request/error logs. In-memory route buffers rebuild with live traffic (~30m for compare/drilldown).
• 1m … 12h / Live
  Range buttons zoom charts. Live pill + Reconnect control the SSE feed. Stream dock shows tick sequence and server/client latency of the ops stream itself.

• Global KPIs
  “Is the API OK right now?”
• Saturation
  “Is the box drowning, or is capacity fine?”
• Window compare
  “Did the last 15 minutes get worse than the 15 before?”
• Charts + deploys
  “When did it change, and was a restart involved?”
• Route click
  “Which path owns the pain?”
• Recent errors
  “What exception / 5xx should I open in Server logs?”