Skip to content

Using Grafana

The ATF - Integration Monitoring (KPI) dashboard (in the Integration folder) shows the health of an environment at a glance: test results, the backend service, the Test Nodes and the mobile devices. Its panels are built from the Prometheus metrics listed in Using Prometheus Metrics in Grafana Dashboards.

Working with the dashboard

  • Tenant (top left) selects the environment. The business panels (test results, Test Nodes, devices) follow it.
  • The time range picker (for example Last 24 hours) sets the period of every panel; Refresh reloads the data, and the dashboard also refreshes itself every minute.
  • Hover over the ⓘ next to a panel title to read what the panel measures.
  • Click a series in a legend to show only that series; click it again to show all.
  • Rows can be collapsed and expanded by clicking their title.

Health KPIs and test results

The top row gives the headline numbers for the selected period: Test Pass Rate (24h), Backend Up, Node Availability, API Error Rate (5xx), API Latency (avg), DB Pool Saturation, Avg Call Setup Time and Msg Queue Discards (range). Colours follow the thresholds of each tile (green is healthy, orange and red need attention).

Test Results & SLO shows how many test cases passed and failed over time (Test Result Rate (SUCCESS / FAILURE)) and the pass rate against the 95% target (Pass Rate Trend (SLO 95%)).

Health KPIs and the Test Results & SLO row

Service health

Service Health - RED (Rate / Errors / Duration) follows the backend API: requests per second by outcome (Request Rate (by outcome)), average and maximum response time (Response Time (avg & max)) and the share of each HTTP status code (HTTP Status Code Distribution).

Service Health: request rate, response time and HTTP status codes

Capacity & Saturation shows the database connection pool (DB Connection Pool (Hikari)), the Task Executor Pool, JVM Heap Usage, CPU Usage and Backend Uptime. A connection pool that stays full is an early sign of slow or failing API calls.

Reliability and scheduled jobs

Reliability & Scheduled Jobs / Deep Health covers the backend runtime (GC Pause Rate & Avg Pause, JVM Live Threads, Open File Descriptors), the Test Nodes' log uploads (Agent Log Upload Rate (by node)) and messages a node dropped (Message Queue Discards (by node)).

Reliability & Scheduled Jobs: GC, threads, file descriptors, log uploads and queue discards

Two tables list the slowest scheduled jobs (Scheduled Jobs - Avg Duration (Top 15)) and the jobs that failed (Scheduled Job Errors (Top 15)).

Fleet status and problem tests

Fleet Status & Problem Tests shows which Test Nodes are up (Test Node Status), the test cases that failed most often in the period (Top Failing Test Cases (range)) and the least stable ones (Flakiest Tests; the score goes from 0, stable, to 1, maximally flaky).

Scheduled job tables and the Fleet Status & Problem Tests row

Mobile devices

Mobile Device KPIs - Phones / MCM Modems counts the devices (Total Mobile Devices, Phones (active), MCM Modems (Quectel EG25)) and shows their condition: Avg Battery (reporting devices), Max Device Temperature and Device Recoveries (range), then Device Battery Level, Device Temperature, Signal Strength - RSRP and Signal Quality - RSRQ per device. Devices that do not report a battery level or temperature (MCM modems, idle devices) are left out of the averages.

Mobile Device KPIs: device counts, battery, temperature and signal

Test Nodes

Test Node (Computer) KPIs - Mac Mini / MiniPC shows how many Test Nodes there are and how many are up (Total Nodes, Nodes Up), how reliably they carry out commands from the platform (Command Success Rate, Command Errors (range), Node Command Rate (by result), Command Success Rate by Node) and which commands failed on which node (Failed Node Commands (command/node)).

Test Node KPIs: node counts and command success

Adding your own panel

  1. Open Dashboards, click New and choose New dashboard (or open an existing dashboard and click Edit).
  2. In the Add pane on the right, click Panel.
  3. Choose the Prometheus data source.
  4. In the query builder, select a metric (for example test_case_results_total) and add label filters (for example tenant = integration, or result = FAILURE). The metrics and their labels are listed in Using Prometheus Metrics in Grafana Dashboards.
  5. Give the panel a Title in the options pane, pick a visualization, and check the preview.
  6. Click Save and choose the folder for the dashboard.