Performance and System Monitoring¶
The Testing Automation Engine includes a pre-installed instance of the third-party tool Prometheus for monitoring and performance analytics. This setup provides critical insights into system and process performance metrics.
Table of Contents¶
- Performance Metrics Table
- Meaning of Parameters
- Key Features of Prometheus
- Data Centralization with Remote Write
- KPI Export and Grafana Integration
- Benefits of Prometheus-Grafana Setup
Performance Metrics Table¶
| Metric | Parameters | Description |
|---|---|---|
| device_battery_level | Device Test_Node instance job tenant | Monitors the battery level of devices during testing. |
| device_network_operator | Device Test_Node instance job tenant | Tracks the network operator used by devices during testing. The value is a numeric ID of the operator, not its name; the operator name of each device is shown on the Mobile ID's page and the Dashboard Status page. |
| device_radio_access_type | Device Test_Node instance job tenant | Indicates the radio access type used by devices during testing. The value is a numeric ID of the radio technology, not its name. |
| device_signal_rsrp | Device Test_Node instance job tenant | Reference Signal Received Power (RSRP): Measures average LTE reference signal strength. |
| device_signal_rsrq | Device Test_Node instance job tenant | Reference Signal Received Quality (RSRQ): Indicates LTE signal quality using RSRP and RSSI. |
| device_signal_rssnr | Device Test_Node instance job tenant | Signal-to-Interference-plus-Noise Ratio (SINR): Reflects LTE signal clarity and resistance to interference. |
| device_signal_rssi | Device Test_Node instance job tenant | Received Signal Strength Indicator (RSSI): Represents total received signal power including noise. |
| data_speed | Device Test_Case instance job tenant | Download speed in Mbps, measured by the SPEED_TEST function. |
| data_ping | Device Test_Case instance job tenant | Round-trip time in ms, measured by the PING function. |
| data_jitter | Device Test_Case instance job tenant | Jitter in ms, measured by the PING function. |
| data_packet_loss | Device Test_Case instance job tenant | Packet loss in %, measured by the PING function. |
| device_temperature | Device Test_Node instance job tenant | Monitors the temperature of devices during testing for stability analysis. |
| call_duration | Device Test_Case instance job tenant | Call duration: the time between the moment ANSWER_CALL reports the call as connected and the moment CALL_DISCONNECT reports it as disconnected. |
| call_setup_time | Device Test_Case instance job tenant | Call setup time: from the outgoing SIP INVITE to the incoming 180 Ringing, read from the Android SIP log of MAKE_CALL. Android phones only, and only when the SIP log lines are available; there is no value for iPhones or MCM modems. |
| test_case_flakiness_score | testcase testcase_id instance job tenant | Flakiness of a test case from 0 (stable) to 1 (maximally flaky), over its last 10 runs. Only test cases that ran in the last 30 days are included. The Dashboard shows it as "Top Flaky Test Cases (30d)". |
| test_case_results_total | instance job tenant result testcase | Tracks the total number of results for each test case in the system. |
| testnode_command_results_total | command instance job tenant result testnode | Tracks the total results of commands executed on test nodes. |
| testnode_up_status | instance job tenant name | Tracks the status of test nodes to determine if they are up and running. |
| total_mobiledevices | instance job tenant | Tracks the total number of mobile devices involved in testing. |
| total_testcases | instance job tenant | Tracks the total number of test cases available in the system. |
| total_testexecutions | instance job tenant | Tracks the total number of test executions performed in the system (direct runs of single test cases are not counted). |
| total_testnodes | instance job tenant | Tracks the total number of test nodes available in the system. |
| total_testsuites | instance job tenant | Tracks the total number of test suites defined in the system. |
| ts_disabled_rate | Test_Suite instance job tenant | Calculates the disabled rate of test suites. |
| ts_success_rate | Test_Suite instance job tenant | Calculates the success rate of test suites. |
| ts_tc_success_count_total ts_tc_fail_count_total ts_tc_disabled_count_total ts_tc_total_count_total | Test_Suite instance job tenant | Counters of the test case results of a test suite: successful, failed, disabled and all test cases, added up over the suite's executions. |
| device_recovery_total | testnode instance job tenant | Counter of device failures that a test node reported as recovered. |
| msg_queue_discard_cnt_total | testnode instance job tenant | Counter of messages a test node reported as discarded from its message queue. |
| agent_conn_errors_total | testnode instance job tenant | Counter of connection errors reported by the agent of a test node. |
| agent_log_lines_uploaded_total | testnode_id testnode_name instance job tenant | Counter of agent log lines uploaded by a test node. |
The signal metrics (device_signal_rsrp, device_signal_rsrq, device_signal_rssnr, device_signal_rssi) are exported only when the device reports that value. Counters are exported with the Prometheus _total suffix.
Meaning of Parameters¶
| Parameter | Description | Explanation |
|---|---|---|
| command | The command executed on a test node. | The function or operation executed on a test node, such as SPEED_TEST or DATA_ROAMING. Together with result it counts successful and failed executions. |
| Device | Refers to the physical or virtual device being tested. | This could include smartphones, IoT devices, or emulated devices used during testing. The value is the Device ID of the Mobile ID. |
| instance | The scrape target. | Added by Prometheus: the address of the TEO backend instance the metric was collected from. |
| job | The scrape job. | Added by Prometheus: the name of the scrape job that collected the metric. |
| name | Identifier for the test node. | Provides unique naming or labels for nodes being used during execution. |
| result | Outcome of a test case or command (pass/fail). | Tracks success, failure, or exceptions generated during testing. |
| tenant | Indicates the client or organization for whom testing is performed. | Represents multi-tenancy structures where tests align with a particular client’s environment or needs. |
| testcase | Specific test scenario or script. | Defines individual test scripts that validate particular features or use cases. |
| Test_Node | Physical or virtual node where tests are executed. | Nodes are computing resources (e.g., virtual machines, Docker containers) used to run tests. |
| Test_Suite | A group of test cases executed together. | Collections of related tests for a specific functionality or module to validate grouped execution. |
| Test_Case | Label of the executed test case. | The name of the test case; used by the data_*, call_duration and call_setup_time metrics. |
Key Features of Prometheus in the Testing Automation Engine¶
-
Comprehensive Metrics:
- CPU utilization
- Memory allocation
- Disk usage
- Input/output (I/O) operations
- System processes
-
Detailed Test Execution Insights: Prometheus delivers metrics that evaluate the performance and health of test execution processes.
-
Key Performance Indicators (KPIs): The engine includes built-in KPIs for various test execution environments, enabling granular monitoring.
Data Centralization with Remote Write¶
Prometheus’ remote write capability allows metrics from multiple nodes in the testing engine cluster to be transmitted to an external Prometheus server. This enables:
-
Centralized Data Collection: All metrics from distributed nodes are aggregated into a single Prometheus instance.
-
Custom Alerting: Users can define alerts based on custom KPI thresholds, ensuring timely notifications for anomalies.
KPI Export and Grafana Integration¶
Metrics collected by Prometheus can be exported using the KPI shipping capability to an external Prometheus instance. This enables:
-
Data Aggregation: Metrics from all test environment virtual machines (VMs) are consolidated.
-
Visualization with Grafana:
- The external Prometheus instance acts as the primary data source for Grafana.
- Grafana creates dashboards for real-time monitoring and performance analytics.
-
Robust Alerting: Alerts can be defined using user-configured thresholds or pre-exported KPIs, supporting efficient issue detection.
Benefits of the Prometheus-Grafana Setup¶
- Real-time visualization and monitoring for test execution processes.
- Improved system health and performance analytics.
- Customizable dashboards and alerts tailored to testing requirements.