System Testing for hybrid systems
System testing is a pivotal phase in the development of hybrid embedded/cloud systems. It ensures that each system, embedded and cloud, meets its requirements and performs as expected, both independently and when integrated. This stage involves extensive validation under realistic conditions, employing a mix of manual and automated tests to uncover potential issues before deployment.
For a deeper understanding of hybrid embedded/cloud systems, you can read the article Hybrid Embedded/Cloud Systems. Additionally, for an overview of the test levels necessary to ensure thorough test coverage, refer to the article Test Levels for hybrid systems.
This article will delve into two closely related aspects: system testing, where individual systems are evaluated independently, and system integration testing, where all components are fully integrated to verify their interactions. For challenges encountered during integration, you may find the article Integration challenges for hybrid systems helpful.
System Testing
System testing evaluates each system independently to ensure it meets its requirements and behaves as expected under various conditions. This step is critical for identifying issues at the system level before integrating the components.
HIL configuration and setup
Hardware In the Loop (HIL) testing is a cornerstone of system testing, especially for the embedded side. For more details about HIL testing for hybrid systems, you can read the article Hardware In the Loop (HIL) testing for hybrid systems. Configuring an HIL system involves selecting suitable hardware and software solutions. Available tools range from proprietary, closed environments with prebuilt automation frameworks to open and flexible solutions that allow testers to use open-source tools. For example, communication between ECUs and external devices often happens over buses like CAN. Hardware tools like the PEAK CAN USB adapter and open-source solutions such as python-can can help interact programmatically with CAN bus communication, providing cost-effective ways to set up the required testing environment.
Testing the embedded system
Before integrating with the cloud, the embedded system can be tested independently using mocked cloud responses. For instance, if communication with the cloud occurs via REST APIs, we can simulate the responses to validate how the ECU processes them. Edge cases, such as handling API timeouts, errors, or unexpected data, can be effectively tested at this stage.
Testing the Cloud System
On the cloud side, the system can be tested as a black box through its REST API without involving the embedded system. This allows testers to simulate workflows and real-use scenarios, ensuring that the cloud software behaves correctly in isolation. Security, data consistency, and error-handling mechanisms can also be validated during this phase.
By automating many of these tests and integrating them into a dedicated CI pipeline, regression testing becomes a seamless part of the development lifecycle.
System Integration Testing
System integration testing is the final step where the embedded and cloud systems are tested together to ensure their interactions work as intended in a production-like environment. Unlike the testing activities highlighted in the previous sections, system integration testing doesn’t rely on mocks. Instead, the full setup, including the HIL system and ECU, is used to simulate realistic scenarios. This allows for the validation of real-world interactions between the embedded system and the cloud.
For instance, the HIL system can simulate edge cases like network latency or unexpected changes in the external environment. This helps verify critical aspects such as:
Timing and sequencing: ensuring operations are executed in the correct order.
Data consistency: validating that data shared between the ECU and cloud remains consistent.
Security: testing authentication, encryption, and secure communication.
The system integration phase provides the ideal environment to address integration challenges such as those outlined in the Integration challenges for hybrid systems article. This includes validating workflows, interaction patterns, and distributed business logic to ensure smooth collaboration between systems.
Final thoughts
System testing, conducted during the development process before production, is vital for ensuring that hybrid embedded/cloud systems operate reliably in real-world conditions. By first independently validating each system and then verifying their interactions during integration testing, teams can uncover potential issues early and prevent costly rework later in the development cycle.
The use of HIL systems adds significant value, enabling testers to simulate complex, real-world scenarios and validate system behavior under challenging conditions. By combining manual and automated testing approaches, organizations can achieve high-quality standards while streamlining the development process.
