Test levels for hybrid systems

When developing hybrid embedded/cloud systems, testing becomes a critical part of ensuring the system operates reliably and as intended. This involves not just testing the embedded and cloud components individually but also ensuring that they work seamlessly together. Let’s explore the testing levels that are typically involved.

If you are not familiar with hybrid embedded/cloud systems, refer to the previous article Hybrid Embedded/Cloud systems. Similarly, if you're new to HIL (Hardware-in-the-Loop) testing, check out the article Hardware In the Loop (HIL) testing for hybrid systems.

This article delves into the testing strategies and levels for these hybrid systems, illustrating how HIL testing fits into the broader picture.

The Testing Pyramid: a familiar framework

The testing pyramid is a well-known concept in software testing. At the base, we have unit tests: fast, code-level tests that are relatively inexpensive to develop and execute. Moving up, integration tests verify how different modules work together, and at the top, system tests evaluate the entire system’s functionality. While tests become more comprehensive as we move up the pyramid, they also become more time-consuming and costly to develop and maintain, and they require a more complex test environment to run.

The usage of HIL across the test levels

This concept applies equally to HIL testing. HIL allows for real-time simulation of the environment an embedded system (like an ECU) will interact with. It enables manual exploratory testing, simulating vehicle-like conditions before the actual vehicle is available, as well as automated testing, which can be integrated into CI/CD pipelines to ensure quick feedback loops.

Two systems, two testing streams

In hybrid systems, we’re essentially dealing with two distinct systems: the embedded system and the cloud platform. Each of these systems undergoes its own testing at the unit, integration, and system levels.

For example:

• Embedded testing: tests at the unit and integration level validate ECU functionality, like processing sensor inputs or executing real-time decisions.

• Cloud testing: tests here might include verifying API endpoints, validating cloud business logic, and ensuring scalability under varying loads.

These tests often proceed in parallel since the embedded and cloud systems are typically developed independently. However, the real challenge, and value, comes during the system integration testing phase.

System Integration Testing: where it all comes together

Once both systems are developed and tested independently, they are brought together for system integration testing. This critical phase ensures that the embedded and cloud components interact as intended. Testing at this level focuses on validating:

• Data flows between the ECU and the cloud.

• Interaction patterns between the ECU and the cloud platform, e.g., via a REST API.

• Distributed business logic, ensuring decisions split across the embedded and cloud systems execute correctly.

This phase often involves HIL testing to replicate real-world conditions and assess how the ECU responds to cloud interactions in real-time.

HIL testing across levels

HIL testing is predominantly applied at the system and system integration levels of the testing pyramid. Here's how it contributes:

• System testing: HIL testing simulates real-world scenarios, allowing engineers to evaluate the ECU's functionality as it interacts with realistic, simulated environments. For example, it can assess how the ECU responds to complex situations like sensor faults or varying network conditions.

• System integration testing: At this level, HIL testing ensures the seamless collaboration between the embedded ECU and the cloud platform. It validates data flows, interaction patterns, and distributed business logic under realistic conditions, reducing reliance on expensive on-road testing.

Final thoughts

Understanding the test levels for hybrid embedded/cloud systems and dedicating adequate effort to both manual and automated testing throughout the development lifecycle is essential for building reliable and robust solutions. By thoroughly testing each system independently and integrating them effectively, teams can identify and resolve issues early, ensuring smooth real-world performance. Tools like HIL testing allow teams to simulate complex environments, validate real-time responses, and accelerate development cycles while maintaining high-quality standards.