Embedded Testing

Six Aspects of Testing Complex Embedded Systems

Depending on the type of the granule, and according to common usage in the industry, it is common to consider testing from six different aspects to evaluate whether the application's behavior, performance, and robustness match expected criteria.

These aspects are:

  1. Software unit testing

  2. Software integration testing

  3. Software validation testing

  4. System unit testing

  5. System integration testing

  6. System validation testing

What follows is a discussion of each of these six aspects in relation to testing complex embedded systems.

1. Software Unit Testing

The GuT is either an isolated C function or a C++ class. Depending on the purpose of the GuT, the test case consists of either:

  • Data-intensive testing: applying a large range of data variation for function parameter values, or

  • Scenario-based testing: exercising different C++ method invocation sequences to perform all possible use cases as found in the requirements.

Points of Observation are returned value parameters, object property assessments, and source code coverage. White-box testing is used for testing units, meaning that the tester must be familiar with the content of the GuT. Unit testing is thus the responsibility of the developer. Since it is not easy to track down trivial errors in a complex embedded system, every effort should be made to locate and remove them at the unit-test level.

2. Software Integration Testing

The GuT is now a set of functions or a cluster of classes. The essence of integration testing is the validation of the interface. The same type of Points of Control applies as for unit testing (data-intensive main function call or method-invocation sequences), while Points of Observation focus on interactions between lower-level granules using information flow diagrams.

As soon as the GuT starts to be meaningful, that is when an end-to-end test scenario can be applied to the GuT. First, performance tests can be run that should provide a good indication about the validity of the architecture. As for functional testing, the earlier the better. Each forthcoming step will then include performance testing.White-box testing is also the method used during that step. Software integration testing is the responsibility of the developer.

3. Software Validation Testing

The GuT is all the user code inside a component. This includes all the code developed by the team, but only the code itself. This aspect of testing can be considered one of the activities that occur toward the end of each software integration or build cycle. Partial use-case instances -- also called partial scenarios -- begin to drive the test implementation. The test implementation is less aware of and influenced by the implementation details of the GuT. Points of Observation include resource usage evaluation since the GuT is a significant part of the overall system. Again (and finally), we consider this step as white-box testing. Software validation testing is still the responsibility of the developer.

4. System Unit Testing

The GuT is now a full system component -- that is, the user code as tested during software validation testing plus all RTOS- and platform-related pieces that allow the code to run at minimum on a simulated target: tasking mechanisms, communications, interrupts, and so on. The Point of Control protocol is no longer a call to a function or a method invocation, but rather a message sent/received using the RTOS message queues, for example.

The similarities in their message-passing paradigms implies that the distinction between test drivers and stubs can, from the perspective of the GuT, be considered irrelevant at this stage. We will call them Virtual Testers because each one can replace and act as another system component vis-?is the GuT. "Simulator" or "tester" are synonyms for "virtual tester." Virtual tester technology should be versatile enough to adapt to a large number of RTOS and networking protocols. From now on, test scripts usually: bring the GuT into the desired initial state; then generate ordered sequences of samples of messages; and validate messages received by comparing (1) message content against expected messages and (2) date of reception against timing constraints. The test script is distributed and deployed over the various virtual testers. System resources are monitored to assess the system's ability to sustain embedded system execution. For this aspect, grey-box testing is the preferred testing method. In most cases, only a knowledge of the interface (the API) to the GuT is required to implement and execute appropriate tests. Depending on the organization, system unit testing is either the responsibility of the developer or of a dedicated system integration team.

5. System Integration Testing

The GuT starts from a set of components within a single node and eventually encompasses all system nodes up to a set of distributed nodes. The PCOs are a mix of RTOS- and network-related communication protocols, such as RTOS events and network messages. In addition to a component, a Virtual Tester can also play the role of a node. As for software integration, the focus is on validating the various interfaces. Grey-box testing is the preferred testing method. System integration testing is typically the responsibility of the system integration team.

6. System Validation Testing

The GuT is now a complete implementation subsystem or the complete embedded system. The objectives of this final aspect are several:

  • Meet external-actor functional requirements. Note that an external-actor might either be a device in a telecom network (say if our embedded system is an Internet Router), or a person (if the system is a consumer device), or both (an Internet Router that can be administered by an end user).

  • Perform final non-functional testing such as load and robustness testing. Virtual testers can be duplicated to simulate load, and be programmed to generate failures in the system.

  • Ensure interoperability with other connected equipment. Check conformance to applicable interconnection standards.

Deciding How to Address These Aspects

Now that we have defined these six aspects, how do we use them? There are plenty of criteria that are used to decide:

  • Whether all these aspects apply to your systems.

  • Whether all the aspects should be performed for all or only some parts of the systems.

  • The order in which the aspects should be applied to the selected parts.

The answers to these questions depend heavily on the kind of embedded system you are developing: safety-critical or not, time-to-market, few or millions deployed, and so on.

Additional Requirements for a Complex Systems Testing Technology

In order to address the challenge of testing complex embedded systems, the testing technology must add the following capabilities:

  • Manage multiple types of Points of Control in order to stimulate the GuT. Do this using: function call; method invocation; and message passing or RPC though different language bindings such as Ada, C, C++ or Java.

  • Offer a wide variety of Points of Observation such as: parameters and global variable inspections; assertion, information, and control path tracing; and code coverage recording or resource usage monitoring. Each of these Points of Observation should provide expected vs. actual assessment capabilities.