When traceability detects what verification does not detect

By Paul Graykowski, Arteris IP

During a complex design process, some errors will inevitably escape even the most expert system-on-chip (SoC) design teams. A disciplined V-model process (diagram below) will protect against many of these errors in the concept-to-design process (left arm) and in the verification-to-validation process (right arm). Within the industry, teams have developed significant automation tools and methodologies for the right arm. The existing process is very good at testing what designers have been instructed to build from design specs. There are also great point tools for the left arm. But the automation of the methodology has not been so strong because the system design tools and areas of use cover a wider range. The downward movement of the left arm still depends on manual translation from one step to another, with the risk of misunderstanding and forgetting. Overall, the V process is inefficient at testing system requirements up to the last right arm step – validation, although it is late enough to find basic errors.

Click to enlarge

The traceability of the path must follow, from system requirements to validation.

The Design Specification as an Oracle System

The tests depend on a reference specification which claims to offer a precise definition of one or more requirements. In testing jargon, this is called an oracle, an ultimate arbiter of truth. Oracles can be design or use case specifications or other concrete definition of requirements. The accuracy with which an oracle builder adheres to the original system’s goals depends on the precision, clarity, and definition of the input the designer receives. Naturally, a problem anywhere along this path will propagate. Worse still, he will broadcast with all the authority of an oracle. Every step beyond this point will see this flaw as a non-negotiable requirement until a wider range test hopefully reveals a disconnect.

It’s common in verification articles to talk about the cost of finding a bug increasing exponentially as it is discovered. The solution is always to shift to the left by doing more testing early – static verification, formal verification, emulation, etc. Speeding up all of these tests is important. Checkers detect disconnects between low-level oracles and the design, but cannot detect oracle issues. Problems typically come from two sources: ambiguity in natural language specifications and schedule pressure forcing incomplete or removed requirements. Whatever the cause, no amount of early checking will detect such issues.

Aids to verification of requirements Correct design by construction

W. Edwards Deming made this observation decades ago:Inspection to improve quality is too late, inefficient, costly. Quality does not come from inspection, but from improving the production process.Or in design terms, it’s better to design the first time accurately than to depend on verification to correct the implementation. Verification is always essential to confirm appropriate behavior. The same statement could be made for updates and bug fixes.

High-level design languages ​​are attracting some interest but are still low-level compared to ultimate system design and do not have a better connection to system-level requirements. A superior approach is to compare directly with requirements, minimizing error-prone intermediate levels of interpretation. Designers can compare next-higher-level requirements in the left arm, leveraging low-level details of implementation and intuition about higher-level system goals. This comparison encourages correct design by construction. Verifiers benefit equally and independently when creating tests. Both can tap into an oracle much closer to the original requirements when implementations are created and modified.

How can the SoC team know that oracles are reliable? The elaboration of requirements in the left arm of the V is ultimately what provides information on which designers will implement the SoC and on which verifiers will build their test suites. Traceability provides links from system specifications through low-level requirements to implementation, unit testing, and validation testing at the system and full system level. These links ensure that architects, designers, and reviewers can track each requirement to ensure it is interpreted correctly. This process provides more eyes on each oracle, design and verification artifact, and more opportunities to detect misinterpretations.

The Untapped Value of Traceability in Design, Verification and Validation

Traceability is most often associated with compliance testing for safety in automotive and other markets. But it can offer much more in the precision of implementing complex requirements in any area as a method to encourage correct design by construction. It also completes the left shift of the check with a focus on system specs. Requirements traceability puts oracles front and center for designers and verifiers. The verification guarantees the correctness of the design to these oracles.

You want to know more ? Discover the Harmony Trace product.