V&V and SQA: Part 1, Definitions and Characterization
I’m going to post a series of short summaries of some of the central aspects of verification, validation ( V&V ) and software quality assurance ( SQA ) for production-grade computer software. These subjects have received significant investigations starting in the 1980s ( more or less ) and have reached maturation and have been successfully applied to a wide range of scientific and engineering software. I don’t intend to give a complete exposition of the subjects, the field is much too big.
This initial post will provide working definitions of the concepts and characterization of the components of the software of interest.
References
A basic reference is the recent book by Pat Roache. For those who might want to do some Googles, Pat is P. J. Roache, or Patrick Roache or Patrick J. Roache. His book is available from Hermosa Publishers and Amazon. A First Edition of the book is also available. Additional references can be found by search at www.osti.gov using Oberkampf as AUTHOR and SAND as Identifier Numbers. The group at Sandia National Laboratories has been especially active in the work. Many of the reports by Sandia will be available for download from the OSTI site. Additional information relative to applications of the concepts will be found by searching OSTI All Fields for ‘ASC verification validation SQA’. And by Google Scholar searches using authors and keywords from these sources.
These references will be applicable for all the posts.
Background and Definition of Terms
See the references above for complete discussions of all aspects of the subjects. A short summary is given in the following paragraphs.
Generally, computer software designed for applications to inherently complex multi-physics, multi-scale physical phenomena and processes is itself somewhat complex; coupling in the physical domain leads to coupling in the coding domain. Care in the use of the codes and interpretations and applications of the results from the code are critically important considerations. Users of such complex software must generally be highly trained and qualified and certified to apply it to analyses of interest. Training for users is usually the work of the code developers and the expert users among the application community.
The V&V and SQA activities discussed in these notes is a formal process. The primary product of the process is documentation that the process has been applied to the software. The V&V and SQA documentation must be in a format that allows any organization to easily, (1) verify that the process has been correctly applied to the software and, (2) understand the conclusions and results obtained from applying the V&V and SQA processes to the software.
Four important characteristics are associated with formal V&V and SQA procedures that will differentiate formal processes from similar activities that usually occur during software development and application.
The first characteristic is independence of the group that performs V&V and SQA from the group that develops the software.
The second is the systematic nature of formal V&V and SQA procedures.
The third is the documentation level of a formal process.
And the forth characteristic is the extensive application of the processes and procedures to all aspects of the software; from basic differential equation modeling to applications of the software to the areas for which it was designed.
The products of the independent V&V and SQA review will ensure that results obtained with the software can be reproduced by parties not associated with the software. Independent V&V and SQA will also help ensure that results of application of the software to its intended analyses are reliable and sufficiently representative of the physical phenomena and processes modelled by the software.
A complete and detailed discussion of the extensive procedures and processes that have evolved into modern V&V and SQA in use today are beyond the scope of these discussions. A rough outline and brief descriptions are as follows ( see Roache for more detailed discussions ):
1. Construction and verification of the specification documents for the software and its intended areas of applications.
2. Verification of the software coding; a software quality assurance (SQA) activity.
3. Verification of the solution methods used in the codes, including the order of accuracy and convergence of numerical methods applied to the discrete equations in the code; code verification.
4. Verification of calculations by the software; calculation verification.
5. Validation, at several levels of the modeling for intended applications; individual model validation, subsystem validation, and top-level system validation.
6. Qualification of the software for use in its intended application areas.
7. Qualification of the users of the software.
8. Quantification of the uncertainties in both the validation experimental data bases and the code-calculated results.
9. Certification and accreditation of the software and its users.
10. Quality assurance procedures for maintenance of the software.
All of these, and more, have been the subjects of investigations and developments relative to software verification and validation. The book by Roache is an excellent single-source reference for additional information on most of these topics.
The focus of these notes are Items (1), (2), (5) and (10). These are good starting points for software that has yet to undergo V&V and SQA. We start the discussions with definitions of the terms as they will mean in this report. These definitions are given within the framework of activities associated with the mathematical models and methods in computer software. Various nomenclature and meanings continue to be in use in the literature; the following are the meanings in these notes.
A brief description of Verification is:
Verification is a procedure for ensuring that the equations are solved correctly.
And of Validation:
Validation is the process of ensuring that the correct equations have been solved.
And Software Quality Assurance:
Are methods and procedures to ensure that the code and its applications maintain verification and validation status throughout its life cycle.
The primary focus areas of verification are the correctness and accuracy of solution methods of the mathematical models and the coding of these. Verification is primarily a mathematical procedure. The primary focus of validation are the physical application areas for which the software is intended to be used. Validation is primarily concerned with modeling of physical phenomena and processes and experimental data about these. These terms have been identified in recognition of the facts that problems may be present in either of the focus areas; bugs in the software coding and oversights in the development of the models and methods.
Verification can be understood in the sense that we are all familiar with the difficulties associated with coding correctly any system of equations and their solution methods. Generally, verification is a procedure that is applied each time changes are made to the models, methods and application procedures.
Validation can be understood in the sense that the equations of the models and methods can be programmed and solved correctly and accurately but the correct equations for the models of physical phenomena and processes might not have been chosen. Generally, validation is an on-going process over the lifetime of the software.
In general, verification, validation, and software quality assurance procedures are designed to ensure that software is error-free and that it is applicable to the application areas for which it has been designed. This assurance must be attained before uncertainties associated with production-level applications of the software can be determined by calculations in the intended application areas. Otherwise, such uncertainties may be due to errors in the coding, the basic mathematical model, or the numerical solution methods, or inapplicability of the software to the problem areas of interest. It cannot be overly stressed that these activities must always precede applications of the software.
Characterization of the Software
Software for real-world complex phenomena and processes is generally comprised of the following models and methods components:
1. Fundamental basic model equations from continuum mechanics such as the Navier-Stokes for mass, momentum and energy conservation, heat conduction, radiative energy transport, chemical-reaction laws, Boltzmann, and many others. These include the constitutive equations for the behavior and properties of the associated materials; equation of state, thermo-physical and transport properties and basic material properties.
2. Engineering models and empirical correlations of experimental data needed to close the basic model equations; heat transfer and friction factor correlations, mass exchange coefficients, for examples.
3. Special purpose models for phenomena and processes that are too complex or insufficiently understood, or computational too expensive, to model from basic principles.
4. Models for phenomena and processes occurring in complex engineering equipment, if a physical system of interest includes hardware. Note that these might include elements of Items 1 through 3, above.
5. Analytical and numerical solution methods for all the equations that comprise the models.
6. Auxiliary functional methods for installation, pre-processing code input and post-processing output, analyses of calculated results, and other user-aids.
7. Non-functional aspects of the software include its ease of, or fitness for, maintainability, extensibility and portability.
All of the above are generally incorporated into computer software for use and application to the analyses for which the models and methods were designed to be applied. For real-world models of inherently complex physical phenomena and processes the software itself will generally be complex and somewhat difficult to accurately apply and the calculated results somewhat difficult to understand.
Note that Item 1 above is a complete characterization of the continuous equations for a computational physics problem. The equation system will be closed. The material and its behavior are described without recourse to descriptions of previous states that the material has attained.
The resulting equations that are used to model the physical phenomena and processes usually form a large system of coupled, non-linear partial and ordinary differential equations plus a very large number of algebraic equations. Almost all complex physical phenomena are non-linear with a multitude of temporal and spatial scales ( multi-scale ) and interactions ( multi-physics ).
Generally, numerical solution methods via finite-difference, finite-element, spectral, and other discrete-approximation approaches, are about the only alternative for solving the system of equations. These approximations applied to the continuous and algebraic equations of the model will give systems of coupled, nonlinear algebraic equations which are enormous in size.
The objectives of V&V and SQA activities are to ensure that: (1) all of the above are correctly coded into the software, (2) the models and methods are applicable to the application areas for which the software was designed, and (3) production-grade software is rigorously maintained under SQA procedures.
The V&V activities and SQA procedures are based upon a set of requirements that must be met for release of computer software for production use. The activities are directed toward ensuring that sufficient, and accurate, information is available for independent replication of important results. The requirements cover the areas of documentation, verification, validation, and quality assurance.
The requirements for release of software to production applications under each of these categories will be the subjects of this series of posts. Technical evaluation criteria and associated success metrics are given along with presentation of the requirements.
No comments yet.
Models Methods Software 
Leave a Reply