V&V and SQA: Part 3, Verification
Verification Activities
The focus of verification is the actual coding of the software with objectives to determine: (1) that the coding corresponds to the equations given in the specification document, (2) the order of accuracy of the numerical methods, and (3) the order of convergence of the numerical methods. In general, the latter two objectives are purely mathematical and go to the heart of the coding of the solution methods. Several of the procedures that are used to pursue these objectives are given in the following discussions.
Coding Review. A review of the coding and comparison of the equations in the code with the specifications provides one method of ensuring the correctness of the coding. In general, a line-by-line review of all the coding is not cost effective, and does not in itself ensure the correctness of the coding. A sampling of the coding, with the focus being on the more important routines and functions, is used instead. If the results of the sampled review indicate that the coding might contain extensive problems, the line-by-line review can be expanded to the extent necessary to determine the correctness of the coding. The expert judgement of the independent reviewers will be used to determine a level of confidence in the coding.
Additional approaches that complement and supplement code review are more useful and productive. Generally some of these methods are almost always applied during the development stages of the software by developers of the codes. Some calculation-based verification activities for complex software include:
The Method of Exact Solutions (MES). Calculations of highly specialized problems which have analytical solutions can be used to ensure that parts of the coding are correct. Various classes of exact solutions can be used including null tests, symmetry tests, and thought problems. Generally, however, problems having analytical solutions test only small, un-coupled, parts of the mathematical equations in the code. Analytical solutions for the intended application areas will generally not be available. For modern comprehensive models of complex real-world phenomena and processes, only a small fraction of the coding can be tested by this method.
Comparison with other Calculations. The results of the subject code can be compared with results from other codes. The software used to calculate the comparison, however, must have been subjected to V&V and be maintained under an approved SQA program. Again this is a very limited test and it is generally difficult to ensure that the codes are identical.
The Method of Manufactured Solutions (MMS). The Method of Manufactured Solutions has proven to be very effective relative to proving the verification of the code. Use of a manufactured solution, in contrast to an analytical solution, has the advantage that the solution can be designed to make all the terms in the equations important. For example, the MMS can be used to determine the presence of many common coding mistakes in the code. The MMS is used to demonstrate the order of accuracy of the numerical solution methods used in the code and the order of convergence for the discrete approximations employed. Comparison of the order of accuracy and order of convergence of the discrete approximations produced by the code with the theoretical values ensures verification of the coding of the numerical methods.
The importance of the Method of Manufactured Solutions relative to Verification can’t be over stated. This Google Scholar search will lead to several publications. The References given in Part 1 of these notes, and repeated at the end here can also be used as a starting point for finding additional information.
The Method of Exact Solutions and Method of Manufactured Solutions have the advantage that the results are quantitative in nature so that concrete numbers are determined by these methods.
Verification of Non-Functional Requirements. The structure and coding of the software can also be reviewed relative to non-functional requirements such as maintainability and extensibility. Again, a certain amount of expert judgement will necessarily be involved as a basis of the success metric for this objective.
Verification Technical Evaluation Criteria and Success Metrics
Verification is an independent, thorough and systematic effort to ensure that the computer program is performing correct calculations and processing data as specified in the code specification documents. The first step in this activity is to assure that all individual models and methods are completely and correctly documented. This step was discussed in the previous posts.
The next verification step is to ensure proper coding of each individual model. At the development and maintenance stages four methods of checking that models and methods have been coded correctly can be applied. These are:
1. An independent visual check of the coding by someone other than the original programmer.
2. Specifying input (driving the model) to produce known results.
3. Editing all required data entering a model and hand calculating the output.
4. Comparing results with output from other verified codes.
5. Application of the MES and MMS procedures to check the coding.
Independent verification beyond the development level can approach the coding on two levels; 1) an overview of an entire code, and 2) detailed review of the coding for specific models and methods. If the code is small, a detailed review of the entire code may be possible. The overall review can check for things such as undefined variables, undeclared externals, and compiler warnings and errors. The detailed coding review can focus on a line-by-line comparison of code with the equations given in the specification documents.
A criterion that can be used to determine that the coding is correct is:
Requirement II: Part 1. Verification Requirement. Is the coding correct with respect to the code specification document?
Technical Evaluation Criteria and Success Metric. A positive response to the following is required:
Does the coding correspond to the specifications for the software?
Specification Document. Volume 1 of the computer code manuals will be the specification document for this requirement.
Requirement II: Part 2. Verification Requirement. This requirement states that the numerical solution method must be shown to be consistent, stable, and convergent. For some solution methods, these characteristics can be demonstrated analytically. For most engineering software, however, demonstration of stability and convergence is accomplished by numerical experiments with the software. It can be very difficult to devise tests that will unequivocally prove stability and convergence of the solution method. However, it is possible to devise a set of calculations that will exercise the important aspects of the code and demonstrate that the basic solution method will produce stable and convergent results. The Method of Exact Solutions and the Method of Manufactured Solutions are important tools relative to testing these requirements.
Technical Evaluation Criteria and Success Metric. A positive response to one or more of the following is required:
a. Key parameters of a test calculation show decreasing variance as time and space increments are decreased (stability and convergence).
b. Test calculation results compare reasonably with known solutions as the temporal and spatial increments are refined.
c. Null transients, and other simple, degenerate test calculations are correct.
d. Results of using the MES and MMS verification methods agree with the theoretical expectations for the solution methods.
The null transient referred to above is a transient calculation of a problem with fixed boundary conditions that are consistent with the initial values in the calculational domain. Such a calculation should remain null for as long as it is executed. Another simple, degenerate test calculation is a check for symmetry in calculated results if the domain grid or the boundary conditions are changed between two calculations. These simple tests are useful for locating major errors in the coding.
Specification Document. The numerical solution methods as given in Volume 1 of the code manuals are considered the specifications.
Requirement II: Part 3. Verification Requirement. The code is correctly solving the equation set of the models. This requirement is easily satisfied if analytical solutions for all the model equations are available. Usually, for real-world models of inherently complex physical phenomena and processes, only a very specialized subset of the general equations can be solved analytically.
Results of calculations of other approved codes that are applicable to the physical problem of interest, especially other calculations of rigorous benchmark problems, can also be used to verify that the code is correctly solving the model equations. And, again, the MES and MMS procedures can be used to determine that the equations are correctly solved.
Technical Evaluation Criteria and Success Metric. A positive response to one or more of the following is required:
a. The calculated results are in agreement with analytical solutions.
b. The calculated results are in agreement with other calculated results.
c. The calculated results are correct based on the expertise and scientific and engineering judgment of the verification team.
d. Calculated results using the MES and MMS procedures show that the equations are solved correctly.
In addition to verification with analytical MES solutions and MMS, the discreet form of the model equations can be verified as follows. All the variables in the discreet equations can be printed out and the discreet form of the equations verified by checking that the printed values satisfy the equations. For a transient code, the variables can be printed at two successive time steps and the time-step form of the equations verified. For scalar equations such as mass and energy conservation this is a relatively straightforward process. Vector equations such as momentum balances require a lot of information from several spatial locations. While somewhat more complicated than scalar equations, the discreet form of momentum balances can be verified.
Specification Document. Volume 1 of the computer code manuals, the Model Theory and Solution Methods Manual contains the equations that are coded and will be the specification document for this requirement.
Verification of Non-Functional Requirements. The structure and coding of the software can also be reviewed relative to non-functional requirements such as maintainability and extensibility. Again, a certain amount of expert judgement will necessarily be involved as a basis of the success metric for this objective.
References
A basic reference is the recent book by Pat Roache. For those who might want to do some Googles, Pat is P. J. Roache, or Patrick Roache or Patrick J. Roache. His book is available from Hermosa Publishers and Amazon. A First Edition of the book is also available. Additional references can be found by search at www.osti.gov using Oberkampf as AUTHOR and SAND as Identifier Numbers. The group at Sandia National Laboratories has been especially active in the work. Many of the reports by Sandia will be available for download from the OSTI site. Additional information relative to applications of the concepts will be found by searching OSTI All Fields for ‘ASC verification validation SQA’. And by Google Scholar searches using authors and keywords from these sources.
Appropriate use of MMS in the above searches will prove to be very useful for those looking for additional information.
No comments yet.
Models Methods Software 
Leave a Reply