Models Methods Software

Dan Hughes

V&V and NNSA Advanced Strategic Computing at LANL

[Updated September 26]
I think the name of the program originally known as the Advanced Strategic Computing Initiative ( ASCI ) is now known as Advanced Simulation & Computing ( ASC ).

[Updated August 26]

I have uploaded an excellent summary, developed by Los Alamos, of the ASC V&V Program at LANL. It is here.

A couple of quotes.

Confidence in simulation extrapolation comes via confidence in physics & numerics models, not calibration to experimental data.

Having “good agreement” between calculations and observations is not sufficient to establish scientifically credible predictive capability.

More and more it seems that the Climate Change Community remains the only holdout among all compute-intensive enterprises relative to application of rigorous, independent V&V to computer software.


It’s been a while since I looked into the general activities in IV&V in scientific and engineering computing. Today I did a quick search over at and got some good hits.

But first, relative to the two quotations in the original post, I failed to mention my usual mantra; Verification must always precede Validation. In the absence of Verification, Validation is not possible.

Reviewing today’s hits indicates to me that IV&V has been established to be a priority of the highest order, as it necessarily should be, in the Advanced Simulation & Computing Program (ASC) work underway at LANL (Los Alamos), LLNL (Livermore), and SNL (Sandia).

This Sandia report SAND 2008-5517 (Unlimited Release Printed January 2009) provides an extensive specification for the overall general SQA requirements. So far as I know there is no document of a similar nature for any piece of software in all the Climate Change Community.

This report, Enhanced Verification Test Suite for Physics Simulation Codes ( LA-14379 Issued: September 2008; aka LLNL-TR-411291; aka SAND2008-7813 Unlimited Release Printed April 2009 ) is an excellent report on Verification.

The report deals strictly with Verification and reiterates that Verification is:

Verification deals with mathematical correctness of the numerical algorithms in a code, while Validation deals with physical correctness of a simulation in a regime of interest. This document is about Verification.

The report open with this quotation:

In an age of spreading pseudoscience and anti-rationalism, it behooves those of us who believe in the good of science and engineering to be above reproach whenever possible. Public confidence is further eroded with every error we make… As Robert Laughlin noted in this magazine, ‘there is a serious danger of this power [of simulations] being misused, either by accident or through deliberate deception.’ Our intellectual and moral traditions will be served well by conscientious attention to verification of codes, verification of calculations, and validation, including the attention given to building new codes or modifying existing codes with specific features that enable these activities. Patrick Roache [Roa04]

where [Roa04] is this paper: Roache, P., “Building PDE Codes to be Verifiable and Validatable,” Comput. Sci. Engrng. 6, pp. 30–38 (2004).

The Introduction in this report provides excellent descriptions of the fundamental and critical necessity for Verification and the how-tos for implementing the Verification process. It should be required reading for everyone involved in production of software the results of which have the potential to influence the establishment of policies that affect the health and safety of the public. Reading the Introduction is a good way to get a firm handle on the concepts of Verification.

I would like to put all of the Introduction here, there is much good info in it, but it’s kind of long. I especially like this:

What is Verification?
Verification is the process of demonstrating that numerical solutions of the discretized algorithms in simulation software are the correct solutions of the corresponding continuum equations. Consequently, verification represents an important aspect of the development, assessment, and application of simulation software for physics and engineering. An essential element of the verification process is the quantitative analysis of simulation code performance on well-defined problems. The outcome of such analyses provides hard evidence of mathematical consistency between the mathematical statements of the physics models and their discrete analogues as implemented with numerical algorithms in the simulation codes.

and this part:

Code Verification
Verification can be summarized as the analysis of whether the numerical solutions of the discrete algorithms provide accurate solutions of the corresponding continuum equations. Distinct numerical schemes based on the identical continuum equations can produce radically different quantitative (and qualitative) results; therefore, while one may obtain nominally correct solutions of the discretized schemes, those results might be inaccurate solutions of the underlying continuum equations. Consequently, verification analysis constitutes a critically important aspect of the development, assessment, and application of simulation software for physics and engineering. It is important to distinguish between verification for the purpose of proving code correctness, and for providing algorithmic assessment; both activities have high value, but differ in details and tenor. An essential element of any verification process is the quantitative analysis of the simulation code performance on well-defined problems. The outcome of such analyses provides defensible evidence of mathematical consistency between the mathematical statements of the physics models and their discrete analogues as implemented with numerical algorithms in the simulation codes.


August 25, 2009 - Posted by | Calculation Verification, MMS, Verification | , , , , , ,


  1. Hi Dan,

    Thanks for the link to the ASC V&V program summary. It was very interesting. Is there any overview document like this that describes what the climate model ensemble IV&V program plan would need to look like — assuming the Climate Change Community finally decides to commit to IV&V?

    I am aware of your background article: Discussions of Application of Verification, Validation, and Quality Assurance Procedures to Climate Modeling Software. Have you done anything more detailed and specific? Perhaps something that discusses SQA remediation? (It is much more difficult to IV&V a software development process after-the-fact rather than during development.) Having some expertise in the area and some free time, I was thinking about investigating the issues myself.

    Comment by George Crews | August 25, 2009 | Reply

  2. Hello George,

    Appendix A in that document outlines a method that has been successfully applied, several times, to large, complex engineering software after-the-fact. The general concepts given there are made specific for the particular models and methods and applications under review. Because the results of applying the method to software are almost always considered to be Quality related items, they are Controlled documents and not usually available for distribution to outside parties.

    The documentation of the models, methods, coding, and applications are taken to be the specification documents for the software and IV&V tasks are developed from the table of contents of the documents. The specification documents are taken to, roughly, be the blueprints and the objective of the IV&V exercise is to determine that the as-built product corresponds to the blueprints. Unfortunately, many codes that have evolved over decades of time do not have sufficient documentation for the process to even get started.

    Let me know if you want to pursue this further; maybe I can offer additional information for a specific case.

    Comment by Dan Hughes | August 26, 2009 | Reply

  3. […] This previous post mentioned V&V and SQA at Los Alamos National Laboratory ( LANL ) within the framework of the Advanced Simulation & Computing ( ASC ) Program for the National Nuclear Security Administration ( NNSA ) […]

    Pingback by More on V&V and SQA at LANL for ASC and the NNSA « Models Methods Software | December 14, 2010 | Reply

  4. Marvelous, what a webpage it is! This blog provides helpful information to us,
    keep it up.

    Comment by Katharina | October 10, 2014 | Reply

  5. Please assistance us by finishing the quick survey.

    Comment by gt racing 2 hack | October 18, 2014 | Reply

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: