EPA HQ and Software Quality Assurance
I sent a slightly revised copy of the comment on the Proposed CO2 Ruling, given in this post, to Ms. Lisa jackson, Administrator of EPA. The letter to Ms. jackson is shown below. I received a reply from Rona Birnbaum, Chief, Climate Science & Impacts Branch, Climate Change Division. (The actual signing of the letter was a task delegated to someone whose name I can’t read.)
As in all previous cases in which I have attempted to convey the critical necessity of Independent Verification and Validation to persons outside the software development community, I failed again. It is obvious that Ms. jackson, Rona Birnbaum, or whoever read and responded to the letter, have no idea what I’m talking about.
Here is my letter to Ms. Jackson:
April 27, 2009
Ms. Lisa Jackson, Administrator
Environmental Protection Agency
Ariel Rios Building
1200 Pennsylvania Avenue NW
Washington, DC 20460.
SUBJECT: Software Verification and Software Quality Assurance
Dear Ms. Jackson:
I have over 40 years experience in development of mathematical models of physical processes in the multi-phase thermal sciences, analytical and numerical solution methods for these models, and incorporation of these into large, complex computer codes designed for applications to complex engineering problems. I have developed software with the BASIC, FORTRAN, F77, and F90/F95 computer languages, among others, on machines using the Unix, Windows NT, and Mac OS operating systems. I have an extensive and continuous peer-reviewed publication and presentation record and have made major contributions to writing computer code manuals. My experience and expertise have direct relationships to development of almost all engineering and scientific models, methods, and software.
Recently my work has focused on documentation and Verification and Validation of software maintained under software quality assurance (SQA) plans. I will focus on related aspects of the Proposed Findings of Docket ID No. EPA-HQ-OAR-2008-0508.
The Intergovernmental Panel for Climate Change (IPCC) Fourth Assessment Report (AR4, IPCC, 2007) states:
“A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.” I have added the emphasis on the models.
Do the numbers from these “large number of simulations available from a broader range of models” GCM calculations have any meaning. My answer is No.
One crucial and necessary first step is that application of Verification procedures have shown that the numbers produced by the software accurately reflect both (1) the original intent of the continuous equations for the models, and (2) the numerical solution methods applied to the discrete approximations to the continuous equations. That is, Verification shows that the equations have been solved correctly. Verification procedures are designed to answer the question, Do the calculated numbers actually satisfy the coded discrete equations and do the solutions of the discrete equations converge to solution of the continuous equations. Neither of these extremely critical properties has been demonstrated for any GCM. None of the GCM codes, and very likely none of any of the enormous number of other computer codes, used in the IPCC processes have been Verified to be correct. Equally important, none of the Journals in which the papers reviewed by the IPCC process are published have editorial policies that require that the software on which papers are based to be Verified.
All software can be Verified. Objective technical criteria and associated success metrics can be developed and applied in a manner that provides assurances about the correctness of the coding of the equations and their numerical solutions. Lack of Verification leaves open the potential that the numbers from the software are simply results of “bugs” in the coding.
The present-day software development community, in all kinds of applications and organizations, is keenly aware that lack of SQA policies and procedures, and successful applications of these to the software, leaves open a significant potential for problems to exist in the software. The computer models and associated software developed for analyses of the Earth’s climate seem to be the sole exception to the understanding of the absolute necessity for application of Verification and SQA procedures to these very complex and extremely important issues. The more complex the physical situation and thus the computer codes, the more likely that bugs are present. Likewise, the more critical the decision that affects the health and safety of the public, the more important that software be Verified.
The situation in CO2 and Climate Science is in stark contrast to that in several engineering societies. These professional societies and their Journal editorial boards have put into place technical requirements on the Verification of the software before the paper can be considered for publication. If the requirements have not been met the paper will not be published; in some cases the paper will be rejected out-of-hand and not be sent out for review. Papers for which the basis is a single calculation on a single grid with no investigations of convergence and other stopping criteria are typically sent back to the authors.
Some of these professional organizations and associated Journals include: The American Society of Mechanical Engineers (ASME) Journal of Heat Transfer and Journal of Fluids Engineering; The American Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft and Rockets; and the International Journal of Numerical Methods for Fluid Flow. Other professional societies and journals are sure to follow the lead of these. References for the editorial polices for these journals are as follows.
The ASME Journal of Heat Transfer, Editorial Board, “Journal of Heat Transfer Editorial Policy Statement on Numerical Accuracy,” ASME Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.
The ASME Journal of Fluids Engineering, C. J. Freitas, “Editorial Policy Statement on the Control of Numerical Accuracy,” ASME J. Fluids Eng., 115, pp. 339–340, 1993. http://tinyurl.com/9fs98 and J. Fluids Eng. 130, 2008. http://timyurl.com/8phpar
The AIAA Journal of Spacecraft and Rockets, AIAA, Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty, AIAA Journal, Vol. 32, No. 1, p. 3, 1994.
The International Journal of Numerical Methods in Fluids, P. M. Gresho and C. Taylor, “Editorial,” International Journal of Numerical Methods in Fluids, Vol. 19, p. iii, 1994.
The results of my direct correspondence about these matters with several of the US agencies and personnel involved in GCM development and applications, and analyses of CO2 and Climate Science, with a strict focus on software Verification and SQA, have shown that Verification and SQA are not considered necessary aspects of any analyses of these important issues. My correspondence has been with appropriate personnel at NASA and GFDL, among others. Correspondence with editors of several of the high-impact Journals in Climate Science, Journals such as Nature, Science, and those published by the AGU, AMS, AAAS, for examples, indicate the same results. These Journals do not have Verification and SQA requirements in place for the papers submitted for publication. This is not an acceptable position for Journals the intent of which is to publish archival papers.
I think it is certain that the EPA has in place mandatory Verification and SQA requirements for all the software applied to all analyses of all other matters that come before the Agency. So far as I am aware, there are no precedents whatsoever for public policy decisions to be based on software for which no Verification and SQA procedures have been applied.
The EPA should not allow decisions on probably the most important issue to come before the Agency to set this type of precedent. No other Agency of the US Government, none whatsoever, has ever allowed and will never allow such a precedent to be set. It is simply unthinkable.
Thank you for your attention to these matters.
Sincerely,
Dan Hughes
Here is the EPA response:
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
WASHINGTON, D.C. 20460
OFFICE OF
AIR AND RADIATION
Mr. Dan Hughes
324 Plank Road
Porter Corners, New York 12859
Dear Mr. Hughes:
Thank you for your letter dated April 27, 2009 to Administrator Lisa Jackson concerning software verification and software quality assurance. My office has been asked to respond. I appreciate your interest in this matter.
Referencing the Environmental Protection Agency’s (EPA’s) Mandatory Reporting of Greenhouse Gases; Proposed Rule (Docket ID NO. EPA-HQ-OAR-2008-0508), your letter argues that Global Circulation Model (GCM) calculations are flawed. First, the GCMs EPA has relied on for technical analysis of climate change have their foundation in widely accepted physical principles. Second, they have an ability to accurately reproduce key features of past and current climate. Third, they have undergone an extensive peer-review process and been validated by numerous scientific bodies. For all these reasons, GCMs that have been thoroughly vetted by the scientific community provide valuable and insightful information about climate change.
While serving as a highly useful tool. GCM results constitute just one part of the large and growing body of scientific evidence about climate change that is guiding EPA’s evolving policy response to the problem. Additional strands of data EPA uses to guide policy include observed changes in temperature, precipitation and other key climatic features, as well as observed impacts of climate change on terrestrial and oceanic systems.
For more information about the strengths and limitations of climate models, please see “Climate Models: An Assessment of Strengths and Limitations.” This report was published in 2008 by the U.S. Climate Change Science Program and is accessible from http://www.climatescience.gov/Library/sap/sap3-l/final-report/default.htm. If you would like for your letter to be considered as an official comment on the proposed Mandatory Reporting rule or would like to submit additional comments, please follow the instructions for submitting written comments available at http://www.epa.gov/climatechange/emissions/ghgrulemaking.html. lf you would like to make an official comment EPA’s Proposed Endangerment Cause or Contribute Findings for Greenhouse Gases Under Section 202(a) of the Clean Air Act (Docket ID N0. EPA-HQ-OAR-2009-0171), please follow the instructions available at http://www.epa.gov/climatechange/endangerment.html.
Again, thank you for your letter. I appreciate the opportunity to be of service and trust the information provided is helpful.
Sincerely,
Rona Birnbaum
Chief, Climate Science & impacts Branch
Climate Change Division
Models Methods Software 
I’d guess that they know exactly what you’re talking about…they just don’t care.
1. “Widely accepted physical principles” — many of which relating to climate they still don’t know how to model
2. “ability to accurately reproduce key features of past and current climate” — anyone can replicate the past with enough degrees of freedom. So what? And that reproduction of the present isn’t going so well.
3. “they have undergone an extensive peer-review process and been validated by numerous scientific bodies” — given how worthless peer-review is, is that a plus? And apparently these scientific bodies have managed to validate the models without, you know, actually validating the models.
What a worthless pile of garbage. Their response actually demonstrates how weak their case is. They’d have been better off to ignore you.