EPA Docket ID No. EPA-HQ-OAR-2008-0508
My contribution to the comments regarding the Proposed Findings by the EPA regarding CO2 are given in this post. I will send hard copy of the letter to the EPA.
April 27, 2009
Environmental Protection Agency
Attention Docket ID No. EPA-HQ-OAR-2008-0508
EPA Docket Center (EPA/DC), Mail Code 6102T
1200 Pennsylvania Avenue, NW
Washington, DC 20460.
SUBJECT: Proposed Findings Docket ID No. EPA-HQ-OAR-2008-0508
Madams and Sirs:
I have over 40 years experience in development of mathematical models of physical processes in the multi-phase thermal sciences, analytical and numerical solution methods for these models, and incorporation of these into large, complex computer codes designed for applications to complex engineering problems. I have developed software with the BASIC, FORTRAN, F77, and F90/F95 computer languages, among others, on machines using the Unix, Windows NT, and Mac OS operating systems. I have an extensive and continuous peer-reviewed publication and presentation record and have made major contributions to writing computer code manuals. My experience and expertise have direct relationships to development of almost all engineering and scientific models, methods, and software.
Recently my work has focused on documentation and Verification and Validation of software maintained under software quality assurance (SQA) plans. I will focus on related aspects of the Proposed Findings of Docket ID No. EPA-HQ-OAR-2008-0508.
The Intergovernmental Panel for Climate Change (IPCC) Fourth Assessment Report (AR4, IPCC, 2007) states:
“A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.” I have added the emphasis on the models.
Do the numbers from these “large number of simulations available from a broader range of models” GCM calculations have any meaning. My answer is No.
One crucial and necessary first step is that application of Verification procedures have shown that the numbers produced by the software accurately reflect both (1) the original intent of the continuous equations for the models, and (2) the numerical solution methods applied to the discrete approximations to the continuous equations. That is, Verification shows that the equations have been solved correctly. Verification procedures are designed to answer the question, Do the calculated numbers actually satisfy the coded discrete equations and do the solutions of the discrete equations converge to solution of the continuous equations. Neither of these extremely critical properties has been demonstrated for any GCM. None of the GCM codes, and very likely none of any of the enormous number of other computer codes, used in the IPCC processes have been Verified to be correct. Equally important, none of the Journals in which the papers reviewed by the IPCC process are published have editorial policies that require that the software on which papers are based to be Verified.
All software can be Verified. Objective technical criteria and associated success metrics can be developed and applied in a manner that provides assurances about the correctness of the coding of the equations and their numerical solutions. Lack of Verification leaves open the potential that the numbers from the software are simply results of “bugs” in the coding.
The present-day software development community, in all kinds of applications and organizations, is keenly aware that lack of SQA policies and procedures, and successful applications of these to the software, leaves open a significant potential for problems to exist in the software. The computer models and associated software developed for analyses of the Earth’s climate seem to be the sole exception to the understanding of the absolute necessity for application of Verification and SQA procedures to these very complex and extremely important issues. The more complex the physical situation and thus the computer codes, the more likely that bugs are present. Likewise, the more critical the decision that affects the health and safety of the public, the more important that software be Verified to the highest standards.
The situation in CO2 and Climate Science is in stark contrast to that in several engineering societies. These professional societies and their Journal editorial boards have put into place technical requirements on the Verification of the software before the paper can be considered for publication. If the requirements have not been met the paper will not be published; in some cases the paper will be rejected out-of-hand and not be sent out for review. Papers for which the basis is a single calculation on a single grid with no investigations of convergence and other stopping criteria are typically sent back to the authors.
Some of these professional organizations and associated Journals include: The American Society of Mechanical Engineers (ASME) Journal of Heat Transfer and Journal of Fluids Engineering; The American Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft and Rockets; and the International Journal of Numerical Methods for Fluid Flow. Other professional societies and journals are sure to follow the lead of these. References for the editorial polices for these journals are as follows.
The ASME Journal of Heat Transfer, Editorial Board, “Journal of Heat Transfer Editorial Policy Statement on Numerical Accuracy,” ASME Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.
The ASME Journal of Fluids Engineering, C. J. Freitas, “Editorial Policy Statement on the Control of Numerical Accuracy,” ASME J. Fluids Eng., 115, pp. 339–340, 1993.
http://tinyurl.com/9fs98 and J. Fluids Eng. 130, 2008. http://timyurl.com/8phpar
The AIAA Journal of Spacecraft and Rockets, AIAA, Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty, AIAA Journal, Vol. 32, No. 1, p. 3, 1994.
The International Journal of Numerical Methods in Fluids, P. M. Gresho and C. Taylor, “Editorial,” International Journal of Numerical Methods in Fluids, Vol. 19, p. iii, 1994.
Correspondence with editors of several of the high-impact Journals in Climate Science, Journals such as Nature, Science, and those published by the AGU, AMS, AAAS, for examples.
The results of my direct correspondence about these matters with several of the US agencies and personnel involved in GCM development and applications, and analyses of CO2 and Climate Science, with a strict focus on software Verification and SQA, have shown that Verification and SQA are not considered necessary aspects of any analyses of these important issues. My correspondence has been with appropriate personnel at NASA and GFDL, among others.
I think it is certain that the EPA has in place mandatory Verification and SQA requirements for all the software applied to all analyses of all other matters that come before the Agency. So far as I am aware, there are no precedents whatsoever for public policy decisions to be based on software for which no Verification and SQA procedures have been applied.
The EPA should not allow decisions on probably the most important issue to come before the Agency to set this type of precedent. No other Agency of the US Government, none whatsoever, has ever allowed and will never allow such a precedent to be set. It is simply unthinkable. Consider the potential consequences if the FDA, FFA, NRC, among many of others, allowed such an unprofessional approach to computer software to be introduced into the decisions for which they are responsible.
Thank you for your attention to these matters.
Yours truly,
Dan Hughes
Models Methods Software 
[…] sent a slightly revised copy of the comment on the Proposed CO2 Ruling, given in this post, to Ms. Lisa jackson, Administrator of EPA. The letter to Ms. jackson is shown below. I received a […]
Pingback by Models, Methods, Software » Blog Archive » EPA HQ and Software Quality Assurance | June 10, 2009 |