Lack of Transparency and some Vital Technical Issues
I have posted several short discussions of software Verification, Validation (V&V) and Software Quality Assurance (SQA) issues a few times at several blogs. I continue to hope that more complete discussions of significant software engineering issues will follow. So far that has not happened. I have made other direct contacts too, but also without success.
Documentation of the models, methods, and software is the key focus relative to V&V and SQA . My research is conducted mainly on the Web. It is entirely possible that I have missed what I’ve been looking for. But no one has yet provided the information that I know is needed in order to provide creditability to projections/predictions from any and all computer software.
All these codes, and all other aspects of all the software used in the climate-change community, belong to us. The continued lack of open documentation is an affront to all of us who have paid for these products and their applications. The climate-change community must open the documentation windows and let some sunshine in.
I have a few questions for any who have answers. I consider these issues to be essentially show-stoppers as far as use of the results of any of the AOLGCM codes and all supporting codes use in all aspects of climate-change analyses, for either (1) archival peer-reviewed publications, (2) providing true insight into the phenomena and processes that are modeled, and most importantly (3) for decision-making relative to public policies. Any and all professional software developers would absolutely require that all of the issues to be mentioned below be sufficiently addressed and documented before using any software for applications in the analyses areas for which it was designed.
Documentation
In no particular order, as each of the following is very important, can anyone provide documented information about:
(1). Audited Software Quality Assurance (SQA) Plans for any of the computer software that is used in all aspects of climate-change analyses.
(2) Documentation of Maintenance under audited and approved SQA procedures of the ‘frozen’ versions that are used for production-level applications.
(3) Documentation of the Qualifications of the users of the software to apply the software to the analyses that they perform.
(4) Documentation of independent Verification that the source coding is correct relative to the code-specification documents.
(5) Documentation of independent Verification that the equations in the code are solved correctly and the order of convergence of the solutions of the discrete equations to the continuous equations has been determined.
(6) Sufficient information from which the software and its applications and results can be independently replicated by personnel not associated with the software.
(7) Documentation that shows that the codes always calculate physically realistic numbers. For example, the time-rate-of-change of temperature, say, is always consistent with the energy equations and is not the results of numerical instabilities or other numerical solution methods problems.
( 8 ) Documentation in which the mathematical properties (characteristics, proper boundary condition specifications, well- (or ill-) posedness, etc.) of all the continuous equations used in a code have been determined. Do attractors exist, for example.
(9) Documentation in which it has been shown analytically that the system of continuous equations used in any AOLGCM model has the chaotic properties that seem to be invoked by association and not by complete analysis. Strange-looking output from computer codes does not prove that the system of continuous equations possess chaotic characteristics. Output from computer codes might very likely be results of modeling problems, mistakes, solution errors, and/or numerical instabilities.
Invoking/appealing-to an analogy to the Lorenz continuous equations is not appropriate for any other model systems. The Lorenz model equations are a severely truncated approximation of an already overly simplified model. The wide range of physical time constants for the climate system, and potential phase errors in the numerical solutions, almost guarantees that aperiodic behavior will be calculated.
Especially true considering the next item.
(10) Documentation in which it has been determined that the discrete equations and numerical solution method are consistent and stable and thus the convergence of the solution of the discrete equations to the continuous equations is assured. Actually I understand that the large AOLGCM codes are known to be unable to demonstrate independence of the discrete approximations used in the numerical solution methods. The calculated results are in fact known to be functions of the spatial and temporal representations used in the numerical solutions. This characteristic proves that convergence cannot be demonstrated. Consistency and stability remain open questions.
(11) Documentation in which it is shown that the models/codes/calculations have been Validated for applications to the analyses for which it has been designed.
All software, each and every one, that is used for analyses of applications the results of which might influence decisions that affect the health and safety of the public will have addressed all these issues in detail.
If my understanding of the status of these critical issues is correct I can only conclude:
(1) The software used in the climate-change community does not meet the most fundamental requirements of software used in almost all other areas of science and engineering. Almost none of the basic elements of accepted software design and applications for production-level software are applied to climate-change software.
(2) The calculated results cannot be accepted as correct and valid additions to the peer-reviewed literature of technical journals.
(3) The software should never be used in attempts to predict the effects of changes in public policy (fuel sources for energy production, say) on the climate; neither short- or long-range.
(4) The calculated results are highly likely not correct relative to physical reality.
I will say that item (10) is in fact a totally unacceptable characteristic for any engineering and scientific software. The results from any codes that have this property would be rejected for publication by many professional engineering organizations. I can be easily corrected if anyone can point me to calculated results from any other areas of science and engineering in which the fact that the numerical methods are known to be not converged is accepted as being, well, acceptable practice. Buildings, airplanes, bridges, elevators, flight-control systems, nothing in fact, are never designed under this approach.
Actually all professional software development projects require far more than the information that I discuss above. Any textbook on software development can be consulted for a more complete listing and detailed discussions. Almost all the large complex AOLGCM codes have evolved over decades from software that was significantly more simple than the present versions. These codes have not been designed and built ‘from scratch’ on a ‘clean piece of paper’. Newly-built software, designed and constructed under SQA plans and associated procedures require very significantly more documentation and independent review, Verification, and Validation than that mentioned above.
Summary
All calculations from programs in which inherently complex physical phenomena and processes occurring within complex geometries are the focus are generally considered to be suspect and usually taken with a grain of salt. The modeling and calculation of climate change over the spatial and temporal scales for a planet present major challenges to all aspects of mathematical modeling and computer calculations. The number of important systems involved along with the inherent complexity of the phenomena and processes, interacting on multiple time scales over extreme spatial and temporal extents, represent possibly unprecedented challenges. V&V and SQA are absolutely necessary under these situations and are standard operating procedures (SOP) for all major software development projects. There are also absolutely necessary and should be SOP for software the calculated results of which are submitted for publication in archival journals. For calculated results that form the basis of policies that affect the health and safety of the public, the requirements for application of these processes are codified in the laws of the country.
In the absence of established, formal V&V and SQA processes and procedures, calculated results are very much less certain to contain true information and thus do not in fact provide knowledge. Even when these processes are rigorously applied to computer software, mistakes and errors still survive, although to a very much less degree than in the absence of the procedures. For the case of the very complex codes and applications associated with climate change, the chances of mistakes and errors are much larger than for less complex analyses. Consider situations that have occurred within the climate change community that sometimes surface in the literature. Data reduction software is an example.
The potential for mistakes and numerical errors in software in the absence of independent, formal, V&V and SQA procedures are among the reasons that engineering journals have implemented editorial policies. Apparently the organizations have decided that publication of a paper that has not been demonstrated to correctly solve the equations and to be based on the correct equations for its intended applications, has a high potential to not represent physical reality. Additionally, apparently they consider that the consequences of publishing such results will not contribute to advances in understanding and knowledge.
Regulatory agencies that are responsible decisions that affect the health and safety of the public will never, and I’m aware that I should never use always and never, make policy decisions based on computer software that has not been Verified and Validated and maintained and applied under SQA procedures. Observational data will always, of course, be more important than computer calculations alone.
If the science community does not begin to implement formal procedures, and at the same time base information on software that is not maintained and applied formally, regulatory agencies will ignore the calculations. And it is not just the large, complex AOLGCM codes that will eventually be required to implement the processes and procedures. All software, and software users, will have to be demonstrated to be Qualified for applications to the analyses for which the software has been designed. Additionally, it is not just the ‘main’ routines in the large codes that will require these, but also the initial and boundary conditions, the pre- and post-post processing routines ( run-time options, grid generation, processing of the calculated results, etc.), the qualifications of the users, and the procedures for installing the software onto a user’s computer system. That is, all aspects of the codes, users, applications, and results.
No comments yet.
Models Methods Software 
Leave a Reply