Royal Meteorological Society V&V and SQA
Some time ago it was suggested to me something to the effect that most of the people in the Climate Change Community to whom I suggest V&V and SQA issues are critical and require significant attention don’t even know what I’m talking about. Here’s some information that lends support to that observation.
The Fundamental Issue
The Intergovernmental Panel for Climate Change (IPCC) Fourth Assessment Report (AR4, IPCC, 2007) states:
“A major advance of this assessment of climate change projections compared with the TAR is the large number of simulations available from a broader range of models. Taken together with additional information from observations, these provide a quantitative basis for estimating likelihoods for many aspects of future climate change.” [ My bolding. ]
Do the numbers from these “large number of simulations available from a broader range of models” GCM calculations have any Meaning. My answer is No.
One crucial and necessary first step is that application of Verification procedures have shown that the numbers produced by the software accurately reflect both (1) the original intent of the continuous equations for the models, and (2) the numerical solution methods applied to the discrete approximations to the continuous equations. That is, Verification shows that the equations have been solved right. Do the numbers actually satisfy the Verified-to-be-correct-as-coded discrete equations and do the solutions of the discrete equations converge to solution of the continuous equations. Neither of these has been demonstrated for any GCM. I will be pleased to be shown to be wrong on this point.
All software can be Verified. Objective technical criteria and associated success metrics can be developed and applied in a manner that provides assurances about the correctness of the coding of the equations and their numerical solutions. Lack of Verification leaves open the potential that the numbers from the software are simply results of “bugs” in the coding.
The present-day software development community, in all kinds of applications and organizations, is keenly aware that lack of SQA policies and procedures, and successful applications of these to the software, leaves open a significant potential for problems to exist in the software. So far as I am aware, there are no precedents whatsoever for public policy decisions to be based on software for which no SQA procedures have been applied.
Hurst Coefficients: A Rough Draft
I have a rough draft discussion about applications of the Hurst Coefficient methodology to (1) solutions of the Lorenz equations, (2) measured temperature data, and (3) results from calcuations with GCMs. I would like to get some peer-review.
Update October 8, 2010
I have uploaded two files to this new server as the others have been lost to the previous server. Application of the Hurst Coefficient method to the Lorenz 1963 system is here. And some of that plus application to some data is here.
As I’ve mentioned a couple of times, there are many broken links because of the loss of the previous server.
All comments, especially corrections to incorrectos, will be appreciated.
Letters to NASA Quality Assurance Staff
On Tuesday November 11, 2008 at about 11:45 am I sent copies of the following letter to several NASA Quality Assurance personnel. The date has proven to be quite a coincidence as subsequent, and continuing, events have shown. I had started writing the letter a few days before I sent it.
I’ve heard nothing back yet; none of the recipients has acknowledged that the letter was received.
I have not yet located similar Quality Assurance offices in NOAA. All pointers will be appreciated.
Moto Trip Report: Part 1. The Most Beautiful Road in America
The Motorcycle God of Wanderlust and the Goddess of Natural Wonders made me do it.
Mary, the love of my life, and the best lover ever in my life, insisted that this road trip be one for me; go slow, go fast, go where I want to, eat what, where and when I want to, stay in any motel that I want to. As usual there are no reservations and no destinations decided in advance. Look at the Real-Time Road Trip Planner and decide which of the Small-Line Roads to check out. Get to an intersection and decide which road to take. Follow the twisty roads, stop and smell the roses, take in the deep blue skies and fluffy white clouds. There was no other plan. The best Moto Road Trips don’t have Plans.
It was a most excellent Moto Road Trip. About eight weeks with every day spent in Biker Trash Paradise. Rode roads, ran roads, and toured for about 12,000 miles (19,000 km) in Western and Northwestern United States. Saw some beautiful and wondrous sights and met some interesting people. I hope do be able to do lots more of this 🙂
GISS/NASA: We Don’t Verify. We Calculate, you Verify.
Another example of the lack of even the most basic concerns with verification of calculations has been discovered in GISS/NASA work.
The response by GISS/NASA employees is extremely disconcerting. That response seems to me to say that if the application procedure simply produces numbers, we’re ok with that. Verification of the numbers is left to others.
Averaging Planet Earths or Averaging Planet Xs
Update January 23, 2011. I fixed the broken links.
There are several discussions floating around on the subject of comparing GCM-calculated numbers with experimental data. Climate Audit and Lucia both have several threads. There are too many threads to give links; let me know if you need a specific thread. One focus of these discussions is the ensemble-average approach that is considered to be necessary for the comparisons. The ensemble is made up of the results from the various different versions of GCMs that calculate the results. Not only are GCMs different from each other, but the suggested approach is to make perturbations in the initial conditions between the runs.
I have mentioned that the results do not reflect the effects of perturbations in only the initial conditions. Everything about the approach is different between the runs. The GCMs are based on different continuous-equation models, numerical solution methods, application procedures, run-time options, and users, and maybe other aspects. How can we be sure that the results are all appropriate for Earth?
Pattern Matching in GISS/NASA ModelE Coding
In a previous post I gave an illustration of how GISS/NASA employees have implemented new and innovative ways to produce inactive code using the capabilities provided by F90/95. I had run across the following statements in routine DIAG.f:
EWATER(J)=EWATER(J)+EL !+W*(SHV*T(I,J,L)*PK(L,I,J)+GRAV
! * *HSCALE*LOG(P(I,J)/PMID(L,I,J)))
The ‘!’ in the first line is going to be very difficult to remember it exists and correctly maintain. Someone might come along and say, “I wonder what that’s doing in the middle of an executable statement.” and promptly un-do the comment. Or un-do the comment of the second line while overlooking the comment in the first line. That would make a screw up on several levels.
Today I have found many more examples of innovative coding by employees of GISS/NASA. It is clear that the NASA Software Quality Assurance procedures are ignored by GISS/NASA. It is equally clear that there are no Software Quality Assurance procedures being applied to the GISS/NASA ModelE code. None.
Update November 2, 2008 down near the end.
Discussion of RealClimate Comments: Part 0
The persons in charge of the GISS/NASA RealClimate Web site have made it clear that continued discussions of this issue are not welcomed over there. So I’ve moved over here.
Models Methods Software 