“an extra thousand code checkers”
Well, GISS/NASA employee Dr. Gavin Schmidt has done it again. He has given direct proof that the fundamental concepts of software Verification and Validation and other Software Quality Assurance procedures are not in the work universe of GISS/NASA.
I posted this response to a couple of comments on the GISS/NASA RealClimate blog. And I got this reply from GISS/NASA employee Gavin Schmidt which says:
[Response: Dan, repeating the same thing a hundred times does not make it true. If you want to campaign to double the climate modeling budget so that we can employ an extra thousand code checkers, great. But funders have made it absolutely clear that they prefer that we focus on science (we are scientists after all) and that no doubling of funding is on the horizon. The priorities are reversed in the nuclear industry and so they do things differently. If you want to check though our code, go ahead – nightly snapshots of our repository are on our website. All bug fixes will be greatly welcomed! In the meantime, feel free to ignore all modelling and instead base your projections on the tea leaves in your cup. – gavin]
To which I responded. Note that Dr. Schmidt has [edit]ed out part of my response in which I asked a direct question. I’m reconstructing that part here because I did not keep a record. The intended focus was as follows:
BTW, you failed to point out what part(s) of my comment were not true. If it was this part:
The problem, and it’s a critically significant problem, is that we continue to read that Climate Science seems to be the only community that cannot preform these fundamental tasks. The use of “an extra thousand code checkers” serves as validation of my characterization of the GISS/NASA approach to these fundamental issues.
then all that is required is for you to point us to the documentation that indicates that the V&V and SQA procedures have been successfully applied to the GISS/NASA ModelE software.
So, the net effects of the editing are: (1) Dr. Schmidt was unable to identify what parts of my comment were not true, and (2) apparently the V&V and SQA documentation for the ModelE software does not exist.
Note, too, that in the course of these exchanges Dr. Schmidt threw out another one of those GISS/NASA pointless strawmen: “an extra thousand code checkers”. No one other than Dr. Schmidt has ever used any number approaching 1000 as the personnel required for implementation of acceptable Independent V&V and SQA procedures. And in typical GISS/NASA fashion, the strawman is tossed into the discussions at the very time that the NASA employees decide that it’s time to end this line of thought.
Well, this latest episode of pointless information exchanges with GISS/NASA employees on RealClimate did result in some possibly useful information. According to an e-mail that I received from Dr. Schmidt, he indicates that this paper contains the final form of the momentum balance in the vertical direction. Equation 2 on page 612 is reported to be the equation in question. So, the GISS/NASA ModelE model for the vertical momentum balance is the equation for steady-state hydrodynamic equilibrium. The approach usually encountered in hydraulics.
Dr. Schmidt continues to insist that I and others search through the published papers to find information that we ask for. Gavin knows very well that I and others have in fact more than once attempted to do that very thing. For my part I did not begin to ask for assistance until after I had made attempts on my own. Generally, what I have found is that it’s like the Russian eggs that require digging deeper and deeper. Except, with the ModelE documentation you never get to anything meaningful. The information does not exist, or is buried so deep in the eggs that it is not efficient to keep digging. Additionally, if we do it ourselves we can see that the official GISS/NASA response will be that we didn’t find the right egg.
Well, while I’m at it I’ll continue with another example of tossing around strawmen with no other purpose than to distract from the real issues. In this response to comments, we get this from GISS/NASA employee Dr. Gavin Schmidt:
[Response: Ah, but someone has claimed the LHC will destory the world. The consensus of scientists says it’s bollocks. But have you examined their code? checked their calculations? examined their assumptions? Why not? Surely the whole world is at stake! – gavin]
Again, another clear avoidance of the issues.
However, I will agree that sufficient documentation such that Independent evaluations of all aspects of the issues, including numerical results from all computer software, are possible should be a goal of the GISS/NASA scientists. Additionally, associated risk analyses of the possible consequences is also highly desirable. Instead, The Climate Change Community continues to invoke The Precautionary Principle. While GISS/NASA employee Dr. Gavin Schmidt seems to think that the computer software and theoretical analyses for the LHC are candidates for independent review, he continues to insist that the same is not required for the GISS/NASA ModelE models and code and applications.
BTW, is The Precautionary Principle solely an instrument of The Climate Change Community?
Finally, “Surely the whole world is at stake!” hmmm … I think I’ve heard that statement before. Where was that and what was the subject area …. ?
Models Methods Software 
A Danish “invention” has utilized code checking with people who have the aspergers syndrom, a mild for for autism. Maybe thats a way to go…
Seriously
Click weblink: http://hbswk.hbs.edu/item/5869.html
[Thanks for the info Manfred. I have edited your comment to put in the URL.]
Schmidt never misses an occasion to make himself look foolish .
The true physicists unlike himself didn’t say “bollocks” .
They said http://arxiv.org/PS_cache/arxiv/pdf/0806/0806.3381v2.pdf .
And yes despite the fact that quantum gravity is not a fully developped theory (yet) , the paper is highly readable and checkable provided one has some knowledge about quantum mechanics and general relativity .
Assumptions are clearly stated and I have examined them . You can do so too .
You can have a look and I am sure that the paper will be much more up to your standards than the “Nacht und Nebel” obfuscations usual to Schmidt’s writings .
You will also notice that there is no “code” to be checked because we are talking physics here and not video games .
Most importantly you will notice that there is nowhere in the paper an appeal to “consensus” (Everybody says so) or to esoterical multimillion computer “models” (Just trust us , it has physics inside) .
In other words you have here a clear , understandable , verifiable statement with explicit mathematic formulations based on explicit valid physical theories .
Everything that the “climate” amateur scientists are missing .
The answer that you could have given to Schmidt was : “Real scientists can convincingly answer a quantum gravity question in 57 pages , you can surely do at least as well for a much easier question by how much will vary the Earth’s temperature .”
Addendum
The paper makes a strong empirical argument : “The Earth has been bombed by cosmic rays in the TeV range during 4 billions years without catastrophic microblackhole production . Therefore the probability that it could happen in the LHC is 0 .”
They then proceed to explain why it is necessarily so .
And equivalent climatic statement would be : “The Earth has seen a vast range of CO2 concentrations during 4 billions years without catastrophic temperature evolutions . Therefore the probability that it could happen now is 0 .”
Then it should be proceeded to explain why it is necessarily so but I know only one person (R.Lindzen) who did it .
A long while back, I posted a comment on RealClimate asking about SQA procedures for their models. The reply, from Gavin, said, roughly that in science, SQA is performed by someone else replicating the work – or not. If the other team cannot replicate, then one of the two is wrong.
I then asked, would we expect say, Microsoft Word, to be SQA’d by setting up a 2nd team to build a duplicate of MS Word, and then determine the deltas between the two versions to determine the error?
Gavin never responded to that question (but they did post it!) My take away from this is that GISS does not have meaningful software quality assurance procedures in use.
Hi Dan,
I hope I can be a more regular visitor to your blog, as this issue really hits home with me. I’ve been involved with CFD for over 20 years, and have looked at countless codes like Model E. To be fair, looking over the source listing, Model E strikes me as a typical legacy code. That is, it started out as Hansen’s Model I and II in the 1980s then got updated and added to over the years to become what it is today. Hence you see different coding styles, modules authored by diverse groups of people, and sparse comments in the code in general. Unfortunately, the code documentation is also about what I would expect for a 1980’s vintage code (which is to say not much, save for an incomplete description in a few papers). This would likely pass muster if the use of the results were limited to publications in select academic journals or fodder for conference presentations. But it seems that climate models are being used to justify far-reaching changes to our economy and society. And when the head of NASA GISS uses climate models to justify the cancellation of a much needed coal-fired power plant or, worse, vandalizing private property in the name of global warming, we have gone far beyond simple academic pursuits.
This is why I have been very hard-nosed about demanding that NASA (and others) go the extra mile to document their work and ** prove ** to others that their software is actually solving the equations it purports to be solving. I know from experience that it is very easy to develop a code that solves flow over an airfoil very accurately, but falls flat on its face when you apply it to some other problem, like a diffuser. It could be something as innocuous as a term in the turbulence model causing a separation point to shift dramatically.
I look forward to engaging myself in an effort to examine codes like Model E to try to decipher (perhaps even reverse engineer) exactly what they’re doing. Perhaps, though, we could start with something simpler, like the GISSTemp source code.
http://data.giss.nasa.gov/gistemp/sources/
I don’t think Hansen even knows what it does anymore!!
cheers,
Frank
Welcome aboard Frank. Several of us have been around the block more than once with GISSTemp. No joy. And now and then we try to steel ourselves and plunge into that darkness one more time.
I suggest you look around on Climate Audit for those threads. There seems to be one or two low-level efforts still underway, both reverse-engineering GISSTemp and starting from scratch. You can search on Climate Audit and I’ll look them up and send some links and names to you.
Let me know when you want to get started.
Hi Dan,
I’m familiar with the stellar efforts at climate audit. I’ve looked at the code before, but like others realized that it’s another legacy code that no one really understands in full, so one really has to begin by looking at how the temperature data is formatted and how this data flows through the various subroutines.
I’ll go ahead and download and unpack the tar file again. I’ll have to see what other tools are required to get it to run (perhaps it would be good to review the climate audit threads again).
If nothing else it would be nice start by trying to identify all of the key code variables and to begin adding comments as appropriate. Something to pursue over the holidays perhaps…
Frank
[…] an extra thousand coe checkers […]