Models Methods Software

GISS/NASA: We Don’t Verify. We Calculate, you Verify.

Another example of the lack of even the most basic concerns with verification of calculations has been discovered in GISS/NASA work.

The response by GISS/NASA employees is extremely disconcerting. That response seems to me to say that if the application procedure simply produces numbers, we’re ok with that. Verification of the numbers is left to others.

The problem was first revealed at Climate Audit, discussed at Watts Up With That, eventually Real Climate jumped in, and that got Luica’s attention.

I commented on Real Climate and Gavin Schmidt, as usual, left an inline reply. That reply, and comments by GISS/NASA and others there, is the basis for this post.

The general tone of the original post by GISS/NASA employee Schmidt is filled with presumptive motives about the people you discovered the GISS/NASA problem. Additionally, and what is even more amazing, Schmidt, and many people commenting at Real Climate say, in effect, that it’s ok for GISS/NASA to publish results that are not correct. Even additionally amazing, Schmidt says in effect that the problem is not with the code or application procedures. As far as I’m concerned, if I publish results calculated by my application of a code to an analysis, it is my responsibility to ensure the correctness of the numbers. No one should ever be in the position that GISS/NASA has taken; we calculate, it’s the responsibility of others to Verify the calculations.

Here is what I said at Real Climate:

This will undoubtedly be disappointing to many, but they should comfort themselves with the thought that the chances of this error happening again has now been diminished. Which is good, right?

Nope, this is not an accepted way for code / application procedures to be corrected. Finding comfort in the fact that people, whose motives you detest, find your bugs after publishing calculated results is simply wrong. Additionally, all the motives you list are (presumptive) assumptions on your part; naked strawmen.

Some people just want the numbers to be correct. Hard to believe isn’t it.

and here’s what Gavin Schmidt said in reply:

[Response: This has nothing whatsoever to do with the GISTEMP code. Some people just want to use anything that happens to push their agenda. Hard to believe isn’t it. – gavin]

Note that again, as been Schmidt’s practice when faced with hard data about the realities of a situation throws out yet another strawman unrelated to the issues. In this case he conveniently overlooks the words ‘application procedures’ and instead throws out the GISSTemp code. I did not mention any specific code in my comment. But I did explicitly say ‘application procedure’. [As an aside, Gavin’s original post is filled with these types of strawman arguments; each paragraph begins with listing an unrelated issue; naked strawmen, I called them.]

It is also interesting that many of the responses supporting GISS/NASA concepts of verification fall under the category that, “It’s ok for GISS/NASA to rely on other, outside, parties for verification. The problem was found and will now be corrected.” This approach is astounding as far as I’m concerned. It immediately begs the question of just how many numbers reported by GISS/NASA are incorrect simply because other parties were not interested in them. And it almost unbelievably says, We don’t care about verification of calculations at all. Verification is not our problem, it’s everyone else’s problem. Take a look at the comments in the thread and check that my interpretation is correct, or not.

GISS/NASA also continues to presumptively assign motives to anyone who disagrees with what they say. It seems that it is impossible for GISS/NASA to think that many people simply what to be reassured that the published numbers are correct. But as Schmidt’s original post and the many comments clearly show, GISS/NASA is happy with the present situation in which the organization throws out some numbers and then relies on the general public for checking them. Astounding, and completely wrong.

This approach to verification is in accord with the apparent general approach of GISS/NASA to models / methods / software /application procedures / users. While all other software developers know that each of these aspects are separate and distinct problems, thus necessitating individual detailed investigations, GISS/NASA throws them all together into the pot, cranks up the code, makes pretty multi-colored plots, and says, Here is The Truth.

But as in the present example, when The Truth is shown to be wanting, they, and many others, say, Well that’s ok; the problem has been found and will be corrected. Again completely overlooking, and probably hoping no one will notice, that they have explicitly invited Doubt into their data reduction and simulation world. And Doubt will be extrapolated to all other GISS/NASA work. Then to excuse the problem by saying, That’s the way Science works, borders on being incredibly _______. Wrong on so many levels that it’s hard to understand. Probably the most astounding part to me is the acceptance that “successful” application of this approach depends on the interests and qualifications of parties not even remotely related to the GISS/NASA missions !!

Now GISS/NASA has estimated the costs for trying to apply a little quality control to the temperature calculations. What’s so expensive about actually thinking about the numbers for which you are responsible? Looking at the fancy multi-colored plot and saying, Hmmm … that’s strange, it’s hotter than two Hells in Siberia in October; not to mention a couple of other places. That’s not expensive. And actually thinking about the numbers for which you are responsible is part of the job in the first place; the most fundamental part as far as I’m concerned. It’s a job requirement in Engineering.

We Don’t Verify. We Calculate, you Verify. Engineering does not work this way. I am surprised to learn that GISS/NASA ‘Science’ does work this way.

November 12, 2008 - Posted by Dan Hughes | Calculation Verification, Verification | Calculation Verification, Code Verification, Verification