Letters to NASA Quality Assurance Staff
On Tuesday November 11, 2008 at about 11:45 am I sent copies of the following letter to several NASA Quality Assurance personnel. The date has proven to be quite a coincidence as subsequent, and continuing, events have shown. I had started writing the letter a few days before I sent it.
I’ve heard nothing back yet; none of the recipients has acknowledged that the letter was received.
I have not yet located similar Quality Assurance offices in NOAA. All pointers will be appreciated.
Ms. Martha Wetherholt
NASA Software Assurance
300 E Street SW
Washington, DC 20546
Ms. Sue Sekira, Code 611
NASA Software Assurance
Goddard Institute of Space Studies
2880 Broadway
New York, NY 10025
Dr. Michele Rienecker, Code 610.1
Global Modeling & Assimilation Office
Earth Sciences Division
Goddard Space Flight Center
Greenbelt, MD 20771
Mr. Robert Connerton, Code 610
Chief Engineer
Earth Sciences Division
Goddard Space Flight Center
Greenbelt, MD 20771
SUBJECT: NASA Software and the IPCC Summary for Policy Makers
Hello Everyone:
Calculated results from software developed and used by NASA form part of the basis for the IPCC Summary for Policy Makers (SPM) reports. It should be clear, then, that results by the software have the potential to influence public policy that will affect the health and safety of the public. The purpose of this letter is to gain some information regarding the status of the Software Quality Assurance for the NASA software.
The GISS/NASA ModelE code, a General Circulation (or Climate) model, is a NASA product used in the analyses leading to the SPM. Additionally, results from the GISSTemp program frequently appear in news releases reporting on the monthly monitoring of the global average surface temperature. And these results subsequently form part of the basis of statements in the IPCC SPM. Results from these both codes appear frequently in peer-reviewed scientific literature and news releases by NASA.
I have not been successful in locating any information at all about the Software Quality Assurance status of these codes. A Software Quality Assurance Plan, maintained by the organizations that develop the software, would provide a good starting point. I would also expect to find that the plan, and the results of applying the plan to the software, had been Independently audited and approved. While the software and results are the subject of extensive Web sites, the Plans and audit-approved applications of the Plans do not seem to be available from these.
Given the global importance of the IPCC SPM reports it seems that NASA would desire that the codes and reported results be of the highest quality. Accuracy of peer-reviewed papers based on calculations with the software should be, by the very definition of peer-reviewed literature, of archival quality. And press releases, almost all of which receive wide-spread interest, should also be determined to be accurate.
By your positions in the NASA organizations, you are well aware of the importance of SQA in all computer software. Likewise you are keenly aware that lack of SQA policies and procedures, and successful applications of these to the codes, leaves open a significant potential for problems to exist in the software.
I would like to determine if SQA Plans, and results of applications of the Plans, are available from your offices. If such Plans are not available can you provide me the information in which exemptions from Software Quality Assurance for these codes has been determined to be appropriate.
Thank you for your attention to this matter.
Sincerely,
Dan Hughes
Models Methods Software 
I like it!
Now, just a suggestion for something further.
Would it be reasonable to send a copy of this to one or two (or a few) Senators?
I am not familiar with the governmental hierarchy of the USA, being from England, but my experience here is that a carefully worded letter, such as yours, can have a good effect if it is read (and understood) by our elected representatives. This type of person may not be technically equipped to pass judgment on the detailed content but may well be inclined to submit it to further scrutiny to someone who might have such knowledge.
I think also of the Wegman investigation, that got at least some publicity, and of the time when Mann’s work was called in by some politicians after some of his conclusions were deemed to be less than well-founded.
Robin
Dan,
A good letter, and I think it would be good if NASA received a lot more letters from concerned individuals. HOwever, can I suggest that you copy to Michael Griffin, Administrator of NASA at
Public Communications Office
NASA Headquarters
Suite 5K39
Washington, DC 20546-0001
(202) 358-0001 (Office)
(202) 358-3469 (Fax)
or electronically at http://www.nasa.gov/about/contact/ask_nasa_form.html
I think that there used to be a time when you could e:mail Michael Griffin directly, but that is no longer the case. Anyhow, he has a responsibility in regard to NASA’s public profile, and should be aware of the issues.
The more letters NASA recieves, the more attention will they pay.
Does their Quality Management Program have a Corrective and Preventative Action function? If so is it possible to see their Corrective and Preventive Action Reports related to GISTEMP-related software failures?
Actually, does GISS have a Registered Quality Management System? If not, what operations at NASA actually operate under registered systems, some any, all of them? If some, what/who decides what programs are registered and subject to external audit?
Man, this just makes me wonder.
Tuesday November 25, 2008, two weeks and still not a word.
Looks like it’s time for a FOI Request.
I’m working my way up the pay-scale. I sent a hard-copy letter to the NASA Chief Engineer and enclosed the letter that’s in the post.
I remain very pessimistic, however. The wagons have been circled. Public policy will be set, based, in-part, on results calculated by software that has not been Verified. NASA is setting new standards here.
“NASA is setting new standards here.”
And of all the agencies to be doing this…It is shocking and hopefully has reverberations.
Note that NASA has ISO9001 according to their site at http://www.hq.nasa.gov/office/codeq/quality/index.htm.
They have to submit the external audits to maintain ISO9001. They also have a quality control system. therefore you should be able to get info by FOI.
Good luck
Any updates?
I’ve been thinking about this for awhile. One thing I’ve heard expressed many times is the consensus argument. That is, we should believe the climate scientists because they are the “experts”. Hmmmmmm.
So, exactly how are climate scientists “experts” at software engineering? The truth is they are amateurs. All one needs to examine is a small section of ModelE software. There is absolutely no doubt that the software has errors. Not only that, there is no doubt the originators of the code have no idea what errors are present and how they might impact the results generated by this product.
I believe if the code was reviewed by several software experts there would be a consensus of this point of view.
I think the next step after finding out there is no quality assurance would be to have the code reviewed by professsional software quality engineers and present the findings to NASA (and/or congress). Let’s face it, if the consensus argument works for AGW then they would look silly debating a consensus of experts who tell them the heart of AGW is based on error prone methods.
Update on December 19, 2008.
Still not a word from any person or organization. On or about December 6, 2008, I sent hard copy of a letter to the Director of the Geophysical Fluid Dynamics Laboratory. No word from there yet either.
It’s not clear to me that NASA should spend time answering general questions like this from the public. Imagine if every agency had to spend all its time answering every question from the public. Note, I think it’s a reasonable question. I’m just not sure that this kind of demand for info is approriate. Need to think about better way to engage.