Another NASA/GISS ModelE Code Fragment
Using the NASA/GISS ModelE code browser I ran across the MODULE CONSTANT in which several constants are setup as parameters.
The following code fragment is also in the routine:
!@param shv specific heat of water vapour (const. pres.) (J/kg C)
c**** shv is currently assumed to be zero to aid energy conservation in
c**** the atmosphere. Once the heat content associated with water
c**** vapour is included, this can be set to the standard value
c**** Literature values are 1911 (Arakawa), 1952 (Wallace and Hobbs)
c**** Smithsonian Met Tables = 4*rvap + delta = 1858–1869 ????
c real*8,parameter :: shv = 4.*rvap ????
real*8,parameter :: shv = 0.
The variable shv that is set to 0. appears in a few lines of code in routines SURFACE and DIAG. The former routine was the subject of this post.
The inline comments about shv in the coding seem to indicate that the NASA/GISS ModelE numerical solution methods fail to conserve the thermal energy associated with all the physical fluid components that are attempted to be modeled. Because of the lack of concise documentation for the NASA/GISS ModelE models and methods it is not possible to fully understand why accounting for the sensible energy content of water vapor could cause such a major problem.
This situation is in direct contrast to the impression offered by the IPCC that the GCM models are based on the fundamental principles of conservation of mass, momentum, and energy.
But the situation is completely consistent with this post. That is, the model for ‘conservation of energy’ used in AOLGCM codes does not account for and provide for conservation of actual real-world thermal energy. Only a hypothetical model for thermal energy conservation is employed in GCM codes. The calculated internal state of the system being forced by energy addition will very likely encounter states that do not correspond to the state of the physical system. And even if a model equation that corresponds rigorously to the physical world was used, the lack of convergence of the numerical solution methods automatically imposes a lack of conservation of energy that would correspond to the actual physical situation.
When modeling and calculations of the temperature of a complex system is the analysis objective, conservation of thermal energy is a really crucial requirement.
The inline comments about shv also indicate a mysterious lack of understanding of where to find numerical values for the basic fundamental thermophysical properties of water vapor. Other comments, and numerical values of fluid properties set in MODULE CONSANT, indicate that many (maybe all) of the transport properties are set to constant approximate values for standard atmospheric pressure at 0 Celsius.
As an aside, the following lines of code in routine DIAG illustrate the new and innovative ways provided by F90/95 to produce inactive code:
EWATER(J)=EWATER(J)+EL !+W*(SHV*T(I,J,L)*PK(L,I,J)+GRAV
! * *HSCALE*LOG(P(I,J)/PMID(L,I,J)))
This F90/95 approach to inline code documentation will surely lead to future problems. Also note that it costs as much to maintain a line of inactive code as it does for a line of active code. But in the case of lines of code like the above, it will very likely cost more because time will be spent in efforts to find out if the more or less hidden method of ‘commenting out’ code is a mistake or was actually intended. More likely, a non-zero value will be set for shv in MODULE CONSTANT above and the statements in DIAG will not be corrected to make them active. This is a very messy situation that should never have arisen.
Looking around in the SURFACE routine I find 29 lines of code that have been commented out by a ‘c’ in column 1.
Code fragments NASA/GISS ModelE have been noted here and here.
Update january 28, 2008
While reading Stoat, Only In It For The Gold, and Bryan Lawrence over the weekend I was struck by the following fact. Languages don’t write bad coding, people write bad coding. In some case, as shown in the post, very, very bad coding. There is absolutely nothing in any version of the Fortran language that compels that constructs such as shown above be used. They are entirely the result of people writing bad code. And that is possible in any language. Additionally, some languages allow constructs that seem to encourage building code that is almost impossible to understand.
There are a very large number of incorrect impressions given in the discussions on the blogs cited above. I will note only one at this time. Characterization of Fortran as a language from the 1950s is simply not correct. As all programmers who use the language know to be a true fact. Construction of data structures has always been possible in the Fortran language.
It would be interesting to check to see how many of the GCMs used in the various IPCC reports are written in a language other than Fortran. I’ll guess none are without even bothering to check.
Finally, the depth of the naivete relative to the tasks required to replace/upgrade very olde, very poorly constructed, undocumented, and yet very actively used, code displayed in the discussions in the blogs linked above is almost beyond description.
Models Methods Software 
Well, as an author of some of the code that runs in this class of models, I think I know very well what I’m talking about … and I think all of us involved in GCM class models would wish for a better way of doing it, but like Gavin said in comments to an old post of yours, double our budget, and we’ll do something about it.
Anyway, your first point was one of mine: good coders write well in any language …
Bryan, as I mentioned to Michael:
I have always been responsible for obtaining funding for the projects on which I work. Why do you and Gavin think that it’s my (or our collective) responsibility to obtain funding for you. Plus as a practical matter exactly how would that work? If I help you write the proposals do I get a cut of the work action?
Hmm, if you don’t oppose a rewrite (see comments to “a GISS ModelE Code Fragment”), what exactly was indescribably naive, per your update above, then?
We aren’t suggesting you are personally responsible for our funding, obviously. We’re just saying that if you underfund the enterprise, people doing the work will cut the corners they think are best cut. I am at least as frustrated by the quality of the stuff as is anybody who doesn’t actually have to work with it, though my concerns are more about workflow and than about fidelity. (Rigor would be better, but again the history of the effort is of people making compromises as needed to get useful work done.)
The point that interests me as a worker in the field is whether and how matters can improve.
For funding agencies the first question that needs asking is whether the models as they exist have some utility despite their flaws. Since much useful work gets done with them, the answer is surely affirmative. The second question is whether their utility can effectively be further increased incrementally. Here I have grave doubts.
For the general public the main questions are whether the global and regional scale projections have significant value for the planning process. I think the consensus among those scientists that use these models is a qualified yes to the global question and a qualified no to the regional. That is, we have a sufficiently good result for mitigation policy and an embarassingly poor one for adaptation policy.
Is it possible to make sufficiently useful regional projections to guide regional policy on the adaptation side of the equation? I can’t see a strong argument to the contrary, but I don’t think those extant codes I have seen provide a sufficiently strong foundation to do this and I think the workflows are no longer well tuned to the problems at hand.
I would like to know whether you are interested in improving the situation, or only in making the present situation look bad. If the former, discussion of the appropriate level of public funding seems as germane as anything, both as an explanation of the current state of affairs and a component of further progress.
As “a practical matter”, since you ask, your role could work like this. You could say “contrary to opinion widely held in some quarters, computational climate science is underfunded in comparison to the importance of its mission”, try to convince your readers of this, and try to convince your representatives of this.
If you are as clever as you seem to think you are, you could also join the likes of me and Brian in trying to figure out how to do the whole enterprise better rather than limiting yourself to snarking about what is wrong now. (I have not read all your postings yet, but the postings I have read have not been especially constructive. Rather they seem filled with some sort of drummed up outrage about a few smart people doing the best they can with limited resources.)
Hi Dan,
Are we actually being told that, because of resource constraints, software quality was sacrificed — knowingly flawed software has been produced — in the name of “utility?”
Oops. I think someone needs to put more study into software quality assurance. To admit that the NASA/GISS ModelE code is of less than the highest quality is to kill its utility.
Why is it a scientist/engineer/mathematician seems to think that being such automatically makes him/her a great programmer? And, of course, being a great programmer automatically makes him/her great at software quality assurance? It’s not the case!
Good work Dan!
I find micheal’s and bryan’s argument about needing more money quite funny. It usually takes
more money to write and maintain bad code. For grins I did a simple cocomo calculation
on a rewrite of modelE. ( yes Nasa uses cocomo) It wasnt that much.
re: #3
Michael, I have been torn between (1) providing you links to the info on this blog, and (2) ignoring your comment.
The opening words of the first sentence of the last paragraph and the last sentence in that paragraph, however, have won out.
If you cannot stick to the technical issues your comments will be ignored.
re: # 4
It looks that way to me George.
There is a vast universe of differences between research-grade and production-grade software. Especially when the results of the intended application areas for the software might have impacts on the health and safety of the public. Research-grade software has never yet been applied to such analyses. Only production-grade software that has met specified requirements for Documentation, Independent Verification of the coding, Independent Validation in the intended areas of application, maintained under Approved and Audited Software Quality Assurance procedures, for which training and Qualification of the Users of the software in its intended application areas have been provided, and Independent Verification of Application Calculations. The specifics might be different for different organizations and applications, but all these points will be covered for applications that affect the health and safety of the public.
Public health and safety is very, very different. Here are three examples from everyday life; bridges, elevators, flight control systems. There are hundreds more. I would not consider flying on a commercial passenger airplane if I thought that everyone had done the best they could with limited funding was an assessment of the status of the flight control software. As an additional issue I would not fly if the flight control software is based on using an ensemble average of the results calculated by an unspecified number of computer codes the V&V and SQA status of which is not known.
In other words, when software applications get moved into the public sector that creates a whole nother animal. The requirements to operate in that sector must be met. Funding must be provided to meet all the requirements of the intended application areas for the software. It is the responsibility of the organizations that sponsor development of the project to obtain all funding.
Research-grade software cannot operate in this sector of applications. That is not how software the results of which have impacts on the health and safety of the public has ever worked.
I have provided the basis for a plan that has been successfully applied to large, complex, legacy software after the codes had been written. All the codes were written in (shudder) FORTRAN.
As discussed in that document, other posts on this site, and on several other sites around the Web, documentation is the necessary key for successful implementation of the plans. An audited and approved Software Quality Assurance Plan is also an important starting point.
I don’t think it’s necessary to take offense to my last paragraph. We have to endure far worse than this on our end, some of it from your readers. (See e.g., #5, which certainly misrepresents my own stated objectives.)
That said, the document linked from comment #7 is extremely interesting and indeed appears to be written in a constructive spirit. If it is not yet published I would suggest you submit it to peer review, e.g., at
http://www.geoscientific-model-development.net/submission/index.html
I’ll recuse myself from reviewing it, but in fact I’d be favorably inclined toward publication. I have long been an advocate of improved V&V on climate codes, and especially an explicit admirer of the ASCI FLASH program which has done an admirable job of bringing scientific HPC coding practices into the modern age.
I don’t think Fortran is the right language for new climate codes because the purposes of climate codes are very different from the purposes of most other codes on at least three fronts 1) Flexibility; it is necessary to reuse phenomenological descriptions on multiple time and space scales in multiple models 2) modifiability; the primary use case of the models involves modification by people who are not software specialists 3) didactic; the audience for a literate, highly self-documenting and highly legible climate code in the Knuthian sense is far more significant in size and influence than in other fields; this is a new use case currently supported by GISS-E being a published punching bag. (I’ve taken a swipe at it myself.)
Nevertheless, contrary to the implications in your V&V paper, I really don’t think public safety depends or is suggested to depend on confidence in existing climate models. No informed person (and here I include the entire IPCC) is claiming that those are anything other than crudely similar analogous systems on which experiments can be undertaken.
It would be nice if we had models of sufficient fidelity to inform adaptation. That is the goal that drives me and should (and often doesn’t) drive most of the climate modeling field. Informing mitigation is already a done deal for contemporary purposes; the conclusion is simple enough: “if you push something hard enough it will fall over”.
Rather than “public safety,” I would suggest we use the more inclusive term “public risk.” The weather can affect our pocketbooks and general wellbeing in additions to our lives.
It seems that we all agree that a change in the climate will put the public at risk (what could be more obvious) and that mitigation and adaptation are the ways to address risk. But we disagree on the degree of mitigation and adaptation that will be required. This is because we disagree on the magnitude and distribution of the risk.
Is the magnitude of the climate risk large, small, zero, or negative? Are the people facing the greatest climate risk the same ones that are getting most of the economic benefits that are driving climate risk?
The reason we disagree on the public risk is because we have no consensus in our confidence in the climate (and economic) models. Surely we need something better “than crudely similar analogous systems on which experiments can be undertaken”? Who would volunteer?
I would have thought that independent verification and validation, much along the lines Dan is suggesting, would be the clear answer. I must be missing something.
While I am one of those who considers the risk of climate change terrifyingly underconstrained and potentially very large, I certainly agree that a significant effort toward improving the extant models is in order.
What you are “missing”, if anything, might be that 1) to some extent nobody knows how to do it 2) to the extent that anyone knows how to do it, those aren’t the people doing it 3) indeed the people who think about the theory have emerged from a pure science background and have very little idea of how applied science works at all and 4) insufficient funds.
Regarding the last point, many of my own complaints in trying to cope with these wretched systems trace back to grossly inadequate documentation, which in turn trace back to an abrupt personnel cut from eight people to six within the software engineering group of CCSM, most of which came out of documentation.
However, it may be the case that “unknown unknowns” will swamp our undertstanding for the foreseeable future. Even the most elegant code may be wrong. For instance there is insufficient data for any test of a model of ice sheet dynamics under large warming forcing; it so happens that the most interesting dynamics is in the part that melts away! All we know is that there is precedent for meters of sea level rise per century. We don’t really have a lot of evidence of where it occurred or why. This is something we’d really like to prognosticate, but in practice there is no reasonable verification test.
There’s some question about who the outside auditor would be. Most CFD is initial value dynamic and chaotic flavored, while climate is boundary condition statistical and ergodic flavored. Accordingly people from fields like aeronautics and even weather prediction tend to ask the wrong questions.
This isn’t to defend things like an undocumented half-commented-out mathematical expression; that such things lurk in the code is not something I for one choose to defend. That’s amateurish nonsense anywhere and the fact that nobody has caught it in a code review before this is discouraging to say the least.
Michael, we had a pretty good discussion on several related issues in this thread for a couple of days. The S/N ratio was larger than average, imo. Maybe you’ll have time to drop over and check it out. (That blog seems to be down today (Feb. 7).
Judith Curry had many kind things to say about CCSM and I plan to (maybe) drop ModelE/ModelE1 for a while and check up on that work. I recall being over there before and noticed that the Web pages for the software engineering group seemed to kind of drop off into a black hole. I’ll check again.
Meanwhile GISS/NASA has updated ModelE to ModelE1, so there’s some catching-up to do. Don’t do a global search for ‘hack’ in the ModelE1 source files. The results are not pretty.
A team of people representing experience and expertise in the various physical phenomena and processes areas, numerical solution methods, applications, and users, and that additionally have experience with large complex computer software would be assembled. The team would interact with the development team for the particular code to be reviewed.
The organizations and their contractors/consultants that have developed GCMs and maybe NWP, models/codes would be a good source. And maybe the activities could be partially shoehorned into existing budgets.
And yes, ICs and validation are very hard problems for GCMs. I again suggest that Oberkampf and colleagues at Sandia have tackled many of the hard issues relative to scientific and engineering software. Go to osti.gov, Information Bridge, Basic Search to get a zeroth-order cut at names and keywords and then use Advanced Search to get focused hits. Many of the Sandia reports are available for downloading. Pat Roache’s Verification and Validation of Scientific and Engineering Software book is another excellent reference. Sandia personnel could possibly introduce the processes and procedures into the GCM/NWP communities under some kind of Inter-Laboratory Loan Plan.
I’m already a great admirer of the Sandia group (especially Douglass Post who made a convincing presentation at the CCSM meeting a couple of years back).
Roache remains firmly over my head as a numerical methods sort, but this doesn’t bother me much. Climate isn’t about numerical methods; it’s about huge parameter spaces and huge datasets and the refinement of crude forcing terms and boundary physics. Our numerical cores in fact are tested and validated. That’s not where the problem lies.
If I had much influence, they or someone like them would in fact be involved. Getting to do anything remotely like what I think would help matters on a shoestring is hard enough. There is no way I personally could have the clout to pull together an effort involving Sandia, but I agree that it sounds like a good idea and would be happy to work on a project of that sort with another PI.
If it helps I have referenced Post in some of my long series of unfunded proposals.
[…] Matching in GISS/NASA ModelE Coding In a previous post I gave an illustration of how GISS/NASA employees have implemented new and innovative ways to […]
[…] Another NASA/GISS ModelE Code Fragment […]