Models Methods Software

Dan Hughes

Royal Meteorological Society V&V and SQA

Some time ago it was suggested to me something to the effect that most of the people in the Climate Change Community to whom I suggest V&V and SQA issues are critical and require significant attention don’t even know what I’m talking about. Here’s some information that lends support to that observation.

A couple of days ago (January 6, 2009) I sent the following e-mail and an attachment to Professor Paul Hardaker, the CEO of the Royal Meteorological Society (RMS) which publishes several peer-reviewed journals:

Dear Professor Hardaker,

Important issues related to computer software and archival publications, such as those published by the Royal Meteorological Society, are the subjects of the attached letter. I have more detailed sources of information related to these subjects should you be interested in pursuing the issues.

I look forward to receiving information about how the RMS Journals handle these situations.

Thank you for your attention to this matter.


The attached letter is as follows:

January 6, 2009
Dan Hughes
324 Plank Road
Porter Corners, NY 12859

Professor Paul Hardaker, CEO
Royal Meteorological Society
104 Oxford Road
RG1 7LL United Kingdom

Subject: Verification and Validation of Computer Codes used as Basis for Paper Submittals

Dear Professor Hardaker:
Paper submittals the basis of which are calculations with computer software continue to be important sources of publications in your RMS journals. Some of these papers might at some time be used as audit/checkpoints/benchmarks for other calculations, and others of these papers might become part of the basis for public policy decisions. The software used for these papers needs to be Verified and Validated for the applications to which they are applied. Papers for which the software has not been Verified should not be accepted for publication in archival journals such as those published by the RMS.

The first crucial aspect of such papers should be the status of the Verification of the software. Verification is the process of determining that the coded equations are correctly solved. Several engineering societies and their journal editorial boards have put into place technical requirements on the Verification of the software before the paper can be considered for publication. If the requirements have not been met the paper will not be published; in some cases the paper will be rejected out-of-hand and not be sent out for review. Papers for which the basis is a single calculation on a single grid with no investigations of convergence and other stopping criteria are typically sent back to the authors.

Some of these professional organizations and associated journals include: The American Society of Mechanical Engineers (ASME) Journal of Heat Transfer and Journal of Fluids Engineering; The American Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft and Rockets; and the International Journal of Numerical Methods for Fluid Flow. Other professional societies and journals are sure to follow the lead of these.

References for the editorial polices for these journals are as follows.

The ASME Journal of Heat Transfer:
Editorial Board, “Journal of Heat Transfer Editorial Policy Statement on Numerical Accuracy,” ASME Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.

The ASME Journal of Fluids Engineering:
C. J. Freitas, “Editorial Policy Statement on the Control of Numerical Accuracy,” ASME J. Fluids Eng., 115, pp. 339–340, 1993. and J. Fluids Eng. 130, 2008.

The AIAA Journal of Spacecraft and Rockets:
AIAA, Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty, AIAA Journal, Vol. 32, No. 1, p. 3, 1994.

The International Journal of Numerical Methods in Fluids:
P. M. Gresho and C. Taylor, “Editorial,” International Journal of Numerical Methods in Fluids, Vol. 19, p. iii, 1994.

Complete and detailed discussion of all aspects of Verification, Validation and Software Quality Assurance for scientific and engineering software are given by Roache in the book, “Verification and Validation in Computational Science and Engineering,” published by Hermosa Press and a paper by Oberkampf, et al., “Verification, Validation, and Predictive Capability in Computational Engineering and Physics,” Sandia National Laboratories Report SAND 2003-3769, gov/cgi-bin/techlib/access, 2003. These publications contain extensive reference citations to the published literature on these subjects.

I have been unsuccessful in locating an editorial policy for RMS journals on this important issue. If one exists will you kindly let me know how I can access it. If a policy does not exist do you have plans to address this issue?

In light of the large potential for uninformed applications of calculated results reported by the RMS, especially applications that might impact public policy, I suggest that the editorial board initiate discussions of the issue and begin planning for implementation of a sound basis for review and acceptance of papers the basis of which are numerical calculations with computer software.

Thank you for your attention to this matter. Please do not hesitate to contact me if you need additional information on this subject.


Dan Hughes

I received the following reply from Professor Hardaker on January 7, 2009:

Dear Mr Hughes

Thanks for the note. The verification and validation of source material is a key part of the peer review process and that’s certainly true for the Society’s journals. We place great importance on the robustness of our peer review process. I hope that re-assures you.

Kind Regards

Paul H.

The response does not begin to address the issues that are the subject of my letter. The ‘source material’ for papers that are based on calculations with computer software is the software. All aspects of the software; extensive documentation, formal V&V and SQA procedures and processes, user qualifications, and calculation Verification prior to publication.


January 10, 2009 - Posted by | Calculation Verification, Verification | , , ,

1 Comment »

  1. So they are quite happy not having a policy on V&V and quality assurance!? Good journals have these sorts of policy’s because it improves the quality of results in the papers that end up being published (and written policies are good to teach the young guys what the ‘best practices’ in the field are); ignoring the need for such a policy basically means you aren’t running a ‘premier’ journal.

    I’m really surprised by that response, I would have expected something like “we’re working on it, but it’s an open area of research, here’s a draft policy, what do you think?” The really troubling thing is that this isn’t new stuff; it is well within the state of the practice in many other fields.

    Comment by jstults | January 15, 2010 | Reply

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: