NDACC Validation Protocol

The Network for the Detection of Atmospheric Composition Change (NDACC) Validation Protocol is designed to insure that archived NDACC data are of as high a quality as possible, within the constraints of measurement technology and retrieval theory at the time the data were taken and analyzed. This is essential if changes in atmospheric composition are to be detected at the earliest possible date, with the accuracy and precision necessary to understand the responsible forcings.

Validation is a continuing process. Instruments and their associated data analysis methods must be validated before they are accepted in the NDACC, and must be continuously monitored throughout their use. The requirements for datasets and evaluation have been stated in the NDACC Data Protocol and the NDACC Theory and Analysis Protocol, respectively, and should be referred to in conjunction with this document. Similarly, the NDACC Instrument Intercomparison Protocol should be referenced for details regarding acceptable intercomparison procedures.

This document serves as an outline plan for the entire validation process. The detailed methods whereby individual instruments are to be validated is the responsibility of the separate Instrument Working Groups represented on the NDACC Steering Committee. Instrument-specific appendices are provided along with this document as a further guideline.

Evaluation of New Instruments and Instrument Teams

Quality Criteria

• Independent evaluation of the instrument design and data analysis 
• Blind instrument and data analysis intercomparisons 
• Progress and actions resulting from such intercomparisons 
• Absolute calibration techniques 
• Use of standard spectroscopic data and standard methods of error analysis

Documentation

When an instrument is accepted, the Instrument Investigator should provide and continue to update the following documentation held in the NDACC archive along with the instrument data:

• Instrument description
• Algorithm description
• Validation procedures
• Validation history
• Calibration procedures and history

Evaluation of Continuing Instruments and Instrument Teams

Instrument Working Groups also have provided criteria for the continuing evaluation of NDACC Instruments (see appendices). These criteria, which have been reviewed and approved by the Steering Committee, address the following:

• Investigator responsibilities for continuing instrument evaluation and routine data archiving
• Participation in regular blind intercomparison campaigns
• Intercomparisons with traveling instruments and laboratory standards
• Absolute calibration techniques
• Use of standard spectroscopic data and standard methods of error analysis
• Analysis method updates to incorporate new scientific standards

The experimental Investigator is primarily responsible for the everyday quality assurance of his/her own data, and for keeping the archives updated; however, from time to time, the Instrument Working Groups should hold validation exercises. Reports of continuing evaluation procedures are to be deposited in the NDACC archive and submitted to the Steering Committee, where appropriate. Instruments found deficient by these procedures should be corrected, if possible, and any affected data in the archive flagged appropriately. If communications between the PI and the Working Group Representatives indicate that the correction of deficiencies is not possible, then the PI will be notified in writing that the instrument is being withdrawn as an active NDACC measurement capability.

Acceptance of a New Generic Measurement Method for NDACC

For a new generic measurement method to be accepted for use within the NDACC, the Steering Committee must be provided with convincing documentation of the technique's validity, including end-to-end simulations of the measurement and data analysis methods, along with correlative experimental evidence from the actual instruments.

Changes to Instruments and Data Processing

Any changes to an instrument should (as far as possible) not affect the nature of the results obtained. If an instrument is improved, it should be possible to process the results so that they are strictly comparable with results obtained previously. If this is not feasible, then the change in the data characteristics should be fully documented, and the documentation deposited in the NDACC data archive.

If the data processing method is changed, then the entire archived dataset should be reprocessed using the new method to maintain comparability. The investigator should retain the old version of the data. It is obvious that some instruments generate more data than others; therefore, the methodology for updating the archive should be established by the specific Working Group for each instrument type.

Contents of the NDACC Data Archive

The data archive should contain enough information to enable the user to assess properly the quality of the data for many years hence. Thus, all relevant documentation should be held on-line along with the data. This documentation should include (but is not limited to):

• Instrument and data processing descriptions, as specified above
• Reports of validation exercises
• History of intercomparisons for the instruments and the data analysis

All archived data should be flagged to indicate the instrument and data processing versions used, as well as the extent to which either or both have been validated. These flags are, in effect, references to detailed documentation in the archive that describes the data quality.

Version:  March 24, 2009