NDACC Validation Protocol

The Network for the Detection of Atmospheric Composition Change (NDACC) Validation Protocol is designed to ensure that archived NDACC data are as high quality as is possible within the constraints of measurement technology and retrieval theory at the time the data were taken and analyzed. This is essential if changes in stratospheric composition are to be detected reliably at the earliest possible date.

Validation is a continuing process. Instruments and their associated data analysis methods must be validated before they are accepted in the NDACC and must be continuously monitored throughout their use. The requirements for datasets and evaluation have been stated in the Data Protocol and the Theory and Analysis Protocol and should be referred to inconjunction with this document. Similarly, the NDACC Instrument Intercomparisons Protocol should be referenced for details regarding acceptable intercomparison procedures.

This document serves as an outline plan for the entire validation process. The detailed methods whereby individual instruments are to be validated is the responsibility of the separate Instrument Groups organized by the Science Team members on the NDACC Steering Committee. Instrument-specific appendices are provided in this document as a further guideline.

Quality Criteria

An instrument proposed to the Steering Committee for NDACC acceptance will be referred to the appropriate Instrument Group for scientific evaluation. The Instrument Groups will be responsible for recommending acceptance of new instruments to the Steering Committee on the basis of a previously established acceptance criteria.

Each Instrument Group has provided acceptance criteria for a new instrument (see appendices). These criteria address issues such as:

• Independent evaluation of the instrument design and data analysis
• Blind instrument and data analysis intercomparisons
• Progress and actions resulting from such intercomparisons
• Absolute calibration techniques
• Use of standard spectroscopic data and standard methods of error analysis

The Steering Committee will review and approve the criteria set up by the Instrument Groups.

Documentation

When an instrument is accepted, the Instrument Investigator should provide and continue to update the following documentation held in the NDACC on-line archive along with the instrument data:

• Instrument description
• Algorithm description
• Validation procedures
• Validation history
• Calibration procedures and history

Instrument Groups also have provided criteria for the continuing evaluation of Primary and Complementary Instruments (see appendices). These criteria address the following:

• Principal and Complementary Investigator (PI and CI) responsibilities for continuing instrument evaluation and routine data archiving
• Participation in regular blind intercomparison campaigns
• Intercomparisons with traveling instruments and laboratory standards
• Absolute calibration techniques
• Use of standard spectroscopic data and standard methods of error analysis
• Analysis method updates to incorporate new scientific standards

The Steering Committee will review and approve the criteria set up by the Instrument Groups.

The experimental Investigator is primarily responsible for the everyday quality assurance of his/her own data and for keeping the archives updated; however, from time to time the Instrument Groups should hold validation exercises. Reports of continuing evaluation procedures are to be deposited in the NDACC archive and submitted to the Steering Committee where appropriate. Instruments found deficient by these procedures should be corrected if possible, and any affected data in the archive flagged appropriately. If correction is not possible, then the instrument should be withdrawn from the NDACC.

For a new generic measurement method to be accepted for use within the NDACC, the Steering Committee must be provided with convincing documentation of the technique's validity, including end-to-end simulations of the measurement and data analysis methods, along with correlative experimental evidence from the actual instruments.

Any changes to an instrument should (as far as possible) not affect the nature of the results obtained. If an instrument is improved, it should bepossible to process the results so that they are strictly comparable with results obtained previously. If this is not feasible, then the change in the data characteristics should be fully documented and the documentation deposited in the NDACC data archive.

If the data processing method is changed, then the entire archived dataset should be processed using the new method to maintain comparability. The old version of the data should be retained by the PI or CI. It is obvious that some instruments generate more data than others; therefore, the methodology for updating the archive should be established by the specific Instrument Group for each instrument type.

The data archive should contain enough information to enable the user to assess properly the quality of the data for many years hence. Thus, all relevant documentation should be held on-line along with the data. This documentation should include:

• Instrument and data processing descriptions as specified above.
• Reports of validation exercises
• History of intercomparisons for the instruments as well as the data analysis

All archived data should be flagged to indicate the instrument and data processing versions used as well as the extent to which either or both have been validated. These flags are, in effect, references to detailed documentation in the archive that describes the data quality.