NDACC Instrument Intercomparisons Protocol

Introduction

Blind instrument intercomparisons are an essential element in the certification of NDACC instrumentation. This document describes the method recommended by the NDACC Steering Committee for instrument intercomparisons leading to the validation and certification of instruments for use at NDACC stations. It describes the roles of the various parties participating in the intercomparison and the rules that should be followed for a successful exercise (i.e., one that is accepted by the scientific community as being objective and thorough).

The formal instrument intercomparison could be preceded by a formal or informal data analysis intercomparison, since a full instrument intercomparison is really an intercomparison of the instruments and their associated analysis procedures. In addition, an initial informal instrument intercomparison could precede the formal one. This would avoid the situation of formal (first-time) intercomparisons resulting in little communication among investigators and an insufficient learning experience (at least during the campaign). Thus, a formal campaign following the informal ones could be considered the natural evolution of developing measurement systems.

The intercomparison protocol described below is applicable to campaign-style exercises. Certainly, more frequent informal intercomparisons and collaborations are encouraged at any time.

Goal

In order to provide a basis for decisions on instrument certification, it is necessary that the organization and conduct of instrument intercomparisons be aligned to some fixed measurement (retrieval) goals. For most exercises, this can be stated as: to intercompare the named atmospheric species or parameters measured simultaneously by participants who do not see each other's results during the campaign, and whose data are submitted on a prescribed timetable to a referee during the course of the campaign.

Participation

Two levels of participation are possible:

• Formal – The participating individual or group agrees to have its results published as submitted by the final submission date. This is the only mode for NDACC certification.
• Informal – No results from the participating individual or group are published, and discussions with groups at the site are permitted. This is a way for new groups to learn and to evolve within the measurement community.

Results

The results are presented to the scientific community by an impartial referee who formalizes and monitors the campaign. The referee is resident at the site of the campaign, and has authority over campaign activities. The referee collects the data at preset times, compares and analyzes it, and prepares it for publication in a refereed journal or report, as well as for presentation at a participant workshop and to the NDACC Steering Committee.

Intercomparison Definition and Structure

Definitions

• Organizers: Those persons who bring together some portion of the community for the intercomparison on behalf of the Steering Committee. They are presumably members of the NDACC science community, and have interacted with the community prior to the decision to organize a formal intercomparison. 
• Referee: The person who has the direct responsibility for handling all of the data during the intercomparison campaign, and who has direct control of the campaign. This person must make decisions affecting each of the participants, and is responsible for insuring that the intercomparison is blind.

Structure

• The organizers, in cooperation with the referee, must specify in detail prior to the beginning of the campaign the species to be measured, the method(s) to be used, the times of operation, the data formats, and the schedule for data submission in the field and after the campaign.
• Each participant must submit to the referee, prior to the campaign, a detailed description of the instrument and the analysis technique. The instrument should not be changed during the campaign.
• The organizers, in cooperation with the referee, must specify in detail the calibration techniques to be followed during the intercomparison, and the frequency and method of their use. This responsibility includes the collection of instrument calibration requirements prior to the campaign so that the calibration method can be used effectively during the campaign, and a plan for reaction to the calibration results during the campaign so that the results are truly comparable by the referee during the campaign.
• Participants must submit a detailed description of spectral ranges, spectral lines, etc. to be used to insure that every group is making comparable measurements.
• The blindness of any (formal) intercomparison must not be compromised by any investigator seeing other data sets prior to the submission of the final data. If the community wants a formal intercomparison, it must be willing to forego the benefits of investigator interaction at the observing site before and during the campaign. Such benefits may be had via informal exercises, as discussed earlier.

Referee's Role

• To insure as far as possible that an impartial and blind intercomparison is achieved.
• To be unbiased and tactful, but tough and decisive, when necessary.
• To coordinate the observations to insure simultaneity, and to maximize equality among the participating instruments.
• To recognize observing or data analysis practices that could introduce differences in the results that are not primarily due to instrumental differences, and to advise the affected group(s) accordingly.
• To mediate and, as far as possible, resolve problems, and to hold discussions with all participants as the need arises.
• To mediate and resolve problems that arise after the final data submission date (i.e., with investigators who may want to change data or otherwise modify their results).
• To record all instructions to participants, and keep a good log of observations.
• To examine the primary data sets quickly, and to advise (only) any participants if their results show marked differences from the overall results. This avoids trivial errors spoiling a group's contribution for more than a day or two; however, this option must be exercised with caution.
• To be on-site during the campaign. This encourages impartiality and helps insure to the community that the campaign was conducted properly.
• To coordinate participation in a post-campaign workshop that is open to the community.
• To coordinate the publication of the results in a refereed journal or report. All participants are to review the report prior to publication, and to be included as authors of this publication.
• To coordinate the campaign with the organizers at the chosen site.
• To organize meetings as required during the campaign.
• To maintain the master clock for synchronization of the data.

Data Submission

• Data should be turned in to the referee on a regular, predefined basis (typically 24 hours after being taken) so the referee can begin to compile and review the results. However, participants could be allowed (at the discretion of the referee) to change their data in the field (during the campaign) in response to any findings or errors they make. This possibility recognizes that field campaigns are stressful, that equipment may arrive damaged, and that mistakes may be made while moving into an operational mode in the field.
• A "data submission date" for final data should be set prior to the inception of the campaign. Participants can change their data prior to this final date and after the closing of the campaign, but they must submit an explanation that is acceptable to the referee. They also should recognize that the explanation might be published at the discretion of the referee as part of the campaign paper. This allows the correction of obvious mistakes and final tuning of the results. This final submission date should be no more than six weeks after the closing of the campaign.
• No results from any other group are to be seen by any participant prior to the release of the data at the workshop.

Exceptions

For some intercomparison campaigns, ancillary data taken at the campaign site (or elsewhere) may be required for proper data analysis. In such cases, the organizers/referee will need to determine the optimal schedule for data submission.

The length of some campaigns may make it impossible for the referee to be onsite for the duration. Should this be the case, a plan will be required to prevent compromising the blindness of the campaign.

For some campaigns, it may be desirable to require final data submission at a workshop held at the end of the campaign. However, this must be the final data submission to the referee, with no changes allowed afterwards, since the blindness of the intercomparison must not be compromised under any conditions for the campaign to maintain its objectivity before the community.

Auxiliary Data

The organizers/referee should determine, prior to the campaign, those auxiliary data that are required, and should invite the appropriate persons to provide these measurements.

Post-Campaign Workshop

A workshop open to the community should be held some months after the campaign. The referee should organize this workshop. The results should be presented in detail, with ample opportunity for the participants to discuss them publicly.

Future Instrument Validation

NDACC-approved instruments could be used for the certification of new instruments that become available at the same site. This recognizes the problem that major intercomparison campaigns will not occur very often since they are expensive and time-consuming, and require participation by many members of the research community. In some cases, a "certified" traveling instrument could be used. A systematic blind approach for testing would need to be employed for impartial judging of the quality of the proposed traveling instrument.

Instrument Specific Requirements

Appendices addressing instrument specific requirements are provided for the NDACC Validation Protocol.

Version:  March 24, 2009