NDACC Instrument Intercomparisons Protocol

Blind instrument intercomparisons are an essential element in the certification of NDACC instrumentation. This document describes the method recommended by the NDACC Steering Committee for instrument intercomparisons leading to the validation and certification of instruments for use at NDACC primary and complementary measurement sites. It describes the roles of the various parties to the campaign and the rules and regulations that should be followed for a successful exercise (i.e. one that is accepted by the scientific community as being objective and thorough).

The formal instrument intercomparison could be preceded by a formal or informal data analysis intercomparison since a full instrument intercomparison is really an intercomparison of the instruments and their associated analysis procedures. In addition, an initial informal instrument intercomparison could precede the formal one. This would avoid the situation of formal (first time) intercomparisons resulting in little communication among investigators and insufficient learning experience (at least during the campaign). Thus, a formal campaign following the informal ones could be considered the natural evolution of developing measurement systems.

The intercomparison protocol described below is applicable to campaign style exercises. Certainly, more frequent informal intercomparisons and collaborations are encouraged at anytime.

In order to provide a basis for decisions on instrument certification, it is necessary that the organization and conduct of instrument intercomparisons be aligned to some fixed measurement (retrieval) goals. For most exercises, this can be stated as: to intercompare stratospheric [species listed] measured simultaneously by participants who do not see each other's results during the campaign and whose data is submitted on a regular timetable to a referee during the course of the campaign.

There are two levels of participation:

Formal - Participating group agrees to have its results published as submitted by the final submission date. This is the only mode for NDACC certification.

Informal - No results from the participating group are published, and discussion rights with groups at the site are permitted. This is a way for new groups to learn and to evolve within the measurement community.

The results are presented to the scientific community by an impartial referee who formalizes and monitors the campaign. The referee is resident at the site of the campaign and has authority over campaign activities there. The referee collects the data at preset times, compares and analyses it, and prepares it for publication in a refereed journal or report as well as for presentation at a participant workshop and to the NDACC Steering Committee.

Definitions -

Organizers: Those persons who bring together some portion of the community for the intercomparison on behalf of the Steering Committee. They are presumably members of the NDACC science community and have interacted with the community prior to the decision to organize a formal intercomparison.

Referee: The person with the direct responsibility for handling all of the data during the intercomparison campaign and who has direct control of the campaign. This person must make decisions affecting each of the participating groups and is responsible for insuring that the intercomparison is blind.

Structure -

a) The organizers, in cooperation with the referee, must specify in detail prior to the beginning of the campaign the species to be measured, the method(s) to be used, the times of operation, the data formats, and the schedule for data submission in the field and after the campaign.

b) Each participants must submit to the referee, prior to the campaign, a detailed description of the instrument and the analysis technique. The instrument should not be changed during the campaign.

c) The organizers, in cooperation with the referee, must specify in detail the calibration techniques to be followed during the intercomparison and the frequency and method of their use. This responsibility includes



- the collection of instrument calibration requirements prior to the campaign so the calibration method can be effectively used during the campaign

- a plan for reaction to the calibration results during the campaign so that the results are truly comparable by the referee during the campaign



d) Participants must submit a detailed description of spectral ranges, spectral lines, etc. to be used to insure that every group is making comparable measurements

e) The blindness of any (formal) intercomparison must not be compromised by any investigator seeing other data sets prior to the submission of the final data. If the community wants a formal intercomparison, it must be willing to forego the benefits of investigator interaction at the observing site before and during the campaign. Such benefits may be had via informal exercises as discussed earlier.

a) To ensure as far as possible that an impartial and blind intercomparison is achieved

b) To be unbiased and tactful, but tough and decisive, when necessary

c) To coordinate the observations to ensure simultaneity and to maximize equality among the participating instruments

d) To recognize observing or data analysis practices that could introduce differences in the results that are not primarily due to instrumental differences; to advise the affected group accordingly

e) To mediate and, as far as possible, resolve problems; to hold discussions with all participants as the need arises

f) To mediate and resolve problems that arise after the final data submission date (i.e., with investigators who may want to change data or otherwise modify their results)

g) To record all instructions to participants and keep a good log of observations

h) To examine the primary data sets quickly and to advise (only) any participants if their results show marked differences from the overall results. This avoids silly errors spoiling a group's contribution for more than a day or two. This option must be exercised with caution.

i) To be on site during the campaign. This encourages impartiality and helps ensure to the community that the campaign is being conducted properly.

j) To coordinate participation in a workshop following the campaign, which should be open to the community

k) To coordinate the publication of the results in a refereed journal or report. All participants are to review the report prior to publication and to be included as authors of this publication.

l) To coordinate the campaign with the organizers at the chosen site

m) To organize meetings as required during the campaign

n) To maintain the master clock for synchronization of the data

a) Data should be turned in to the referee on a regular basis (typically 24 hours after being taken) so the referee can begin to compile and review the results. However, participants could be allowed (at the discretion of the referee) to change their data in the field (during the campaign) in response to findings or errors they make. This possibility recognizes that field campaigns are stressful, that equipment may arrive damaged, and that mistakes may be made while moving into an operational mode in the field.

b) A "data submission date" for final data should be set prior to the inception of the campaign. Participants can change their data prior to this final date and after the closing of the campaign, but they must submit an explanation that is acceptable to the referee. They should also recognize that the explanation may be published at the discretion of the referee as part of the campaign paper. This allows connection of obvious mistakes and final tuning of the results. This final submission date should be no more than 6 weeks after the closing of the campaign.

c) No results from any other group are to be seen by any participant prior to the release of the data at the workshop.

For some intercomparison campaigns, ancillary data taken at the campaign site (or elsewhere) may be required for proper data analysis. In such cases, the organizers/referee will need to determine the optimal schedule for data submission.

The length of some campaigns may make it impossible for the referee to always be in residence. Should this be the case, a plan will be required that will not compromise the blindness of the campaign.

For some campaigns it may be desirable to require final data submission at a workshop held at the end of the campaign. However, this must be the final data submission to the referee with no changes allowed afterwards since the blindness of the intercomparison must not be compromised under any conditions for the campaign to maintain its objectivity before the community.

The organizers/referee should determine, prior to the campaign, those auxiliary data that are required and should invite the appropriate persons to provide these measurements.

A workshop open to the community should be held some months after the campaign. This workshop should be organized by the referee and the results presented in detail with ample opportunity for the participants to publicly discuss them.

Primary NDACC instruments are located at both primary and complementary NDACC sites and could be used for the certification of new instruments that become available. This recognizes the problem that major intercomparison campaigns will not occur very often since they are expensive and time consuming and require participation by many members of the research community. In some cases, a "certified" traveling instrument could be used. Clearly a systematic blind approach for testing would need to be employed for impartial judging of the quality of the proposed instrument.

An appendix addressing instrument specific requirements is given in the Network for the Detection of Stratospheric Change: Validation Policy document.