spacer link to MAST page spacer logo image spacer
 
link to STScI page


Overview of On-the-Fly Reprocessing

Introduction

The On-The-Fly Reprocessing (OTFR) system reconstructs FITS data files from original telemetry and calibrates data at the time that a user's request for the data is processed. Through this system, users obtain data calibrated with up-to-date reference files, parameters, and software. Only the telemetry files (called POD files) need to be stored for instruments supported by OTFR. The On-The-Fly Reprocessing system was released March, 2001 for STIS and WFPC2; September 26, 2001 for NICMOS; and March 2002 for ACS. Late in July 2007, a final reprocessing effort was completed for STIS data taken prior to August 4, 2004 and OTFR was turned off for these data.

Advantages of OFTR

By deploying OFTR, we only need to store the the original telemetry in the HST archive. Doing so significantly reduced the volume of data stored in the archive (Hanisch et al 1997). For WFPC2, we achieved a savings in storage space by more than a factor of ten. The raw data occupies less storage and compresses much better than the calibrated data. The storage requirements for the HST archive have rapidly increased with STIS,NICMOS, and ACS, and will become even greater with COS and WFC3.

Another motivation comes from the scientific need to recalibrate data to take advantage of various improvements. A large fraction (over 90%) of the calibrated data in the HST archive could be improved by recalibration, although the improvements are not always significant. In the past, instruments that undergo evolution of calibration files or calibration software often required that users carry out their own recalibrations at their home sites. With OTFR, the STScI carries out the recalibration.

The elements for improved calibrations are listed below.

  • Improved calibration files. For example, the ideal dark current reference files for correcting WFPC2 observations are those that are obtained at nearly the same time as the science data. Therefore, the original calibration, using nominal dark reference files, is not optimal. The OTFR system helps solve this problem by permitting users to obtain the calibrated data with the appropriate dark file as soon as it is available. In addition, other types of improved calibration files are sometimes generated that supersede previous calibration files. Such files are used by OTFR soon after they are created.
  • Improved parameter values. Improvements to the data can come about as better values for header keyword parameters are determined by the instrument teams. The OTFR system incorporates the improved header structure in a clean way.
  • Improved software. Improvements are made to the calibration software (e.g., calwp2 and calstis) to correct problems, improve algorithms, or add new capabilities.
  • Handling associations. OTFR generates associated data products in the indentical process as the OPUS pipeline.

Dataflow

In broad outline, the following steps are carried out.

  • A user specifies the desired calibrated datasets through the usual retrieval mechanisms in Starview or the archive Web interface. The user is informed of the initial status by email.
  • The relevant POD files are retrieved from the HST archive and passed to the OTFR system.
  • The raw FITS files are constructed from the telemetry (POD files).
  • The headers are updated with the best values of calibration files, calibration switches, and other parameters.
  • The data are calibrated.
  • The data are sent back to the HST archive system.
  • The HST archive system returns the data to the user.
  • The HST archive system sends a final email message about the completion status to the user.

Elements of the OTFR System

  • User interfaces

    The STScI archive Web interface to the archive may be used to retrieve OTFR data. This is transparent to the user. Starview, the Java application interface to the STScI archive, also provides capabilities for requesting OTFR processing (Travisano and Richon 1997). StarView currently is the only archive interface which allows the user to review the details of the OTFR changes and reference file updates, by employing the instrument OTFR screens.

  • Data Archiving and Distribution System (DADS)

    DADS, the STScI data archive system, provides raw data to the OTFR pipeline (Travisano and Richon 1997). DADS also sends the newly calibrated and/or raw data to users.

  • Calibration Database System (CDBS)

    The calibration database system tracks all calibration files used for calibrating HST data (Lubow et al 1997). The system contains a tool called bestref that recommends the calibration files for each existing dataset. These recommendations are updated as new calibration files become available.

  • Calibration Software

    The calxxx software is used to calibrate the data. It is provided through STSDAS.

  • OPUS

    To coordinate the processing activities, the OTFR system employs OPUS blackboards (Rose 1998). They allow for parallel processing of tasks distributed over multiple CPUs. The OPUS system operates by means of scripts that transfer the data from the archive system, perform the getref/upref processing, the actual calibration processing, and the return of the data to the HST archive system.

References

Hanisch, R.J., Abney, F., Donahue, M., Gardner, L., Hopkins, E., Kennedy, H., Kyprianou, M., Pollizzi, J., Postman, M., Richon, J., Swade, D., Travisano, J., and White, R. 1997, "HARP: The Hubble Archive Re-Engineering Project Summary Report", A.S.P. Conference Series, Vol. 125, Gareth Hunt and H. E. Payne, eds., p. 294.

Lubow, S., Cox, C., Hurt, L., and Simon, B. 1997, Redesign of CDBS, STScI Report

Lubow, S. and Pollizzi, J. 1999, "On-The-Fly Calibration at STScI", ADASS VIII, ASP Conference Series, Vol. 172, D.M. Mehringer, R.L. Plante, and D.A. Roberts, eds., p. 187

Rose, J. 1998, "OPUS-97: A Generalized Operational Pipeline System", ADASS VII, A.S.P. Conference Series, Vol. 145, 1998, R. Albrecht, R.N. Hook and H.A. Bushouse, eds., p.344

Travisano, J.J. and Richon, J. G. 1997, "The Evolution of the HST Archive", Archive Re-Engineering Project Summary Report", ADASS VI, A.S.P. Conference Series, Vol. 125, Gareth Hunt and H. E. Payne, eds., p. 286.