IUEDAC IDL-Based FITS Routines

This document is intended to provide a description of the options available for reading/writing IUE data with IDL-based IUEDAC (formally IUERDAF) software. This document includes the following sections:

I.
Introduction (history, data types, documentation)
II.
Caveats
III.
Cookbook examples of reading/writing FITS files of IUE data
IV.
Description of the tasks used in these examples

(For a list of acronym definitions click here.)

I. Introduction

A. History

The IUE mission had a long life time, starting from a era before the exchange of astronomical data in FITS format became a standard and extending to times of rapid development of computer power and numerical analysis techniques. Such factors required the development of two data processing packages, the first, IUESIPS, for data reduction following the observations, and NEWSIPS, for final archiving of the data at the close of the mission. The NEWSIPS-generated data products are to be preferred, but MAST will continue also to support analysis of the corresponding IUESIPS data.

The initial writing of IUE data in IUESIPS were done in "GO format," a protocol based on the VICAR file format system (see "Original IUE VICAR Label") used by the instrument development team at RAL and adopted for the IUE project. VICAR data consists of headers written in EBCDIC and binary and data in binary and required VICAR-specific software to read them. IUEDAC programs were written in IDL for convenient reading of archived VICAR files and for manipulating the IUE data arrays in the IDL environment.

Historically, most astronomers used IUEDAC software to read IUESIPS data; a very few users developed their own software). The use of FITS formats in NEWSIPS permitted a wider range of options. For example, several IUE data- reading tasks became available in the European package MIDAS. More recently, the IRAF/STSDAS group has written generalized FITS table-reading tasks which can be used for reading of IUE data as well as a 'iuetools' package which permits the selection of specific extracted arrays from MXLO/MXLO FITS files. The output files can then be read by standard IRAF and STSDAS tasks; see the companion document "reading IUE data files in IRAF/STSDAS."

B. Conversion from GO to RDAF format

One may convert IUESIPS GO files to "RDAF" format (*.lab and *.dat file pairs) by invoking iue IUEDAC routine,'gotordaf'. (Then one can read them by invoking 'iuespec'.)

C. Types of Data Format

The following FITS file types are used for IUE data files: p>

FITS table and image formats have been the basis for developing various programs listed below. Note also that the same file format is assumed for both tape and disk files. Thus, disk files are not separated into separate header and data files.

D. Other Documentation

The reader may wish to consult the following documentation:

II. Caveats

A. Use of IDL for running IUEDAC routines

Because IUEDAC routines are written in IDL, the user must have IDL (licensed by RSI) on his/her computer. IUEDAC routines generally run on IDL versions 3-5 and have been modified to comply with Y2K dates. IUEDAC was written in a Unix environment, but it is also supported in VMS, MS windows, and MacOs.

B. Running the routines in the VMS Operating System

A few comments should be made regarding the use of FITS disk files on VMS computers. First, VMS does not support IEEE floating point format. Although routines are available for converting IEEE format to VMS format, some IEEE special values exist which have no counterpart under VMS. (The typical solution is to run a routine to find the special characters and then prompt the user for a replacement value.) IUEDAC software calls a routine which converts the IEEE special characters to -999.9 (although it may not flag all the allowed special characters). Second, when files are transferred from Unix to vms systems using FTP (using binary mode), the result is usually a 512-byte fixed-length record file. These files must then be either converted to 2880-byte fixed-length record files, or read using RMS block mode which ignores all file attributes. The IUEDAC FITS routines typically use the RMS-block mode for reading FITS files, while the writing routines generate 2880-byte fixed length records.

III. Examples

Examples of how to use the existing IDL routines are included below. With the exception of the IUESIPS-specific reader "extfits.pro" noted below, IDL FITS-reading routines convert the FITS headers into IDL string arrays, and the data into IDL variables. When reading FITS files containing one or more extensions, the user must specify which data is to be extracted. Be sure that FITS files have been transferred properly! (e.g. by first typing "binary" in the ftp process.)

A. Reading and Interpreting FITS Files:

  1. Displaying contents of FITS files, IUEFHRD.pro To display the format of an unknown FITS file called guess.fit: Write the primary header as "mhead", the first extension header (if it exists) as "exthead", and the starting record number of each header and data section found in a vector called "params":

    iuefhrd,'guess.fit',params,mhead,exthead

  2. A generalized FITS reader, IFITSRD.pro (see also Section IV)

    a. To read headers and primary array file arr.fit: Store main header in string array "mheader", extension header in "exheader" and primary data array as "arr":

    IDL>  ifitsrd,'arr.fit',-1,mheader,exheader,arr       ; read primary array

    b. To read headers and image extension data as eps from same file:

    IDL>  ifitsrd,'arr.fit',0,mheader,exheader,eps       ; read image extension

    In general, 'ifitsrd' calls 'iuefhrd', which lists all the parameter fields in the binary table (this may be as many as 20, but is usually less than this). Including even one of the p-fields (e.g. "p1") in the command will resulting in a listing of all these fields. Thus the action of reading the file will give the user the correspondence between the p-fields and the arrays or scalar values they represent.

    c. Reading of data arrays.

    Assume "guess.fit" has a primary array, an image extension and a binary table extension. Write the headers and the first 5 columns from the first 2 rows of the binary table, assuming the binary extension data starts in record 12:

    IDL>  ifitsrd,'guess.fit',[1,2],mhead,exhead,p1,p2,p3,p4,p5,erec=12

    Note that one uses "ifitsrd" for either for image (SILO/SIHI) or extracted data files. Because "ifitsrd" is a generalized reader, it addresses arrays by their running row numbers.

  3. FITS-reader for NEWSIPS extracted spectral data arrays, READMX.pro "readmx" reads IUE merged extracted spectra (MXLO, MXHI) and returns the primary FITS header, wavelengths and absolute fluxes. The nu flags, net, background, and ripple-corrected fluxes may be optionally returned.

    a. To read a low-dispersion image:

    IDL> readmx,'swp32525.mxlo',main,wave,flux,flags,sigma,bkgrd,net

    b. To read the region of a high dispersion spectrum contiaining MgII:

    IDL> readmx,'lwp12345.mxhi',head,wave,flux,quality,sigma,wrange=[2790,2810]

    c. To read echelle orders 100--103 of this image:

    IDL> readmx,'lwp12345.mxhi',h,wave,flux,quality,sigma,orange=[100,103]

    Notice that because "readmx" is written for IUE data, it recognizes echelle orders or wavelengths rather than row numbers.

B. Writing FITS Files

These routines write FITS files from string arrays (headers) and variables (data arrays) brought into the user's IDL memory buffer by previous read commands. Both IUESIPS= and NEWSIPS-generated data may be written.

  1. To generate a FITS primary array file called arr.fit from an IDL variable ARR:

    IDL>  arr = indgen(3,4,5)         ; IDL variable
    IDL>  keygen,arr,header,in='data'         ; generate FITS keywords
    IDL>  hdugen,header,arr,fname='arr.fit'         ; create primary array file

  2. To append a variable "eps" as an image extension to the above file:

    IDL>  keygen,eps,epshd,in='data'         ; create header for eps array
    IDL>  imxgen,'arr.fit',epshd,eps         ; append to arr.fit

  3. Writing IDL variables to a binary table extension file with IFITSWRT,pro.

    a. To write W,F,NU, and SIGMA vectors to a FITS binary table file called 'data.fits' (default name) using the string array HDR as the primary array, and defining TTYPE and TUNIT keywords:

    IDL>  ifitswrt,hdr,w,f,nu,sigma,p1t='wavelength',p1u='angstrom', $

    p2t='flux',p2u='ergs/cm2/sec/A',p3t='quality',p4t='sigmas'

    b. Write X and Y vectors to a FITS (binary table) file called "gauss.fits" (no header array exists):

    IDL>  ifitswrt,0,x,y,ofn='gauss.fits'

    c. Write header, wavelength, flux, and nu vectors as a binary table extension to a FITS file called "data.fits" (default name).

    IDL>  ifitswrt,hdr,w,f,nu

  4. Writing IUESIPS files in GO format to FITS files

    IDL> gotordaf,`swp12345.mehi1'          ; first convert GO to rdaf-format files
    IDL> extfits,'imaget',1,3          ; here "imaget" is a root name like swp12345hlg.

    The output file will be "swp12345hlg.fits"

IV. Summary of FITS Reading and Writing Routines

A. General FITS Readers:

IFITSRD
- general FITS reader, but written with IUE data in mind. IFITSRD can read primary arrays, binary table extensions, image extensions, and ascii table extension FITS formats. The various data fields are given by a series of "p-arguments" (see Example 2c above), all of which are printed to the screen when the IFITSRD command is given. In general, one may not know what data fields these p-arguments correspond to until they are printed out. Since IFITSRD is a generalized FITS reader, these p's will differ for different types of files and, obviously, for different satellite missions. IFITSRD uses the following routines, which are also general FITS readers designed with IUE files in mind:

IUEFHRD
- reads a FITS primary and extension header. If file format unknown, run this before IFITSRD

IUEARRD
- reads FITS primary array or image extension

IUE3DRD
- reads one row of a FITS table Output is 1-20 fields (IDL variables).

IUE3DROW
- reads 1 row of a FITS table into an IDL structure

IUEATRD
- reads one row of a FITS ASCII table Output is 1-20 fields (IDL variables)

B. FITS Writers

  1. General Writers:

    IFITSWRT
    - writes 1 row to a binary table extension using up to 15 input parameters. An input string array header can be specified for creating the primary FITS header. Parameters may be strings, scalars, or vectors or arrays (but currently not variable-length arrays.) Additional rows can be appended to existing binary tables.

  2. Writers for IUE data:

    SAVFITS
    - writes IUE data from IDLSAV files to fits files
    IMXGEN
    - Appends input header and data to input FITS file as image extn
    HDUGEN
    - creates FITS primary array file from input IDL data and header
    MXCON
    - converts NEWSIPS MXLOs to IRAF-compatible primary FITS files.
    (The output file is compatible with IRAF fits readers.)

  3. IUESIPS-specific FITS writers:

    IMFITS
    - specific to IUESIPS image files (RI, PI, LBL, FES) Converts images to primary array FITS images. FITS files. Using IMXGEN (see below), epsilon arrays can (optionally) be appended to PI or ELBL files as an image extension.
    EXTFITS
    - specific to IUESIPS extracted files (MELO, MEHI) Converts "lab, dat" files generated by GOTORDAF to FITS files. Output is FITS primary array or binary table extensions. Each row of the binary table denotes data for a spectral order.

  4. Miscellaneous Low Level Routines:

    Routines designated with a "*" were originally borrowed from the ASTRON library and were developed by UIT, HRS and others. Most are modified forms of borrowed versions suitable for IUE data.

    *STPAR
    - returns the value of a FITS keyword.
    *ADDPAR
    - adds or modifies a FITS keyword in an IDL string array.
    *DELPAR
    - deletes a keyword from an IDL string array.
    *FLAGNAN
    - flags IEEE NaN (not a number) values
    PSTRA
    - write an IDL string array (e.g., a FITS header) into an ASCII disk file for storage or printing.
    IUEFHMOD
    - modifies a keyword in an existing FITS file.
    FITSCON
    - converts FITS binary table data types (i.e. IEEE big-indian format) to formats compatible with the current cpu.
    TRANS_BYTES
    - translates internal data formats between different machine architectures.
    SWAP_BYTES
    - low level routine for swapping bytes.
    FITSLAB
    - converts an IUE VICAR label to FITS keywords as currently planned for the IUE final archive.
    KEYGEN
    - converts the IUE scale factor record to FITS keywords, or generates the required FITS keywords necessary to describe the input data set.
    FDATE
    - converts year and cumulative day to FITS date format.
    *DATECONV
    - allows conversion between various date formats, including both old and new FITS date-format conventions.

  5. Miscellaneous Text Files:

    The files below are FITS header files which are used by some of the FITS writing routines to create FITS headers:   3DHD1.TXT;    3DHD2.TXT;    MAIN_HD.TXT;     IMAGE_HD.TXT;    3DMELO.TXT;    3DMELO2.TXT;    3DMEHI.TXT;    3DMEHI2.TXT;    BINHD.TXT;    GENHD.TXT.